Examples of org.apache.hadoop.hbase.util.ByteBloomFilter$MetaWriter

org.apache.hadoop.hbase.util.ByteBloomFilter
ne-lab.org">European Commission One-Lab Project 034819. Bloom filters are very sensitive to the number of elements inserted into them. For HBase, the number of entries depends on the size of the data stored in the column. Currently the default region size is 256MB, so entry count ~= 256MB / (average value size for column). Despite this rule of thumb, there is no efficient way to calculate the entry count after compactions. Therefore, it is often easier to use a dynamic bloom filter that will add extra space instead of allowing the error rate to grow. ( http://www.eecs.harvard.edu/~michaelm/NEWWORK/postscripts/BloomFilterSurvey .pdf ) m denotes the number of bits in the Bloom filter (bitSize) n denotes the number of elements inserted into the Bloom filter (maxKeys) k represents the number of hash functions used (nbHash) e represents the desired false positive rate for the bloom (err) If we fix the error rate (e) and know the number of entries, then the optimal bloom size m = -(n * ln(err) / (ln(2)^2) ~= n * ln(err) / ln(0.6185) The probability of false positives is minimized when k = m/n ln(2). @see BloomFilter The general behavior of a filter @see Space/Time Trade-Offs in Hash Coding with Allowable Errors

        ByteBuffer b = getMetaBlock(BLOOM_FILTER_META_KEY, false);
        if (b != null) {
          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }
          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType==BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + name);
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here


      if (bloomType != BloomType.NONE && conf != null) {
        float err = conf.getFloat("io.hfile.bloom.error.rate", (float)0.01);
        int maxFold = conf.getInt("io.hfile.bloom.max.fold", 7);


        this.bloomFilter = new ByteBloomFilter(maxKeys, err,
            Hash.getHashType(conf), maxFold);
        this.bloomFilter.allocBloom();
        this.bloomType = bloomType;
      } else {
        this.bloomFilter = null;

View Full Code Here

        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);
        int tooBig = conf.getInt(IO_STOREFILE_BLOOM_MAX_KEYS, 128*1000*1000);
        
        if (maxKeys < tooBig) { 
          try {
            bloom = new ByteBloomFilter(maxKeys, err,
                Hash.getHashType(conf), maxFold);
            bloom.allocBloom();
            bt = bloomType;
          } catch (IllegalArgumentException iae) {
            LOG.warn(String.format(

View Full Code Here

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);
        int tooBig = conf.getInt(IO_STOREFILE_BLOOM_MAX_KEYS, 128*1000*1000);
        
        if (maxKeys < tooBig) { 
          try {
            bloom = new ByteBloomFilter(maxKeys, err,
                Hash.getHashType(conf), maxFold);
            bloom.allocBloom();
            bt = bloomType;
          } catch (IllegalArgumentException iae) {
            LOG.warn(String.format(

View Full Code Here

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

        if (bloomType == BloomType.ROWCOL) {
          err /= 2;
        }
        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);


        this.bloomFilter = new ByteBloomFilter(maxKeys, err,
            Hash.getHashType(conf), maxFold);
        this.bloomFilter.allocBloom();
        this.bloomType = bloomType;
      } else {
        this.bloomFilter = null;

View Full Code Here

          if (bloomFilterType == BloomType.NONE) {
            throw new IOException("valid bloom filter type not found in FileInfo");
          }




          this.bloomFilter = new ByteBloomFilter(b);
          LOG.info("Loaded " + (bloomFilterType== BloomType.ROW? "row":"col")
                 + " bloom filter metadata for " + reader.getName());
        }
      } catch (IOException e) {
        LOG.error("Error reading bloom filter meta -- proceeding without", e);

View Full Code Here

          }


          //Printing bloom information
          ByteBuffer b = reader.getMetaBlock("BLOOM_FILTER_META", false);
          if (b!= null) {
            BloomFilter bloomFilter = new ByteBloomFilter(b);
            System.out.println("BloomSize: " + bloomFilter.getByteSize());
            System.out.println("No of Keys in bloom: " +
                bloomFilter.getKeyCount());
            System.out.println("Max Keys for bloom: " +
                bloomFilter.getMaxKeys());
          } else {
            System.out.println("Could not get bloom data from meta block");
          }
        }
        if (printBlocks) {

View Full Code Here

        int maxFold = conf.getInt(IO_STOREFILE_BLOOM_MAX_FOLD, 7);
        int tooBig = conf.getInt(IO_STOREFILE_BLOOM_MAX_KEYS, 128*1000*1000);
        
        if (maxKeys < tooBig) { 
          try {
            bloom = new ByteBloomFilter(maxKeys, err,
                Hash.getHashType(conf), maxFold);
            bloom.allocBloom();
            bt = bloomType;
          } catch (IllegalArgumentException iae) {
            LOG.warn(String.format(

View Full Code Here

0 1

TOP

Related Classes of org.apache.hadoop.hbase.util.ByteBloomFilter$MetaWriter

org.apache.hadoop.hbase.io.hfile.HFile

org.apache.hadoop.hbase.regionserver.StoreFile$Reader

org.apache.hadoop.hbase.regionserver.StoreFile$Writer

java.util.Random

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.