Examples of com.cloudera.util.bloom.BloomSet

com.cloudera.util.bloom.BloomSet
This is an interface for a bloom set. Currently this shim wraps the o.a.h.util.bloom.BloomFilter with a more generic API but uses inefficient calls. This is only for test code so for now. Operations of this are woefully inefficient computationally and this should be eventually replaced with a version that does not rely on hadoop's Writeable-tied version.

  /** Test it with larger number of events and larger number of slots. */
  @Test
  @Ignore("Takes too long to run")
  public void testBloomSetCompare100Mx10M() {
    // generally we want about 9-10 bits per entry.
    BloomSet b1 = new BloomSet(1000000000, 2); // 1B bits ~= 125MB
    BloomSet b2 = new BloomSet(1000000000, 2);


    for (int i = 0; i < 100000000; i++) { // 100M "entries"
      if (i != 234000)
        b1.addInt(i); // drop one that is included in the other set.


      if (i <= 10000000)
        b2.addInt(i);


      // only add the first 10M to the second hash
    }


    assertFalse(b1.contains(b2)); // b1 doesn't have all b2 has!
    assertFalse(b2.contains(b1));
  }

View Full Code Here

  /**
   * {@inheritDoc}
   */
  @Override
  synchronized public void open() throws IOException {
    bloom = new BloomSet(size, hashes);
    state = BloomCheckState.UNKNOWN;
    successCount = 0;
    failCount = 0;
    super.open();
    reportSink.open();

View Full Code Here

  public void append(Event e) throws IOException {
    byte[] data = e.get(BloomGeneratorDeco.A_BLOOMSETDATA);
    // if has BloomSet Data is present
    if (data != null) {
      // process and then drop the event.
      BloomSet subset = new BloomSet(data);
      boolean contained = bloom.contains(subset);


      if (LOG.isDebugEnabled()) {
        LOG.debug("received bloom set: " + Arrays.toString(subset.getBytes()));
        LOG.debug("local bloom set:    " + Arrays.toString(bloom.getBytes()));
      }


      synchronized (this) {
        switch (state) {

View Full Code Here

  }


  /** {@inheritDoc} */
  @Override
  public void open() throws IOException {
    bloom = new BloomSet(size, hashes);
    super.open();
  }

View Full Code Here

0 1

TOP

Related Classes of com.cloudera.util.bloom.BloomSet

com.cloudera.flume.handlers.debug.BloomCheckDecorator

com.cloudera.flume.handlers.debug.BloomGeneratorDeco

com.cloudera.flume.handlers.debug.TestBloomSetDecos

org.apache.hadoop.util.bloom.BloomFilter

org.apache.hadoop.util.bloom.Key

java.nio.ByteBuffer

java.io.DataOutputStream

java.io.ByteArrayOutputStream

java.io.ByteArrayInputStream

java.io.DataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.