Examples of org.apache.solr.util.OpenBitSet

org.apache.solr.util.OpenBitSet
An "open" BitSet implementation that allows direct access to the array of words storing the bits.
Unlike java.util.bitet, the fact that bits are packed into an array of longs is part of the interface. This allows efficient implementation of other algorithms by someone other than the author. It also allows one to efficiently implement alternate serialization or interchange formats.
OpenBitSet is faster than java.util.BitSet in most operations and *much* faster at calculating cardinality of sets and results of set operations. It can also handle sets of larger cardinality (up to 64 * 2**32-1)
The goals of OpenBitSet are the fastest implementation possible, and maximum code reuse. Extra safety and encapsulation may always be built on top, but if that's built in, the cost can never be removed (and hence people re-implement their own version in order to get better performance). If you want a "safe", totally encapsulated (and slower and limited) BitSet class, use java.util.BitSet.

Performance Results
Test system: Pentium 4, Sun Java 1.5_06 -server -Xbatch -Xmx64M
BitSet size = 1,000,000
Results are java.util.BitSet time divided by OpenBitSet time.

cardinality intersect_count union nextSetBit get iterator

50% full 3.36 3.96 1.44 1.46 1.99 1.58

1% full 3.31 3.90 1.04 0.99

Test system: AMD Opteron, 64 bit linux, Sun Java 1.5_06 -server -Xbatch -Xmx64M
BitSet size = 1,000,000
Results are java.util.BitSet time divided by OpenBitSet time.

cardinality intersect_count union nextSetBit get iterator

50% full 2.50 3.50 1.00 1.03 1.12 1.25

1% full 2.51 3.49 1.00 1.02

@version $Id$

	cardinality	intersect_count	union	nextSetBit	get	iterator
50% full	3.36	3.96	1.44	1.46	1.99	1.58
1% full	3.31	3.90		1.04		0.99

	cardinality	intersect_count	union	nextSetBit	get	iterator
50% full	2.50	3.50	1.00	1.03	1.12	1.25
1% full	2.51	3.49		1.00		1.02

  public int unionSize(DocSet other) {
    return this.size() + other.size() - this.intersectionSize(other);
  }


  public DocSet andNot(DocSet other) {
    OpenBitSet newbits = (OpenBitSet)(this.getBits().clone());
    newbits.andNot(other.getBits());
    return new BitDocSet(newbits);
  }

View Full Code Here

public class BitDocSet extends DocSetBase {
  final OpenBitSet bits;
  int size;    // number of docs in the set (cached for perf)


  public BitDocSet() {
    bits = new OpenBitSet();
  }

View Full Code Here

    }
  }


  @Override
   public DocSet andNot(DocSet other) {
    OpenBitSet newbits = (OpenBitSet)(bits.clone());
     if (other instanceof OpenBitSet) {
       newbits.andNot(((BitDocSet)other).bits);
     } else {
       DocIterator iter = other.iterator();
       while (iter.hasNext()) newbits.clear(iter.nextDoc());
     }
     return new BitDocSet(newbits);
  }

View Full Code Here

     return new BitDocSet(newbits);
  }


  @Override
   public DocSet union(DocSet other) {
     OpenBitSet newbits = (OpenBitSet)(bits.clone());
     if (other instanceof BitDocSet) {
       newbits.union(((BitDocSet)other).bits);
     } else {
       DocIterator iter = other.iterator();
       while (iter.hasNext()) newbits.set(iter.nextDoc());
     }
     return new BitDocSet(newbits);
  }

View Full Code Here

        } else {
          // calculate the maximum value that this term may have
          // and if it is >= smallestCount, then flag for refinement
          long maxCount = sfc.count;
          for (int shardNum=0; shardNum<rb.shards.length; shardNum++) {
            OpenBitSet obs = dff.counted[shardNum];
            if (!obs.get(sfc.termNum)) {
              // if missing from this shard, add the max it could be
              maxCount += dff.maxPossible(sfc,shardNum);
            }
          }
          if (maxCount >= smallestCount) {
            // TODO: on a tie, we could check the term values
            needRefinement = true;
          }
        }


        if (needRefinement) {
          // add a query for each shard missing the term that needs refinement
          for (int shardNum=0; shardNum<rb.shards.length; shardNum++) {
            OpenBitSet obs = dff.counted[shardNum];
            if (!obs.get(sfc.termNum) && dff.maxPossible(sfc,shardNum)>0) {
              dff.needRefinements = true;
              if (query==null) query = dff.makeQuery(sfc);
              toRefine[shardNum].add(query);
            }
          }

View Full Code Here


  void add(int shardNum, NamedList shardCounts, int numRequested) {
    int sz = shardCounts.size();
    int numReceived = sz;


    OpenBitSet terms = new OpenBitSet(termNum+sz);


    long last = 0;
    for (int i=0; i<sz; i++) {
      String name = shardCounts.getName(i);
      long count = ((Number)shardCounts.getVal(i)).longValue();
      if (name == null) {
        missingCount += count;
        numReceived--;
      } else {
        ShardFacetCount sfc = counts.get(name);
        if (sfc == null) {
          sfc = new ShardFacetCount();
          sfc.name = name;
          sfc.termNum = termNum++;
          counts.put(name, sfc);
        }
        sfc.count += count;
        terms.fastSet(sfc.termNum);
        last = count;
      }
    }


    // the largest possible missing term is 0 if we received less

View Full Code Here

    if (pos < scratch.length) {
      scratch[pos]=doc;
    } else {
      // this conditional could be removed if BitSet was preallocated, but that
      // would take up more memory, and add more GC time...
      if (bits==null) bits = new OpenBitSet(maxDoc);
      bits.fastSet(doc);
    }


    pos++;
  }

View Full Code Here

   * Inefficient base implementation.
   *
   * @see BitDocSet#getBits
   */
  public OpenBitSet getBits() {
    OpenBitSet bits = new OpenBitSet();
    for (DocIterator iter = iterator(); iter.hasNext();) {
      bits.set(iter.nextDoc());
    }
    return bits;
  };

View Full Code Here

    if (other instanceof HashDocSet) {
      return other.intersection(this);
    }


    // Default... handle with bitsets.
    OpenBitSet newbits = (OpenBitSet)(this.getBits().clone());
    newbits.and(other.getBits());
    return new BitDocSet(newbits);
  }

View Full Code Here

    newbits.and(other.getBits());
    return new BitDocSet(newbits);
  }


  public DocSet union(DocSet other) {
    OpenBitSet newbits = (OpenBitSet)(this.getBits().clone());
    newbits.or(other.getBits());
    return new BitDocSet(newbits);
  }

View Full Code Here

0 1 2 3 4

TOP

Related Classes of org.apache.solr.util.OpenBitSet

org.apache.solr.handler.component.DistribFieldFacet

org.apache.solr.handler.component.FacetComponent

org.apache.solr.search.BitDocSet

org.apache.solr.search.DocSetBase

org.apache.solr.search.DocSetHitCollector

org.apache.solr.search.DocSetPerf

org.apache.solr.search.SolrIndexSearcher

org.apache.solr.search.TestDocSet

org.apache.solr.tst.TestRequestHandler

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.solr.util.OpenBitSet

Performance Results

Related Classes of org.apache.solr.util.OpenBitSet