Examples of it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor

Package it.unimi.dsi.mg4j.search.visitor

Examples of it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor

it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor
A visitor collecting information about terms appearing in a {@link it.unimi.dsi.mg4j.search.DocumentIterator}.
The purpose of this visitor is that of exploring before iteration the structure of a {@link DocumentIterator} to count how many terms are actually used, and set up somepreliminary access data. More precisely, we count the distinct pairs index/term appearing in all leaves of nonzero frequency (the latter condition is used to skip empty iterators). For this visitor to work, all leaves of nonzero frequency must return a non-null value on a call to {@link it.unimi.dsi.mg4j.index.IndexIterator#term()}.
During the visit, we keep track of which index/term pair have been already seen. Each pair is assigned an distinct offset—a number between zero and the overall number of distinct pairs—which is stored into each index iterator {@linkplain it.unimi.dsi.mg4j.index.IndexIterator#id() id}and is used afterwards to access quickly data about the pair. Note that duplicate index/term pairs get the same offset. The overall number of distinct pairs is returned by {@link #numberOfPairs()} after a visit.
During the visit, the indices actually appearing in some nonzero-frequency leaf are gathered; they are accessible as a vector returned by {@link #indices()}, and the map from positions in this vector to indices is inverted by {@link #indexMap()}. If you need to force some index to appear in {@link #indices()}, there's a special {@link #prepare(ReferenceSet)} method.
The offset assigned to each pair index/term is returned by {@link #offset(Index,String)}. Should you need to know the terms associated to each index, they are returned by {@link #terms(Index)}.
The after a term collection, usually counters are set up by a visit of {@link it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor}.

   * @param termMap a map from terms to positions in <code>frequencies</code>, or <code>null</code> if <code>frequencies</code> is <code>null</code>.
   * @param frequencies the frequencies, or <code>null</code> for Boldi's variant.
   */
  public BM25FScorer( final double k1, final Reference2DoubleMap<Index> b, final StringMap<? extends CharSequence> termMap, final LongList frequencies ) {
    this.termMap = termMap;
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.bByIndex = b;
    this.frequencies = frequencies;

View Full Code Here

   * @param frequencies the frequencies, or <code>null</code> for Boldi's variant.
   * @param b the <var>b</var> parameter, specified as a map from indices to values.
   */
  public BM25FScorer( final double k1, final StringMap<? extends CharSequence> termMap, final LongList frequencies, final Object2DoubleMap<String> b ) {
    this.termMap = termMap;
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.bByName = b;
    this.frequencies = frequencies;

View Full Code Here

  private final CounterSetupVisitor counterSetupVisitor;
  /** The term collection visitor used to estimate counts. */
  private final TermCollectionVisitor termCollectionVisitor;
  
  public CountScorer() {
    termCollectionVisitor = new TermCollectionVisitor();
    counterSetupVisitor = new CounterSetupVisitor( termCollectionVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( counterSetupVisitor );
  }

View Full Code Here

  /** Creates a BM25 scorer using specified <var>k</var><sub>1</sub> and <var>b</var> parameters.
   * @param k1 the <var>k</var><sub>1</sub> parameter.
   * @param b the <var>b</var> parameter.
   */
  public BM25Scorer( final double k1, final double b ) {
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.b = b;
    k1Plus1 = k1 + 1;

View Full Code Here

  private int[] size;
  /** An array indexed by offsets that caches the inverse document-frequency part of the formula, multiplied by the index weight. */
  private double[] weightedIdfPart;
  
  public TfIdfScorer() {
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
  }

View Full Code Here

  }


  
  public void testCounterSetup() throws Exception {


    TermCollectionVisitor termVisitor = new TermCollectionVisitor();
    CounterSetupVisitor setupVisitor = new CounterSetupVisitor( termVisitor );
    CounterCollectionVisitor counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );


    it.unimi.dsi.mg4j.query.nodes.Query query = simpleParser.parse( "this" );
    DocumentIteratorBuilderVisitor documentIteratorBuilderVisitor = new DocumentIteratorBuilderVisitor( null, index0, Integer.MAX_VALUE );


    DocumentIterator indexIterator = query.accept( documentIteratorBuilderVisitor );


    indexIterator.nextDocument();


    termVisitor.prepare();
    indexIterator.accept( termVisitor );
    setupVisitor.prepare();
    indexIterator.accept( setupVisitor );


    // assertEquals( 3, indexIterator.frequency() );

View Full Code Here

TOP

Related Classes of it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor

it.unimi.dsi.mg4j.index.Index

it.unimi.dsi.mg4j.search.score.BM25FScorer

it.unimi.dsi.mg4j.search.score.BM25Scorer

it.unimi.dsi.mg4j.search.score.CountScorer

it.unimi.dsi.mg4j.search.score.ScorerTest

it.unimi.dsi.mg4j.search.score.TfIdfScorer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.