Examples of it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor

Package it.unimi.dsi.mg4j.search.visitor

Examples of it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor

it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor
A visitor collecting the counts of terms in a {@link it.unimi.dsi.mg4j.search.DocumentIterator}tree.
Note: in fact, the documentation of this class clarifies also the usage of {@link it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor} and{@link it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor}.
Several scoring schemes, such as {@linkplain it.unimi.dsi.mg4j.search.score.BM25Scorer BM25}or cosine-based measures, require the counts (number of occurrences in a given document) of terms in the query. Since we do not want restrict the ability of the user to specify sophisticated constraints such as term proximity, order, consecutivity, etc., we prefer not to use a bag-of-words query model, in which the user simply inputs a number of terms (in that case, of course, the definition of the count of each term is trivial). Rather, we provide a group of visitors that make it possible to retrieve counts for each term appearing in the query.
Since MG4J provides multi-index queries, each count is actually associated to a pair index/term (e.g., the count of class in the main text might be different from the count of class in the title). Moreover, we must be careful to define a sensible semantics, as when logical operators alternate there might be occurrences of a term in a query whose count might give misleading information (in particular if the same term appear several times).
Thus, we define a true path on the query tree (which parallels the composite tree of the associated {@link it.unimi.dsi.mg4j.document.DocumentIterator}) as a path from the root that passes only through nodes whose associated subquery evaluates to true (in the Boolean sense). A counter-collection visitor records in the counter arrays only the counts of index/term pairs appearing at the end of a true path.
For instance, in a query like a OR (title:b AND c) in a document that contains a and c in the main text, but does not contain b in the title, only the count of a will be taken into consideration. In the same way, for a query whose outmost operation is a negation no counter will ever be written.
Preparing and using counters

Instance of this class are useful only in connection with a {@link it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor} (and, in turn, with a{@link it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor}). More precisely, there are three phases:
1. after creating the {@link it.unimi.dsi.mg4j.search.DocumentIterator}, {@linkplain it.unimi.dsi.mg4j.search.visitor.DocumentIteratorVisitor#prepare() prepare}a {@link it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor} and perform a{@linkplain it.unimi.dsi.mg4j.search.DocumentIterator#accept(DocumentIteratorVisitor) visit} to gather term information and possibly cache some data about the {@linkplain it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor#terms(Index) terms appearing in the iterator};
2. {@linkplain it.unimi.dsi.mg4j.search.visitor.DocumentIteratorVisitor#prepare() prepare}a {@link it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor} based on the previous{@link it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor}, and perform a visit to read frequencies and prepare counters;
3. start iterating: after each call to {@link it.unimi.dsi.mg4j.search.DocumentIterator#nextDocument() nextDocument()}, {@linkplain it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor#clear() clear the counters}, perform a {@linkplain it.unimi.dsi.mg4j.search.DocumentIterator#acceptOnTruePaths(DocumentIteratorVisitor) visit along true paths} using an instance of this class and inspect the data gathered in the{@linkplain it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor} (see, for example, the source code of{@link it.unimi.dsi.mg4j.search.score.CountScorer}).
Note that all visitors are reusable: just {@link it.unimi.dsi.mg4j.search.visitor.DocumentIteratorVisitor#prepare() prepare()} them before usage, butbe careful as a {@link it.unimi.dsi.mg4j.search.visitor.CounterSetupVisitor} must be prepared and visitedafter the associated {@link it.unimi.dsi.mg4j.search.visitor.TermCollectionVisitor}has been prepared and visited. The {@link it.unimi.dsi.mg4j.search.visitor.DocumentIteratorVisitor#prepare() prepare()}method of this class is a no-op, so it is not necessary to call it.

   */
  public BM25FScorer( final double k1, final Reference2DoubleMap<Index> b, final StringMap<? extends CharSequence> termMap, final LongList frequencies ) {
    this.termMap = termMap;
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.bByIndex = b;
    this.frequencies = frequencies;
    k1Plus1 = k1 + 1;
    bByName = null;

View Full Code Here

   */
  public BM25FScorer( final double k1, final StringMap<? extends CharSequence> termMap, final LongList frequencies, final Object2DoubleMap<String> b ) {
    this.termMap = termMap;
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.bByName = b;
    this.frequencies = frequencies;
    k1Plus1 = k1 + 1;
    bByIndex = null;

View Full Code Here

  private final TermCollectionVisitor termCollectionVisitor;
  
  public CountScorer() {
    termCollectionVisitor = new TermCollectionVisitor();
    counterSetupVisitor = new CounterSetupVisitor( termCollectionVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( counterSetupVisitor );
  }

View Full Code Here

   * @param b the <var>b</var> parameter.
   */
  public BM25Scorer( final double k1, final double b ) {
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
    this.k1 = k1;
    this.b = b;
    k1Plus1 = k1 + 1;
  }

View Full Code Here

  private double[] weightedIdfPart;
  
  public TfIdfScorer() {
    termVisitor = new TermCollectionVisitor();
    setupVisitor = new CounterSetupVisitor( termVisitor );
    counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );
  }

View Full Code Here

  
  public void testCounterSetup() throws Exception {


    TermCollectionVisitor termVisitor = new TermCollectionVisitor();
    CounterSetupVisitor setupVisitor = new CounterSetupVisitor( termVisitor );
    CounterCollectionVisitor counterCollectionVisitor = new CounterCollectionVisitor( setupVisitor );


    it.unimi.dsi.mg4j.query.nodes.Query query = simpleParser.parse( "this" );
    DocumentIteratorBuilderVisitor documentIteratorBuilderVisitor = new DocumentIteratorBuilderVisitor( null, index0, Integer.MAX_VALUE );


    DocumentIterator indexIterator = query.accept( documentIteratorBuilderVisitor );


    indexIterator.nextDocument();


    termVisitor.prepare();
    indexIterator.accept( termVisitor );
    setupVisitor.prepare();
    indexIterator.accept( setupVisitor );


    // assertEquals( 3, indexIterator.frequency() );
    assertEquals( 3, setupVisitor.frequency[ 0 ] );


    assertEquals( 1, indexIterator.nextDocument() );
    counterCollectionVisitor.prepare();
    indexIterator.accept( counterCollectionVisitor );
    assertEquals( 1, setupVisitor.count[ 0 ] );
    assertEquals( 2, indexIterator.nextDocument() );
    counterCollectionVisitor.prepare();
    indexIterator.accept( counterCollectionVisitor );
    assertEquals( 1, setupVisitor.count[ 0 ] );
  }

View Full Code Here

TOP

Related Classes of it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor

it.unimi.dsi.mg4j.search.score.BM25FScorer

it.unimi.dsi.mg4j.search.score.BM25Scorer

it.unimi.dsi.mg4j.search.score.CountScorer

it.unimi.dsi.mg4j.search.score.ScorerTest

it.unimi.dsi.mg4j.search.score.TfIdfScorer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor

Preparing and using counters

Related Classes of it.unimi.dsi.mg4j.search.visitor.CounterCollectionVisitor