Examples of org.terrier.structures.CollectionStatistics

org.terrier.structures.CollectionStatistics
This class provides basic statistics for the indexed collection of documents, such as the average length of documents, or the total number of documents in the collection.
After indexing, statistics are saved in the PREFIX.log file, along with the classes that should be used for the Lexicon, the DocumentIndex, the DirectIndex and the InvertedIndex. This means that an index knows how it was build and how it should be opened again. @author Gianni Amati, Vassilis Plachouras, Craig Macdonald

          if (documentPointers.length > 0)
            assertEquals("Number of pointers for docid " + docid + " is incorrect", documentPointers[docid], docpointers);
          assertEquals("Document length for docid "+docid+" is incorrect", documentLengths[docid], doclen);
        }
      }
      CollectionStatistics cs = index.getCollectionStatistics();
      assertEquals("Number of documents is incorrect", cs.getNumberOfDocuments(), docid + 1);
      assertEquals("Number of pointers is incorrect", cs.getNumberOfPointers(), pointers);
      assertEquals("Number of tokens is incorrect", cs.getNumberOfTokens(), tokens);
      if (numberOfTerms > 0)
      {
        assertEquals("Not all termIds found in direct index", termIds.size(), numberOfTerms);
      }
    }

View Full Code Here

        assertEquals("Numebr of pointers for docid " + docid + " is incorrect", documentPointers[docid], docpointers);
      assertEquals("Document length for docid "+docid+" is incorrect", documentLengths[docid], doclen);
      docid++;
    }
    
    CollectionStatistics cs = index.getCollectionStatistics();
    assertEquals("Number of documents is incorrect", cs.getNumberOfDocuments(), docid);
    assertEquals("Number of pointers is incorrect", cs.getNumberOfPointers(), pointers);
    assertEquals("Number of tokens is incorrect", cs.getNumberOfTokens(), tokens);
    if (numberOfTerms > 0)
    {
      assertEquals("Not all termIds found in direct index", termIds.size(), numberOfTerms);
    }
  }

View Full Code Here

    di = null;
  }
  
  protected void checkCollectionStatistics(Index index)
  {
    final CollectionStatistics cs = index.getCollectionStatistics();
    assertEquals("Number of documents doesn't match", DOCUMENT_LENGTHS.length, cs.getNumberOfDocuments());
    assertEquals("Number of tokens doesn't match", StaTools.sum(DOCUMENT_LENGTHS), cs.getNumberOfTokens());
    assertEquals("Average document length doesn't match", StaTools.mean(DOCUMENT_LENGTHS), cs.getAverageDocumentLength(), 0.0d);
    assertEquals("Number of pointers doesnt match", NUMBER_POINTERS, cs.getNumberOfPointers());
    assertEquals("Number of unique terms doesn't match", NUMBER_UNIQUE_TERMS, cs.getNumberOfUniqueTerms());
  }

View Full Code Here

TOP

Related Classes of org.terrier.structures.CollectionStatistics

org.terrier.tests.HadoopShakespeareEndToEndTest$CheckDirectHSplits

org.terrier.tests.ShakespeareEndToEndTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.