Examples of org.apache.lucene.index.pruning.CarmelTopKTermPruningPolicy

org.apache.lucene.index.pruning.CarmelTopKTermPruningPolicy

l.acm.org/citation.cfm?id=383958">Static index pruning for information retrieval systems, D. Carmel at al, ACM SIGIR 2001 .

The claim of this pruning technique is, quoting from the above paper:

Prune the index in such a way that a human "cannot distinguish the difference" between the results of a search engine whose index is pruned and one whose index is not pruned.

For indexes with a large number of terms this policy might be too slow. In such situations, the uniform pruning approach in {@link CarmelUniformTermPruningPolicy} will be faster, though it mightproduce inferior search quality, as that policy does not pose a theoretical guarantee on resulted search quality.

TODO implement also CarmelTermPruningDeltaTopPolicy

  public void testCarmelTopKPruning() throws Exception {
    IndexReader in = IndexReader.open(sourceDir, true);
    // validate full scores - without pruning, just to make sure we test the right thing
    validateDocScores(fullScores, in, false, false); // validate both docs and scores
    // prune reader
    CarmelTopKTermPruningPolicy tfp = new CarmelTopKTermPruningPolicy(in, null, K, EPSILON, R, null);
    PruningReader tfr = new PruningReader(in, null, tfp);
    
    // create the pruned index
    RAMDirectory targetDir = new RAMDirectory();
    WhitespaceAnalyzer analyzer = new WhitespaceAnalyzer(TEST_VERSION_CURRENT);

Examples of org.apache.lucene.index.pruning.CarmelTopKTermPruningPolicy

Related Classes of org.apache.lucene.index.pruning.CarmelTopKTermPruningPolicy