Builds an inverted index saving term-block information. It optionally saves term-field information as well.
Algorithm:
- While there are terms left:
- Read M term ids from lexicon, in lexicographical order
- Read the occurrences of these M terms into memory from the direct file
- Write the occurrences of these M terms to the inverted file
- Rewrite the lexicon, removing block frequencies, and adding inverted file offsets
- Write the collection statistics
Lexicon term selection: There are two strategies of selecting the number of terms to read from the lexicon. The trade-off here is to read a small enough number of terms into memory such that the occurrences of all those terms from the direct file can fit in memory. On the other hand, the less terms that are read implies more iterations, which is I/O expensive, as the entire direct file has to be read for every iteration.
The two strategies are:
- Read a fixed number of terms on each iterations - this corresponds to the property invertedfile.processterms
- Read a fixed number of occurrences (pointers) on each iteration. The number of pointers can be determined using the sum of frequencies of each term from the lexicon. This corresponds to the property invertedfile.processpointers.
By default, the 2nd strategy is chosen, unless the
invertedfile.processpointers has a zero value specified.
Properties:
- invertedfile.processterms - the number of terms to process in each iteration. Defaults to 25,000
- invertedfile.processpointers - the number of pointers to process in each iteration. Defaults to 2,000,000, which specifies that invertedfile.processterms should be read from the lexicon, regardless of the number of pointers.
@author Douglas Johnson & Vassilis Plachouras & Craig Macdonald