Examples of org.apache.lucene.codecs.lucene42.Lucene42TermVectorsFormat

• org.apache.lucene.codecs.lucene42.Lucene42RWCodec
  google.com/p/lz4/">LZ4 compression algorithm,
• binary data is written using fixed-size blocks of {@link PackedInts packed ints}.

Term vectors are stored using two files

• a data file where terms, frequencies, positions, offsets and payloads are stored,
• an index file, loaded into memory, used to locate specific documents in the data file.

File formats

1. A vector data file (extension .tvd). This file stores terms, frequencies, positions, offsets and payloads for every document. Upon writing a new segment, it accumulates data into memory until the buffer used to store terms and payloads grows beyond 4KB. Then it flushes all metadata, terms and positions to disk using LZ4 compression for terms and payloads and {@link BlockPackedWriter blocks of packed ints} for positions.

   Here is a more detailed description of the field data file format:

   • VectorData (.tvd) --> <Header>, PackedIntsVersion, ChunkSize, <Chunk>^ChunkCount
   • Header --> {@link CodecUtil#writeHeader CodecHeader}
   • PackedIntsVersion --> {@link PackedInts#VERSION_CURRENT} as a {@link DataOutput#writeVInt VInt}
   • ChunkSize is the number of bytes of terms to accumulate before flushing, as a {@link DataOutput#writeVInt VInt}
   • ChunkCount is not known in advance and is the number of chunks necessary to store all document of the segment
   • Chunk --> DocBase, ChunkDocs, < NumFields >, < FieldNums >, < FieldNumOffs >, < Flags >, < NumTerms >, < TermLengths >, < TermFreqs >, < Positions >, < StartOffsets >, < Lengths >, < PayloadLengths >, < TermAndPayloads >
   • DocBase is the ID of the first doc of the chunk as a {@link DataOutput#writeVInt VInt}
   • ChunkDocs is the number of documents in the chunk
   • NumFields --> DocNumFields^ChunkDocs
   • DocNumFields is the number of fields for each doc, written as a {@link DataOutput#writeVInt VInt} if ChunkDocs==1 and as a {@link PackedInts} array otherwise
   • FieldNums --> FieldNumDelta^{TotalDistincFields}, a delta-encoded list of the sorted unique field numbers present in the chunk
   • FieldNumOffs --> FieldNumOff^TotalFields, as a {@link PackedInts} array
   • FieldNumOff is the offset of the field number in FieldNums
   • TotalFields is the total number of fields (sum of the values of NumFields)
   • Flags --> Bit < FieldFlags >
   • Bit is a single bit which when true means that fields have the same options for every document in the chunk
   • FieldFlags --> if Bit==1: Flag^{TotalDistinctFields} else Flag^TotalFields
   • Flag: a 3-bits int where:
     • the first bit means that the field has positions
     • the second bit means that the field has offsets
     • the third bit means that the field has payloads
   • NumTerms --> FieldNumTerms^TotalFields
   • FieldNumTerms: the number of terms for each field, using {@link BlockPackedWriter blocks of 64 packed ints}
   • TermLengths --> PrefixLength^TotalTerms SuffixLength^TotalTerms
   • TotalTerms: total number of terms (sum of NumTerms)
   • PrefixLength: 0 for the first term of a field, the common prefix with the previous term otherwise using {@link BlockPackedWriter blocks of 64 packed ints}
   • SuffixLength: length of the term minus PrefixLength for every term using {@link BlockPackedWriter blocks of 64 packed ints}
   • TermFreqs --> TermFreqMinus1^TotalTerms
   • TermFreqMinus1: (frequency - 1) for each term using {@link BlockPackedWriter blocks of 64 packed ints}
   • Positions --> PositionDelta^{TotalPositions}
   • TotalPositions is the sum of frequencies of terms of all fields that have positions
   • PositionDelta: the absolute position for the first position of a term, and the difference with the previous positions for following positions using {@link BlockPackedWriter blocks of 64 packed ints}
   • StartOffsets --> (AvgCharsPerTerm^{TotalDistinctFields}) StartOffsetDelta^TotalOffsets
   • TotalOffsets is the sum of frequencies of terms of all fields that have offsets
   • AvgCharsPerTerm: average number of chars per term, encoded as a float on 4 bytes. They are not present if no field has both positions and offsets enabled.
   • StartOffsetDelta: (startOffset - previousStartOffset - AvgCharsPerTerm * PositionDelta). previousStartOffset is 0 for the first offset and AvgCharsPerTerm is 0 if the field has no positions using {@link BlockPackedWriter blocks of 64 packed ints}
   • Lengths --> LengthMinusTermLength^TotalOffsets
   • LengthMinusTermLength: (endOffset - startOffset - termLength) using {@link BlockPackedWriter blocks of 64 packed ints}
   • PayloadLengths --> PayloadLength^{TotalPayloads}
   • TotalPayloads is the sum of frequencies of terms of all fields that have payloads
   • PayloadLength is the payload length encoded using {@link BlockPackedWriter blocks of 64 packed ints}
   • TermAndPayloads --> LZ4-compressed representation of < FieldTermsAndPayLoads >^TotalFields
   • FieldTermsAndPayLoads --> Terms (Payloads)
   • Terms: term bytes
   • Payloads: payload bytes (if the field has payloads)

2. An index file (extension .tvx).

   • VectorIndex (.tvx) --> <Header>, <ChunkIndex>
   • Header --> {@link CodecUtil#writeHeader CodecHeader}
   • ChunkIndex: See {@link CompressingStoredFieldsIndexWriter}