Examples of org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat

Package org.apache.lucene.codecs.lucene40

Examples of org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat

org.apache.lucene.codecs.lucene40.Lucene40RWCodec
Lucene 4.0 Term Vectors format.
Term Vector support is an optional on a field by field basis. It consists of 3 files.
1. The Document Index or .tvx file.
  
  For each document, this stores the offset into the document data (.tvd) and field data (.tvf) files.
  
  DocumentIndex (.tvx) --> Header,<DocumentPosition,FieldPosition> ^NumDocs
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - DocumentPosition --> {@link DataOutput#writeLong UInt64} (offset in the .tvd file)
  - FieldPosition --> {@link DataOutput#writeLong UInt64} (offset in the .tvf file)
2. The Document or .tvd file.
  
  This contains, for each document, the number of fields, a list of the fields with term vector info and finally a list of pointers to the field information in the .tvf (Term Vector Fields) file.
  
  The .tvd file is used to map out the fields that have term vectors stored and where the field information is in the .tvf file.
  
  Document (.tvd) --> Header,<NumFields, FieldNums, FieldPositions> ^NumDocs
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - NumFields --> {@link DataOutput#writeVInt VInt}
  - FieldNums --> <FieldNumDelta> ^NumFields
  - FieldNumDelta --> {@link DataOutput#writeVInt VInt}
  - FieldPositions --> <FieldPositionDelta> ^NumFields-1
  - FieldPositionDelta --> {@link DataOutput#writeVLong VLong}
3. The Field or .tvf file.
  
  This file contains, for each field that has a term vector stored, a list of the terms, their frequencies and, optionally, position, offset, and payload information.
  
  Field (.tvf) --> Header,<NumTerms, Flags, TermFreqs> ^NumFields
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - NumTerms --> {@link DataOutput#writeVInt VInt}
  - Flags --> {@link DataOutput#writeByte Byte}
  - TermFreqs --> <TermText, TermFreq, Positions?, PayloadData?, Offsets?> ^NumTerms
  - TermText --> <PrefixLength, Suffix>
  - PrefixLength --> {@link DataOutput#writeVInt VInt}
  - Suffix --> {@link DataOutput#writeString String}
  - TermFreq --> {@link DataOutput#writeVInt VInt}
  - Positions --> <PositionDelta PayloadLength?>^TermFreq
  - PositionDelta --> {@link DataOutput#writeVInt VInt}
  - PayloadLength --> {@link DataOutput#writeVInt VInt}
  - PayloadData --> {@link DataOutput#writeByte Byte}^{NumPayloadBytes}
  - Offsets --> < {@link DataOutput#writeVInt VInt}, {@link DataOutput#writeVInt VInt}>^TermFreq
  Notes:
  - Flags byte stores whether this term vector has position, offset, payload. information stored.
  - Term byte prefixes are shared. The PrefixLength is the number of initial bytes from the previous term which must be pre-pended to a term's suffix in order to form the term's bytes. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y".
  - PositionDelta is, if payloads are disabled for the term's field, the difference between the position of the current occurrence in the document and the previous occurrence (or zero, if this is the first occurrence in this document). If payloads are enabled for the term's field, then PositionDelta/2 is the difference between the current and the previous position. If payloads are enabled and PositionDelta is odd, then PayloadLength is stored, indicating the length of the payload at the current term position.
  - PayloadData is metadata associated with a term position. If PayloadLength is stored at the current position, then it indicates the length of this payload. If PayloadLength is not stored, then this payload has the same length as the payload at the previous position. PayloadData encodes the concatenated bytes for all of a terms occurrences.
  - Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.

 */
@Deprecated
public class AppendingCodec extends FilterCodec {


  public AppendingCodec() {
    super("Appending", new Lucene40Codec());
  }

View Full Code Here

    throw new UnsupportedOperationException("this codec can only be used for reading");
  }


  @Override
  public FieldsProducer fieldsProducer(SegmentReadState state) throws IOException {
    PostingsReaderBase postings = new Lucene40PostingsReader(state.directory, state.fieldInfos, state.segmentInfo, state.context, state.segmentSuffix);
    
    boolean success = false;
    try {
      FieldsProducer ret = new AppendingTermsReader(
                                                    state.directory,
                                                    state.fieldInfos,
                                                    state.segmentInfo,
                                                    postings,
                                                    state.context,
                                                    state.segmentSuffix,
                                                    state.termsIndexDivisor);
      success = true;
      return ret;
    } finally {
      if (!success) {
        postings.close();
      }
    }
  }

View Full Code Here

 */


class AppendingRWPostingsFormat extends AppendingPostingsFormat {
  @Override
  public FieldsConsumer fieldsConsumer(SegmentWriteState state) throws IOException {
    PostingsWriterBase docsWriter = new Lucene40PostingsWriter(state);
    boolean success = false;
    try {
      FieldsConsumer ret = new AppendingTermsWriter(state, docsWriter, BlockTreeTermsWriter.DEFAULT_MIN_BLOCK_SIZE, BlockTreeTermsWriter.DEFAULT_MAX_BLOCK_SIZE);
      success = true;
      return ret;
    } finally {
      if (!success) {
        docsWriter.close();
      }
    }
  }

View Full Code Here

 * Read-write version of AppendingCodec for testing
 */
public class AppendingRWCodec extends FilterCodec {


  public AppendingRWCodec() {
    super("Appending", new Lucene40RWCodec());
  }

View Full Code Here

    
    dir.close();
  }
  
  public void testDisableImpersonation() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec() };
    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(oldCodecs[random().nextInt(oldCodecs.length)]);
    IndexWriter writer = new IndexWriter(dir, conf);

View Full Code Here

    dir.close();
  }
  
  @Test
  public void testUpdateOldSegments() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec(), new Lucene45RWCodec() };
    Directory dir = newDirectory();
    
    boolean oldValue = OLD_FORMAT_IMPERSONATION_IS_ACTIVE;
    // create a segment with an old Codec
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));

View Full Code Here

    dir.close();
  }
  
  @Test
  public void testUpdateOldSegments() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec(), new Lucene45RWCodec() };
    Directory dir = newDirectory();
    
    boolean oldValue = OLD_FORMAT_IMPERSONATION_IS_ACTIVE;
    // create a segment with an old Codec
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));

View Full Code Here

    
    dir.close();
  }
  
  public void testUpdateOldSegments() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec(), new Lucene45RWCodec() };
    Directory dir = newDirectory();
    
    boolean oldValue = OLD_FORMAT_IMPERSONATION_IS_ACTIVE;
    // create a segment with an old Codec
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));

View Full Code Here

    
    dir.close();
  }
  
  public void testDisableImpersonation() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec() };
    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(oldCodecs[random().nextInt(oldCodecs.length)]);
    IndexWriter writer = new IndexWriter(dir, conf);

View Full Code Here

    dir.close();
  }
  
  @Test
  public void testUpdateOldSegments() throws Exception {
    Codec[] oldCodecs = new Codec[] { new Lucene40RWCodec(), new Lucene41RWCodec(), new Lucene42RWCodec(), new Lucene45RWCodec() };
    Directory dir = newDirectory();
    
    boolean oldValue = OLD_FORMAT_IMPERSONATION_IS_ACTIVE;
    // create a segment with an old Codec
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));

View Full Code Here

0 1

TOP

Related Classes of org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat

org.apache.lucene.analysis.MockAnalyzer

org.apache.lucene.codecs.appending.AppendingCodec

org.apache.lucene.codecs.appending.AppendingPostingsFormat

org.apache.lucene.codecs.appending.AppendingRWCodec

org.apache.lucene.codecs.appending.AppendingRWPostingsFormat

org.apache.lucene.codecs.blocktree.BlockTreeTermsReader

org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter

org.apache.lucene.codecs.BlockTreeTermsReader

org.apache.lucene.codecs.BlockTreeTermsWriter

org.apache.lucene.codecs.Codec

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.