Examples of org.apache.lucene.codecs.lucene49.Lucene49NormsFormat

Package org.apache.lucene.codecs.lucene49

Examples of org.apache.lucene.codecs.lucene49.Lucene49NormsFormat

org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat
Lucene 4.9 Score normalization format.
Encodes normalization values with these strategies:
- Uncompressed: when values fit into a single byte and would require more than 4 bits per value, they are just encoded as an uncompressed byte array.
- Constant: when there is only one value present for the entire field, no actual data is written: this constant is encoded in the metadata
- Table-compressed: when the number of unique values is very small (< 64), and when there are unused "gaps" in the range of values used (such as {@link SmallFloat}), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking ( {@link PackedInts}).
- Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see {@link BlockPackedWriter}. This is only used when norms of larger than one byte are present.
Files:
1. .nvd: Norms data
2. .nvm: Norms metadata
1. The Norms metadata or .nvm file.
  
  For each norms field, this stores metadata, such as the offset into the Norms data (.nvd)
  
  Norms metadata (.dvm) --> Header,<Entry>^NumFields,Footer
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - Entry --> FieldNumber,Type,Offset
  - FieldNumber --> {@link DataOutput#writeVInt vInt}
  - Type --> {@link DataOutput#writeByte Byte}
  - Offset --> {@link DataOutput#writeLong Int64}
  - Footer --> {@link CodecUtil#writeFooter CodecFooter}
  FieldNumber of -1 indicates the end of metadata.
  
  Offset is the pointer to the start of the data in the norms data (.nvd), or the singleton value for Constant
  
  Type indicates how Numeric values will be compressed:
  - 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block.
  - 1 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.
  - 2 --> constant. When there is a single value for the entire field.
  - 3 --> uncompressed: Values written as a simple byte[].
2. The Norms data or .nvd file.
  
  For each Norms field, this stores the actual per-document data (the heavy-lifting)
  
  Norms data (.nvd) --> Header,<Uncompressed | TableCompressed | DeltaCompressed>^NumFields,Footer
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - Uncompressed --> {@link DataOutput#writeByte Byte}^maxDoc
  - TableCompressed --> PackedIntsVersion,Table,BitPackedData
  - Table --> TableSize, {@link DataOutput#writeLong int64}^TableSize
  - BitpackedData --> {@link PackedInts}
  - DeltaCompressed --> PackedIntsVersion,BlockSize,DeltaCompressedData
  - DeltaCompressedData --> {@link BlockPackedWriter BlockPackedWriter(blockSize=16k)}
  - PackedIntsVersion,BlockSize,TableSize --> {@link DataOutput#writeVInt vInt}
  - Footer --> {@link CodecUtil#writeFooter CodecFooter}
@lucene.experimental

    savedCodec = Codec.getDefault();
    // currently only these codecs that support random access ordinals
    int victim = random().nextInt(2);
    switch(victim) {
      case 0:  Codec.setDefault(TestUtil.alwaysDocValuesFormat(new DirectDocValuesFormat()));
      default: Codec.setDefault(TestUtil.alwaysDocValuesFormat(new Lucene49DocValuesFormat()));
    }
  }

View Full Code Here

    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    
    Document doc = new Document();

View Full Code Here

    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setMergePolicy(NoMergePolicy.INSTANCE); // disable merges to simplify test assertions.
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    Document doc = new Document();
    doc.add(new StringField("id", "d0", Store.NO));

View Full Code Here

    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    
    Document doc = new Document();

View Full Code Here

    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setMergePolicy(NoMergePolicy.INSTANCE); // disable merges to simplify test assertions.
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    Document doc = new Document();
    doc.add(new StringField("id", "d0", Store.NO));

View Full Code Here


  };


  public void testRamBytesUsed() throws IOException {
    Directory dir = newDirectory();
    IndexWriterConfig cfg = new IndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random())).setCodec(TestUtil.alwaysDocValuesFormat(new Lucene49DocValuesFormat()));
    RandomIndexWriter iw = new RandomIndexWriter(random(), dir, cfg);
    final int maxDoc = TestUtil.nextInt(random(), 10, 1000);
    final int maxTermLength = TestUtil.nextInt(random(), 1, 4);
    for (int i = 0; i < maxDoc; ++i) {
      Document d = new Document();

View Full Code Here

0 1 2

TOP

Related Classes of org.apache.lucene.codecs.lucene49.Lucene49NormsFormat

org.apache.lucene.codecs.asserting.AssertingCodec

org.apache.lucene.codecs.cheapbastard.CheapBastardCodec

org.apache.lucene.codecs.compressing.CompressingCodec

org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat

org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat2

org.apache.lucene.index.BaseStoredFieldsFormatTestCase

org.apache.lucene.index.BinaryDocValues

org.apache.lucene.index.FieldInfo

org.apache.lucene.index.NumericDocValues

org.apache.lucene.index.RandomAccessOrds

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.