Examples of Lucene49DocValuesFormat

org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat
Lucene 4.9 DocValues format.
Encodes the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) with these strategies:
{@link DocValuesType#NUMERIC NUMERIC}:
- Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see {@link DirectWriter}.
- Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as {@link SmallFloat}), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking ( {@link DirectWriter}).
- GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
- Monotonic-compressed: when all numbers are monotonically increasing offsets, they are written as blocks of bitpacked integers, encoding the deviation from the expected delta.
{@link DocValuesType#BINARY BINARY}:
- Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed directly with multiplication ( {@code docID * length}).
- Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written as Monotonic-compressed numerics.
- Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed numerics.
{@link DocValuesType#SORTED SORTED}:
- Sorted: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, along with the per-document ordinals written using one of the numeric strategies above.
{@link DocValuesType#SORTED_SET SORTED_SET}:
- SortedSet: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.
{@link DocValuesType#SORTED_NUMERIC SORTED_NUMERIC}:
- SortedNumeric: a value list and per-document index into this list are written using the numeric strategies above.
Files:
1. .dvd: DocValues data
2. .dvm: DocValues metadata
1. The DocValues metadata or .dvm file.
  
  For DocValues field, this stores metadata, such as the offset into the DocValues data (.dvd)
  
  DocValues metadata (.dvm) --> Header,<Entry>^NumFields,Footer
  - Entry --> NumericEntry | BinaryEntry | SortedEntry | SortedSetEntry | SortedNumericEntry
  - NumericEntry --> GCDNumericEntry | TableNumericEntry | DeltaNumericEntry
  - GCDNumericEntry --> NumericHeader,MinValue,GCD,BitsPerValue
  - TableNumericEntry --> NumericHeader,TableSize, {@link DataOutput#writeLong Int64}^TableSize,BitsPerValue
  - DeltaNumericEntry --> NumericHeader,MinValue,BitsPerValue
  - MonotonicNumericEntry --> NumericHeader,PackedVersion,BlockSize
  - NumericHeader --> FieldNumber,EntryType,NumericType,MissingOffset,DataOffset,Count,EndOffset
  - BinaryEntry --> FixedBinaryEntry | VariableBinaryEntry | PrefixBinaryEntry
  - FixedBinaryEntry --> BinaryHeader
  - VariableBinaryEntry --> BinaryHeader,AddressOffset,PackedVersion,BlockSize
  - PrefixBinaryEntry --> BinaryHeader,AddressInterval,AddressOffset,PackedVersion,BlockSize
  - BinaryHeader --> FieldNumber,EntryType,BinaryType,MissingOffset,MinLength,MaxLength,DataOffset
  - SortedEntry --> FieldNumber,EntryType,BinaryEntry,NumericEntry
  - SortedSetEntry --> EntryType,BinaryEntry,NumericEntry,NumericEntry
  - SortedNumericEntry --> EntryType,NumericEntry,NumericEntry
  - FieldNumber,PackedVersion,MinLength,MaxLength,BlockSize,ValueCount --> {@link DataOutput#writeVInt VInt}
  - EntryType,CompressionType --> {@link DataOutput#writeByte Byte}
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - MinValue,GCD,MissingOffset,AddressOffset,DataOffset,EndOffset --> {@link DataOutput#writeLong Int64}
  - TableSize,BitsPerValue --> {@link DataOutput#writeVInt vInt}
  - Footer --> {@link CodecUtil#writeFooter CodecFooter}
  Sorted fields have two entries: a BinaryEntry with the value metadata, and an ordinary NumericEntry for the document-to-ord metadata.
  
  SortedSet fields have three entries: a BinaryEntry with the value metadata, and two NumericEntries for the document-to-ord-index and ordinal list metadata.
  
  SortedNumeric fields have two entries: A NumericEntry with the value metadata, and a numeric entry with the document-to-value index.
  
  FieldNumber of -1 indicates the end of metadata.
  
  EntryType is a 0 (NumericEntry) or 1 (BinaryEntry)
  
  DataOffset is the pointer to the start of the data in the DocValues data (.dvd)
  
  EndOffset is the pointer to the end of the data in the DocValues data (.dvd)
  
  NumericType indicates how Numeric values will be compressed:
  - 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block.
  - 1 -->, gcd-compressed. When all integers share a common divisor, only quotients are stored using blocks of delta-encoded ints.
  - 2 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.
  BinaryType indicates how Binary values will be stored:
  - 0 --> fixed-width. All values have the same length, addressing by multiplication.
  - 1 -->, variable-width. An address for each value is stored.
  - 2 --> prefix-compressed. An address to the start of every interval'th value is stored.
  MinLength and MaxLength represent the min and max byte[] value lengths for Binary values. If they are equal, then all values are of a fixed size, and can be addressed as DataOffset + (docID * length). Otherwise, the binary values are of variable size, and packed integer metadata (PackedVersion,BlockSize) is written for the addresses.
  MissingOffset points to a byte[] containing a bitset of all documents that had a value for the field. If its -1, then there are no missing values.
  Checksum contains the CRC32 checksum of all bytes in the .dvm file up until the checksum. This is used to verify integrity of the file on opening the index.
2. The DocValues data or .dvd file.
  
  For DocValues field, this stores the actual per-document data (the heavy-lifting)
  
  DocValues data (.dvd) --> Header,<NumericData | BinaryData | SortedData>^NumFields,Footer
  - NumericData --> DeltaCompressedNumerics | TableCompressedNumerics | GCDCompressedNumerics
  - BinaryData --> {@link DataOutput#writeByte Byte}^DataLength,Addresses
  - SortedData --> {@link FST FST<Int64>}
  - DeltaCompressedNumerics,TableCompressedNumerics,GCDCompressedNumerics --> {@link DirectWriter PackedInts}
  - Addresses --> {@link MonotonicBlockPackedWriter MonotonicBlockPackedInts(blockSize=16k)}
  - Footer --> {@link CodecUtil#writeFooter CodecFooter}
@lucene.experimental

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

        new AssertingPostingsFormat(),
        new MemoryPostingsFormat(true, random.nextFloat()),
        new MemoryPostingsFormat(false, random.nextFloat()));
    
    addDocValues(avoidCodecs,
        new Lucene49DocValuesFormat(),
        new MemoryDocValuesFormat(),
        new SimpleTextDocValuesFormat(),
        new AssertingDocValuesFormat());


    Collections.shuffle(formats, random);

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat


  protected IndexWriterConfig newIndexWriterConfig(Random random) {
    final IndexWriterConfig indexWriterConfig = LuceneTestCase.newIndexWriterConfig(random, LuceneTestCase.TEST_VERSION_CURRENT, new MockAnalyzer(random));
    //TODO can we randomly choose a doc-values supported format?
    if (needsDocValues())
      indexWriterConfig.setCodec( TestUtil.alwaysDocValuesFormat(new Lucene49DocValuesFormat()));;
    return indexWriterConfig;
  }

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

    savedCodec = Codec.getDefault();
    // currently only these codecs that support random access ordinals
    int victim = random().nextInt(2);
    switch(victim) {
      case 0:  Codec.setDefault(TestUtil.alwaysDocValuesFormat(new DirectDocValuesFormat()));
      default: Codec.setDefault(TestUtil.alwaysDocValuesFormat(new Lucene49DocValuesFormat()));
    }
  }

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    
    Document doc = new Document();

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setMergePolicy(NoMergePolicy.INSTANCE); // disable merges to simplify test assertions.
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    Document doc = new Document();
    doc.add(new StringField("id", "d0", Store.NO));

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    
    Document doc = new Document();

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setMergePolicy(NoMergePolicy.INSTANCE); // disable merges to simplify test assertions.
    conf.setCodec(new Lucene49Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return new Lucene49DocValuesFormat();
      }
    });
    IndexWriter writer = new IndexWriter(dir, conf);
    Document doc = new Document();
    doc.add(new StringField("id", "d0", Store.NO));

View Full Code Here

Examples of org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat


  };


  public void testRamBytesUsed() throws IOException {
    Directory dir = newDirectory();
    IndexWriterConfig cfg = new IndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random())).setCodec(TestUtil.alwaysDocValuesFormat(new Lucene49DocValuesFormat()));
    RandomIndexWriter iw = new RandomIndexWriter(random(), dir, cfg);
    final int maxDoc = TestUtil.nextInt(random(), 10, 1000);
    final int maxTermLength = TestUtil.nextInt(random(), 1, 4);
    for (int i = 0; i < maxDoc; ++i) {
      Document d = new Document();

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.