Examples of org.apache.lucene.codecs.lucene45.Lucene45DocValuesFormat

Package org.apache.lucene.codecs.lucene45

Examples of org.apache.lucene.codecs.lucene45.Lucene45DocValuesFormat

org.apache.lucene.codecs.lucene45.Lucene45Codec
Lucene 4.5 DocValues format.
Encodes the four per-document value types (Numeric,Binary,Sorted,SortedSet) with these strategies:
{@link DocValuesType#NUMERIC NUMERIC}:
- Delta-compressed: per-document integers written in blocks of 16k. For each block the minimum value in that block is encoded, and each entry is a delta from that minimum value. Each block of deltas is compressed with bitpacking. For more information, see {@link BlockPackedWriter}.
- Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as {@link SmallFloat}), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking ( {@link PackedInts}).
- GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
{@link DocValuesType#BINARY BINARY}:
- Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed directly with multiplication ( {@code docID * length}).
- Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written in blocks of 16k, with the current absolute start for the block, and the average (expected) delta per entry. For each document the deviation from the delta (actual - expected) is written.
- Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written in blocks of 16k, with the current absolute start for the block, and the average (expected) delta per entry. For each chunk the deviation from the delta (actual - expected) is written.
{@link DocValuesType#SORTED SORTED}:
- Sorted: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, along with the per-document ordinals written using one of the numeric strategies above.
{@link DocValuesType#SORTED_SET SORTED_SET}:
- SortedSet: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.
Files:
1. .dvd: DocValues data
2. .dvm: DocValues metadata
1. The DocValues metadata or .dvm file.
  
  For DocValues field, this stores metadata, such as the offset into the DocValues data (.dvd)
  
  DocValues metadata (.dvm) --> Header,<Entry>^NumFields
  - Entry --> NumericEntry | BinaryEntry | SortedEntry | SortedSetEntry
  - NumericEntry --> GCDNumericEntry | TableNumericEntry | DeltaNumericEntry
  - GCDNumericEntry --> NumericHeader,MinValue,GCD
  - TableNumericEntry --> NumericHeader,TableSize, {@link DataOutput#writeLong Int64}^TableSize
  - DeltaNumericEntry --> NumericHeader
  - NumericHeader --> FieldNumber,EntryType,NumericType,MissingOffset,PackedVersion,DataOffset,Count,BlockSize
  - BinaryEntry --> FixedBinaryEntry | VariableBinaryEntry | PrefixBinaryEntry
  - FixedBinaryEntry --> BinaryHeader
  - VariableBinaryEntry --> BinaryHeader,AddressOffset,PackedVersion,BlockSize
  - PrefixBinaryEntry --> BinaryHeader,AddressInterval,AddressOffset,PackedVersion,BlockSize
  - BinaryHeader --> FieldNumber,EntryType,BinaryType,MissingOffset,MinLength,MaxLength,DataOffset
  - SortedEntry --> FieldNumber,EntryType,BinaryEntry,NumericEntry
  - SortedSetEntry --> EntryType,BinaryEntry,NumericEntry,NumericEntry
  - FieldNumber,PackedVersion,MinLength,MaxLength,BlockSize,ValueCount --> {@link DataOutput#writeVInt VInt}
  - EntryType,CompressionType --> {@link DataOutput#writeByte Byte}
  - Header --> {@link CodecUtil#writeHeader CodecHeader}
  - MinValue,GCD,MissingOffset,AddressOffset,DataOffset --> {@link DataOutput#writeLong Int64}
  - TableSize --> {@link DataOutput#writeVInt vInt}
  Sorted fields have two entries: a BinaryEntry with the value metadata, and an ordinary NumericEntry for the document-to-ord metadata.
  
  SortedSet fields have three entries: a BinaryEntry with the value metadata, and two NumericEntries for the document-to-ord-index and ordinal list metadata.
  
  FieldNumber of -1 indicates the end of metadata.
  
  EntryType is a 0 (NumericEntry) or 1 (BinaryEntry)
  
  DataOffset is the pointer to the start of the data in the DocValues data (.dvd)
  
  NumericType indicates how Numeric values will be compressed:
  - 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block.
  - 1 -->, gcd-compressed. When all integers share a common divisor, only quotients are stored using blocks of delta-encoded ints.
  - 2 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.
  BinaryType indicates how Binary values will be stored:
  - 0 --> fixed-width. All values have the same length, addressing by multiplication.
  - 1 -->, variable-width. An address for each value is stored.
  - 2 --> prefix-compressed. An address to the start of every interval'th value is stored.
  MinLength and MaxLength represent the min and max byte[] value lengths for Binary values. If they are equal, then all values are of a fixed size, and can be addressed as DataOffset + (docID * length). Otherwise, the binary values are of variable size, and packed integer metadata (PackedVersion,BlockSize) is written for the addresses.
  MissingOffset points to a byte[] containing a bitset of all documents that had a value for the field. If its -1, then there are no missing values.
2. The DocValues data or .dvd file.
  
  For DocValues field, this stores the actual per-document data (the heavy-lifting)
  
  DocValues data (.dvd) --> Header,<NumericData | BinaryData | SortedData>^NumFields
  - NumericData --> DeltaCompressedNumerics | TableCompressedNumerics | GCDCompressedNumerics
  - BinaryData --> {@link DataOutput#writeByte Byte}^DataLength,Addresses
  - SortedData --> {@link FST FST<Int64>}
  - DeltaCompressedNumerics --> {@link BlockPackedWriter BlockPackedInts(blockSize=16k)}
  - TableCompressedNumerics --> {@link PackedInts PackedInts}
  - GCDCompressedNumerics --> {@link BlockPackedWriter BlockPackedInts(blockSize=16k)}
  - Addresses --> {@link MonotonicBlockPackedWriter MonotonicBlockPackedInts(blockSize=16k)}
  SortedSet entries store the list of ordinals in their BinaryData as a sequences of increasing {@link DataOutput#writeVLong vLong}s, delta-encoded.
@lucene.experimental

    Directory directory = newDirectory();
    // we don't use RandomIndexWriter because it might add more docvalues than we expect !!!!1
    IndexWriterConfig iwc = newIndexWriterConfig(TEST_VERSION_CURRENT, analyzer);
    final DocValuesFormat fast = DocValuesFormat.forName("Lucene45");
    final DocValuesFormat slow = DocValuesFormat.forName("SimpleText");
    iwc.setCodec(new Lucene45Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        if ("dv1".equals(field)) {
          return fast;
        } else {

View Full Code Here

        dvFormat = DocValuesFormat.forName("Lucene45");
      } else {
        dvFormat = DocValuesFormat.forName(TEST_DOCVALUESFORMAT);
      }
      
      codec = new Lucene45Codec() {       
        @Override
        public PostingsFormat getPostingsFormatForField(String field) {
          return format;
        }

View Full Code Here

    }
    dir.close();
  }
  
  public void testSameCodecDifferentInstance() throws Exception {
    Codec codec = new Lucene45Codec() {
      @Override
      public PostingsFormat getPostingsFormatForField(String field) {
        if ("id".equals(field)) {
          return new Pulsing41PostingsFormat(1);
        } else if ("date".equals(field)) {

View Full Code Here

    };
    doTestMixedPostings(codec);
  }
  
  public void testSameCodecDifferentParams() throws Exception {
    Codec codec = new Lucene45Codec() {
      @Override
      public PostingsFormat getPostingsFormatForField(String field) {
        if ("id".equals(field)) {
          return new Pulsing41PostingsFormat(1);
        } else if ("date".equals(field)) {

View Full Code Here

    // (and maybe their params, too) to infostream on flush and merge.
    // otherwise in a real debugging situation we won't know whats going on!
    if (LuceneTestCase.VERBOSE) {
      System.out.println("forcing postings format to:" + format);
    }
    return new Lucene45Codec() {
      @Override
      public PostingsFormat getPostingsFormatForField(String field) {
        return format;
      }
    };

View Full Code Here

    // (and maybe their params, too) to infostream on flush and merge.
    // otherwise in a real debugging situation we won't know whats going on!
    if (LuceneTestCase.VERBOSE) {
      System.out.println("forcing docvalues format to:" + format);
    }
    return new Lucene45Codec() {
      @Override
      public DocValuesFormat getDocValuesFormatForField(String field) {
        return format;
      }
    };

View Full Code Here

 */
public class TestAllFilesHaveCodecHeader extends LuceneTestCase {
  public void test() throws Exception {
    Directory dir = newDirectory();
    IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));
    conf.setCodec(new Lucene45Codec());
    // riw should sometimes create docvalues fields, etc
    RandomIndexWriter riw = new RandomIndexWriter(random(), dir, conf);
    Document doc = new Document();
    // these fields should sometimes get term vectors, etc
    Field idField = newStringField("id", "", Field.Store.NO);

View Full Code Here

  private final StoredFieldsFormat storedFields = new AssertingStoredFieldsFormat();
  private final DocValuesFormat docValues = new AssertingDocValuesFormat();
  private final NormsFormat norms = new AssertingNormsFormat();


  public AssertingCodec() {
    super("Asserting", new Lucene45Codec());
  }

View Full Code Here

  // these go to disk for all docvalues/norms datastructures
  private final DocValuesFormat docValues = new DiskDocValuesFormat();
  private final NormsFormat norms = new DiskNormsFormat();


  public CheapBastardCodec() {
    super("CheapBastard", new Lucene45Codec());
  }

View Full Code Here

  
  public void testWriteReadMerge() throws IOException {
    // get another codec, other than the default: so we are merging segments across different codecs
    final Codec otherCodec;
    if ("SimpleText".equals(Codec.getDefault().getName())) {
      otherCodec = new Lucene45Codec();
    } else {
      otherCodec = new SimpleTextCodec();
    }
    Directory dir = newDirectory();
    IndexWriterConfig iwConf = newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random()));

View Full Code Here

0 1 2 3

TOP

Related Classes of org.apache.lucene.codecs.lucene45.Lucene45DocValuesFormat

org.apache.lucene.codecs.asserting.AssertingCodec

org.apache.lucene.codecs.cheapbastard.CheapBastardCodec

org.apache.lucene.codecs.compressing.CompressingCodec

org.apache.lucene.codecs.diskdv.DiskDocValuesFormat

org.apache.lucene.codecs.diskdv.DiskNormsFormat

org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat

org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat2

org.apache.lucene.index.BaseStoredFieldsFormatTestCase

org.apache.lucene.index.BinaryDocValues

org.apache.lucene.index.RandomAccessOrds

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.