Examples of parquet.column.values.rle.RunLengthBitPackingHybridEncoder

parquet.column.values.rle.RunLengthBitPackingHybridEncoder

lways bit-pack a multiple of 8 values at a time, so we only store the number of values / 8 bit-pack-count := (number of values in this run) / 8 bit-packed-values := bit packed back to back, from LSB to MSB rle-run :=

rle-header := varint-encode( (number of times repeated) << 1) repeated-value := value that is repeated, using a fixed-width of round-up-to-next-byte(bit-width)} NOTE: this class is only responsible for creating and returning the {@code }portion of the above grammar. The {@code } portion is done by{@link RunLengthBitPackingHybridValuesWriter}

Only supports values >= 0 // TODO: is that ok? Should we make a signed version? @author Alex Levenson

      int maxDicId = getDictionarySize() - 1;
      if (DEBUG) LOG.debug("max dic id " + maxDicId);
      int bitWidth = BytesUtils.getWidthFromMaxInt(maxDicId);


      // TODO: what is a good initialCapacity?
      RunLengthBitPackingHybridEncoder encoder = new RunLengthBitPackingHybridEncoder(bitWidth, 64 * 1024);
      IntIterator iterator = encodedValues.iterator();
      try {
        while (iterator.hasNext()) {
          encoder.writeInt(iterator.next());
        }
        // encodes the bit width
        byte[] bytesHeader = new byte[] { (byte) bitWidth };
        BytesInput rleEncodedBytes = encoder.toBytes();
        if (DEBUG) LOG.debug("rle encoded bytes " + rleEncodedBytes.size());
        BytesInput bytes = concat(BytesInput.from(bytesHeader), rleEncodedBytes);
        if (firstPage && ((bytes.size() + dictionaryByteSize) > rawDataByteSize)) {
          fallBackToPlainEncoding();
        } else {

Related Classes of parquet.column.values.rle.RunLengthBitPackingHybridEncoder

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.