Examples of net.agkn.hll.HLL

net.agkn.hll.HLL
inria.fr/flajolet/Publications/FlFuGaMe07.pdf"> 'HyperLogLog' data structure and algorithm is used, which combines both probabilistic and non-probabilistic techniques to improve the accuracy and storage requirements of the original algorithm.
More specifically, initializing and storing a new {@link HLL} willallocate a sentinel value symbolizing the empty set ( {@link HLLType#EMPTY}). After adding the first few values, a sorted list of unique integers is stored in a {@link HLLType#EXPLICIT} hash set. When configured, accuracy canbe sacrificed for memory footprint: the values in the sorted list are "promoted" to a " {@link HLLType#SPARSE}" map-based HyperLogLog structure. Finally, when enough registers are set, the map-based HLL will be converted to a bit-packed " {@link HLLType#FULL}" HyperLogLog structure.
This data structure is interoperable with the implementations found at:
- postgresql-hll, and
- js-hll
when properly serialized. @author timon

        final FileWriter output = openOutput(schemaVersion, "sparse_sparse", TestType.UNION);


        final Random random = new Random(SEED);


        // the accumulator, starts empty
        final HLL hll = newHLL(HLLType.EMPTY);
        final HLL emptyHLL = newHLL(HLLType.EMPTY);


        cumulativeUnionLine(output, hll, emptyHLL, schemaVersion);


        for(int i=0; i<SPARSE_THRESHOLD; i++) {
            // make a SPARSE set and populate with cardinality 1
            final HLL sparseHLL = newHLL(HLLType.SPARSE);
            final int registerIndex = Math.abs(random.nextInt()) % REGISTER_COUNT;
            final int registerValue = ((Math.abs(random.nextInt()) % REGISTER_MAX_VALUE) + 1);
            final long rawValue = constructHLLValue(LOG2M, registerIndex, registerValue);
            sparseHLL.addRaw(rawValue);


            cumulativeUnionLine(output, hll, sparseHLL, schemaVersion);
        }


        output.flush();

View Full Code Here

        final FileWriter output = openOutput(schemaVersion, "probabilistic_probabilistic", TestType.UNION);


        final Random random = new Random(SEED);


        // the accumulator, starts empty
        final HLL hll = newHLL(HLLType.EMPTY);
        final HLL emptyHLL = newHLL(HLLType.EMPTY);
        cumulativeUnionLine(output, hll, emptyHLL, schemaVersion);


        for(int i=0; i<1000/*number of rows to generate*/; i++) {
            // make a FULL set and populate with
            final HLL fullHLL = newHLL(HLLType.FULL);
            final int elementCount = random.nextInt(10000/*arbitrary maximum cardinality*/);
            for(int j=0;j<elementCount;j++) {
                fullHLL.addRaw(random.nextLong());
            }


            cumulativeUnionLine(output, hll, fullHLL, schemaVersion);
        }

View Full Code Here

        final FileWriter output = openOutput(schemaVersion, "comprehensive", TestType.UNION);


        final Random random = new Random(SEED);


        // the accumulator, starts empty
        final HLL hll = newHLL(HLLType.EMPTY);
        final HLL emptyHLL = newHLL(HLLType.EMPTY);


        cumulativeUnionLine(output, hll, emptyHLL, schemaVersion);


        for(int i=0; i<1000/*number of rows to generate*/; i++) {
            final HLL randomHLL = generateRandomHLL(random);
            cumulativeUnionLine(output, hll, randomHLL, schemaVersion);
        }


        output.flush();
        output.close();

View Full Code Here

                break;
            default:
                throw new RuntimeException("We should never be here.");
        }


        final HLL hll = newHLL(HLLType.EMPTY);
        for(int i=0; i<cardinalityBaseline; i++) {
            hll.addRaw(random.nextLong());
        }
        for(int i=0; i<random.nextInt(cardinalityCap - cardinalityBaseline); i++) {
            hll.addRaw(random.nextLong());
        }


        return hll;
    }

View Full Code Here

0 1

TOP

Related Classes of net.agkn.hll.HLL

it.unimi.dsi.fastutil.ints.Int2ByteOpenHashMap

it.unimi.dsi.fastutil.longs.LongOpenHashSet

net.agkn.hll.IntegrationTestGenerator

net.agkn.hll.serialization.HLLMetadata

net.agkn.hll.serialization.HLLSerializationTest

net.agkn.hll.serialization.IHLLMetadata

net.agkn.hll.serialization.ISchemaVersion

net.agkn.hll.serialization.IWordDeserializer

net.agkn.hll.serialization.IWordSerializer

net.agkn.hll.util.BitVector

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.