Examples of QuantileDigest

io.airlift.stats.QuantileDigest
eerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.132.7343, a data structure for approximating quantiles by trading off error with memory requirements.

The size of the digest is adjusted dynamically to achieve the error bound and requires O(log2(U) / maxError) space, where U is the number of bits needed to represent the domain of the values added to the digest. The error is defined as the discrepancy between the real rank of the value returned in a quantile query and the rank corresponding to the queried quantile.

Thus, for a query for quantile q that returns value v, the error is |rank(v) - q * N| / N, where N is the number of elements added to the digest and rank(v) is the real rank of v

This class also supports exponential decay. The implementation is based on the ideas laid out in http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.3978

Examples of io.airlift.stats.QuantileDigest


                // skip null values
                if (!values.isNull() && !weights.isNull() && sampleWeight > 0) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.digest.estimatedInMemorySizeInBytes();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest


                // skip null values
                if (!values.isNull() && sampleWeight > 0) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest


                // skip null values
                if (!values.isNull() && (masks == null || masks.getBoolean())) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest


                // skip null values
                if (!values.isNull() && !weights.isNull() && (masks == null || masks.getBoolean())) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.digest.estimatedInMemorySizeInBytes();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest


                // skip null values
                if (!values.isNull(position) && (masks == null || masks.getBoolean(position))) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

Examples of io.airlift.stats.QuantileDigest

                if (!intermediates.isNull(position)) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice(position).getInput();

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.