Examples of io.airlift.stats.QuantileDigest$SizeOf

io.airlift.stats.QuantileDigest
eerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.132.7343, a data structure for approximating quantiles by trading off error with memory requirements.

The size of the digest is adjusted dynamically to achieve the error bound and requires O(log2(U) / maxError) space, where U is the number of bits needed to represent the domain of the values added to the digest. The error is defined as the discrepancy between the real rank of the value returned in a quantile query and the rank corresponding to the queried quantile.

Thus, for a query for quantile q that returns value v, the error is |rank(v) - q * N| / N, where N is the number of elements added to the digest and rank(v) is the real rank of v

This class also supports exponential decay. The implementation is based on the ideas laid out in http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.3978


                // skip null values
                if (!values.isNull() && !weights.isNull() && sampleWeight > 0) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.digest.estimatedInMemorySizeInBytes();

View Full Code Here

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here


                // skip null values
                if (!values.isNull() && sampleWeight > 0) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here


                // skip null values
                if (!values.isNull() && (masks == null || masks.getBoolean())) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here


                // skip null values
                if (!values.isNull() && !weights.isNull() && (masks == null || masks.getBoolean())) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.digest.estimatedInMemorySizeInBytes();

View Full Code Here

                if (!intermediates.isNull()) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice().getInput();

View Full Code Here


                // skip null values
                if (!values.isNull(position) && (masks == null || masks.getBoolean(position))) {
                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    sizeOfValues -= currentValue.getDigest().estimatedInMemorySizeInBytes();

View Full Code Here

                if (!intermediates.isNull(position)) {
                    long groupId = groupIdsBlock.getGroupId(position);


                    DigestAndPercentile currentValue = digests.get(groupId);
                    if (currentValue == null) {
                        currentValue = new DigestAndPercentile(new QuantileDigest(0.01));
                        digests.set(groupId, currentValue);
                        sizeOfValues += currentValue.getDigest().estimatedInMemorySizeInBytes();
                    }


                    SliceInput input = intermediates.getSlice(position).getInput();

View Full Code Here

0 1 2 3 4

TOP

Related Classes of io.airlift.stats.QuantileDigest$SizeOf

com.facebook.presto.operator.aggregation.ApproximateDoublePercentileAggregations

com.facebook.presto.operator.aggregation.ApproximateLongPercentileAggregations

com.facebook.presto.operator.aggregation.ApproximatePercentileAggregation$ApproximatePercentileGroupedAccumulator

com.facebook.presto.operator.aggregation.ApproximatePercentileWeightedAggregation$ApproximatePercentileWeightedGroupedAccumulator

com.facebook.presto.operator.aggregation.DoubleApproximatePercentileAggregation

com.facebook.presto.operator.aggregation.DoubleApproximatePercentileWeightedAggregation

com.facebook.presto.operator.aggregation.LongApproximatePercentileAggregation

com.facebook.presto.operator.aggregation.LongApproximatePercentileWeightedAggregation

com.google.common.util.concurrent.AtomicDouble

java.util.concurrent.atomic.AtomicInteger

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.