Examples of com.clearspring.analytics.stream.cardinality.HyperLogLogPlus

Package com.clearspring.analytics.stream.cardinality

Examples of com.clearspring.analytics.stream.cardinality.HyperLogLogPlus

com.clearspring.analytics.stream.cardinality.HyperLogLogPlus
c.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf

Brief HyperLogLog++ Overview
Uses 64 bit hashing instead of 32 Has two representation modes: sparse and normal
'normal' is approximately the same as regular hyperloglog (still uses 64 bits)
'sparse' handles lower cardinality values with a highly accurate but poorly scaling strategy and leverages data compression to compete with 'normal' for as long as possible (sparse has the advantage on accuracy per unit of memory at low cardinality but quickly falls behind).

    
    Object result = udafEvaluator.terminate(agg);
    Assert.assertNotNull(result);
    
    byte[] b = ((JavaBinaryObjectInspector) finalOutputOi).getPrimitiveJavaObject(result);
    HyperLogLogPlus hll = HyperLogLogPlus.Builder.build( b );
    Long cardEst = hll.cardinality();
    
    LOG.info("cardEst = " + cardEst);
    
    int actualUniques = h.keySet().size();
    LOG.info("actualUniques = " + actualUniques);

View Full Code Here

    
      byte[] bref1 = this.binary1Inspector.getPrimitiveJavaObject(blobObj1);
      byte[] bref2 = this.binary2Inspector.getPrimitiveJavaObject(blobObj2);
    
      if(bref1 != null && bref2 != null) {
        HyperLogLogPlus hll1 = HyperLogLogPlus.Builder.build( bref1 );
        HyperLogLogPlus hll2 = HyperLogLogPlus.Builder.build( bref2 );
      
        ICardinality merged = hll1.merge(hll2);
      return merged.getBytes();
      } else {
        return null;

View Full Code Here

    return precision != 0;
  }


  public void init(int precision) {
    this.precision = precision;
    hll = new HyperLogLogPlus(precision);
  }

View Full Code Here

      ///HyperLogLogPlus hll = HyperLogLogPlus.Builder.build( bref.getData() );
      
      byte[] bref = this.binaryInspector.getPrimitiveJavaObject(blobObj);
      if(bref == null)
        return null;
      HyperLogLogPlus hll = HyperLogLogPlus.Builder.build( bref );
      
      
      return hll.cardinality();
    } catch(Exception e) {
      LOG.error("Error", e);
      throw new HiveException(e);
    }

View Full Code Here

    this.outputOption = outputOption;
  }


  @Override
  public ICardinality initialize() {
    return new HyperLogLogPlus(precision);
  }

View Full Code Here

TOP

Related Classes of com.clearspring.analytics.stream.cardinality.HyperLogLogPlus

brickhouse.udf.hll.CombineHyperLogLogUDF

brickhouse.udf.hll.EstimateCardinalityUDF

brickhouse.udf.hll.HLLBuffer

brickhouse.udf.hll.HyperLogLogUDAFTest

com.liveramp.cascading_ext.combiner.lib.HyperLogLogExactAggregator

java.io.ByteArrayInputStream

java.io.DataOutputStream

java.io.ByteArrayOutputStream

java.io.DataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.