Examples of cascading.operation.Aggregator

cascading.operation.Aggregator
An Aggregator takes the set of all values associated with a unique grouping and returns zero or more values. {@link cascading.operation.aggregator.MaxValue}, {@link cascading.operation.aggregator.MinValue}, {@link cascading.operation.aggregator.Count}, and {@link cascading.operation.aggregator.Average} are good examples.
Aggregator implementations should be reentrant. There is no guarantee an Aggregator instance will be executed in a unique vm, or by a single thread. The {@link #start(cascading.flow.FlowProcess,AggregatorCall)}method provides a mechanism for maintaining a 'context' object to hold intermediate values.
Note {@link TupleEntry} instances are reused internally so should not be stored. Instead use the TupleEntry or Tuplecopy constructors to make safe copies.
Since Aggregators can be chained, and Cascading pipelines all operation results, any Aggregators coming ahead of the current Aggregator must return a value before the {@link #complete(cascading.flow.FlowProcess,AggregatorCall)}method on this Aggregator is called. Subsequently, if any previous Aggregators return more than one Tuple result, this complete() method will be called for each Tuple emitted.
Thus it is a best practice to implement a {@link Buffer} when emitting more than one, or zero Tuple results. @see AggregatorCall @see OperationCall

    }


  @Test
  public void testMinValue()
    {
    Aggregator aggregator = new MinValue();


    Tuple[] arguments = new Tuple[]{new Tuple( new Double( 1.0 ) ), new Tuple( new Double( 3.0 ) ),
                                    new Tuple( new Double( 2.0 ) ), new Tuple( new Double( 4.0 ) ),
                                    new Tuple( new Double( -5.0 ) )};

View Full Code Here

    }


  @Test
  public void testMaxValueNonNumber()
    {
    Aggregator aggregator = new MaxValue();


    Tuple[] arguments = new Tuple[]{new Tuple( 'a' ), new Tuple( 'b' ),
                                    new Tuple( 'c' ), new Tuple( 'd' ),
                                    new Tuple( 'e' )};

View Full Code Here

    }


  @Test
  public void testMinValueNonNumber()
    {
    Aggregator aggregator = new MinValue();


    Tuple[] arguments = new Tuple[]{new Tuple( 'a' ), new Tuple( 'b' ),
                                    new Tuple( 'c' ), new Tuple( 'd' ),
                                    new Tuple( 'e' )};

View Full Code Here

    Pipe pruned = new Each( joined, new Fields( "id2" ), new Identity(), Fields.RESULTS );
//    pruned = new Checkpoint( pruned );
    Pipe merged = new Merge( pruned, rhs );
    Pipe grouped = new GroupBy( merged, new Fields( "id2" ) );
//    Pipe grouped = new GroupBy( Pipe.pipes(  pruned, people  ), new Fields( "id2" ) );
    Aggregator count = new Count( new Fields( "count" ) );
    Pipe counted = new Every( grouped, count );


    String testJoinMerge = "testJoinMergeGroupBy/" + ( ( joined instanceof CoGroup ) ? "cogroup" : "hashjoin" );
    Tap sink = getPlatform().getDelimitedFile( Fields.ALL, true, "\t", null, getOutputPath( testJoinMerge ), SinkMode.REPLACE );

View Full Code Here

    // test that selector against incoming creates proper outgoing
    pipe = new Each( pipe, new Fields( 1 ), new Identity() );


    pipe = new GroupBy( pipe, new Fields( 0 ) );


    Aggregator counter = new Count();


    pipe = new Every( pipe, new Fields( 0 ), counter, new Fields( 0, 1 ) );


    Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );

View Full Code Here


    pipe = new Each( pipe, Fields.ALL, new RegexFilter( "a|b|c" ) );


    pipe = new GroupBy( pipe, new Fields( 0 ) );


    Aggregator counter = new Count();


    pipe = new Every( pipe, new Fields( 0 ), counter, new Fields( 0, 1 ) );


    Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );

View Full Code Here


    pipe = new Each( pipe, new Fields( -1 ), parser, new Fields( -1 ) );


    pipe = new GroupBy( pipe, new Fields( 0 ) );


    Aggregator counter = new Count();


    pipe = new Every( pipe, new Fields( 0 ), counter, new Fields( 0, 1 ) );


    Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );

View Full Code Here

0 1

TOP

Related Classes of cascading.operation.Aggregator

cascading.BasicPipesPlatformTest

cascading.JoinFieldedPipesPlatformTest

cascading.operation.aggregator.AggregatorsTest

com.manning.hip.ch12.cascading.PopularLogResources

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.