Examples of cascading.tuple.TupleEntryCollector

cascading.tuple.TupleEntryCollector
Interface TupleEntryCollector is used to allow {@link cascading.operation.BaseOperation} instances to emitone or more result {@link Tuple} values.
The general rule in Cascading is if you are handed a Tuple, you cannot change or cache it. Attempts at modifying such a Tuple will result in an Exception. Preventing caching is harder, see below.
If you create the Tuple, you can re-use or modify it.
When calling {@link #add(Tuple)} or {@link #add(TupleEntry)}, you are passing a Tuple to the down stream pipes and operations. Since no downstream operation may modify or cache the Tuple instance, it is safe to re-use the Tuple instance when {@code add()} returns.
That said, Tuple copies do get cached in order to perform specific operations in the underlying platforms. Currently only a shallow copy is made (via the {@link Tuple} copy constructor). Thus, any mutable type or collectionplaced inside a Tuple will not be copied, but will likely be cached if a copy of the Tuple passed downstream is copied.
So any subsequent changes to that nested type or collection will be reflected in the cached copy, a likely source of hard to find errors.
There is currently no way to specify that a deep copy must be performed when making a Tuple copy.

  final Tap trap;
  final String trapName;


  static TupleEntryCollector getTrapCollector( Tap trap, FlowProcess flowProcess )
    {
    TupleEntryCollector trapCollector = trapCollectors.get( trap );


    if( trapCollector == null )
      {
      try
        {

View Full Code Here


    function = each.getFunction();


    operationCall.setArguments( argumentsEntry );


    operationCall.setOutputCollector( new TupleEntryCollector( getOperationDeclaredFields() )
    {
    @Override
    protected void collect( TupleEntry input ) throws IOException
      {
      Tuple outgoing = outgoingBuilder.makeResult( incomingEntry.getTuple(), input.getTuple() );

View Full Code Here

    {
    super.initialize();


    buffer = every.getBuffer();


    outputCollector = new TupleEntryCollector( getOperationDeclaredFields() )
    {
    @Override
    protected void collect( TupleEntry resultEntry ) throws IOException
      {
      Tuple outgoing = outgoingBuilder.makeResult( incomingEntry.getTuple(), resultEntry.getTuple() );

View Full Code Here

    {
    super.initialize();


    aggregator = every.getAggregator();


    outputCollector = new TupleEntryCollector( getOperationDeclaredFields() )
    {
    @Override
    protected void collect( TupleEntry resultEntry ) throws IOException
      {
      Tuple outgoing = outgoingBuilder.makeResult( incomingEntry.getTuple(), resultEntry.getTuple() );

View Full Code Here

    }


  @Override
  public TupleEntryCollector openSystemIntermediateForWrite() throws IOException
    {
    return new TupleEntryCollector( Fields.size( 2 ) )
    {
    @Override
    protected void collect( TupleEntry tupleEntry )
      {
      try

View Full Code Here

    {
    String tapPath = getOutputPath( "tapreplace" );


    Tap tap = getPlatform().getTextFile( tapPath, SinkMode.KEEP );


    TupleEntryCollector collector = tap.openForWrite( getPlatform().getFlowProcess() ); // casting for test


    for( int i = 0; i < 100; i++ )
      collector.add( new Tuple( "string", "" + i, i ) );


    collector.close();


    tap = getPlatform().getTextFile( tapPath, SinkMode.REPLACE );


    collector = tap.openForWrite( getPlatform().getFlowProcess() ); // casting for test


    for( int i = 0; i < 100; i++ )
      collector.add( new Tuple( "string", "" + i, i ) );


    collector.close();
    }

View Full Code Here


    Hfs temp = new Hfs( new TextLine(), statePath, SinkMode.REPLACE );


    try
      {
      TupleEntryCollector writer = temp.openForWrite( new HadoopFlowProcess( conf ) );


      writer.add( new Tuple( stepState ) );


      writer.close();
      }
    catch( IOException exception )
      {
      throw new FlowException( "unable to write step state to Hadoop FS: " + temp.getIdentifier() );
      }

View Full Code Here

      this.partitionEntry.setTuple( partitionTuple );
      }


    TupleEntryCollector getCollector( String path )
      {
      TupleEntryCollector collector = collectors.get( path );


      if( collector != null )
        return collector;


      try

View Full Code Here

        }
      }


    public void closeCollector( String path )
      {
      TupleEntryCollector collector = collectors.get( path );
      if( collector == null )
        return;
      try
        {
        collector.close();


        flowProcess.increment( Counters.Paths_Closed, 1 );
        }
      catch( Exception exception )
        {

View Full Code Here


  private void writeFileTo( String path ) throws IOException
    {
    Hfs tap = new Hfs( new TextLine( new Fields( "offset", "line" ) ), getOutputPath( path ) );


    TupleEntryCollector collector = tap.openForWrite( getPlatform().getFlowProcess() );


    collector.add( new Tuple( 1, "1" ) );


    collector.close();
    }

View Full Code Here

0 1 2 3 4 5

TOP

Related Classes of cascading.tuple.TupleEntryCollector

bixo.datum.ScoredUrlDatumTest

bixo.examples.crawl.CreateUrlDatumFromOutlinksFunction

bixo.examples.crawl.DemoCrawlTool

bixo.examples.crawl.LatestUrlDatumBufferTest

bixo.examples.webmining.CreateLinkDatumFromOutlinksFunction

bixo.examples.webmining.CreateResultsFunction

bixo.examples.webmining.DemoWebMiningWorkflow

bixo.operations.FilterAndScoreByUrlAndRobotsTest

bixo.operations.MakeFetchSetsBuffer

bixo.pipes.AbstractFetchPipeTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.