Examples of cascading.cascade.CascadeConnector

cascading.cascade.CascadeConnector
Class CascadeConnector is used to construct a new {@link Cascade} instance from a collection of {@link cascading.flow.Flow} instance.
Note order is not significant when adding passing Flow instances to the {@code connect}method. This connector will order them based on their dependencies, if any.
One Flow is dependent on another if the first sinks (produces) output that the second Flow sources (consumes) as input. A sink and source are considered equivalent if the fully qualified identifier, typically {@link Tap#getFullIdentifier(Object)}from either are {@code equals()}.

Note that checkpoint sink Taps from an upstream Flow may be the sources to downstream Flow instances.
The {@link CascadeDef} is a convenience class for dynamically defining a Cascade that can be passed to the{@link CascadeConnector#connect(CascadeDef)} method.
Use the {@link CascadeProps} fluent helper class to create global properties to pass to the CascadeConnectorconstructor. @see CascadeDef @see CascadeProps

    Tap sink2 = getPlatform().getTextFile( getOutputPath( "flowstats2" ), SinkMode.REPLACE );


    Flow flow1 = getPlatform().getFlowConnector().connect( "stats1 test", source, sink1, pipe );
    Flow flow2 = getPlatform().getFlowConnector().connect( "stats2 test", source, sink2, pipe );


    Cascade cascade = new CascadeConnector().connect( flow1, flow2 );


    cascade.complete();


    CascadeStats cascadeStats = cascade.getCascadeStats();

View Full Code Here


    Tap nextSink = new Hfs( new TextLine(), getOutputPath( "glob2" ), SinkMode.REPLACE );


    Flow nextFlow = getPlatform().getFlowConnector( getProperties() ).connect( "second", sink, nextSink, concatPipe );


    Cascade cascade = new CascadeConnector().connect( concatFlow, nextFlow );


    cascade.complete();


    validateLength( concatFlow, 10 );
    }

View Full Code Here


    Tap nextSink = new Hfs( new TextLine(), getOutputPath( "globmultiource2" ), SinkMode.REPLACE );


    Flow nextFlow = getPlatform().getFlowConnector( getProperties() ).connect( "second", sink, nextSink, concatPipe );


    Cascade cascade = new CascadeConnector().connect( concatFlow, nextFlow );


    cascade.complete();


    validateLength( concatFlow, 10 );
    }

View Full Code Here

    String sinkPath3 = getOutputPath( "flow3" );
    FileOutputFormat.setOutputPath( thirdConf, new Path( remove( sinkPath3, true ) ) );


    Flow thirdMR = new MapReduceFlow( thirdConf, true );


    CascadeConnector cascadeConnector = new CascadeConnector();


    // pass out of order
    Cascade cascade = cascadeConnector.connect( firstFlow, secondFlow, thirdMR, firstMR, secondMR );


    cascade.complete();


    validateLength( new Hfs( new TextLine(), sinkPath3 ).openForRead( new HadoopFlowProcess( defaultConf ) ), 10 );
    }

View Full Code Here

0 1

TOP

Related Classes of cascading.cascade.CascadeConnector

cascading.cascade.hadoop.RiffleCascadePlatformTest

cascading.cascade.planner.FlowGraph

cascading.cascade.planner.IdentifierGraph

cascading.flow.hadoop.MapReduceFlowPlatformTest

cascading.scheme.hadoop.WritableSequenceFilePlatformTest

cascading.stats.hadoop.CascadingStatsLocalHadoopErrorPlatformTest

cascading.stats.hadoop.HadoopStatsPlatformTest

cascading.stats.local.CascadingStatsPlatformTest

cascading.stats.tez.TezStatsPlatformTest

cascading.tap.hadoop.HadoopTapPlatformTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.