Examples of cascading.pipe.SubAssembly

cascading.pipe.SubAssembly

Subclasses of SubAssembly encapsulate complex assemblies of {@link Pipe}s so they my be reused in the same manner a Pipe is used.

That is, a typical SubAssembly subclass will accept a 'previous' Pipe instance, and a few arguments for configuring the resulting sub-assembly.

The previous pipe (or pipes) must be passed on the super constructor, or set via {@link #setPrevious(Pipe)}. This allows the current SubAssembly to become the parent of any Pipe instances between the previous and the tails, exclusive of the previous, and inclusive of the tails.

Subsequently all tail Pipes must be set via the {@link #setTails(Pipe)} method.

Note if the SubAssembly represents a split in the pipeline process, all the 'tails' of the assembly must be passed to {@link #setTails(Pipe)}. It is up the the developer to provide any other access to the tails so they may be chained into any subsequent Pipes.

Any {@link cascading.property.ConfigDef} values on this SubAssembly will be honored by child Pipe instances via the{@link cascading.pipe.Pipe#getParent()} back link described above.

    // connect the pipe assembly to the tap instances
    Flow importPagesFlow = flowConnector.connect( "import pages", localPagesSource, importedPages, importPipe );


    // a predefined pipe assembly that splits the stream into two named "url pipe" and "word pipe"
    // these pipes could be retrieved via the getTails() method and added to new pipe instances
    SubAssembly wordCountPipe = new WordCountSplitAssembly( "wordcount pipe", "url pipe", "word pipe" );


    // create Hadoop sequence files to store the results of the counts
    Tap sinkUrl = new Hfs( new SequenceFile( new Fields( "url", "word", "count" ) ), urlsPath );
    Tap sinkWord = new Hfs( new SequenceFile( new Fields( "word", "count" ) ), wordsPath );

  }


  @Test
  public void testMultiCombiner() throws IOException {
    Pipe pipe = new Pipe("pipe");
    SubAssembly multiCombiner = MultiCombiner.assembly(pipe, def1, def2, def3, def4, def5);
    Pipe[] tails = multiCombiner.getTails();


    Map<String, Tap> sinks = Maps.newHashMap();


    for (CombinerDefinition def : Lists.<CombinerDefinition>newArrayList(def1, def2, def3, def4, def5)) {
      Tap output = getTupleOutputTap("testMultipleTails", def.getName(), def.getGroupFields().append(def.getOutputFields()));

Examples of cascading.pipe.SubAssembly

Related Classes of cascading.pipe.SubAssembly