Examples of cascading.flow.FlowNode

cascading.flow.FlowNode
Class FlowNode represents the smallest parallelizable unit of work. It is a child to a {@link cascading.flow.FlowStep} and may have many siblings within the FlowStep.
A FlowNode is commonly executed as one or more slices, where a slice is a JVM executing against a portion of data.
Most slices within a FlowNode are identical, except for the sub-set of data they will be processing against.
But on some platforms, like MapReduce, a slice is executing a single flow pipeline. Thus a FlowNode may consist of some set of pipelines (or pipeline graph). One pipeline per 'streamed' input source Tap.
In a MapReduce model (like Apache Hadoop MapReduce) a FlowNode can by the Map or Reduce side of a job (where a job is a FlowStep).
In a DAG model (like Apache Tez), a FlowNode is a 'vertex', and the 'DAG' is a FlowStep.


    Iterator<FlowNode> iterator = nodeGraph.getOrderedTopologicalIterator(); // ordering of nodes for consistent remote debugging


    while( iterator.hasNext() )
      {
      FlowNode flowNode = iterator.next();


      Vertex vertex = createVertex( flowProcess, initializedConfig, flowNode );
      dag.addVertex( vertex );


      vertexMap.put( flowNode, vertex );
      }


    LinkedList<ProcessGraph.ProcessEdge> processedEdges = new LinkedList<>();


    for( ProcessGraph.ProcessEdge processEdge : nodeGraph.edgeSet() )
      {
      if( processedEdges.contains( processEdge ) )
        continue;


      FlowNode edgeTargetFlowNode = nodeGraph.getEdgeTarget( processEdge );


      FlowElement flowElement = processEdge.getFlowElement();
      List<FlowNode> sourceNodes = nodeGraph.getElementSourceProcesses( flowElement );


      EdgeProperty edgeProperty = createEdgeProperty( initializedConfig, processEdge );


      Vertex targetVertex = vertexMap.get( edgeTargetFlowNode );


      if( sourceNodes.size() == 1 || flowElement instanceof CoGroup || flowElement instanceof Boundary ) // todo: create group vertices around incoming ordinal
        {
        FlowNode edgeSourceFlowNode = nodeGraph.getEdgeSource( processEdge );
        Vertex sourceVertex = vertexMap.get( edgeSourceFlowNode );


        LOG.debug( "adding edge between: {} and {}", sourceVertex, targetVertex );


        dag.addEdge( Edge.create( sourceVertex, targetVertex, edgeProperty ) );
        }
      else if( flowElement instanceof GroupBy || flowElement instanceof Merge ) // merge - source nodes > 1
        {
        List<String> sourceVerticesIDs = new ArrayList<>();
        List<Vertex> sourceVertices = new ArrayList<>();


        for( FlowNode edgeSourceFlowNode : sourceNodes )
          {
          sourceVerticesIDs.add( edgeSourceFlowNode.getID() );
          sourceVertices.add( vertexMap.get( edgeSourceFlowNode ) );
          processedEdges.add( nodeGraph.getEdge( edgeSourceFlowNode, edgeTargetFlowNode ) );
          }


        VertexGroup vertexGroup = dag.createVertexGroup( edgeTargetFlowNode.getID(), sourceVertices.toArray( new Vertex[ sourceVertices.size() ] ) );

View Full Code Here

      String reduceNodeState = jobConf.getRaw( "cascading.flow.step.node.reduce" );


      if( reduceNodeState == null )
        reduceNodeState = readStateFromDistCache( jobConf, jobConf.get( FlowStep.CASCADING_FLOW_STEP_ID ), "reduce" );


      FlowNode node = deserializeBase64( reduceNodeState, jobConf, BaseFlowNode.class );


      streamGraph = new HadoopReduceStreamGraph( currentProcess, node, Util.getFirst( node.getSourceElements() ) );


      group = (HadoopGroupGate) streamGraph.getHeads().iterator().next();


      for( Duct head : streamGraph.getHeads() )
        LOG.info( "sourcing from: " + ( (ElementDuct) head ).getFlowElement() );


      for( Duct tail : streamGraph.getTails() )
        LOG.info( "sinking to: " + ( (ElementDuct) tail ).getFlowElement() );


      for( Tap trap : node.getTraps() )
        LOG.info( "trapping to: " + trap );
      }
    catch( Throwable throwable )
      {
      reportIfLocal( throwable );

View Full Code Here

      String mapNodeState = jobConf.getRaw( "cascading.flow.step.node.map" );


      if( mapNodeState == null )
        mapNodeState = readStateFromDistCache( jobConf, jobConf.get( FlowStep.CASCADING_FLOW_STEP_ID ), "map" );


      FlowNode node = deserializeBase64( mapNodeState, jobConf, BaseFlowNode.class );


      Tap source = BaseFlowStep.getTapForID( node.getSourceTaps(), jobConf.get( "cascading.step.source" ) );


      streamGraph = new HadoopMapStreamGraph( currentProcess, node, source );


      for( Duct head : streamGraph.getHeads() )
        LOG.info( "sourcing from: " + ( (ElementDuct) head ).getFlowElement() );


      for( Duct tail : streamGraph.getTails() )
        LOG.info( "sinking to: " + ( (ElementDuct) tail ).getFlowElement() );


      for( Tap trap : node.getTraps() )
        LOG.info( "trapping to: " + trap );
      }
    catch( Throwable throwable )
      {
      reportIfLocal( throwable );

View Full Code Here

    {
    for( ElementGraph nodeSubGraph : nodeSubGraphs )
      {
      List<? extends ElementGraph> pipelineGraphs = pipelineSubGraphsMap.get( nodeSubGraph );


      FlowNode flowNode = flowPlanner.createFlowNode( flowElementGraph, nodeSubGraph, pipelineGraphs );


      addVertex( flowNode );
      }


    bindEdges();

View Full Code Here

TOP

Related Classes of cascading.flow.FlowNode

cascading.flow.hadoop.FlowMapper

cascading.flow.hadoop.FlowReducer

cascading.flow.planner.process.FlowNodeGraph

cascading.flow.tez.Hadoop2TezFlowStep

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.