Examples of cascading.flow.tez.Hadoop2TezFlowProcess

cascading.flow.tez.Hadoop2TezFlowProcess
Class HadoopFlowProcess is an implementation of {@link cascading.flow.FlowProcess} for Hadoop. Use this interface to get directaccess to the Hadoop JobConf and Reporter interfaces.
Be warned that coupling to this implementation will cause custom {@link cascading.operation.Operation}s to fail if they are executed on a system other than Hadoop. @see cascading.flow.FlowSession


    Set<FlowElement> accumulated = new HashSet<>( node.getSourceElements() );


    accumulated.remove( streamedSource );


    Hadoop2TezFlowProcess tezProcess = (Hadoop2TezFlowProcess) flowProcess;
    TezConfiguration conf = tezProcess.getConfiguration();


    for( FlowElement flowElement : accumulated )
      {
      LOG.info( "using accumulated source: " + flowElement );


      if( flowElement instanceof Tap )
        {
        Tap source = (Tap) flowElement;


        // allows client side config to be used cluster side
        String property = conf.getRaw( "cascading.node.accumulated.source.conf." + Tap.id( source ) );


        if( property == null )
          throw new IllegalStateException( "accumulated source conf property missing for: " + source.getIdentifier() );


        conf = getSourceConf( tezProcess, conf, property );
        }
      else
        {
        conf = (TezConfiguration) inputConfigMap.get( FlowElements.id( flowElement ) );
        }


      FlowProcess flowProcess = conf == null ? tezProcess : new Hadoop2TezFlowProcess( tezProcess, conf );


      handleHead( flowElement, flowProcess );
      }
    }

View Full Code Here

    {
    LOG.info( "calling {}#start() on: {}", logicalInput.getClass().getSimpleName(), getSource() );


    logicalInput.start();


    Hadoop2TezFlowProcess tezFlowProcess;


    if( flowProcess instanceof FlowProcessWrapper )
      tezFlowProcess = (Hadoop2TezFlowProcess) ( (FlowProcessWrapper) flowProcess ).getDelegate();
    else
      tezFlowProcess = (Hadoop2TezFlowProcess) flowProcess;


    TezConfiguration configuration = tezFlowProcess.getConfiguration();


    try
      {
      reader = (MRReader) logicalInput.getReader();
      }

View Full Code Here

    }


  @Override
  public FlowProcess getFlowProcess()
    {
    return new Hadoop2TezFlowProcess( FlowSession.NULL, null, getConfiguration() );
    }

View Full Code Here

TOP

Related Classes of cascading.flow.tez.Hadoop2TezFlowProcess

cascading.flow.tez.stream.element.TezSourceStage

cascading.flow.tez.stream.graph.Hadoop2TezStreamGraph

cascading.platform.tez.Hadoop2TezPlatform

org.apache.tez.dag.api.TezConfiguration

cascading.CascadingException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.