Examples of org.apache.tez.dag.api.GroupInputEdge

org.apache.tez.dag.api.GroupInputEdge
A composite edge that represents a common connection between a group of input vertices and a single output vertex. This can be used to perform e.g. an efficient union of the data produced by the input vertices. The output vertex tasks see a unified/merged view of the data from all the input vertices.

        JobConf parentConf = workToConf.get(unionWorkItems.get(0));


        // now hook up the children
        for (BaseWork v: children) {
          // finally we can create the grouped edge
          GroupInputEdge e = utils.createEdge(group, parentConf,
               workToVertex.get(v), work.getEdgeProperty(w, v), work.getVertexType(v));


          dag.addEdge(e);
        }
      } else {

View Full Code Here

      default:
        mergeInputClass = TezMergedLogicalInput.class;
        break;
    }


    return new GroupInputEdge(group, w, createEdgeProperty(edgeProp),
        new InputDescriptor(mergeInputClass.getName()));
  }

View Full Code Here

      default:
        mergeInputClass = TezMergedLogicalInput.class;
        break;
    }


    return new GroupInputEdge(group, w, createEdgeProperty(edgeProp),
        new InputDescriptor(mergeInputClass.getName()));
  }

View Full Code Here

    org.apache.tez.dag.api.VertexGroup uv12 = dag.createVertexGroup(groupName1, v1, v2);
    OutputDescriptor outDesc = new OutputDescriptor("output.class");
    uv12.addOutput("uvOut", outDesc, TotalCountingOutputCommitter.class);
    v3.addOutput("uvOut", outDesc, TotalCountingOutputCommitter.class);
    
    GroupInputEdge e1 = new GroupInputEdge(uv12, v3,
        new EdgeProperty(DataMovementType.SCATTER_GATHER, 
            DataSourceType.PERSISTED, SchedulingType.SEQUENTIAL,
            new OutputDescriptor("dummy output class"),
            new InputDescriptor("dummy input class")), 
            new InputDescriptor("merge.class"));

View Full Code Here

0 1

TOP

Related Classes of org.apache.tez.dag.api.GroupInputEdge

org.apache.hadoop.hive.ql.exec.tez.DagUtils

org.apache.hadoop.hive.ql.exec.tez.TezTask

org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder

org.apache.tez.dag.app.dag.impl.TestDAGImpl

org.apache.tez.dag.history.utils.TestDAGUtils

org.apache.tez.mapreduce.examples.UnionExample

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.