Examples of cascading.flow.hadoop.HadoopFlowConnector

cascading.flow.hadoop.HadoopFlowConnector
Use the HadoopFlowConnector to link source and sink {@link Tap} instances with an assembly of {@link Pipe} instances intoan executable {@link HadoopFlow} for execution on an Apache Hadoop cluster. @see cascading.property.AppProps @see cascading.flow.FlowConnectorProps @see cascading.flow.FlowDef @see cascading.flow.hadoop.MapReduceFlow

    FlowDef flowDef = new FlowDef()
        .addSource("input", inputTap)
        .addSink(p, outputTap)
        .addTail(p);


    new HadoopFlowConnector().connect(flowDef).complete();
  }

View Full Code Here


    Pipe inPipe = new Pipe("in");
    Pipe p = new Each(inPipe, new Fields("value"), new ExpandProto(Example.Person.class), new Fields("id", "name", "email", "position"));


    Hfs sink = new Hfs(new TextLine(), "/tmp/output");
    new HadoopFlowConnector().connect(inTap, sink, p).complete();


    TupleEntryIterator iter = sink.openForRead(new HadoopFlowProcess());
    List<Tuple> results = new ArrayList<Tuple>();
    while (iter.hasNext()) {
      results.add(iter.next().getTupleCopy());

View Full Code Here

    Map<String, Tap> sources = new HashMap<String, Tap>();
    sources.put("source1", ExampleFixtures.SOURCE_TAP_1);
    sources.put("source2", ExampleFixtures.SOURCE_TAP_2);


    //  set some default properties and set the flow step strategy
    Flow f = new HadoopFlowConnector(BloomProps.getDefaultProperties()).connect("Example BloomJoin", sources, sink, joined);
    f.setFlowStepStrategy(new BloomAssemblyStrategy());


    f.complete();


    //  Take a look at the output tuples

View Full Code Here

    }


  @Override
  public FlowConnector getFlowConnector()
    {
    return new HadoopFlowConnector( HadoopUtil.createProperties( getPlannerConfig() ) );
    }

View Full Code Here

0 1 2

TOP

Related Classes of cascading.flow.hadoop.HadoopFlowConnector

cascading.flow.hadoop.planner.HadoopPlanner

cascading.flow.hadoop.planner.MapReduceHadoopRuleRegistry

cascading.lingual.platform.hadoop.HadoopPlatformBroker

cascading.pattern.Main

cascading.platform.hadoop.HadoopPlatform

com.backtype.support.CascadingUtils

com.liveramp.cascading_ext.example.BloomJoinExampleWithoutCascadingUtil

com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest

com.squareup.cascading2.function.ExtractProtoTest

com.squareup.cascading2.function.TestExpandProto

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.