Examples of cascading.flow.FlowConnector

cascading.flow.FlowConnector
Class FlowConnector is the base class for all platform planners.
See the {@link FlowDef} class for a fluent way to define a new Flow.
Use the FlowConnector to link source and sink {@link Tap} instances with an assembly of {@link Pipe} instances intoan executable {@link cascading.flow.Flow}.
FlowConnector invokes a planner for the target execution environment.
For executing Flows in local memory against local files, see {@link cascading.flow.local.LocalFlowConnector}.
For Apache Hadoop, see the {@link cascading.flow.hadoop.HadoopFlowConnector}. Or if you have a pre-existing custom Hadoop job to execute, see {@link cascading.flow.hadoop.MapReduceFlow}, which doesn't require a planner.
Note that all {@code connect} methods take a single {@code tail} or an array of {@code tail} Pipe instances. "tail"refers to the last connected Pipe instances in a pipe-assembly. Pipe-assemblies are graphs of object with "heads" and "tails". From a given "tail", all connected heads can be found, but not the reverse. So "tails" must be supplied by the user.
The FlowConnector and the underlying execution framework (Hadoop or local mode) can be configured via a {@link Map} or {@link Properties} instance given to the constructor.
This properties map must be populated before constructing a FlowConnector instance. Many planner specific properties can be set through the {@link FlowConnectorProps} fluent interface.
Some planners have required properties. Hadoop expects {@link AppProps#setApplicationJarPath(java.util.Map,String)} or{@link AppProps#setApplicationJarClass(java.util.Map,Class)} to be set.
Any properties set and passed through the FlowConnector constructor will be global to all Flow instances created through the that FlowConnector instance. Some properties are on the {@link FlowDef} and would only be applicable to theresulting Flow instance.
These properties are used to influence the current planner and are also passed down to the execution framework to override any default values. For example when using the Hadoop planner, the number of reducers or mappers can be set by using platform specific properties.
Custom operations (Functions, Filter, etc) may also retrieve these property values at runtime through calls to {@link cascading.flow.FlowProcess#getProperty(String)} or {@link FlowProcess#getStringProperty(String)}.
Most applications will need to call {@link cascading.property.AppProps#setApplicationJarClass(java.util.Map,Class)} or{@link cascading.property.AppProps#setApplicationJarPath(java.util.Map,String)} so thatthe correct application jar file is passed through to all child processes. The Class or path must reference the custom application jar, not a Cascading library class or jar. The easiest thing to do is give setApplicationJarClass the Class with your static main function and let Cascading figure out which jar to use.
Note that Map is compatible with the {@link Properties} class, so properties can be loaded atruntime from a configuration file.
By default, all {@link cascading.operation.Assertion}s are planned into the resulting Flow instance. This can be changed for a given Flow by calling {@link FlowDef#setAssertionLevel(cascading.operation.AssertionLevel)} or globallyvia {@link FlowConnectorProps#setAssertionLevel(cascading.operation.AssertionLevel)}.
Also by default, all {@link cascading.operation.Debug}s are planned into the resulting Flow instance. This can be changed for a given flow by calling {@link FlowDef#setDebugLevel(cascading.operation.DebugLevel)} or globally via{@link FlowConnectorProps#setDebugLevel(cascading.operation.DebugLevel)}.
As of version 3.0, custom {@link cascading.flow.planner.rule.RuleRegistry} instances can be provided to customizea given planner. @see cascading.flow.local.LocalFlowConnector @see cascading.flow.hadoop.HadoopFlowConnector

        Properties cfg = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);


        FlowDef flow = new FlowDef().addSource(copy, sourceTap())
                .addTailSink(copy, new EsTap("cascading-hadoop/cascade-connector"));


        FlowConnector connector = new HadoopFlowConnector(cfg);
        Flow[] flows = new Flow[] { connector.connect(flow) };


        CascadeConnector cascadeConnector = new CascadeConnector(cfg);
        cascadeConnector.connect(flows).complete();
    }

View Full Code Here

        Properties cfg = new TestSettings().getProperties();


        FlowDef flow = new FlowDef().addSource(copy, sourceTap()).addTailSink(copy,
                new EsTap("cascading-local/cascade-connector"));


        FlowConnector connector = new LocalFlowConnector(cfg);
        Flow[] flows = new Flow[] { connector.connect(flow) };


        CascadeConnector cascadeConnector = new CascadeConnector(cfg);
        cascadeConnector.connect(flows).complete();
    }

View Full Code Here

            Tap urlSource = _platform.makeTap(_platform.makeTextScheme(), _inputFilePath);
            Pipe importPipe = new Each("url importer", new Fields("line"), new CreateUrlFromTextFunction(new SimpleUrlNormalizer(), new SimpleUrlValidator()));


            Tap urlSink = _platform.makeTap(_platform.makeBinaryScheme(CrawlDbDatum.FIELDS), _destDirPath, SinkMode.REPLACE);


            FlowConnector flowConnector = _platform.makeFlowConnector();
            Flow flow = flowConnector.connect(urlSource, urlSink, importPipe);
            flow.complete();
        } catch (Exception e) {
            throw e;
        }
    }

View Full Code Here

        Pipe resultsPipe = new GroupBy("results pipe", Pipe.pipes(fetchedPipe, unfetchedPipe), 
                        new Fields(UrlDatum.URL_FN));
        resultsPipe = new Every(resultsPipe, new LatestUrlDatumBuffer(), Fields.RESULTS);




        FlowConnector flowConnector = platform.makeFlowConnector();
        Flow flow = flowConnector.connect(sources, resultSink, resultsPipe);
        flow.complete();
        
        // verify that the resulting pipe has the latest tuple
        
        Tap testSink = platform.makeTap(platform.makeBinaryScheme(UrlDatum.FIELDS), resultsPath);

View Full Code Here

        if (!options.isLocalPlatformMode()) {
            sinkMap.put(writableSeqFileDataPipe.getName(), writableSeqFileSink);
            tailPipes.add(writableSeqFileDataPipe);
        }
        
        FlowConnector flowConnector = platform.makeFlowConnector();
        Flow flow = flowConnector.connect(inputSource, sinkMap, tailPipes.toArray(new Pipe[tailPipes.size()]));


        return flow;
    }

View Full Code Here

            
            BasePath inputPath = platform.makePath(input);
            Tap sourceTap = platform.makeTap(platform.makeTextScheme(), inputPath);
            Tap sinkTap = platform.makeTap(platform.makeBinaryScheme(UrlDatum.FIELDS), crawlDBPath, SinkMode.REPLACE);
            
            FlowConnector flowConnector = platform.makeFlowConnector();
            Flow flow = flowConnector.connect(sourceTap, sinkTap, importPipe);
            flow.complete();
        }


        return crawlDBPath.getAbsolutePath();
    }

View Full Code Here

        fetcherPolicy.setCrawlDelay(5 * 1000L);
        BaseFetcher fetcher = new SimpleHttpFetcher(2, fetcherPolicy, userAgent);
        BaseScoreGenerator scorer = new FixedScoreGenerator();
        FetchPipe fetchPipe = new FetchPipe(pipe, scorer, fetcher, 1);


        FlowConnector flowConnector = platform.makeFlowConnector();
        Flow flow = flowConnector.connect(in, FetchPipe.makeSinkMap(status, content), fetchPipe);
        flow.complete();
        
        // Test for all valid fetches.
        Tap validate = platform.makeTap(platform.makeBinaryScheme(StatusDatum.FIELDS), statusPath);
        TupleEntryIterator tupleEntryIterator = validate.openForRead(platform.makeFlowProcess());

View Full Code Here

        UserAgent userAgent = new FirefoxUserAgent();
        BaseFetcher fetcher = new SimpleHttpFetcher(10, userAgent);
        BaseScoreGenerator scorer = new FixedScoreGenerator();
        FetchPipe fetchPipe = new FetchPipe(pipe, scorer, fetcher, 1);


        FlowConnector flowConnector = platform.makeFlowConnector();


        Flow flow = flowConnector.connect(in, FetchPipe.makeSinkMap(status, content), fetchPipe);
        flow.complete();
        
        // Test for 10 good fetches.
        Tap validate = platform.makeTap(platform.makeBinaryScheme(FetchedDatum.FIELDS), contentPath);
        TupleEntryIterator tupleEntryIterator = validate.openForRead(platform.makeFlowProcess());

View Full Code Here

            sinkTapMap.put(ANALYZER_PIPE_NAME, analyzerSinkTap);
            
            LOGGER.info("Running fetch job with " + options);


            // Finally we can run it.
            FlowConnector flowConnector = platform.makeFlowConnector();
            Flow flow = flowConnector.connect(sourceTap, sinkTapMap, splitterPipe, mboxPageStatusPipe, analysisPipe);
            flow.writeDOT("build/goodFlow.dot");
            flow.complete();
        } catch (Throwable t) {
            System.err.println("Exception running AnalyzeEmail: " + t.getMessage());
            t.printStackTrace(System.err);

View Full Code Here

            BasePath filePath = platform.makePath(filename);
            TextLine textLineLocalScheme = new TextLine(new Fields("url"));
            Tap sourceTap = platform.makeTap(textLineLocalScheme, filePath, SinkMode.KEEP);
            SinkTap sinkTap = new NullSinkTap(new Fields("url"));
            
            FlowConnector flowConnector = platform.makeFlowConnector();
            Flow flow = flowConnector.connect(sourceTap, sinkTap, pipe);


            flow.complete();
        } catch (Exception e) {
            System.err.println("Exception running tool: " + e.getMessage());
            e.printStackTrace(System.err);

View Full Code Here

0 1 2 3

TOP

Related Classes of cascading.flow.FlowConnector

bixo.examples.crawl.DemoCrawlWorkflow

bixo.examples.crawl.LatestUrlDatumBufferTest

bixo.examples.crawl.UrlImporter

bixo.examples.webmining.DemoWebMiningWorkflow

bixo.fetcher.FetcherTest

bixo.pipes.AbstractFetchPipeTest

bixo.pipes.ParsePipeTest

bixo.pipes.RunFakeFetchPipe

bixo.tools.LengthenUrlsTool

cascading.flow.hadoop.BuildJobsHadoopPlatformTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.