Examples of Tap

cascading.tap.Tap
A Tap represents the physical data source or sink in a connected {@link cascading.flow.Flow}.
That is, a source Tap is the head end of a connected {@link Pipe} and {@link Tuple} stream, anda sink Tap is the tail end. Kinds of Tap types are used to manage files from a local disk, distributed disk, remote storage like Amazon S3, or via FTP. It simply abstracts out the complexity of connecting to these types of data sources.
A Tap takes a {@link Scheme} instance, which is used to identify the type of resource (text file, binary file, etc).A Tap is responsible for how the resource is reached.
By default when planning a Flow, Tap equality is a function of the {@link #getIdentifier()} and {@link #getScheme()}values. That is, two Tap instances are the same Tap instance if they sink/source the same resource and sink/source the same fields.
Some more advanced taps, like a database tap, may need to extend equality to include any filtering, like the {@code where} clause in a SQL statement so two taps reading from the same SQL table aren't considered equal.
Taps are also used to determine dependencies between two or more {@link Flow} instances when used with a{@link cascading.cascade.Cascade}. In that case the {@link #getFullIdentifier(Object)} value is used and the Schemeis ignored.
com.vst.model.Tap
Created by IntelliJ IDEA. User: And Lilia Date: 07.04.2009 Time: 18:32:46 To change this template use File | Settings | File Templates. @hibernate.subclass discriminator-value = "tap"
org.springframework.xd.test.fixtures.Tap
Test fixture that creates a tap. @author Glenn Renfro

Examples of cascading.tap.Tap


    private static final String INPUT = TestUtils.sampleArtistsDat(CascadingHadoopSuite.configuration);


    @Test
    public void testWriteToES() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/artists", new Fields("name", "url", "picture"));
        Pipe pipe = new Pipe("copy");


        FlowDef flowDef = FlowDef.flowDef().addSource(pipe, in).addTailSink(pipe, out);
        StatsUtils.proxy(new HadoopFlowConnector(HdpBootstrap.asProperties(CascadingHadoopSuite.configuration)).connect(flowDef)).complete();
    }

View Full Code Here

Examples of cascading.tap.Tap

        assertThat(RestUtils.getMapping("cascading-hadoop/artists").skipHeaders().toString(), is("artists=[name=STRING, picture=STRING, url=STRING]"));
    }


    @Test
    public void testWriteToESWithAlias() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/alias", "", new Fields("name", "url", "picture"));
        Pipe pipe = new Pipe("copy");


        // rename "id" -> "garbage"
        pipe = new Each(pipe, new Identity(new Fields("garbage", "name", "url", "picture", "ts")));

View Full Code Here

Examples of cascading.tap.Tap


    @Test
    public void testIndexPattern() throws Exception {
        Properties props = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);


        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/pattern-{id}", new Fields("id", "name", "url", "picture"));
        Pipe pipe = new Pipe("copy");
        StatsUtils.proxy(new HadoopFlowConnector(props).connect(in, out, pipe)).complete();
    }

View Full Code Here

Examples of cascading.tap.Tap


    @Test
    public void testIndexPatternWithFormat() throws Exception {
        Properties props = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);


        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/pattern-format-{ts:YYYY-MM-dd}", new Fields("id", "name", "url", "picture", "ts"));
        Pipe pipe = new Pipe("copy");
        StatsUtils.proxy(new HadoopFlowConnector(props).connect(in, out, pipe)).complete();
    }

View Full Code Here

Examples of cascading.tap.Tap

    private static final String INPUT = TestUtils.sampleArtistsJson(CascadingHadoopSuite.configuration);


    @Test
    public void testWriteToES() throws Exception {
        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/artists");


        Pipe pipe = new Pipe("copy");
        build(cfg(), in, out, pipe);
    }

View Full Code Here

Examples of cascading.tap.Tap

    public void testIndexAutoCreateDisabled() throws Exception {
        Properties cfg = cfg();
        cfg.setProperty(ConfigurationOptions.ES_INDEX_AUTO_CREATE, "false");


        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/non-existing", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg, in, out, pipe);
    }

View Full Code Here

Examples of cascading.tap.Tap

        build(cfg, in, out, pipe);
    }


    @Test
    public void testIndexPattern() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/pattern-{number}", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg(), in, out, pipe);
    }

View Full Code Here

Examples of cascading.tap.Tap

        build(cfg(), in, out, pipe);
    }


    @Test
    public void testIndexPatternWithFormat() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/pattern-format-{@timestamp:YYYY-MM-dd}", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg(), in, out, pipe);
    }

View Full Code Here

Examples of cascading.tap.Tap

    private static final String INPUT = TestUtils.sampleArtistsDat();


    @Test
    public void testWriteToES() throws Exception {
        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-local/artists", new Fields("name", "url", "picture"));


        Pipe pipe = new Pipe("copy");
        build(new TestSettings().getProperties(), in, out, pipe);
    }

View Full Code Here

Examples of cascading.tap.Tap

    }


    @Test
    public void testWriteToESWithAlias() throws Exception {
        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-local/alias", new Fields("name", "url", "picture"));
        Pipe pipe = new Pipe("copy");


        // rename "id" -> "garbage"
        pipe = new Each(pipe, new Identity(new Fields("garbage", "name", "url", "picture", "ts")));

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.