Examples of cascading.tuple.Fields

cascading.tuple.Fields
Class Fields represents the field names in a {@link Tuple}. A tuple field may be a literal String value representing a name, or it may be a literal Integer value representing a position, where positions start at position 0. A Fields instance may also represent a set of field names and positions.
Fields are used as both declarators and selectors. A declarator declares that a given {@link Tap} or{@link cascading.operation.Operation} returns the given field names, for a set of values the size ofthe given Fields instance. A selector is used to select given referenced fields from a Tuple. For example;
Fields fields = new Fields( "a", "b", "c" );
This creates a new Fields instance with the field names "a", "b", and "c". This Fields instance can be used as both a declarator or a selector, depending on how it's used.
Or For example;
Fields fields = new Fields( 1, 2, -1 );
This creates a new Fields instance that can only be used as a selector. It would select the second, third, and last position from a given Tuple instance, assuming it has at least four positions. Since the original field names for those positions will carry over to the new selected Tuple instance, if the original Tuple only had three positions, the third and last positions would be the same, and would throw an error on there being duplicate field names in the selected Tuple instance.
Additionally, there are eight predefined Fields sets used for different purposes; {@link #NONE}, {@link #ALL}, {@link #GROUP}, {@link #VALUES}, {@link #ARGS}, {@link #RESULTS}, {@link #UNKNOWN}, {@link #REPLACE}, and {@link #SWAP}.
The {@code NONE} Fields set represents no fields.
The {@code ALL} Fields set is a "wildcard" that represents all the current available fields.
The {@code GROUP} Fields set represents all the fields used as grouping values in a previous {@link cascading.pipe.Splice}. If there is no previous Group in the pipe assembly, the GROUP represents all the current field names.
The {@code VALUES} Fields set represent all the fields not used as grouping fields in a previous Group.
The {@code ARGS} Fields set is used to let a given Operation inherit the field names of its argument Tuple. This Fields setis a convenience and is typically used when the Pipe output selector is {@code RESULTS} or {@code REPLACE}.
The {@code RESULTS} Fields set is used to represent the field names of the current Operations return values. This Fieldsset may only be used as an output selector on a Pipe. It effectively replaces in the input Tuple with the Operation result Tuple.
The {@code UNKNOWN} Fields set is used when Fields must be declared, but how many and their names is unknown. This allowsfor arbitrarily length Tuples from an input source or some Operation. Use this Fields set with caution.
The {@code REPLACE} Fields set is used as an output selector to inline replace values in the incoming Tuple withthe results of an Operation. This is a convenience Fields set that allows subsequent Operations to 'step' on the value with a given field name. The current Operation must always use the exact same field names, or the {@code ARGS}Fields set.
The {@code SWAP} Fields set is used as an output selector to swap out Operation arguments with its results. Neitherthe argument and result field names or size need to be the same. This is useful for when the Operation arguments are no longer necessary and the result Fields and values should be appended to the remainder of the input field names and Tuple.

    private static final String INPUT = TestUtils.sampleArtistsDat(CascadingHadoopSuite.configuration);


    @Test
    public void testWriteToES() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/artists", new Fields("name", "url", "picture"));
        Pipe pipe = new Pipe("copy");


        FlowDef flowDef = FlowDef.flowDef().addSource(pipe, in).addTailSink(pipe, out);
        StatsUtils.proxy(new HadoopFlowConnector(HdpBootstrap.asProperties(CascadingHadoopSuite.configuration)).connect(flowDef)).complete();
    }

View Full Code Here

    }


    @Test
    public void testWriteToESWithAlias() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/alias", "", new Fields("name", "url", "picture"));
        Pipe pipe = new Pipe("copy");


        // rename "id" -> "garbage"
        pipe = new Each(pipe, new Identity(new Fields("garbage", "name", "url", "picture", "ts")));


        Properties props = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);
        props.setProperty("es.mapping.names", "url:address");
        StatsUtils.proxy(new HadoopFlowConnector(props).connect(in, out, pipe)).complete();
    }

View Full Code Here

    @Test
    public void testIndexPattern() throws Exception {
        Properties props = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);


        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/pattern-{id}", new Fields("id", "name", "url", "picture"));
        Pipe pipe = new Pipe("copy");
        StatsUtils.proxy(new HadoopFlowConnector(props).connect(in, out, pipe)).complete();
    }

View Full Code Here

    @Test
    public void testIndexPatternWithFormat() throws Exception {
        Properties props = HdpBootstrap.asProperties(CascadingHadoopSuite.configuration);


        Tap in = sourceTap();
        Tap out = new EsTap("cascading-hadoop/pattern-format-{ts:YYYY-MM-dd}", new Fields("id", "name", "url", "picture", "ts"));
        Pipe pipe = new Pipe("copy");
        StatsUtils.proxy(new HadoopFlowConnector(props).connect(in, out, pipe)).complete();
    }

View Full Code Here

        CascadeConnector cascadeConnector = new CascadeConnector(cfg);
        cascadeConnector.connect(flows).complete();
    }


    private Tap sourceTap() {
        return new Hfs(new TextDelimited(new Fields("id", "name", "url", "picture", "ts")), INPUT);
    }

View Full Code Here

        Properties cfg = cfg();
        cfg.setProperty(ConfigurationOptions.ES_INDEX_AUTO_CREATE, "false");


        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/non-existing", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg, in, out, pipe);
    }

View Full Code Here

    }


    @Test
    public void testIndexPattern() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/pattern-{number}", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg(), in, out, pipe);
    }

View Full Code Here

    }


    @Test
    public void testIndexPatternWithFormat() throws Exception {
        Tap in = sourceTap();
        Tap out = new EsTap("json-cascading-hadoop/pattern-format-{@timestamp:YYYY-MM-dd}", new Fields("line"));
        Pipe pipe = new Pipe("copy");
        build(cfg(), in, out, pipe);
    }

View Full Code Here

        props.put(ConfigurationOptions.ES_INPUT_JSON, "true");
        return props;
    }


    private Tap sourceTap() {
        return new Hfs(new TextDelimited(new Fields("line")), INPUT);
    }

View Full Code Here


    @Test
    public void testWriteToES() throws Exception {
        // local file-system source
        Tap in = sourceTap();
        Tap out = new EsTap("cascading-local/artists", new Fields("name", "url", "picture"));


        Pipe pipe = new Pipe("copy");
        build(new TestSettings().getProperties(), in, out, pipe);
    }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of cascading.tuple.Fields

bixo.datum.FetchSetDatum

cascading.BasicPipesPlatformTest

cascading.BasicTrapPlatformTest

cascading.BufferPipesPlatformTest

cascading.cascade.CascadePlatformTest

cascading.cascade.hadoop.RiffleCascadePlatformTest

cascading.cascade.ParallelCascadePlatformTest

cascading.CoGroupFieldedPipesPlatformTest

cascading.DistanceUseCasePlatformTest

cascading.FieldedPipesPlatformTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.