Examples of cascading.tuple.Fields

cascading.tuple.Fields
Class Fields represents the field names in a {@link Tuple}. A tuple field may be a literal String value representing a name, or it may be a literal Integer value representing a position, where positions start at position 0. A Fields instance may also represent a set of field names and positions.
Fields are used as both declarators and selectors. A declarator declares that a given {@link Tap} or{@link cascading.operation.Operation} returns the given field names, for a set of values the size ofthe given Fields instance. A selector is used to select given referenced fields from a Tuple. For example;
Fields fields = new Fields( "a", "b", "c" );
This creates a new Fields instance with the field names "a", "b", and "c". This Fields instance can be used as both a declarator or a selector, depending on how it's used.
Or For example;
Fields fields = new Fields( 1, 2, -1 );
This creates a new Fields instance that can only be used as a selector. It would select the second, third, and last position from a given Tuple instance, assuming it has at least four positions. Since the original field names for those positions will carry over to the new selected Tuple instance, if the original Tuple only had three positions, the third and last positions would be the same, and would throw an error on there being duplicate field names in the selected Tuple instance.
Additionally, there are eight predefined Fields sets used for different purposes; {@link #NONE}, {@link #ALL}, {@link #GROUP}, {@link #VALUES}, {@link #ARGS}, {@link #RESULTS}, {@link #UNKNOWN}, {@link #REPLACE}, and {@link #SWAP}.
The {@code NONE} Fields set represents no fields.
The {@code ALL} Fields set is a "wildcard" that represents all the current available fields.
The {@code GROUP} Fields set represents all the fields used as grouping values in a previous {@link cascading.pipe.Splice}. If there is no previous Group in the pipe assembly, the GROUP represents all the current field names.
The {@code VALUES} Fields set represent all the fields not used as grouping fields in a previous Group.
The {@code ARGS} Fields set is used to let a given Operation inherit the field names of its argument Tuple. This Fields setis a convenience and is typically used when the Pipe output selector is {@code RESULTS} or {@code REPLACE}.
The {@code RESULTS} Fields set is used to represent the field names of the current Operations return values. This Fieldsset may only be used as an output selector on a Pipe. It effectively replaces in the input Tuple with the Operation result Tuple.
The {@code UNKNOWN} Fields set is used when Fields must be declared, but how many and their names is unknown. This allowsfor arbitrarily length Tuples from an input source or some Operation. Use this Fields set with caution.
The {@code REPLACE} Fields set is used as an output selector to inline replace values in the incoming Tuple withthe results of an Operation. This is a convenience Fields set that allows subsequent Operations to 'step' on the value with a given field name. The current Operation must always use the exact same field names, or the {@code ARGS}Fields set.
The {@code SWAP} Fields set is used as an output selector to swap out Operation arguments with its results. Neitherthe argument and result field names or size need to be the same. This is useful for when the Operation arguments are no longer necessary and the result Fields and values should be appended to the remainder of the input field names and Tuple.

    }




    @Test
    public void testReadFromESWithFields() throws Exception {
        Tap in = new EsTap("cascading-hadoop/artists", query, new Fields("url", "name"));
        Pipe pipe = new Pipe("copy");
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertSizeEquals(2));
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertNotNull());


        // print out

View Full Code Here

        build(cfg(), in, out, pipe);
    }


    @Test
    public void testReadFromESAliasedField() throws Exception {
        Tap in = new EsTap("cascading-hadoop/alias", query, new Fields("address"));
        Pipe pipe = new Pipe("copy");
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertNotNull());


        // print out
        Tap out = new HadoopPrintStreamTap(Stream.NULL);

View Full Code Here

        build(cfg(), in, out, pipe);
    }


    @Test
    public void testReadFromESWithFieldAlias() throws Exception {
        Tap in = new EsTap("cascading-hadoop/alias", query, new Fields("url"));
        Pipe pipe = new Pipe("copy");
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertNotNull());


        // print out
        Tap out = new HadoopPrintStreamTap(Stream.NULL);

View Full Code Here

        RestUtils.refresh("cascading-hadoop");


        Properties cfg = cfg();
        cfg.setProperty("es.mapping.names", "nested:data.map.key");


        Tap in = new EsTap("cascading-hadoop/nestedmap", new Fields("nested"));
        Pipe pipe = new Pipe("copy");
        pipe = new Each(pipe, new FilterNotNull());
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertSizeLessThan(2));


        // print out

View Full Code Here

        RestUtils.refresh(indexPrefix + "cascading-local");


        Properties cfg = cfg();
        cfg.setProperty("es.mapping.names", "nested:data.map.key");


        Tap in = new EsTap(indexPrefix + "cascading-local/nestedmap", new Fields("nested"));
        Pipe pipe = new Pipe("copy");
        pipe = new Each(pipe, new FilterNotNull());
        pipe = new Each(pipe, AssertionLevel.STRICT, new AssertSizeLessThan(2));


        // print out

View Full Code Here

            }
        }


        static Tuple coerceToString(SinkCall<?, ?> sinkCall) {
            TupleEntry entry = sinkCall.getOutgoingEntry();
            Fields fields = entry.getFields();
            Tuple tuple = entry.getTuple();


            if (fields.hasTypes()) {
                Type types[] = new Type[fields.size()];
                for (int index = 0; index < fields.size(); index++) {
                    Type type = fields.getType(index);
                    if (type instanceof CoercibleType<?>) {
                        types[index] = String.class;
                    }
                    else {
                        types[index] = type;

View Full Code Here

      throw new IllegalArgumentException("may only have one key field, found: " + keyField.print());
    }
  }


  private void setSourceSink(Fields keyFields, Fields[] columnFields) {
    Fields allFields = keyFields;


    if (columnFields.length != 0) {
      allFields = Fields.join(keyFields, Fields.join(columnFields)); // prepend
    }

View Full Code Here

    result.add(keyWritable);


    for (int i = 0; i < this.familyNames.length; i++) {
      String familyName = this.familyNames[i];
      byte[] familyNameBytes = Bytes.toBytes(familyName);
      Fields fields = this.valueFields[i];
      for (int k = 0; k < fields.size(); k++) {
        String fieldName = (String) fields.get(k);
        byte[] fieldNameBytes = Bytes.toBytes(fieldName);
        byte[] cellValue = row.getValue(familyNameBytes, fieldNameBytes);
        if (cellValue == null) {
            cellValue = new byte[0];
        }

View Full Code Here

    Tuple key = tupleEntry.selectTuple(keyField);
    ImmutableBytesWritable keyBytes = (ImmutableBytesWritable) key.getObject(0);
    Put put = new Put(keyBytes.get());


    for (int i = 0; i < valueFields.length; i++) {
      Fields fieldSelector = valueFields[i];
      TupleEntry values = tupleEntry.selectEntry(fieldSelector);


      for (int j = 0; j < values.getFields().size(); j++) {
        Fields fields = values.getFields();
        Tuple tuple = values.getTuple();


        ImmutableBytesWritable valueBytes = (ImmutableBytesWritable) tuple.getObject(j);
        put.add(Bytes.toBytes(familyNames[i]), Bytes.toBytes((String) fields.get(j)), valueBytes.get());
      }
    }


    outputCollector.collect(null, put);
  }

View Full Code Here

    columns = new String[size];


    int count = 0;


    for (int i = 0; i < fieldsArray.length; i++) {
      Fields fields = fieldsArray[i];


      for (int j = 0; j < fields.size(); j++) {
        if (familyNames == null) { columns[count++] = hbaseColumn((String) fields.get(j)); } else {
          columns[count++] = hbaseColumn(familyNames[i]) + (String) fields.get(j);
        }
      }
    }


    return columns;

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of cascading.tuple.Fields

bixo.datum.FetchSetDatum

cascading.BasicPipesPlatformTest

cascading.BasicTrapPlatformTest

cascading.BufferPipesPlatformTest

cascading.cascade.CascadePlatformTest

cascading.cascade.hadoop.RiffleCascadePlatformTest

cascading.cascade.ParallelCascadePlatformTest

cascading.CoGroupFieldedPipesPlatformTest

cascading.DistanceUseCasePlatformTest

cascading.FieldedPipesPlatformTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.