Examples of org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField

Package org.apache.pig.piggybank.evaluation

Examples of org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField

org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
This class is similar to MaxTupleBy1stField except that it allows you to specify with field to use (instead of just using 1st field) and to specify ascending or descending. The first parameter in the constructor specifies which field to use and the second parameter to the constructor specifies which extremal to retrieve. Strings prefixed by "min", "least", "desc", "small" and "-", irrespective of capitalization and leading white spacing, specifies the computation of the minimum and all other strings means maximum;
Example1: Invoking the UDF
e.g. Using this udf:
define myMin ExtremalTupleByNthField( '4', 'min' ); T = group G ALL; R = foreach T generate myMin(G); is equivalent to:
T = order G by $3 asc; R = limit G 1; Note above 4 indicates the field with index 3 in the tuple. The 4th field can be any comparable type, so you can use float, int, string, or even tuples. By default constructor, this UDF behaves as MaxTupleBy1stField in that it chooses the max tuple by the comparable in the first field.
Example 2: Default behavior
This class also has a one parameter constructor that specifies the index and takes the max tuple from the bag. define myMax ExtremalTupleByNthField( '3' ); T = group G ALL; R = foreach T generate myMax(G); is equivalent to:
T = order G by $2 desc; R = limit G 1;
Example 3: Choosing a Large Bag or Tuple
Another possible use case is the choosing of larger or smaller bags/tuples. In pig, bags and tuples are comparable and the comparison is based on size. define biggestBag ExtremalTupleByNthField('1', max); R = group TABLE by (key1, key2); G = cogroup L by key1, R by group.key1; V = foreach G generate L, biggestBag(R); This results in each L(eft) bag associated with only the largest bag from the R(ight) table. If all bags in R are of equal size, the comparator continues on to perform element-wise comparison. In case of a complete tie in the comparison, which result is returned is nondeterministic. But because this class is able to compare any comparable we are able to specify a secondary key.
Example 4: Secondary Sort Key
define biggestBag ExtremalTupleByNthField('1', max); G = cogroup L by key1, M by key1, R by key1; V = foreach G generate FLATTEN(L), biggestBag(R.($0, $1, $2, $5)) as best_result_by_0, biggestBag(R.($3, $1, $2, $5)) as best_result_by_3, biggestBag(M.($0, $2)) as best_misc_data; this will generate two sets of results and misc data based on two separate criterion. Since all tuples in the bags have the same size (4, 4, 2 respectively), the tuple comparator continues on and compares the members of tuples until it finds one. best_result_by_0 and best_result_by3 are ordered by 1st and 4th member of the tuples. Within each group, ties are broken by second and third field. Finally, note that the udf implements both Algebraic and Accumulator, so it is relatively efficient because it's a one-pass algorithm.


public class TestExtremalTupleByNthField {


  @Test
  public void testMin() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("3", "min");


    DataBag input = BagFactory.getInstance().newDefaultBag();


    for (int i = 100; i > 0; --i) {
      Tuple t = TupleFactory.getInstance().newTuple();
      t.append(i);
      t.append(" " + i);
      t.append(i);
      input.add(t);
    }


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    Assert.assertEquals(" 1", (String) out.get(1));
  }

View Full Code Here

    Assert.assertEquals(" 1", (String) out.get(1));
  }


  @Test
  public void testMax() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("4", "max");


    DataBag input = BagFactory.getInstance().newDefaultBag();


    for (int i = 0; i < 100; i++) {
      Tuple t = TupleFactory.getInstance().newTuple();
      t.append(i);
      t.append(" " + i);
      t.append(i);
      t.append(i);
      input.add(t);
    }


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    Assert.assertEquals(" 99", (String) out.get(1));
  }

View Full Code Here

    Assert.assertEquals(" 99", (String) out.get(1));
  }


  @Test
  public void testMaxComplexKey() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("3", "max");


    DataBag input = BagFactory.getInstance().newDefaultBag();


    for (int j = 0; j < 3; ++j) {
      for (int i = 0; i < 100; i++) {
        Tuple t = TupleFactory.getInstance().newTuple();
        t.append(-i);
        t.append(" " + j + ", " + i);
        Tuple key = TupleFactory.getInstance().newTuple();
        key.append(j);
        key.append(i);
        t.append(key);
        t.append(-i);
        input.add(t);
      }
    }


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    Assert.assertEquals(" 2, 99", (String) out.get(1));
  }

View Full Code Here

    Assert.assertEquals(" 2, 99", (String) out.get(1));
  }


  @Test
  public void testMinComplexKey() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("3", "min");


    DataBag input = BagFactory.getInstance().newDefaultBag();


    for (int j = 0; j < 3; ++j) {
      for (int i = 0; i < 100; i++) {
        Tuple t = TupleFactory.getInstance().newTuple();
        t.append(-i);
        t.append(" " + j + ", " + i);
        Tuple key = TupleFactory.getInstance().newTuple();
        key.append(j);
        key.append(i);
        t.append(key);
        t.append(-i);
        input.add(t);
      }
    }


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    Assert.assertEquals(" 0, 0", (String) out.get(1));
  }

View Full Code Here

    Assert.assertEquals(" 0, 0", (String) out.get(1));
  }


  @Test
  public void testMinStringey() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("4", "min");


    DataBag input = BagFactory.getInstance().newDefaultBag();
    Tuple t = TupleFactory.getInstance().newTuple();
    t.append("a");
    t.append("a");
    t.append("a");
    t.append("min");
    input.add(t);


    t = TupleFactory.getInstance().newTuple();
    t.append("b");
    t.append("b");
    t.append("b");
    t.append("max");
    input.add(t);


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    // ironically "max" is smaller than "min"
    Assert.assertEquals("b", (String) out.get(1));
  }

View Full Code Here

    Assert.assertEquals("b", (String) out.get(1));
  }


  @Test
  public void testBiggerBag() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("1", "max");


    DataBag input = BagFactory.getInstance().newDefaultBag();
    DataBag dbSmaller = BagFactory.getInstance().newDefaultBag();
    dbSmaller.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has three items")));
    dbSmaller.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has three items")));
    dbSmaller.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has three items")));
    input.add(TupleFactory.getInstance().newTuple(
        Arrays.asList(dbSmaller, "smaller")));


    DataBag dbBigger = BagFactory.getInstance().newDefaultBag();
    dbBigger.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has four items")));
    dbBigger.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has four items")));
    dbBigger.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has four items")));
    dbBigger.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has four items")));
    dbBigger.add(TupleFactory.getInstance().newTuple(
        Arrays.asList("This bag has four items")));
    input.add(TupleFactory.getInstance().newTuple(
        Arrays.asList(dbBigger, "bigger")));


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    // DataBags are ordered by size, so the bigger one will be the one
    // containing 4 items
    Assert.assertEquals("bigger", out.get(1));
  }

View Full Code Here

    Assert.assertEquals("bigger", out.get(1));
  }


  @Test
  public void testBiggerTuple() throws Exception {
    ExtremalTupleByNthField o = new ExtremalTupleByNthField("1", "min");


    DataBag input = BagFactory.getInstance().newDefaultBag();
    Tuple tpSmaller = TupleFactory.getInstance().newTuple();
    tpSmaller.append("This is a smaller tuple.");
    tpSmaller.append("This is a smaller tuple.");
    tpSmaller.append("This is a smaller tuple.");
    input.add(TupleFactory.getInstance().newTuple(
        Arrays.asList(tpSmaller, "smaller")));


    Tuple tpBigger = TupleFactory.getInstance().newTuple();
    tpBigger.append("This is a bigger tuple.");
    tpBigger.append("This is a bigger tuple.");
    tpBigger.append("This is a bigger tuple.");
    tpBigger.append("This is a bigger tuple.");
    input.add(TupleFactory.getInstance().newTuple(
        Arrays.asList(tpBigger, "bigger")));


    Tuple tupleInput = TupleFactory.getInstance().newTuple();
    tupleInput.append(input);


    Tuple out = o.exec(tupleInput);


    // DataBags are ordered by size, so the bigger one will be the one
    // containing 4 items
    Assert.assertEquals("smaller", out.get(1));
  }

View Full Code Here

  }


  @Test
  public void testMaxAccumulated() throws Exception {


    ExtremalTupleByNthField o = new ExtremalTupleByNthField("5", "max");


    for (int j = 0; j < 5; ++j) {
      for (int i = 0; i < 100; i++) {
        Tuple t = TupleFactory.getInstance().newTuple();
        t.append(i + j);
        t.append(" " + (i + j));
        t.append(i + j);
        t.append(i + j);
        t.append(i + j);
        o.accumulate(t);
      }


      Tuple out = o.getValue();
      Assert.assertEquals(" " + (99 + j), (String) out.get(1));
      o.cleanup();
    }
  }

View Full Code Here

TOP

Related Classes of org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField

org.apache.pig.data.DataBag

org.apache.pig.data.Tuple

org.apache.pig.piggybank.test.evaluation.TestExtremalTupleByNthField

org.apache.pig.backend.executionengine.ExecException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField

Example1: Invoking the UDF

Example 2: Default behavior

Example 3: Choosing a Large Bag or Tuple

Example 4: Secondary Sort Key

Related Classes of org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField