Examples of Combiner

ch.epfl.lbd.datamining.structure.Combiner
com.hazelcast.mapreduce.Combiner
The abstract Combiner class is used to build combiners for the {@link Job}.
Those Combiners are distributed inside of the cluster and are running alongside the {@link Mapper} implementations in the same node.
Combiners are called in a threadsafe way so internal locking is not required.

Combiners are normally used to build intermediate results on the mapping nodes to lower the traffic overhead between the different nodes before the reducing phase.
Combiners need to be capable of combining data in multiple chunks to create a more streaming like internal behavior.

A simple Combiner implementation in combination with a {@link Reducer} could looklike that avg-function implementation:
```
 public class AvgCombiner implements Combiner<String, Integer, Tuple<Long, Long>> { private long count; private long amount; public void combine(String key, Integer value) { count++; amount += value; } public Tuple<Long, Long> finalizeChunk() { Tuple<Long, Long> tuple = new Tuple<>(count, amount); count = 0; amount = 0; return tuple; } } public class SumReducer implements Reducer<String, Tuple<Long, Long>, Integer> { private long count; private long amount; public void reduce( String key, Tuple<Long, Long> value ) { count += value.getFirst(); amount += value.getSecond(); } public Integer finalizeReduce() { return amount / count; } } 
```
@param < KeyIn> key type of the resulting keys @param < ValueIn> value type of the incoming values @param < ValueOut> value type of the reduced values @since 3.2
org.apache.accumulo.core.iterators.Combiner
A SortedKeyValueIterator that combines the Values for different versions (timestamp) of a Key within a row into a single Value. Combiner will replace one or more versions of a Key and their Values with the most recent Key and a Value which is the result of the reduce method. An {@link Column}which only specifies a column family will combine all Keys in that column family individually. Similarly, a {@link Column} which specifies acolumn family and column qualifier will combine all Keys in column family and qualifier individually. Combination is only ever performed on multiple versions and not across column qualifiers or column visibilities. Implementations must provide a reduce method: {@code public Value reduce(Key key, Iterator iter)}. This reduce method will be passed the most recent Key and an iterator over the Values for all non-deleted versions of that Key. A combiner will not combine keys that differ by more than the timestamp. This class and its implementations do not automatically filter out unwanted columns from those being combined, thus it is generally recommended to use a {@link Combiner} implementation with the {@link ScannerBase#fetchColumnFamily(Text)} or {@link ScannerBase#fetchColumn(Text,Text)} methods.
org.apache.tez.runtime.library.common.combine.Combiner
Combiner Initialization
The Combiner class is picked up using the TEZ_RUNTIME_COMBINER_CLASS attribute in {@link TezRuntimeConfiguration}Partitioners need to provide a single argument ( {@link TezRawKeyValueIterator}) constructor.

Examples of org.apache.accumulo.core.iterators.Combiner

    tm.put(new Key("r", "count", "b", 3), new Value("1".getBytes()));
    tm.put(new Key("r", "count", "b", 4), new Value("1".getBytes()));
    tm.put(new Key("r", "count", "b", 5), new Value("1".getBytes()));
    tm.put(new Key("r", "count", "c", 6), new Value("1".getBytes()));
    SortedMapIterator smi = new SortedMapIterator(tm);
    Combiner iter = new SummingCombiner();
    IteratorSetting s = new IteratorSetting(10, "s", SummingCombiner.class);
    SummingCombiner.setColumns(s, Collections.singletonList(new IteratorSetting.Column("count")));
    SummingCombiner.setEncodingType(s, LongCombiner.StringEncoder.class);
    iter.init(smi, s.getOptions(), new DefaultIteratorEnvironment());
    Combiner iter2 = new SummingCombiner();
    IteratorSetting s2 = new IteratorSetting(10, "s2", SummingCombiner.class);
    SummingCombiner.setColumns(s2, Collections.singletonList(new IteratorSetting.Column("count","a")));
    SummingCombiner.setEncodingType(s2, LongCombiner.StringEncoder.class);
    iter2.init(iter, s.getOptions(), new DefaultIteratorEnvironment());
    iter2.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(iter2.hasTop());
    assertEquals("2", iter2.getTopValue().toString());
    iter2.next();
    assertTrue(iter2.hasTop());
    assertEquals("3", iter2.getTopValue().toString());
    iter2.next();
    assertTrue(iter2.hasTop());
    assertEquals("1", iter2.getTopValue().toString());
    iter2.next();
    assertFalse(iter2.hasTop());
  }

View Full Code Here

Examples of org.apache.accumulo.core.iterators.Combiner

    // keys that aggregate
    nkv(tm1, 1, 1, 1, 1, false, 4l, encoder);
    nkv(tm1, 1, 1, 1, 2, false, 3l, encoder);
    nkv(tm1, 1, 1, 1, 3, false, 2l, encoder);
    
    Combiner ai = new MaxCombiner();
    
    IteratorSetting is = new IteratorSetting(1, SummingCombiner.class);
    LongCombiner.setEncodingType(is, SummingCombiner.Type.VARLEN);
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("4", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    ai = new MinCombiner();
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("2", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
  }

View Full Code Here

Examples of org.apache.accumulo.core.iterators.Combiner

    // keys that aggregate
    nkv(tm1, 1, 1, 1, 1, false, nal(1l, 2l), encoder);
    nkv(tm1, 1, 1, 1, 2, false, nal(3l, 4l, 5l), encoder);
    nkv(tm1, 1, 1, 1, 3, false, nal(), encoder);
    
    Combiner ai = new SummingArrayCombiner();
    
    IteratorSetting is = new IteratorSetting(1, SummingArrayCombiner.class);
    SummingArrayCombiner.setEncodingType(is, type);
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertBytesEqual(encoder.encode(nal(4l, 6l, 5l)), ai.getTopValue().get());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    is.clearOptions();
    SummingArrayCombiner.setEncodingType(is, encoderClass);
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertBytesEqual(encoder.encode(nal(4l, 6l, 5l)), ai.getTopValue().get());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    is.clearOptions();
    SummingArrayCombiner.setEncodingType(is, encoderClass.getName());
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertBytesEqual(encoder.encode(nal(4l, 6l, 5l)), ai.getTopValue().get());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    is.clearOptions();
    SummingArrayCombiner.setEncodingType(is, SummingCombiner.VAR_LEN_ENCODER.getClass().getName());
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    try {
      ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
      Assert.fail();
    } catch (IllegalArgumentException e) {}
    
    is.clearOptions();
    SummingArrayCombiner.setEncodingType(is, BadEncoder.class.getName());
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    try {
      ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
      Assert.fail();
    } catch (IllegalArgumentException e) {}
  }

View Full Code Here

Examples of org.apache.accumulo.core.iterators.Combiner

    // keys that aggregate
    nkv(tm1, 1, 1, 1, 1, false, 2l, encoder);
    nkv(tm1, 1, 1, 1, 2, false, 3l, encoder);
    nkv(tm1, 1, 1, 1, 3, false, 4l, encoder);
    
    Combiner ai = new SummingCombiner();
    
    IteratorSetting is = new IteratorSetting(1, SummingCombiner.class);
    LongCombiner.setEncodingType(is, VarLenEncoder.class);
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    // try seeking to the beginning of a key that aggregates
    
    ai.seek(nr(1, 1, 1, 3), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    // try seeking the middle of a key the aggregates
    ai.seek(nr(1, 1, 1, 2), EMPTY_COL_FAMS, false);
    
    assertFalse(ai.hasTop());
    
    // try seeking to the end of a key the aggregates
    ai.seek(nr(1, 1, 1, 1), EMPTY_COL_FAMS, false);
    
    assertFalse(ai.hasTop());
    
    // try seeking before a key the aggregates
    ai.seek(nr(1, 1, 1, 4), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
  }

View Full Code Here

Examples of org.apache.accumulo.core.iterators.Combiner

    
    // keys that do not aggregate
    nkv(tm1, 2, 2, 1, 1, false, 2l, encoder);
    nkv(tm1, 2, 2, 1, 2, false, 3l, encoder);
    
    Combiner ai = new SummingCombiner();
    
    IteratorSetting is = new IteratorSetting(1, SummingCombiner.class);
    LongCombiner.setEncodingType(is, FixedLenEncoder.class.getName());
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 2), ai.getTopKey());
    assertEquals("3", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 1), ai.getTopKey());
    assertEquals("2", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    // seek after key that aggregates
    ai.seek(nr(1, 1, 1, 2), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 2), ai.getTopKey());
    assertEquals("3", encoder.decode(ai.getTopValue().get()).toString());
    
    // seek before key that aggregates
    ai.seek(nr(1, 1, 1, 4), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 2), ai.getTopKey());
    assertEquals("3", encoder.decode(ai.getTopValue().get()).toString());
    
  }

View Full Code Here

Examples of org.apache.accumulo.core.iterators.Combiner

    
    // keys that do not aggregate
    nkv(tm1, 2, 2, 1, 1, false, 2l, encoder);
    nkv(tm1, 2, 2, 1, 2, false, 3l, encoder);
    
    Combiner ai = new SummingCombiner();
    
    IteratorSetting is = new IteratorSetting(1, SummingCombiner.class);
    LongCombiner.setEncodingType(is, FixedLenEncoder.class.getName());
    Combiner.setColumns(is, Collections.singletonList(new IteratorSetting.Column("cf001")));
    
    ai.init(new SortedMapIterator(tm1), is.getOptions(), null);
    
    SortedKeyValueIterator<Key,Value> ai2 = ai.deepCopy(null);
    SortedKeyValueIterator<Key,Value> ai3 = ai.deepCopy(null);
    ai.seek(new Range(), EMPTY_COL_FAMS, false);
    
    assertTrue(ai.hasTop());
    assertEquals(nk(1, 1, 1, 3), ai.getTopKey());
    assertEquals("9", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 2), ai.getTopKey());
    assertEquals("3", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertTrue(ai.hasTop());
    assertEquals(nk(2, 2, 1, 1), ai.getTopKey());
    assertEquals("2", encoder.decode(ai.getTopValue().get()).toString());
    
    ai.next();
    
    assertFalse(ai.hasTop());
    
    // seek after key that aggregates
    ai2.seek(nr(1, 1, 1, 2), EMPTY_COL_FAMS, false);
    
    assertTrue(ai2.hasTop());

View Full Code Here

Examples of org.apache.tez.runtime.library.common.combine.Combiner

          TezRuntimeConfiguration.TEZ_RUNTIME_IFILE_READAHEAD_BYTES_DEFAULT);
    } else {
      this.ifileReadAheadLength = 0;
    }
    
    Combiner combiner = TezRuntimeUtils.instantiateCombiner(conf, inputContext);
    
    FileSystem localFS = FileSystem.getLocal(this.conf);
    LocalDirAllocator localDirAllocator = 
        new LocalDirAllocator(TezRuntimeFrameworkConfigs.LOCAL_DIRS);

View Full Code Here

Examples of org.apache.tez.runtime.library.common.combine.Combiner

      clazz = (Class<? extends Combiner>) conf.getClassByName(className);
    } catch (ClassNotFoundException e) {
      throw new IOException("Unable to load combiner class: " + className);
    }
    
    Combiner combiner = null;
    
      Constructor<? extends Combiner> ctor;
      try {
        ctor = clazz.getConstructor(TaskContext.class);
        combiner = ctor.newInstance(taskContext);

View Full Code Here

Examples of org.apache.tez.runtime.library.common.combine.Combiner

          TezJobConfig.TEZ_RUNTIME_IFILE_READAHEAD_BYTES_DEFAULT);
    } else {
      this.ifileReadAheadLength = 0;
    }
    
    Combiner combiner = TezRuntimeUtils.instantiateCombiner(conf, inputContext);
    
    FileSystem localFS = FileSystem.getLocal(this.conf);
    LocalDirAllocator localDirAllocator = 
        new LocalDirAllocator(TezJobConfig.LOCAL_DIRS);

View Full Code Here

Examples of org.apache.tez.runtime.library.common.combine.Combiner

          TezJobConfig.TEZ_RUNTIME_IFILE_READAHEAD_BYTES_DEFAULT);
    } else {
      this.ifileReadAheadLength = 0;
    }
    
    Combiner combiner = TezRuntimeUtils.instantiateCombiner(conf, inputContext);
    
    FileSystem localFS = FileSystem.getLocal(this.conf);
    LocalDirAllocator localDirAllocator = 
        new LocalDirAllocator(TezJobConfig.LOCAL_DIRS);

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.