Examples of com.hazelcast.mapreduce.Combiner

com.hazelcast.mapreduce.Combiner
The abstract Combiner class is used to build combiners for the {@link Job}.
Those Combiners are distributed inside of the cluster and are running alongside the {@link Mapper} implementations in the same node.
Combiners are called in a threadsafe way so internal locking is not required.

Combiners are normally used to build intermediate results on the mapping nodes to lower the traffic overhead between the different nodes before the reducing phase.
Combiners need to be capable of combining data in multiple chunks to create a more streaming like internal behavior.

A simple Combiner implementation in combination with a {@link Reducer} could looklike that avg-function implementation:
```
 public class AvgCombiner implements Combiner<String, Integer, Tuple<Long, Long>> { private long count; private long amount; public void combine(String key, Integer value) { count++; amount += value; } public Tuple<Long, Long> finalizeChunk() { Tuple<Long, Long> tuple = new Tuple<>(count, amount); count = 0; amount = 0; return tuple; } } public class SumReducer implements Reducer<String, Tuple<Long, Long>, Integer> { private long count; private long amount; public void reduce( String key, Tuple<Long, Long> value ) { count += value.getFirst(); amount += value.getSecond(); } public Integer finalizeReduce() { return amount / count; } } 
```
@param < KeyIn> key type of the resulting keys @param < ValueIn> value type of the incoming values @param < ValueOut> value type of the reduced values @since 3.2

        class TestCombinerFactory
                implements CombinerFactory {


            @Override
            public Combiner newCombiner(Object key) {
                return new Combiner() {
                    @Override
                    public void combine(Object value) {
                    }


                    @Override
                    public Object finalizeChunk() {
                        return null;
                    }
                };
            }
        }


        class CreationTask
                implements Runnable {
            private final CountDownLatch latchStart;
            private final CountDownLatch latchEnd;
            private final AtomicReferenceArray<Combiner> array;
            private final DefaultContext<Integer, Integer> defaultContext;
            private final int index;


            CreationTask(CountDownLatch latchStart, CountDownLatch latchEnd, AtomicReferenceArray<Combiner> array,
                         DefaultContext<Integer, Integer> defaultContext, int index) {
                this.latchStart = latchStart;
                this.latchEnd = latchEnd;
                this.array = array;
                this.defaultContext = defaultContext;
                this.index = index;
            }


            @Override
            public void run() {
                try {
                    latchStart.await();
                    Combiner combiner = defaultContext.getOrCreateCombiner(1);
                    array.set(index, combiner);


                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    latchEnd.countDown();
                }
            }
        }


        int threadCount = 20;


        AtomicReferenceArray<Combiner> combiners = new AtomicReferenceArray<Combiner>(threadCount);
        DefaultContext<Integer, Integer> context = new DefaultContext<Integer, Integer>(new TestCombinerFactory(), null);


        CountDownLatch latchStart = new CountDownLatch(1);
        CountDownLatch latchEnd = new CountDownLatch(threadCount);


        for (int i = 0; i < threadCount; i++) {
            Thread t = new Thread(new CreationTask(latchStart, latchEnd, combiners, context, i));
            t.start();
        }


        latchStart.countDown();
        latchEnd.await(1, TimeUnit.MINUTES);


        for (int i = 0; i < threadCount - 1; i++) {
            Combiner c1 = combiners.get(i);
            Combiner c2 = combiners.get(i + 1);
            assertTrue("Returned combiners are not identical: " + c1 + " -> " + c2, c1 == c2);
        }
    }

View Full Code Here

        class TestCombinerFactory
                implements CombinerFactory {


            @Override
            public Combiner newCombiner(Object key) {
                return new Combiner() {
                    @Override
                    public void combine(Object value) {
                    }


                    @Override

View Full Code Here


            @Override
            public void run() {
                try {
                    latchStart.await();
                    Combiner combiner = defaultContext.getOrCreateCombiner(1);
                    array.set(index, combiner);


                } catch (Exception e) {
                    e.printStackTrace();
                } finally {

View Full Code Here

TOP

Related Classes of com.hazelcast.mapreduce.Combiner

com.hazelcast.mapreduce.impl.task.CreationTask

com.hazelcast.mapreduce.impl.task.TestCombinerFactory

com.hazelcast.mapreduce.impl.task.ThreadsafeCombinerTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.