Examples of Reducer

com.hazelcast.mapreduce.Reducer
The abstract Reducer class is used to build reducers for the {@link Job}.
Reducers may be distributed inside of the cluster but there is always only one Reducer per key.
Reducers are called in a threadsafe way so internal locking is not required.

Due to the fact that there is only one Reducer per key mapped values needs to be transmitted to one of the cluster nodes. To reduce the traffic costs between the nodes a {@link Combiner} implementation can be added to the call which runs alongsidethe mapper to pre-reduce mapped values into intermediate results.

A simple Reducer implementation could look like that sum-function implementation:
```
 public class SumReducer implements Reducer<String, Integer, Integer> { private int sum = 0; public void reduce( String key, Integer value ) { sum += value; } public Integer finalizeReduce() { return sum; } } 
```
@param < KeyIn> key type of the resulting keys @param < ValueIn> value type of the incoming values @param < ValueOut> value type of the reduced values @since 3.2
loop.Reducer
Takes an AST generated by the parser and strips it of unnecessary elements. Also optimizes the AST to be in its most compact form. This is sometimes referred to as reducing the "parse tree" to an AST (abstract syntax tree). Nothing done in this phase affects the semantics of the program.
org.apache.hadoop.hive.contrib.mr.Reducer
Simple reducer interface.
org.apache.hadoop.mapred.Reducer
ess int noValues = 0; while (values.hasNext()) { V value = values.next(); // Increment the no. of values for this key ++noValues; // Process the <key, value> pair (assume this takes a while) // ... // ... // Let the framework know that we are alive, and kicking! if ((noValues%10) == 0) { reporter.progress(); } // Process some more // ... // ... // Output the <key, value> output.collect(key, value); } // Increment the no. of <key, list of values> pairs processed ++noKeys; // Increment counters reporter.incrCounter(NUM_RECORDS, 1); // Every 100 keys update application-level status if ((noKeys%100) == 0) { reporter.setStatus(reduceTaskId + " processed " + noKeys); } } }
@see Mapper @see Partitioner @see Reporter @see MapReduceBase
org.apache.hadoop.mapreduce.Reducer
Reduces a set of intermediate values which share a key to a smaller set of values.
Reducer implementations can access the {@link Configuration} for the job via the {@link JobContext#getConfiguration()} method.
Reducer has 3 primary phases:
1. Shuffle
  
  The Reducer copies the sorted output from each {@link Mapper} using HTTP across the network.
2. Sort
  
  The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key).
  
  The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.
  
  SecondarySort
  
  To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce.The grouping comparator is specified via {@link Job#setGroupingComparatorClass(Class)}. The sort order is controlled by {@link Job#setSortComparatorClass(Class)}.
  For example, say that you want to find duplicate web pages and tag them all with the url of the "best" known example. You would set up the job like:
  - Map Input Key: url
  - Map Input Value: document
  - Map Output Key: document checksum, url pagerank
  - Map Output Value: url
  - Partitioner: by checksum
  - OutputKeyComparator: by checksum and then decreasing pagerank
  - OutputValueGroupingComparator: by checksum
3. Reduce
  
  In this phase the {@link #reduce(Object,Iterable,Context)}method is called for each <key, (collection of values)> in the sorted inputs.
  
  The output of the reduce task is typically written to a {@link RecordWriter} via {@link Context#write(Object,Object)}.
The output of the Reducer is not re-sorted.

Example:
```
 public class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Key key, Iterable values,  Context context) throws IOException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.collect(key, result); } } 
```
@see Mapper @see Partitioner
rationals.transformations.Reducer

Examples of com.hazelcast.mapreduce.Reducer

        }
    }


    private void reduceChunk(Map<Key, Chunk> chunk) {
        for (Map.Entry<Key, Chunk> entry : chunk.entrySet()) {
            Reducer reducer = supervisor.getReducerByKey(entry.getKey());
            if (reducer != null) {
                Chunk chunkValue = entry.getValue();
                if (chunkValue instanceof List) {
                    for (Object value : (List) chunkValue) {
                        reducer.reduce(value);
                    }
                } else {
                    reducer.reduce(chunkValue);
                }
            }
        }
    }

View Full Code Here

Examples of com.hazelcast.mapreduce.Reducer

        }
        return result;
    }


    public <KeyIn, ValueIn, ValueOut> Reducer<KeyIn, ValueIn, ValueOut> getReducerByKey(Object key) {
        Reducer reducer = reducers.get(key);
        if (reducer == null && configuration.getReducerFactory() != null) {
            reducer = configuration.getReducerFactory().newReducer(key);
            Reducer oldReducer = reducers.putIfAbsent(key, reducer);
            if (oldReducer != null) {
                reducer = oldReducer;
            } else {
                reducer.beginReduce(key);
            }

View Full Code Here

Examples of com.hazelcast.mapreduce.Reducer

        }
        return result;
    }


    public <KeyIn, ValueIn, ValueOut> Reducer<ValueIn, ValueOut> getReducerByKey(Object key) {
        Reducer reducer = reducers.get(key);
        if (reducer == null && configuration.getReducerFactory() != null) {
            reducer = configuration.getReducerFactory().newReducer(key);
            Reducer oldReducer = reducers.putIfAbsent(key, reducer);
            if (oldReducer != null) {
                reducer = oldReducer;
            } else {
                reducer.beginReduce();
            }

View Full Code Here

Examples of com.hazelcast.mapreduce.Reducer

        }
    }


    private void reduceChunk(Map<Key, Chunk> chunk) {
        for (Map.Entry<Key, Chunk> entry : chunk.entrySet()) {
            Reducer reducer = supervisor.getReducerByKey(entry.getKey());
            if (reducer != null) {
                Chunk chunkValue = entry.getValue();
                if (chunkValue instanceof CombinerResultList) {
                    for (Object value : (List) chunkValue) {
                        reducer.reduce(value);
                    }
                } else {
                    reducer.reduce(chunkValue);
                }
            }
        }
    }

View Full Code Here

Examples of com.hazelcast.mapreduce.Reducer

        }
        return result;
    }


    public <KeyIn, ValueIn, ValueOut> Reducer<ValueIn, ValueOut> getReducerByKey(Object key) {
        Reducer reducer = reducers.get(key);
        if (reducer == null && configuration.getReducerFactory() != null) {
            reducer = configuration.getReducerFactory().newReducer(key);
            Reducer oldReducer = reducers.putIfAbsent(key, reducer);
            if (oldReducer != null) {
                reducer = oldReducer;
            } else {
                reducer.beginReduce();
            }

View Full Code Here

Examples of loop.Reducer

    return null;
  }


  public void reduceAll() {
    for (ClassDecl classDecl : classes.values()) {
      new Reducer(classDecl).reduce();
    }
    for (FunctionDecl functionDecl : functions.values()) {
      new Reducer(functionDecl).reduce();
    }


    if (initializer != null) {
      List<Node> reduced = new ArrayList<Node>(initializer.size());
      for (Node node : initializer) {
        reduced.add(new Reducer(node).reduce());
      }
      initializer = reduced;
    }
  }

View Full Code Here

Examples of org.apache.hadoop.hive.contrib.mr.Reducer

/**
 * Example Reducer (WordCount). 
 */
public final class WordCountReduce {
  public static void main(final String[] args) throws Exception {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      public void reduce(String key, Iterator<String[]> records, Output output) throws Exception {
        int count = 0;
        
        while (records.hasNext()) {
          // note we use col[1] -- the key is provided again as col[0]

View Full Code Here

Examples of org.apache.hadoop.hive.contrib.mr.Reducer

  private WordCountReduce() {
    // prevent instantiation
  }


  public static void main(final String[] args) throws Exception {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      public void reduce(String key, Iterator<String[]> records, Output output)
          throws Exception {
        int count = 0;


        while (records.hasNext()) {

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.Reducer

  
  private void combineAndSpill(
      RawKeyValueIterator kvIter,
      Counters.Counter inCounter) throws IOException {
    JobConf job = jobConf;
    Reducer combiner = ReflectionUtils.newInstance(combinerClass, job);
    Class<K> keyClass = (Class<K>) job.getMapOutputKeyClass();
    Class<V> valClass = (Class<V>) job.getMapOutputValueClass();
    RawComparator<K> comparator = 
      (RawComparator<K>)job.getOutputKeyComparator();
    try {
      CombineValuesIterator values = new CombineValuesIterator(
          kvIter, comparator, keyClass, valClass, job, Reporter.NULL,
          inCounter);
      while (values.more()) {
        combiner.reduce(values.getKey(), values, combineCollector,
                        Reporter.NULL);
        values.nextKey();
      }
    } finally {
      combiner.close();
    }
  }

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of Reducer

Shuffle

Sort

SecondarySort

Reduce

Examples of com.hazelcast.mapreduce.Reducer

Examples of com.hazelcast.mapreduce.Reducer

Examples of com.hazelcast.mapreduce.Reducer

Examples of com.hazelcast.mapreduce.Reducer

Examples of com.hazelcast.mapreduce.Reducer

Examples of loop.Reducer

Examples of org.apache.hadoop.hive.contrib.mr.Reducer

Examples of org.apache.hadoop.hive.contrib.mr.Reducer

Examples of org.apache.hadoop.mapred.Reducer

Examples of org.apache.hadoop.mapred.Reducer