Examples of org.apache.hadoop.mapreduce.Reducer$Context

org.apache.hadoop.mapreduce.Reducer
Reduces a set of intermediate values which share a key to a smaller set of values.
Reducer implementations can access the {@link Configuration} for the job via the {@link JobContext#getConfiguration()} method.
Reducer has 3 primary phases:
1. Shuffle
  
  The Reducer copies the sorted output from each {@link Mapper} using HTTP across the network.
2. Sort
  
  The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key).
  
  The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.
  
  SecondarySort
  
  To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce.The grouping comparator is specified via {@link Job#setGroupingComparatorClass(Class)}. The sort order is controlled by {@link Job#setSortComparatorClass(Class)}.
  For example, say that you want to find duplicate web pages and tag them all with the url of the "best" known example. You would set up the job like:
  - Map Input Key: url
  - Map Input Value: document
  - Map Output Key: document checksum, url pagerank
  - Map Output Value: url
  - Partitioner: by checksum
  - OutputKeyComparator: by checksum and then decreasing pagerank
  - OutputValueGroupingComparator: by checksum
3. Reduce
  
  In this phase the {@link #reduce(Object,Iterable,Context)}method is called for each <key, (collection of values)> in the sorted inputs.
  
  The output of the reduce task is typically written to a {@link RecordWriter} via {@link Context#write(Object,Object)}.
The output of the Reducer is not re-sorted.

Example:
```
 public class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Key key, Iterable values,  Context context) throws IOException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.collect(key, result); } } 
```
@see Mapper @see Partitioner

     * </ul>
     */
    public static void execute(final String uri, final Evaluator evaluator) {
        try {
            String documentURI = getCanonicalURI(uri);
            Context rootCtx = evaluator.newContext(null);
            Tracer trc = new Tracer();
            SCXML doc = SCXMLReader.read(new URL(documentURI));
            if (doc == null) {
                System.err.println("The SCXML document " + uri
                        + " can not be parsed!");
                System.exit(-1);
            }
            System.out.println(SCXMLWriter.write(doc));
            SCXMLExecutor exec = new SCXMLExecutor(evaluator, null, trc);
            EventDispatcher ed = new SimpleScheduler(exec);
            exec.setEventdispatcher(ed);
            exec.setStateMachine(doc);
            exec.addListener(doc, trc);
            exec.registerInvokerClass("scxml", SimpleSCXMLInvoker.class);
            exec.setRootContext(rootCtx);
            exec.go();
            BufferedReader br = new BufferedReader(new
                InputStreamReader(System.in));
            String event;
            while ((event = br.readLine()) != null) {
                event = event.trim();
                if (event.equalsIgnoreCase("help") || event.equals("?")) {
                    System.out.println("Enter a space-separated list of "
                        + "events");
                    System.out.println("To populate a variable in the "
                        + "current context, type \"name=value\"");
                    System.out.println("To quit, enter \"quit\"");
                    System.out.println("To reset state machine, enter "
                        + "\"reset\"");
                } else if (event.equalsIgnoreCase("quit")) {
                    break;
                } else if (event.equalsIgnoreCase("reset")) {
                    exec.reset();
                } else if (event.indexOf('=') != -1) {
                    int marker = event.indexOf('=');
                    String name = event.substring(0, marker);
                    String value = event.substring(marker + 1);
                    rootCtx.setLocal(name, value);
                    System.out.println("Set variable " + name + " to "
                        + value);
                } else if (event == null || event.trim().length() == 0
                           || event.equalsIgnoreCase("null")) {
                    TriggerEvent[] evts = {new TriggerEvent(null,

View Full Code Here

        else if (!transition.isNoEventsTransition()) {
            return false;
        }
        if (transition.getCond() != null) {
            Boolean result = Boolean.FALSE;
            Context context = exctx.getScInstance().getContext(transition.getParent());
            context.setLocal(Context.NAMESPACES_KEY, transition.getNamespaces());
            try {
                if ((result = exctx.getEvaluator().evalCond(context, transition.getCond())) == null) {
                    result = Boolean.FALSE;
                    if (exctx.getAppLog().isDebugEnabled()) {
                        exctx.getAppLog().debug("Treating as false because the cond expression was evaluated as null: '"
                                + transition.getCond() + "'");
                    }
                }
            }
            catch (SCXMLExpressionException e) {
                exctx.getInternalIOProcessor().addEvent(new TriggerEvent(TriggerEvent.ERROR_EXECUTION, TriggerEvent.ERROR_EVENT));
                exctx.getErrorReporter().onError(ErrorConstants.EXPRESSION_ERROR, "Treating as false due to error: "
                        + e.getMessage(), transition);
            }
            finally {
                context.setLocal(Context.NAMESPACES_KEY, null);
            }
            return result;
        }
        return true;
    }

View Full Code Here

     * @param scInstance the state machine instance holding the system context
     * @param event The event being stored
     * @param internal Flag indicating the event was received internally or externally
     */
    public void setSystemEventVariable(final SCInstance scInstance, final TriggerEvent event, boolean internal) {
        Context systemContext = scInstance.getSystemContext();
        EventVariable eventVar = null;
        if (event != null) {
            String eventType = internal ? EventVariable.TYPE_INTERNAL : EventVariable.TYPE_EXTERNAL;


            final int triggerEventType = event.getType();
            if (triggerEventType == TriggerEvent.ERROR_EVENT || triggerEventType == TriggerEvent.CHANGE_EVENT) {
                eventType = EventVariable.TYPE_PLATFORM;
            }


            // TODO: determine sendid, origin, originType and invokeid based on context later.
            eventVar = new EventVariable(event.getName(), eventType, null, null, null, null, event.getPayload());
        }
        systemContext.setLocal(SCXMLSystemContext.EVENT_KEY, eventVar);
    }

View Full Code Here

        Evaluator eval = exctx.getEvaluator();
        for (TransitionalState ts : statesToInvoke) {
            if (ts.getInvokes().isEmpty()) {
                continue;
            }
            Context context = scInstance.getContext(ts);
            for (Invoke i : ts.getInvokes()) {
                String src = i.getSrc();
                if (src == null) {
                    String srcexpr = i.getSrcexpr();
                    Object srcObj;
                    try {
                        context.setLocal(Context.NAMESPACES_KEY, i.getNamespaces());
                        srcObj = eval.eval(context, srcexpr);
                        context.setLocal(Context.NAMESPACES_KEY, null);
                        src = String.valueOf(srcObj);
                    } catch (SCXMLExpressionException see) {
                        exctx.getInternalIOProcessor().addEvent(new TriggerEvent(TriggerEvent.ERROR_EXECUTION, TriggerEvent.ERROR_EVENT));
                        exctx.getErrorReporter().onError(ErrorConstants.EXPRESSION_ERROR, see.getMessage(), i);
                    }
                }
                String source = src;
                PathResolver pr = i.getPathResolver();
                if (pr != null) {
                    source = i.getPathResolver().resolvePath(src);
                }
                Invoker inv;
                try {
                    inv = exctx.newInvoker(i.getType());
                } catch (InvokerException ie) {
                    exctx.getInternalIOProcessor().addEvent(new TriggerEvent("failed.invoke."+ts.getId(), TriggerEvent.ERROR_EVENT));
                    continue;
                }
                List<Param> params = i.params();
                Map<String, Object> args = new HashMap<String, Object>();
                for (Param p : params) {
                    String argExpr = p.getExpr();
                    Object argValue = null;
                    context.setLocal(Context.NAMESPACES_KEY, p.getNamespaces());
                    // Do we have an "expr" attribute?
                    if (argExpr != null && argExpr.trim().length() > 0) {
                        try {
                            argValue = eval.eval(context, argExpr);
                        } catch (SCXMLExpressionException see) {
                            exctx.getInternalIOProcessor().addEvent(new TriggerEvent(TriggerEvent.ERROR_EXECUTION, TriggerEvent.ERROR_EVENT));
                            exctx.getErrorReporter().onError(ErrorConstants.EXPRESSION_ERROR, see.getMessage(), i);
                        }
                    } else {
                        // No. Does value of "name" attribute refer to a valid
                        // location in the data model?
                        try {
                            argValue = eval.evalLocation(context, p.getName());
                            if (argValue == null) {
                                // Generate error, 4.3.1 in WD-scxml-20080516
                                exctx.getInternalIOProcessor().addEvent(new TriggerEvent(ts.getId() + ERR_ILLEGAL_ALLOC, TriggerEvent.ERROR_EVENT));
                            }
                        } catch (SCXMLExpressionException see) {
                            exctx.getInternalIOProcessor().addEvent(new TriggerEvent(TriggerEvent.ERROR_EXECUTION, TriggerEvent.ERROR_EVENT));
                            exctx.getErrorReporter().onError(ErrorConstants.EXPRESSION_ERROR, see.getMessage(), i);
                        }
                    }
                    context.setLocal(Context.NAMESPACES_KEY, null);
                    args.put(p.getName(), argValue);
                }
                String invokeId = exctx.setInvoker(i, inv);
                inv.setInvokeId(invokeId);
                inv.setParentIOProcessor(exctx.getExternalIOProcessor());

View Full Code Here

  public void testUseOfWritableRegisteredComparator() throws IOException {
    
    // this test should use the comparator registered inside TestWritable
    // to output the keys in reverse order
    MapReduceDriver<TestWritable,Text,TestWritable,Text,TestWritable,Text> driver 
      = MapReduceDriver.newMapReduceDriver(new Mapper(), new Reducer());
    
    driver.withInput(new TestWritable("A1"), new Text("A1"))
      .withInput(new TestWritable("A2"), new Text("A2"))
      .withInput(new TestWritable("A3"), new Text("A3"))
      .withKeyGroupingComparator(new TestWritable.SingleGroupComparator())

View Full Code Here


  @Test
  public void testClosedFormMapReduce() throws IOException {
    
    MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver 
      = MapReduceDriver.newMapReduceDriver(new StatefulMapper(), new Reducer());
    
    mapReduceDriver.addInput(new LongWritable(1L), new Text("hello"));
    mapReduceDriver.addInput(new LongWritable(2L), new Text("schmo"));
    mapReduceDriver.withOutput(new Text("SomeKey"), new IntWritable(2));
    mapReduceDriver.runTest();

View Full Code Here

  /**
   * Tests that method context.getNumReduceTasks() correctly returns value from configuration
   * */
  @Test
  public void testContext_getNumReduceTasks() throws Exception {
    Reducer reducer = new TestReducer();
    ReduceDriver driver = ReduceDriver.newReduceDriver(reducer);
    driver.getConfiguration().setInt("mapred.reduce.tasks", 10);


    List<Text> values = new ArrayList();
    values.add(new Text("bb"));

View Full Code Here

  public void testUseOfWritableRegisteredComparator() throws IOException {


    // this test should use the comparator registered inside TestWritable
    // to output the keys in reverse order
    MapReduceDriver<TestWritable, Text, TestWritable, Text, TestWritable, Text> driver = MapReduceDriver
        .newMapReduceDriver(new Mapper(), new Reducer());


    driver
        .withInput(new TestWritable("A1"), new Text("A1"))
        .withInput(new TestWritable("A2"), new Text("A2"))
        .withInput(new TestWritable("A3"), new Text("A3"))

View Full Code Here


  @Test
  public void testClosedFormMapReduce() throws IOException {


    MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver
      = MapReduceDriver.newMapReduceDriver(new StatefulMapper(), new Reducer());


    mapReduceDriver.addInput(new LongWritable(1L), new Text("hello"));
    mapReduceDriver.addInput(new LongWritable(2L), new Text("schmo"));
    mapReduceDriver.withOutput(new Text("SomeKey"), new IntWritable(2));
    mapReduceDriver.runTest();

View Full Code Here

    context.getConfiguration().setClassLoader(basicMapReduceContext.getProgram().getClassLoader());
    basicMapReduceContext.getMetricsCollectionService().startAndWait();


    try {
      String userReducer = context.getConfiguration().get(ATTR_REDUCER_CLASS);
      Reducer delegate = createReducerInstance(context.getConfiguration().getClassLoader(), userReducer);


      // injecting runtime components, like datasets, etc.
      try {
        Reflections.visit(delegate, TypeToken.of(delegate.getClass()),
                          new PropertyFieldSetter(basicMapReduceContext.getSpecification().getProperties()),
                          new MetricsFieldSetter(basicMapReduceContext.getMetrics()),
                          new DataSetFieldSetter(basicMapReduceContext));
      } catch (Throwable t) {
        LOG.error("Failed to inject fields to {}.", delegate.getClass(), t);
        throw Throwables.propagate(t);
      }


      LoggingContextAccessor.setLoggingContext(basicMapReduceContext.getLoggingContext());


      // this is a hook for periodic flushing of changes buffered by datasets (to avoid OOME)
      WrappedReducer.Context flushingContext = createAutoFlushingContext(context, basicMapReduceContext);


      if (delegate instanceof ProgramLifecycle) {
        try {
          ((ProgramLifecycle<BasicMapReduceContext>) delegate).initialize(basicMapReduceContext);
        } catch (Exception e) {
          LOG.error("Failed to initialize mapper with " + basicMapReduceContext.toString(), e);
          throw Throwables.propagate(e);
        }
      }


      delegate.run(flushingContext);
      // sleep to allow metrics to be written
      TimeUnit.SECONDS.sleep(2L);


      // transaction is not finished, but we want all operations to be dispatched (some could be buffered in
      // memory by tx agent

View Full Code Here

0 1

TOP

Related Classes of org.apache.hadoop.mapreduce.Reducer$Context

ca.eandb.jmist.framework.color.Color

ca.eandb.jmist.framework.Intersection

ca.uhn.fhir.model.dstu.composite.CodeableConceptDt

ca.uhn.fhir.model.dstu.composite.PeriodDt

com.asakusafw.utils.java.model.syntax.Expression

com.dianping.cat.message.spi.MessageTree

com.dianping.cat.message.Transaction

com.facebook.presto.hive.shaded.com.google.common.util.concurrent.ThreadFactoryBuilder

com.google.code.or.binlog.impl.event.RotateEvent

com.google.code.or.binlog.impl.event.TableMapEvent

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.hadoop.mapreduce.Reducer$Context

Shuffle

Sort

SecondarySort

Reduce

Related Classes of org.apache.hadoop.mapreduce.Reducer$Context