Examples of org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.OutputCommitter
OutputCommitter describes the commit of task output for a Map-Reduce job.
The Map-Reduce framework relies on the OutputCommitter of the job to:
1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion.
3. Setup the task temporary output.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output.
6. Discard the task commit.
The methods in this class can be called from several different processes and from several different contexts. It is important to know which process and which context each is called from. Each method should be marked accordingly in its documentation. It is also important to note that not all methods are guaranteed to be called once and only once. If a method is not guaranteed to have this property the output committer needs to handle this appropriately. Also note it will only be in rare situations where they may be called multiple times for the same task. @see org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter @see JobContext @see TaskAttemptContext

        int id = sp.getId();
        setVarsInConf(id);
        HCatOutputFormat outFormat = new HCatOutputFormat();
        TaskAttemptContext cntxt = HCatHadoopShims.Instance.get().createTaskAttemptContext
            (conf, new TaskAttemptID(HCatHadoopShims.Instance.get().createTaskID(), id));
        OutputCommitter committer = null;
        RecordWriter<WritableComparable<?>, HCatRecord> writer;
        try {
            committer = outFormat.getOutputCommitter(cntxt);
            committer.setupTask(cntxt);
            writer = outFormat.getRecordWriter(cntxt);
            while (recordItr.hasNext()) {
                HCatRecord rec = recordItr.next();
                writer.write(null, rec);
            }
            writer.close(cntxt);
            if (committer.needsTaskCommit(cntxt)) {
                committer.commitTask(cntxt);
            }
        } catch (IOException e) {
            if (null != committer) {
                try {
                    committer.abortTask(cntxt);
                } catch (IOException e1) {
                    throw new HCatException(ErrorType.ERROR_INTERNAL_EXCEPTION, e1);
                }
            }
            throw new HCatException("Failed while writing", e);
        } catch (InterruptedException e) {
            if (null != committer) {
                try {
                    committer.abortTask(cntxt);
                } catch (IOException e1) {
                    throw new HCatException(ErrorType.ERROR_INTERNAL_EXCEPTION, e1);
                }
            }
            throw new HCatException("Failed while writing", e);

View Full Code Here

      conf.getVertexOutputFormatClass() != null) {
      try {
        LOG.info("Master is ready to commit final job output data.");
        VertexOutputFormat vertexOutputFormat =
          conf.createVertexOutputFormat();
        OutputCommitter outputCommitter =
          vertexOutputFormat.getOutputCommitter(proxy);
        // now we will have our output in OUTDIR if all went well...
        outputCommitter.commitJob(proxy);
        LOG.info("Master has committed the final job output data.");
      } catch (InterruptedException ie) {
        LOG.error("Interrupted while attempting to obtain " +
          "OutputCommitter.", ie);
      } catch (IOException ioe) {

View Full Code Here

      "saveVertices: Done saving vertices.");
    // YARN: must complete the commit the "task" output, Hadoop isn't there.
    if (getConfiguration().isPureYarnJob() &&
      getConfiguration().getVertexOutputFormatClass() != null) {
      try {
        OutputCommitter outputCommitter =
          vertexOutputFormat.getOutputCommitter(getContext());
        if (outputCommitter.needsTaskCommit(getContext())) {
          LoggerUtils.setStatusAndLog(getContext(), LOG, Level.INFO,
            "OutputCommitter: committing task output.");
          // transfer from temp dirs to "task commit" dirs to prep for
          // the master's OutputCommitter#commitJob(context) call to finish.
          outputCommitter.commitTask(getContext());
        }
      } catch (InterruptedException ie) {
        LOG.error("Interrupted while attempting to obtain " +
          "OutputCommitter.", ie);
      } catch (IOException ioe) {

View Full Code Here

    conf.setInt(MRJobConfig.NUM_REDUCES, 0);
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    OutputCommitter committer = mock(OutputCommitter.class);
    CommitterEventHandler commitHandler =
        createCommitterEventHandler(dispatcher, committer);
    commitHandler.init(conf);
    commitHandler.start();

View Full Code Here

    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    CyclicBarrier syncBarrier = new CyclicBarrier(2);
    OutputCommitter committer = new TestingOutputCommitter(syncBarrier, false);
    CommitterEventHandler commitHandler =
        createCommitterEventHandler(dispatcher, committer);
    commitHandler.init(conf);
    commitHandler.start();

View Full Code Here

    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    CyclicBarrier syncBarrier = new CyclicBarrier(2);
    OutputCommitter committer = new TestingOutputCommitter(syncBarrier, true);
    CommitterEventHandler commitHandler =
        createCommitterEventHandler(dispatcher, committer);
    commitHandler.init(conf);
    commitHandler.start();

View Full Code Here

    Configuration conf = new Configuration();
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    OutputCommitter committer = new StubbedOutputCommitter() {
      @Override
      public synchronized void setupJob(JobContext jobContext)
          throws IOException {
        while (!Thread.interrupted()) {
          try {

View Full Code Here

    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    CyclicBarrier syncBarrier = new CyclicBarrier(2);
    OutputCommitter committer = new WaitingOutputCommitter(syncBarrier, true);
    CommitterEventHandler commitHandler =
        createCommitterEventHandler(dispatcher, committer);
    commitHandler.init(conf);
    commitHandler.start();

View Full Code Here

    Configuration conf = new Configuration();
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    OutputCommitter committer = new StubbedOutputCommitter() {
      @Override
      public void setupJob(JobContext jobContext) throws IOException {
        throw new IOException("forced failure");
      }

View Full Code Here

    Configuration conf = new Configuration();
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
    AsyncDispatcher dispatcher = new AsyncDispatcher();
    dispatcher.init(conf);
    dispatcher.start();
    OutputCommitter committer = new StubbedOutputCommitter() {
      @Override
      public synchronized void abortJob(JobContext jobContext, State state)
          throws IOException {
        while (!Thread.interrupted()) {
          try {

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.mapreduce.OutputCommitter

com.asakusafw.runtime.stage.output.StageOutputFormat

com.asakusafw.testdriver.file.FileOutputFormatDriver

com.facebook.giraph.hive.output.HiveApiOutputFormat

com.facebook.hiveio.output.HiveApiOutputFormat

com.hadoop.mapreduce.LzoIndexOutputFormat

com.hadoop.mapreduce.TestLzoTextInputFormat

org.apache.accumulo.examples.wikisearch.ingest.WikipediaMapperTest

org.apache.accumulo.examples.wikisearch.logic.TestQueryLogic

org.apache.accumulo.examples.wikisearch.output.SortingRFileOutputFormat

org.apache.giraph.hive.Helpers

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.