Examples of org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl

Package org.apache.hadoop.mapreduce.lib.jobcontrol

Examples of org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl

org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl
This class encapsulates a set of MapReduce jobs and its dependency. It tracks the states of the jobs by placing them into different tables according to their states. This class provides APIs for the client app to add a job to the group and to get the jobs in the group in different states. When a job is added, an ID unique to the group is assigned to the job. This class has a thread that submits jobs when they become ready, monitors the states of the running jobs, and updates the states of jobs based on the state changes of their depending jobs states. The class provides APIs for suspending/resuming the thread, and for stopping the thread.


      ControlledJob aboveAvgControlledJob = new ControlledJob(
          getAverageJobConf(binningOutputAbove, aboveAverageRepOutput));
      aboveAvgControlledJob.addDependingJob(binningControlledJob);


      JobControl jc = new JobControl("AverageReputation");
      jc.addJob(binningControlledJob);
      jc.addJob(belowAvgControlledJob);
      jc.addJob(aboveAvgControlledJob);


      jc.run();
      code = jc.getFailedJobList().size() == 0 ? 0 : 1;
    }


    FileSystem fs = FileSystem.get(new Configuration());
    fs.delete(countingOutput, true);
    fs.delete(binningOutputRoot, true);

View Full Code Here

    public abstract void addJob(int numReducers, boolean mapComp,
        boolean outComp, Size size, JobControl gridmix);
  }


  public GridMixRunner() throws IOException {
    gridmix = new JobControl("GridMix");
    if (null == config || null == fs) {
      throw new IOException("Bad configuration. Cannot continue.");
    }
  }

View Full Code Here


  public static JobControl createValueAggregatorJobs(String args[],
    Class<? extends ValueAggregatorDescriptor>[] descriptors) 
  throws IOException {
    
    JobControl theControl = new JobControl("ValueAggregatorJobs");
    ArrayList<ControlledJob> dependingJobs = new ArrayList<ControlledJob>();
    Configuration conf = new Configuration();
    if (descriptors != null) {
      conf = setAggregatorDescriptors(descriptors);
    }
    Job job = createValueAggregatorJob(conf, args);
    ControlledJob cjob = new ControlledJob(job, dependingJobs);
    theControl.addJob(cjob);
    return theControl;
  }

View Full Code Here

TOP

Related Classes of org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl

mrdp.ch6.JobControlDriver

org.apache.hadoop.mapreduce.GridMixRunner

org.apache.hadoop.mapreduce.lib.aggregate.ValueAggregatorJob

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.