Examples of JobClient

com.asakusafw.yaess.jobqueue.client.JobClient
An abstract interface of job queue client. @since 0.2.6
eu.stratosphere.nephele.client.JobClient
The job client is able to submit, control, and abort jobs.
This class is thread-safe.
org.apache.flink.runtime.client.JobClient
The job client is able to submit, control, and abort jobs.
org.apache.hadoop.mapred.JobClient
te a new JobConf JobConf job = new JobConf(new Configuration(), MyJob.class); // Specify various job-specific parameters job.setJobName("myjob"); job.setInputPath(new Path("in")); job.setOutputPath(new Path("out")); job.setMapperClass(MyJob.MyMapper.class); job.setReducerClass(MyJob.MyReducer.class); // Submit the job, then poll for progress until the job is complete JobClient.runJob(job);

Job Control

At times clients would chain map-reduce jobs to accomplish complex tasks which cannot be done via a single map-reduce job. This is fairly easy since the output of the job, typically, goes to distributed file-system and that can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such situations the various job-control options are:
1. {@link #runJob(JobConf)} : submits the job and returns only after the job has completed.
2. {@link #submitJob(JobConf)} : only submits the job, then poll the returned handle to the {@link RunningJob} to query status and make scheduling decisions.
3. {@link JobConf#setJobEndNotificationURI(String)} : setup a notificationon job-completion, thus avoiding polling.
@see JobConf @see ClusterStatus @see Tool @see DistributedCache
org.apache.nutch.mapReduce.JobClient
JobClient interacts with the JobTracker network interface. This object implements the job-control interface, and should be the primary method by which user programs interact with the networked job system. @author Mike Cafarella

Examples of com.asakusafw.yaess.jobqueue.client.JobClient

        boolean[] bs = blackList;
        assert cs.length >= 1;
        while (true) {
            synchronized (cs) {
                for (int i = 0; i < cs.length; i++) {
                    JobClient client = cs[location];
                    boolean enabled = blackList[location] == false;
                    location = (location + 1) % cs.length;
                    if (enabled) {
                        return client;
                    }

View Full Code Here

Examples of com.asakusafw.yaess.jobqueue.client.JobClient

        assert context != null;
        assert script != null;
        JobScript job = convert(context, script);
        for (int i = 1, n = clients.count() * 2; i <= n; i++) {
            monitor.checkCancelled();
            JobClient client = clients.get();
            try {
                JobId id = registerWithTimeout(job, client);
                return new JobInfo(job, id, client);
            } catch (IOException e) {
                clients.setError(client);

View Full Code Here

Examples of eu.stratosphere.nephele.client.JobClient

        OptimizedPlan op = compileProgram(jobName);
        
        NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
        JobGraph jobGraph = jgg.compileJobGraph(op);


        JobClient client = this.executor.getJobClient(jobGraph);
        client.setConsoleStreamForReporting(AbstractTestBase.getNullPrintStream());
        JobExecutionResult result = client.submitJobAndWait();
        
        this.latestResult = result;
        return result;
      }
      catch (Exception e) {

View Full Code Here

Examples of org.apache.flink.runtime.client.JobClient

    JobGraph job = getJobGraph(compiledPlan, libraries);
    return run(job, wait);
  }


  public JobExecutionResult run(JobGraph jobGraph, boolean wait) throws ProgramInvocationException {
    JobClient client;
    try {
      client = new JobClient(jobGraph, configuration, this.userCodeClassLoader);
    }
    catch (IOException e) {
      throw new ProgramInvocationException("Could not open job manager: " + e.getMessage());
    }
    
    client.setConsoleStreamForReporting(this.printStatusDuringExecution ? System.out : null);


    try {
      if (wait) {
        return client.submitJobAndWait();
      }
      else {
        JobSubmissionResult result = client.submitJob();
        
        if (result.getReturnCode() != ReturnCode.SUCCESS) {
          throw new ProgramInvocationException("The job was not successfully submitted to the nephele job manager"
            + (result.getDescription() == null ? "." : ": " + result.getDescription()));
        }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

      File wd = new File(".").getAbsoluteFile();
      StreamUtil.unJar(new File(jar_), wd);
    }


    // if jobConf_ changes must recreate a JobClient
    jc_ = new JobClient(jobConf_);
    boolean error = true;
    running_ = null;
    String lastReport = null;
    try {
      running_ = jc_.submitJob(jobConf_);

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    jobConf.set(TMP_DIR_LABEL, tmpDir.toUri().toString());
    LOG.info("srcCount=" + srcCount);
    jobConf.setInt(SRC_COUNT_LABEL, srcCount);
    jobConf.setLong(TOTAL_SIZE_LABEL, cbsize);
    jobConf.setNumMapTasks(getMapCount(cbsize,
        new JobClient(jobConf).getClusterStatus().getTaskTrackers()));
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

      return null;
    }


    try {
      Configuration cfg = job.getConfiguration();
      JobClient jobClient = null;
      try {
        Constructor<JobClient> constr = JobClient.class.getConstructor(Configuration.class);
        jobClient = constr.newInstance(cfg);
      } catch (Exception e) {
        jobClient = new JobClient();
      }
      org.apache.hadoop.mapred.JobID id = getOldJobId(job);
      if (id != null) {
        return jobClient.getJob(id);
      } else {
        return null;
      }
    } catch (IOException e) {
      return null;

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    idbconf.setInfiniDBHome("/usr/local/Calpont");


    jobconf.setMapperClass(InfiniDoopMapper.class);
    jobconf.setNumMapTasks(1);
    jobconf.setNumReduceTasks(2);
    JobClient client = new JobClient();
    client.setConf(jobconf);
    try {
      JobClient.runJob(jobconf);
    } catch (Exception e) {
      e.printStackTrace();
    }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    idbconf.setOutputPath("output2");
    jobconf.setMapperClass(InfiniDoopInputMapper.class);
    jobconf.setNumMapTasks(4);
    jobconf.setNumReduceTasks(1);
    jobconf.set("mapred.textoutputformat.separator", "|");
    JobClient client = new JobClient();


    client.setConf(jobconf);
    try {
      JobClient.runJob(jobconf);
    } catch (Exception e) {
      e.printStackTrace();
    }

View Full Code Here

Examples of org.apache.hadoop.mapred.JobClient

    jobconf.set("mapred.output.dir", "output");
    
        jobconf.setMapperClass(InfiniDoopMapper.class);
        jobconf.setNumMapTasks(1);
        jobconf.setNumReduceTasks(2);
        JobClient client = new JobClient();
        //System.out.println(jobconf.getUser());
        //System.out.println(FileOutputFormat.getOutputPath(jobconf));
        jobconf.setInt("mapred.min.split.size", 2147483647);


        //int i = (job.waitForCompletion(true)? 1: 0);
        //return i;
        client.setConf(jobconf);
    try {
      JobClient.runJob(jobconf);
    } catch (Exception e) {
      e.printStackTrace();
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of JobClient

Job Control

Examples of com.asakusafw.yaess.jobqueue.client.JobClient

Examples of com.asakusafw.yaess.jobqueue.client.JobClient

Examples of eu.stratosphere.nephele.client.JobClient

Examples of org.apache.flink.runtime.client.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient

Examples of org.apache.hadoop.mapred.JobClient