Package org.commoncrawl.util.shared

Examples of org.commoncrawl.util.shared.JobBuilder


   
    try {
      // parse the command line arguments
      CommandLine cmdLine = parser.parse( options, args );
     
      JobBuilder builder = new JobBuilder("BareBones Job", new Configuration());
     
      LOG.info("Paths:"+ cmdLine.getOptionValues("path"));
      if (cmdLine.hasOption("path")) {
        for (String path : cmdLine.getOptionValues("path")){
          LOG.info("Adding Input Path:" + path);
          builder.input(new Path(path));
        }
      }
      else {
        throw new IOException("No Paths Specified!");
      }
     
      builder.keyValue(NullWritable.class, NullWritable.class);
      builder.mapper(BareBonesJob.class);
      builder.inputFormat(ARCFileInputFormat.class);
      builder.outputFormat(NullOutputFormat.class);
      builder.numReducers(0);
     
      JobConf job = builder.build();
     
      JobClient.runJob(job);
     
    }
    catch (IOException e) {
View Full Code Here

TOP

Related Classes of org.commoncrawl.util.shared.JobBuilder

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.