Examples of org.apache.hadoop.mapreduce.InputFormat

org.apache.hadoop.mapreduce.InputFormat
InputFormat describes the input-specification for a Map-Reduce job.
The Map-Reduce framework relies on the InputFormat of the job to:
1. Validate the input-specification of the job.
2. Split-up the input file(s) into logical {@link InputSplit}s, each of which is then assigned to an individual {@link Mapper}.
3. Provide the {@link RecordReader} implementation to be used to gleaninput records from the logical InputSplit for processing by the {@link Mapper}.
The default behavior of file-based {@link InputFormat}s, typically sub-classes of {@link FileInputFormat}, is to split the input into logical {@link InputSplit}s based on the total size, in bytes, of the input files. However, the {@link FileSystem} blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Clearly, logical splits based on input-size is insufficient for many applications since record boundaries are to respected. In such cases, the application has to also implement a {@link RecordReader} on whom lies theresponsibility to respect record-boundaries and present a record-oriented view of the logical InputSplit to the individual task. @see InputSplit @see RecordReader @see FileInputFormat

                        inputSpecificJob);
                // The above setLocation call could write to the conf within
                // the inputSpecificJob - use this updated conf
                
                // get the InputFormat from it and ask for splits
                InputFormat inpFormat = loadFunc.getInputFormat();
                List<InputSplit> oneInputSplits = inpFormat.getSplits(
                        HadoopShims.createJobContext(inputSpecificJob.getConfiguration(), 
                                jobcontext.getJobID()));
                List<InputSplit> oneInputPigSplits = getPigSplits(
                        oneInputSplits, i, inpTargets.get(i),
                        HadoopShims.getDefaultBlockSize(fs, isFsPath? path: fs.getWorkingDirectory()),

View Full Code Here

    Map<Path, String> formatMap = PangoolMultipleInputs.getInputFormatMap(job);
    Map<Path, String> mapperMap = PangoolMultipleInputs.getInputProcessorFileMap(job);


    for(Map.Entry<Path, String> entry : formatMap.entrySet()) {
      FileInputFormat.setInputPaths(jobCopy, entry.getKey());
      InputFormat inputFormat = DCUtils.loadSerializedObjectInDC(conf, InputFormat.class,
          entry.getValue(), true);
      List<InputSplit> pathSplits = inputFormat.getSplits(jobCopy);
      for(InputSplit pathSplit : pathSplits) {
        splits.add(new TaggedInputSplit(pathSplit, conf, entry.getValue(), mapperMap
            .get(entry.getKey())));
      }
    }

View Full Code Here

    @SuppressWarnings("unchecked")
    public static <K, V> void writePartitionFile(Job job, Sampler<K, V> sampler)
      throws IOException, ClassNotFoundException, InterruptedException {
      LinkedList<K> splits = new LinkedList<K>();
      Configuration conf = job.getConfiguration();
      final InputFormat inf =
        ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
      int numPartitions = job.getNumReduceTasks();
      K[] samples = sampler.getSample(inf, job);
      LOG.info("Using " + samples.length + " samples");
      RawComparator<K> comparator = (RawComparator<K>) job.getGroupingComparator();

View Full Code Here

   */
  @SuppressWarnings("unchecked") // getInputFormat, getOutputKeyComparator
  public static <K,V> void writePartitionFile(Job job, Sampler<K,V> sampler) 
      throws IOException, ClassNotFoundException, InterruptedException {
    Configuration conf = job.getConfiguration();
    final InputFormat inf = 
        ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
    int numPartitions = job.getNumReduceTasks();
    K[] samples = (K[])sampler.getSample(inf, job);
    LOG.info("Using " + samples.length + " samples");
    RawComparator<K> comparator =

View Full Code Here

  @SuppressWarnings("unchecked")
  // getInputFormat, getOutputKeyComparator
  public static <K, V> void writePartitionFile(Job job, Sampler<K, V> sampler) throws IOException, ClassNotFoundException, InterruptedException {
    log.debug("writePartitionFile({},{})", job, sampler);
    Configuration conf = job.getConfiguration();
    @SuppressWarnings("rawtypes")
    final InputFormat inf = ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
    int numPartitions = job.getNumReduceTasks() / 9;
    log.debug("Number of partitions is {} for each index", numPartitions);
    K[] samples = sampler.getSample(inf, job);
    log.info("Using " + samples.length + " samples");

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.mapreduce.InputFormat

bulkimport.BulkImportJobExample$VerboseInputSampler

com.datasalt.pangool.tuplemr.mapred.lib.input.DelegatingInputFormat

com.datasalt.pangool.tuplemr.mapred.lib.output.TestTupleTextInputOutputFormat

com.endgame.binarypig.loaders.AbstractExecutingLoaderTest

com.endgame.binarypig.loaders.BinarySequenceFileLoaderTest

org.apache.hadoop.hbase.mapreduce.hadoopbackport.InputSampler

org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat

org.apache.hadoop.mapreduce.lib.input.MultipleInputs

org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

org.apache.hadoop.mapreduce.lib.partition.InputSampler

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.