Examples of InputFormat

com.thaiopensource.relaxng.input.InputFormat
org.apache.hadoop.mapred.InputFormat
InputFormat describes the input-specification for a Map-Reduce job.
The Map-Reduce framework relies on the InputFormat of the job to:
1. Validate the input-specification of the job.
2. Split-up the input file(s) into logical {@link InputSplit}s, each of which is then assigned to an individual {@link Mapper}.
3. Provide the {@link RecordReader} implementation to be used to gleaninput records from the logical InputSplit for processing by the {@link Mapper}.
The default behavior of file-based {@link InputFormat}s, typically sub-classes of {@link FileInputFormat}, is to split the input into logical {@link InputSplit}s based on the total size, in bytes, of the input files. However, the {@link FileSystem} blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Clearly, logical splits based on input-size is insufficient for many applications since record boundaries are to respected. In such cases, the application has to also implement a {@link RecordReader} on whom lies theresponsibilty to respect record-boundaries and present a record-oriented view of the logical InputSplit to the individual task. @see InputSplit @see RecordReader @see JobClient @see FileInputFormat
org.apache.hadoop.mapreduce.InputFormat
InputFormat describes the input-specification for a Map-Reduce job.
The Map-Reduce framework relies on the InputFormat of the job to:
1. Validate the input-specification of the job.
2. Split-up the input file(s) into logical {@link InputSplit}s, each of which is then assigned to an individual {@link Mapper}.
3. Provide the {@link RecordReader} implementation to be used to gleaninput records from the logical InputSplit for processing by the {@link Mapper}.
The default behavior of file-based {@link InputFormat}s, typically sub-classes of {@link FileInputFormat}, is to split the input into logical {@link InputSplit}s based on the total size, in bytes, of the input files. However, the {@link FileSystem} blocksize of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size.

Clearly, logical splits based on input-size is insufficient for many applications since record boundaries are to respected. In such cases, the application has to also implement a {@link RecordReader} on whom lies theresponsibility to respect record-boundaries and present a record-oriented view of the logical InputSplit to the individual task. @see InputSplit @see RecordReader @see FileInputFormat

Examples of com.thaiopensource.relaxng.input.InputFormat

  }
  
  public static void generateXSDFromDTD(File input, File output) throws Exception {
    ErrorHandlerImpl eh = new ErrorHandlerImpl();
    OutputFormat of = new XsdOutputFormat();
    InputFormat inFormat = new DtdInputFormat();
    SchemaCollection sc = inFormat.load(
        UriOrFile.toUri(input.getAbsolutePath()), new String[0], "xsd", eh);
    
    OutputDirectory od = new LocalOutputDirectory(
        sc.getMainUri(),output,"xml","utf-8",80,4);

View Full Code Here

Examples of com.thaiopensource.relaxng.input.InputFormat

   * @throws Exception
   */
  public static void generateDTDFromXML(File input,File output) throws Exception {
    ErrorHandlerImpl eh = new ErrorHandlerImpl();
    OutputFormat of = new DtdOutputFormat();
    InputFormat inFormat = new XmlInputFormat();
    SchemaCollection sc = inFormat.load(
        UriOrFile.toUri(input.getAbsolutePath()), new String[0], "dtd", eh);
    
    OutputDirectory od = new LocalOutputDirectory(
        sc.getMainUri(),output,"xml","utf-8",80,4);

View Full Code Here

Examples of com.thaiopensource.relaxng.input.InputFormat

   * @throws Exception
   */
  public static void generateXSDFromXML(File input,File output) throws Exception {
    ErrorHandlerImpl eh = new ErrorHandlerImpl();
    OutputFormat of = new XsdOutputFormat();
    InputFormat inFormat = new XmlInputFormat();
    SchemaCollection sc = inFormat.load(
        UriOrFile.toUri(input.getAbsolutePath()), new String[0], "xsd", eh);
    
    OutputDirectory od = new LocalOutputDirectory(
        sc.getMainUri(),output,"xml","utf-8",80,4);

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

      } finally {
        writer.close();
      }


      // try splitting the file in a variety of sizes
      InputFormat format = new SequenceFileInputFormat();
      RecInt key = new RecInt();
      RecBuffer value = new RecBuffer();
      for (int i = 0; i < 3; i++) {
        int numSplits =
          random.nextInt(MAX_LENGTH/(SequenceFile.SYNC_INTERVAL/20))+1;
        InputSplit[] splits = format.getSplits(job, numSplits);


        // check each split
        BitSet bits = new BitSet(length);
        for (int j = 0; j < splits.length; j++) {
          RecordReader reader =
            format.getRecordReader(splits[j], job, Reporter.NULL);
          try {
            int count = 0;
            while (reader.next(key, value)) {
              assertFalse("Key in multiple partitions.", bits.get(key.getData()));
              bits.set(key.getData());

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

        return super.getSplits(job, numSplits);
      }


      // Use HiveInputFormat if any of the paths is not splittable
      Class inputFormatClass = part.getInputFileFormatClass();
      InputFormat inputFormat = getInputFormatFromCache(inputFormatClass, job);


      // Since there is no easy way of knowing whether MAPREDUCE-1597 is present in the tree or not,
      // we use a configuration variable for the same
      if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
        // The following code should be removed, once

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

          .getPartitionDescFromPathRecursively(pathToPartitionInfo, dir,
              IOPrepareCache.get().allocatePartitionDescMap(), true);
      // create a new InputFormat instance if this is the first time to see this
      // class
      Class inputFormatClass = part.getInputFileFormatClass();
      InputFormat inputFormat = getInputFormatFromCache(inputFormatClass, job);
      Utilities.copyTableJobPropertiesToConf(part.getTableDesc(), newjob);


      FileInputFormat.setInputPaths(newjob, dir);
      newjob.setInputFormat(inputFormat.getClass());
      InputSplit[] iss = inputFormat.getSplits(newjob, numSplits / dirs.length);
      for (InputSplit is : iss) {
        result.add(new HiveInputSplit(is, inputFormatClass.getName()));
      }
    }
    return result.toArray(new HiveInputSplit[result.size()]);

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

    }


    pushProjectionsAndFilters(cloneJobConf, inputFormatClass, hsplit.getPath()
      .toString(), hsplit.getPath().toUri().getPath(), nonNative);


    InputFormat inputFormat = getInputFormatFromCache(inputFormatClass,
        cloneJobConf);
    RecordReader innerReader = inputFormat.getRecordReader(inputSplit,
        cloneJobConf, reporter);


    HiveRecordReader<K,V> rr = new HiveRecordReader(innerReader);
    rr.initIOContext(hsplit, job, inputFormatClass, innerReader);
    return rr;

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

    for (Path dir : dirs) {
      PartitionDesc part = getPartitionDescFromPath(pathToPartitionInfo, dir);
      // create a new InputFormat instance if this is the first time to see this
      // class
      Class inputFormatClass = part.getInputFileFormatClass();
      InputFormat inputFormat = getInputFormatFromCache(inputFormatClass, job);
      Utilities.copyTableJobPropertiesToConf(part.getTableDesc(), newjob);


      // Make filter pushdown information available to getSplits.
      ArrayList<String> aliases =
        mrwork.getPathToAliases().get(dir.toUri().toString());
      if ((aliases != null) && (aliases.size() == 1)) {
        Operator op = mrwork.getAliasToWork().get(aliases.get(0));
        if ((op != null) && (op instanceof TableScanOperator)) {
          TableScanOperator tableScan = (TableScanOperator) op;
          pushFilters(newjob, tableScan);
        }
      }


      FileInputFormat.setInputPaths(newjob, dir);
      newjob.setInputFormat(inputFormat.getClass());
      InputSplit[] iss = inputFormat.getSplits(newjob, numSplits / dirs.length);
      for (InputSplit is : iss) {
        result.add(new HiveInputSplit(is, inputFormatClass.getName()));
      }
    }
    return result.toArray(new HiveInputSplit[result.size()]);

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

    // for each dir, get the InputFormat, and do validateInput.
    for (Path dir : dirs) {
      PartitionDesc part = getPartitionDescFromPath(pathToPartitionInfo, dir);
      // create a new InputFormat instance if this is the first time to see this
      // class
      InputFormat inputFormat = getInputFormatFromCache(part
          .getInputFileFormatClass(), job);


      FileInputFormat.setInputPaths(newjob, dir);
      newjob.setInputFormat(inputFormat.getClass());
      ShimLoader.getHadoopShims().inputFormatValidateInput(inputFormat, newjob);
    }
  }

View Full Code Here

Examples of org.apache.hadoop.mapred.InputFormat

      inputFormatClass = Class.forName(inputFormatClassName);
    } catch (Exception e) {
      throw new IOException("cannot find class " + inputFormatClassName);
    }


    InputFormat inputFormat = getInputFormatFromCache(inputFormatClass);
    
    return new HiveRecordReader(inputFormat.getRecordReader(inputSplit, job, reporter));
  }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.