Examples of org.apache.hadoop.mapreduce.lib.input.CombineFileSplit

Package org.apache.hadoop.mapreduce.lib.input

Examples of org.apache.hadoop.mapreduce.lib.input.CombineFileSplit

org.apache.hadoop.mapreduce.lib.input.CombineFileSplit
A sub-collection of input files. Unlike {@link FileSplit}, CombineFileSplit class does not represent a split of a file, but a split of input files into smaller sets. A split may contain blocks from different file but all the blocks in the same split are probably local to some rack
CombineFileSplit can be used to implement {@link RecordReader}'s, with reading one record per file. @see FileSplit @see CombineFileInputFormat


    public static Map<String, List<CombineFileSplitAdapter>> organizeSplitsByLocation(final List<InputSplit> splits) throws IOException {
        Map<String, List<CombineFileSplitAdapter>> locationSplits = new HashMap<String, List<CombineFileSplitAdapter>>();


        for (InputSplit split : splits) {
            CombineFileSplit csplit = (CombineFileSplit) split;


            String location = extractLocation(csplit);


            List<CombineFileSplitAdapter> splitsForLocation = locationSplits.get(location);
            if (splitsForLocation == null) {

View Full Code Here

0 1 2

TOP

Related Classes of org.apache.hadoop.mapreduce.lib.input.CombineFileSplit

com.alexholmes.hadooputils.combine.common.mapreduce.SplitMetricsCombineInputFormat

com.linkedin.camus.sweeper.mapreduce.AvroKeyCombineFileInputFormat

com.linkedin.whiteelephant.parsing.ParseJobsFromLogs$TheMapper

org.apache.hadoop.fs.Path

org.apache.hadoop.mapred.gridmix.InputStriper

org.apache.hadoop.mapred.gridmix.TestCompressionEmulationUtils

org.apache.hadoop.mapred.gridmix.TestFileQueue

org.apache.hadoop.mapred.gridmix.TestGridMixClasses

org.apache.sqoop.mapreduce.ExportInputFormat

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.