Examples of CombineFileSplit

org.apache.hadoop.mapred.lib.CombineFileSplit

A sub-collection of input files. Unlike {@link org.apache.hadoop.mapred.FileSplit}, CombineFileSplit * class does not represent a split of a file, but a split of input files into smaller sets. A split may contain blocks from different file but all the blocks in the same split are probably local to some rack
CombineFileSplit can be used to implement {@link org.apache.hadoop.mapred.RecordReader}'s, with reading one record per file. @see org.apache.hadoop.mapred.FileSplit @see CombineFileInputFormat

org.apache.hadoop.mapreduce.lib.input.CombineFileSplit

A sub-collection of input files. Unlike {@link FileSplit}, CombineFileSplit class does not represent a split of a file, but a split of input files into smaller sets. A split may contain blocks from different file but all the blocks in the same split are probably local to some rack
CombineFileSplit can be used to implement {@link RecordReader}'s, with reading one record per file. @see FileSplit @see CombineFileInputFormat


    public static Map<String, List<CombineFileSplitAdapter>> organizeSplitsByLocation(final List<InputSplit> splits) throws IOException {
        Map<String, List<CombineFileSplitAdapter>> locationSplits = new HashMap<String, List<CombineFileSplitAdapter>>();


        for (InputSplit split : splits) {
            CombineFileSplit csplit = (CombineFileSplit) split;


            String location = extractLocation(csplit);


            List<CombineFileSplitAdapter> splitsForLocation = locationSplits.get(location);
            if (splitsForLocation == null) {

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of CombineFileSplit

Examples of org.apache.hadoop.mapreduce.lib.input.CombineFileSplit