Examples of org.apache.hadoop.hive.contrib.mr.GenericMR

Package org.apache.hadoop.hive.contrib.mr

Examples of org.apache.hadoop.hive.contrib.mr.GenericMR

org.apache.hadoop.hive.contrib.mr.GenericMR
This class attempts to provide a simple framework for writing Hive map/reduce tasks in java. The main benefit is that it deals with grouping the keys together for reduce tasks. Additionally, it deals with all system io... and provides something closer to the hadoop m/r. As an example, here's the wordcount reduce: new GenericMR().reduce(System.in, System.out, new Reducer() { public void reduce(String key, Iterator records, Output output) throws Exception { int count = 0; while (records.hasNext()) { count += Integer.parseInt(records.next()[1]); } output.collect(new String[] { key, String.valueOf(count) }); }});

/**
 * Example Mapper (Identity). 
 */
public final class IdentityMapper {
  public static void main(final String[] args) throws Exception {
    new GenericMR().map(System.in, System.out, new Mapper() {
      @Override
      public void map(final String[] record, final Output output) throws Exception {
        output.collect(record);
      }
    });

View Full Code Here

/**
 * Example Reducer (WordCount). 
 */
public final class WordCountReduce {
  public static void main(final String[] args) throws Exception {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      public void reduce(String key, Iterator<String[]> records, Output output) throws Exception {
        int count = 0;
        
        while (records.hasNext()) {
          // note we use col[1] -- the key is provided again as col[0]

View Full Code Here

 * Example Mapper (Identity).
 */
public final class IdentityMapper {


  public static void main(final String[] args) throws Exception {
    new GenericMR().map(System.in, System.out, new Mapper() {
      @Override
      public void map(final String[] record, final Output output) throws Exception {
        output.collect(record);
      }
    });

View Full Code Here

  private WordCountReduce() {
    // prevent instantiation
  }


  public static void main(final String[] args) throws Exception {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      public void reduce(String key, Iterator<String[]> records, Output output)
          throws Exception {
        int count = 0;


        while (records.hasNext()) {

View Full Code Here

TOP

Related Classes of org.apache.hadoop.hive.contrib.mr.GenericMR

org.apache.hadoop.hive.contrib.mr.example.IdentityMapper

org.apache.hadoop.hive.contrib.mr.example.WordCountReduce

java.util.NoSuchElementException

java.io.BufferedReader

java.io.PrintWriter

java.io.InputStreamReader

java.io.OutputStreamWriter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.