Examples of com.datasalt.pangool.tuplemr.mapred.lib.output.TupleTextOutputFormat$CustomCSVWriter

Package com.datasalt.pangool.tuplemr.mapred.lib.output

Examples of com.datasalt.pangool.tuplemr.mapred.lib.output.TupleTextOutputFormat$CustomCSVWriter

com.datasalt.pangool.tuplemr.mapred.lib.output.TupleTextOutputFormat
We had to almost re-implement CSVWriter for properly supporting null strings. We can't reuse a lot of code due to inheritance / visibility problems.

    mr.setFieldAliases("urlMap", new Aliases().add("url", "nonCanonicalUrl"));
    mr.setGroupByFields("url");
    mr.setOrderBy(new OrderBy().add("url", Order.ASC).addSchemaOrder(Order.ASC));
    mr.setSpecificOrderBy("urlRegister", new OrderBy().add("timestamp", Order.ASC));
    mr.setTupleReducer(new Handler());
    mr.setOutput(new Path(output), new TupleTextOutputFormat(getURLRegisterSchema(), false, '\t',
        NO_QUOTE_CHARACTER, NO_ESCAPE_CHARACTER), ITuple.class, NullWritable.class);


    try {
      mr.createJob().waitForCompletion(true);
    } finally {

View Full Code Here


      // Use Pangool API - parse CSV, etc
      TupleMRBuilder builder = new TupleMRBuilder(conf);
      TupleTextInputFormat parsingInputFormat = new TupleTextInputFormat(schema, skipHeading, false,
          separator.charAt(0), quotes.charAt(0), escape.charAt(0), FieldSelector.NONE, null);
      TupleTextOutputFormat outputFormat = new TupleTextOutputFormat(schema, false, separator.charAt(0),
          quotes.charAt(0), escape.charAt(0));


      builder.addIntermediateSchema(schema);
      builder.addInput(inputP, parsingInputFormat, new IdentityTupleMapper());
      builder.setGroupByFields(groupBy);

View Full Code Here

TOP

Related Classes of com.datasalt.pangool.tuplemr.mapred.lib.output.TupleTextOutputFormat$CustomCSVWriter

au.com.bytecode.opencsv.CSVWriter

com.datasalt.pangool.examples.urlresolution.UrlResolution

com.splout.db.benchmark.IdentityJob

org.apache.hadoop.fs.Path

java.io.PrintWriter

java.io.OutputStreamWriter

java.io.BufferedWriter

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.