Examples of com.datasalt.pangool.tuplemr.OrderBy

com.datasalt.pangool.tuplemr.OrderBy
OrderBy is a convenience builder used by {@link TupleMRConfig} , similar to{@link Criteria}. The main difference is that {@link OrderBy} is mutableusing the concatenation pattern and allows to specify schemaOrder. The OrderBy instances are converted to immutable Criteria objects by {@link TupleMRConfig}.

    Configuration conf = getConf();
    TupleMRConfigBuilder b = new TupleMRConfigBuilder();
    b.addIntermediateSchema(schema1);
    b.addIntermediateSchema(schema2);
    b.setGroupByFields("booleanField", "intField");
    b.setOrderBy(new OrderBy().add("booleanField",Order.ASC).add("intField",Order.DESC).addSchemaOrder(Order.DESC));
    b.setSpecificOrderBy("schema1",new OrderBy().add("strField",Order.DESC));
    b.setSpecificOrderBy("schema2",new OrderBy().add("longField",Order.DESC));
    TupleMRConfig config = b.buildConf();
    TupleMRConfig.set(config, conf);
    comparator.setConf(conf);
  }

View Full Code Here

    
    TupleMRBuilder mr = new TupleMRBuilder(conf, "AvroTweetsJoin");
    mr.addIntermediateSchema(getPangoolTweetSchema());
    mr.addIntermediateSchema(getPangoolRetweetSchema());
    mr.setGroupByFields("tweet_id");
    mr.setOrderBy(new OrderBy().add("tweet_id",Order.ASC).addSchemaOrder(Order.ASC));
    
    mr.addInput(tweetsPath,new AvroInputFormat<Record>(getAvroTweetSchema()),new TweetsMapper());
    mr.addInput(retweetsPath, new HadoopInputFormat(TextInputFormat.class), new RetweetsMapper());
    mr.setOutput(outputPath,new AvroOutputFormat<Record>(getAvroOutputSchema()),
        AvroWrapper.class,NullWritable.class);

View Full Code Here

    TupleMRBuilder mr = new TupleMRBuilder(conf, "Pangool Topical Word Count");
    mr.addIntermediateSchema(getSchema());
    mr.setGroupByFields("my_avro");
    //here the custom comparator that groups by "topic,word" is used. 
    MyAvroComparator customComp = new MyAvroComparator(getAvroSchema(),"topic","word");
    mr.setOrderBy(new OrderBy().add("my_avro",Order.ASC,customComp));
    mr.addInput(new Path(args[0]), new HadoopInputFormat(TextInputFormat.class), new TokenizeMapper());
    // We'll use a TupleOutputFormat with the same schema than the intermediate schema
    mr.setTupleOutput(new Path(args[1]), getSchema());
    mr.setTupleReducer(new CountReducer());
    mr.setTupleCombiner(new CountReducer());

View Full Code Here

    if (replicateAll) {
      tBuilder.replicateToAll();
    } else {
      tBuilder.partitionBy("id");
    }
    tBuilder.insertionSortOrder(new OrderBy().add("id", Criteria.Order.ASC));
    builder.add(tBuilder.build());
    builder.setNPartitions(1);


    // Dummy tabled added only because at least one table with partition must be present
    // in the tablespace.

View Full Code Here

0 1 2 3 4

TOP

Related Classes of com.datasalt.pangool.tuplemr.OrderBy

com.datasalt.pangool.examples.avro.AvroCustomSerializationJob

com.datasalt.pangool.examples.avro.AvroTopicalWordCount

com.datasalt.pangool.examples.avro.AvroTweetsJoin

com.datasalt.pangool.examples.movingaverage.MovingAverage

com.datasalt.pangool.examples.secondarysort.SecondarySort

com.datasalt.pangool.examples.simplesecondarysort.SimpleSecondarySort

com.datasalt.pangool.examples.topicalwordcount.TopicFingerprint

com.datasalt.pangool.examples.topnhashtags.TopNHashTags

com.datasalt.pangool.examples.urlresolution.UrlResolution

com.datasalt.pangool.examples.useractivitynormalizer.UserActivityNormalizer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.