Examples of com.intel.hadoop.graphbuilder.idnormalize.mapreduce.SortDictMR

Package com.intel.hadoop.graphbuilder.idnormalize.mapreduce

Examples of com.intel.hadoop.graphbuilder.idnormalize.mapreduce.SortDictMR

com.intel.hadoop.graphbuilder.idnormalize.mapreduce.SortDictMR
This MapReduce class partitions the dictionary output of HashIdMR based on the hash of the rawId, the key. It can also be used to partition the dictionary based on the hash of the newId, the value, for reverse lookup.
Input directory: list of rawid vid pair. Output directory: $outputdir/


    try {
      HashIdMR job1 = new HashIdMR(graphparser, vidparser, vdataparser);
      job1.run(input + "/vdata", output);


      SortDictMR job2 = new SortDictMR(numParts, true, vidparser);
      job2.run(output + "/vidmap", output + "/temp/partitionedvidmap");


      SortEdgeMR job3 = new SortEdgeMR(numParts, graphparser, vidparser,
          edataparser);
      job3.run(input + "/edata", output + "/temp/partitionededata");

View Full Code Here

 * 
 * @author Haijie Gu
 */
public class SortDictionaryMapReduceTest {
  public static void main(String[] args) throws IOException {
    SortDictMR driver = new SortDictMR(32, true, new StringParser());
    driver.run(args[0], args[1]);
  }

View Full Code Here

TOP

Related Classes of com.intel.hadoop.graphbuilder.idnormalize.mapreduce.SortDictMR

com.intel.hadoop.graphbuilder.job.AbstractIdNormalizationJob

com.intel.hadoop.graphbuilder.test.idnormalize.SortDictionaryMapReduceTest

org.apache.hadoop.fs.Path

org.apache.hadoop.mapred.JobConf

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.