Examples of com.intel.hadoop.graphbuilder.preprocess.mapreduce.EdgeTransformMR

com.intel.hadoop.graphbuilder.preprocess.mapreduce.EdgeTransformMR

kipedia.org/wiki/Tf*idf}transformation can be done with two passes on a doc-word count graph.

First pass computes the term frequency (tf):

reduceEndPoint = SOURCE.
Reduce: sum x y -> x + y.
Apply: divide x z -> x / z.

Second pass computes the final tfidf:

reduceEndPoint = TARGET.
Reduce: count x y -> y + 1.
Apply: tfidf tf doccount -> tf * log_10 (TOTAL_DOC / doccount).

Input directory: list of edges. Output directory: $outputdir contains list of transformed edges.

      throws NotFoundException, InstantiationException, IllegalAccessException,
      CannotCompileException {


    GraphParser graphparser = (GraphParser) graphParserClass().newInstance();


    EdgeTransformMR mr = new EdgeTransformMR(reduceEndPoint, graphparser,
        ParserFactory.getParserByClassName(vidClass().getName()),
        ParserFactory.getParserByClassName(edataClass().getName()));


    Class valueClass =  EdgeTransformJobValueFactory
        .getValueClassByClassName(vidClass().getName(), edataClass().getName());
    mr.setKeyValueClass(vidClass(), valueClass);
    // Distribute new class file to cluster.
    FsUtil.distributedTempClassToClassPath(mr.getConf());
    mr.setFunctionClass(reduceFunction().getClass(), applyFunction().getClass());


    if (userOpts != null) {
      mr.setUserOptions(userOpts);
    }


    try {
      mr.run(input, output);
    } catch (Exception e) {
      e.printStackTrace();
      return false;
    }
    return true;

Examples of com.intel.hadoop.graphbuilder.preprocess.mapreduce.EdgeTransformMR

Related Classes of com.intel.hadoop.graphbuilder.preprocess.mapreduce.EdgeTransformMR