Examples of com.intel.hadoop.graphbuilder.preprocess.mapreduce.CreateGraphMR

com.intel.hadoop.graphbuilder.preprocess.mapreduce.CreateGraphMR

This MapReduce Job creates an initial edge list and vertex list from raw input data, e.g. text xml. The result graph does not contain self edge and duplicate vertex/edges.

The Mapper class parse each input value, provided by the {@code InputFormat}, and output a list of {@code Vertex} and a list of {@code Edge} using a{@code GraphTokenizer}.

The Reducer class applies user defined {@code Functional}s to reduce duplicate edges and vertices. If no such {@code Functional} is provide, itoutputs the first instance and discards the rest with the same identifier. It also discards self edges: v - > v. An option for discard bidirectional edge is provided by {@code cleanBidirectionalEdge(boolean)}.

Input directory: Can take multiple input directories. Output directory structure:

$outputdir/edata contains edge data output
$outputdir/vdata contains vertex data output

@see GraphTokenizer


  public boolean run(GraphTokenizer<VidType, VertexData, EdgeData> tokenizer,
      InputFormat inputformat, String[] inputs, String output)
      throws CannotCompileException, NotFoundException, IOException {
    // Required parameters;
    CreateGraphMR mr = new CreateGraphMR(tokenizer, inputformat);


    Class valueClass = PreprocessJobValueFactory.getValueClassByClassName(
        tokenizer.vidClass().getName(), tokenizer.vdataClass().getName(),
        tokenizer.edataClass().getName());
    // Set value class based on the types of tokenizer.
    mr.setValueClass(valueClass);
    // Distributed the new class file to cluster.
    FsUtil.distributedTempClassToClassPath(mr.getConf());


    // Optional parameters;
    Class vreducerClass = vertexReducer() == null ? null : vertexReducer()
        .getClass();
    Class ereducerClass = edgeReducer() == null ? null : edgeReducer()
        .getClass();
    mr.setFunctionClass(vreducerClass, ereducerClass);
    mr.cleanBidirectionalEdge(cleanBidirectionalEdge());


    // User defined parameters;
    if (userOpts != null) {
      mr.setUserOptions(userOpts);
    }


    try {
      mr.run(inputs, output);
    } catch (Exception e) {
      e.printStackTrace();
      return false;
    }
    return true;

Related Classes of com.intel.hadoop.graphbuilder.preprocess.mapreduce.CreateGraphMR

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.