Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

Package org.apache.mahout.math.hadoop.decomposer

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob
Class for taking the output of an eigendecomposition (specified as a Path location), and verifies correctness, in terms of the following: if you have a vector e, and a matrix m, then let e' = m.timesSquared(v); the error w.r.t. eigenvector-ness is the cosine of the angle between e and e':
```
 error(e,e') = e.dot(e') / (e.norm(2)*e'.norm(2)) 
```
A set of eigenvectors should also all be very close to orthogonal, so this job computes all inner products between eigenvectors, and checks that this is close to the identity matrix.

Parameters used in the cleanup (other than in the input/output path options) include --minEigenvalue, which specifies the value below which eigenvector/eigenvalue pairs will be discarded, and --maxError, which specifies the maximum error (as defined above) to be tolerated in an eigenvector.

If all the eigenvectors can fit in memory, --inMemory allows for a speedier completion of this task by doing so.

                  overshoot,
                  true,
                  seqFiles.toString());


    // now run the verifier to trim down the number of eigenvectors
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigens = new Path(tmp, "verifiedeigens");
    verifier.runJob(conf, seqFiles, input.getRowPath(), verifiedEigens, false, 1.0, 0.0, numEigenVectors);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    return new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), numEigenVectors, input.numRows());
  }

View Full Code Here

      Path lanczosSeqFiles = new Path(outputCalc, "eigenvectors");


      solver.runJob(conf, state, overshoot, true, lanczosSeqFiles.toString());


      // perform a verification
      EigenVerificationJob verifier = new EigenVerificationJob();
      Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
      verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, clusters);


      Path cleanedEigens = verifier.getCleanedEigensPath();
      DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters,
          numDims);
      W.setConf(depConf);
      DistributedRowMatrix Wtrans = W.transpose();
      data = Wtrans.getRowPath();

View Full Code Here

    solver.run(testData, output, tmp, null, sampleData.size(), sampleDimension,
        false, desiredRank);
    Path rawEigenvectors = new Path(output,
        DistributedLanczosSolver.RAW_EIGENVECTORS);
    Configuration conf = new Configuration(config);
    new EigenVerificationJob().run(testData, rawEigenvectors, output, tmp, 0.5,
        0.0, true, conf);
    Path cleanEigenvectors = new Path(output,
        EigenVerificationJob.CLEAN_EIGENVECTORS);
    
    // now multiply the testdata matrix and the eigenvector matrix

View Full Code Here

                  overshoot,
                  true,
                  lanczosSeqFiles.toString());


    // perform a verification
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
    verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, clusters);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters, numDims);
    W.setConf(depConf);
    DistributedRowMatrix Wtrans = W.transpose();
    //    DistributedRowMatrix Wt = W.transpose();

View Full Code Here

                  overshoot,
                  true,
                  seqFiles.toString());


    // now run the verifier to trim down the number of eigenvectors
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigens = new Path(tmp, "verifiedeigens");
    verifier.runJob(conf, seqFiles, input.getRowPath(), verifiedEigens, false, 1.0, numEigenVectors);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    return new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), numEigenVectors, input.numRows());
  }

View Full Code Here

      Path lanczosSeqFiles = new Path(outputCalc, "eigenvectors");


      solver.runJob(conf, state, overshoot, true, lanczosSeqFiles.toString());


      // perform a verification
      EigenVerificationJob verifier = new EigenVerificationJob();
      Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
      verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, clusters);


      Path cleanedEigens = verifier.getCleanedEigensPath();
      DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters,
          numDims);
      W.setConf(depConf);
      DistributedRowMatrix Wtrans = W.transpose();
      data = Wtrans.getRowPath();

View Full Code Here

0 1

TOP

Related Classes of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

org.apache.commons.cli2.builder.GroupBuilder

org.apache.commons.cli2.CommandLine

org.apache.commons.cli2.commandline.Parser

org.apache.commons.cli2.Group

org.apache.commons.cli2.Option

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.Path

org.apache.hadoop.io.IntWritable

org.apache.hadoop.io.Writable

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.