Examples of EigenVerificationJob

org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob
Class for taking the output of an eigendecomposition (specified as a Path location), and verifies correctness, in terms of the following: if you have a vector e, and a matrix m, then let e' = m.timesSquared(v); the error w.r.t. eigenvector-ness is the cosine of the angle between e and e':
```
 error(e,e') = e.dot(e') / (e.norm(2)*e'.norm(2)) 
```
A set of eigenvectors should also all be very close to orthogonal, so this job computes all inner products between eigenvectors, and checks that this is close to the identity matrix.

Parameters used in the cleanup (other than in the input/output path options) include --minEigenvalue, which specifies the value below which eigenvector/eigenvalue pairs will be discarded, and --maxError, which specifies the maximum error (as defined above) to be tolerated in an eigenvector.

If all the eigenvectors can fit in memory, --inMemory allows for a speedier completion of this task by doing so.

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  eigenVectors,
                  eigenValues,
                  lanczosSeqFiles.toString());


    // perform a verification
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
    verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, 0.0, clusters);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters, numDims);
    W.configure(depConf);
    DistributedRowMatrix Wtrans = W.transpose();
    //    DistributedRowMatrix Wt = W.transpose();

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  eigenVectors,
                  eigenValues,
                  seqFiles.toString());


    // now run the verifier to trim down the number of eigenvectors
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigens = new Path(tmp, "verifiedeigens");
    verifier.runJob(conf, seqFiles, input.getRowPath(), verifiedEigens, false, 1.0, 0.0, numEigenVectors);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    return new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), numEigenVectors, input.numRows());
  }

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

    int sampleDimension = sampleData.get(0).get().size();
    // call EigenVerificationJob separately
    solver.run(testData, output, tmp, sampleData.size(), sampleDimension, false, desiredRank);
    Path rawEigenvectors = new Path(output, DistributedLanczosSolver.RAW_EIGENVECTORS);
    JobConf conf = new JobConf(config);
    new EigenVerificationJob().run(testData, rawEigenvectors, output, tmp, 0.5, 0.0, true, conf);
    Path cleanEigenvectors = new Path(output, EigenVerificationJob.CLEAN_EIGENVECTORS);


    // now multiply the testdata matrix and the eigenvector matrix
    DistributedRowMatrix svdT = new DistributedRowMatrix(cleanEigenvectors, tmp, desiredRank - 1, sampleDimension);
    svdT.configure(conf);

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

    solver.run(testData, output, tmp, null, sampleData.size(), sampleDimension,
        false, desiredRank);
    Path rawEigenvectors = new Path(output,
        DistributedLanczosSolver.RAW_EIGENVECTORS);
    Configuration conf = new Configuration(config);
    new EigenVerificationJob().run(testData, rawEigenvectors, output, tmp, 0.5,
        0.0, true, conf);
    Path cleanEigenvectors = new Path(output,
        EigenVerificationJob.CLEAN_EIGENVECTORS);
    
    // now multiply the testdata matrix and the eigenvector matrix

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  overshoot,
                  true,
                  seqFiles.toString());


    // now run the verifier to trim down the number of eigenvectors
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigens = new Path(tmp, "verifiedeigens");
    verifier.runJob(conf, seqFiles, input.getRowPath(), verifiedEigens, false, 1.0, numEigenVectors);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    return new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), numEigenVectors, input.numRows());
  }

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  overshoot,
                  true,
                  lanczosSeqFiles.toString());


    // perform a verification
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
    verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, clusters);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters, numDims);
    W.setConf(depConf);
    DistributedRowMatrix Wtrans = W.transpose();
    //    DistributedRowMatrix Wt = W.transpose();

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  overshoot,
                  true,
                  seqFiles.toString());


    // now run the verifier to trim down the number of eigenvectors
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigens = new Path(tmp, "verifiedeigens");
    verifier.runJob(conf, seqFiles, input.getRowPath(), verifiedEigens, false, 1.0, numEigenVectors);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    return new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), numEigenVectors, input.numRows());
  }

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

      Path lanczosSeqFiles = new Path(outputCalc, "eigenvectors");


      solver.runJob(conf, state, overshoot, true, lanczosSeqFiles.toString());


      // perform a verification
      EigenVerificationJob verifier = new EigenVerificationJob();
      Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
      verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, clusters);


      Path cleanedEigens = verifier.getCleanedEigensPath();
      DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters,
          numDims);
      W.setConf(depConf);
      DistributedRowMatrix Wtrans = W.transpose();
      data = Wtrans.getRowPath();

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

    // call EigenVerificationJob separately
    int desiredRank = 13;
    solver.run(testData, output, tmp, null, sampleData.size(), sampleDimension, false, desiredRank);
    Path rawEigenvectors = new Path(output, DistributedLanczosSolver.RAW_EIGENVECTORS);
    Configuration conf = new Configuration(config);
    new EigenVerificationJob().run(testData, rawEigenvectors, output, tmp, 0.5, 0.0, true, conf);
    Path cleanEigenvectors = new Path(output, EigenVerificationJob.CLEAN_EIGENVECTORS);


    // now multiply the testdata matrix and the eigenvector matrix
    DistributedRowMatrix svdT = new DistributedRowMatrix(cleanEigenvectors, tmp, desiredRank,
        sampleDimension);

View Full Code Here

Examples of org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob

                  overshoot,
                  true,
                  lanczosSeqFiles.toString());


    // perform a verification
    EigenVerificationJob verifier = new EigenVerificationJob();
    Path verifiedEigensPath = new Path(outputCalc, "eigenverifier");
    verifier.runJob(conf, lanczosSeqFiles, L.getRowPath(), verifiedEigensPath, true, 1.0, 0.0, clusters);
    Path cleanedEigens = verifier.getCleanedEigensPath();
    DistributedRowMatrix W = new DistributedRowMatrix(cleanedEigens, new Path(cleanedEigens, "tmp"), clusters, numDims);
    W.setConf(depConf);
    DistributedRowMatrix Wtrans = W.transpose();
    //    DistributedRowMatrix Wt = W.transpose();

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.