Examples of water.MRTask

Map/Reduce style distributed computation. MRTask provides several map and reduce methods that can be overriden to specify a computation. Several instances of this class will be created to distribute the computation over F/J threads and machines. Non-transient fields are copied and serialized to instances created for map invocations. Reduce methods can store their results in fields. Results are serialized and reduced all the way back to the invoking node. When the last reduce method has been called, fields of the initial MRTask instance contains the computation results. Apart from small reduced POJO returned to the calling node, MRtask2 can produce output vector(s) as a result. These will have chunks co-located with the input dataset, however, their number of lines will generally differ, (so they won't be strictly compatible with the original). To produce output vectors, call doAll.dfork version with required number of outputs and override appropriate map call taking required number of NewChunks. MRTask will automatically close the new Appendable vecs and produce an output frame with newly created Vecs.

    // ABt is written into ncs[] with an MRTask on A, and therefore will
    // be similarly partitioned as A.
    //
    // chks.length == A.numCols() (== B.numCols())
    // ncs.length == ABt_cols (B.numRows())
    Frame ABt = new MRTask() {
        public void map(Chunk chks[], NewChunk ncs[]) {
          int chunkSize = chks[0].len();
          Vec B_vecs[] = B.vecs();


          for (int c = 0; c < ncs.length; c++) {


    // Execute MRTask on the new Frame, and fill each cell (initially 0) by
    // computing appropriate values from A and B.
    //
    // chks.length == B.numCols()
    new MRTask() {
      public void map(Chunk chks[]) {
        int chunkSize = chks[0].len();
        long start = chks[0].start();
        long A_rows = A.numRows();
        Vec A_vecs[] = A.vecs();


    // Execute MRTask on the new Frame, and fill each cell (initially 0) by
    // computing appropriate values from A.
    //
    // chks.length == A.numCols()
    new MRTask() {
      public void map(Chunk chks[]) {
        int chunkSize = chks[0].len();
        long start = chks[0].start();
        Vec A_vecs[] = A.vecs();
        long A_rows = A.numRows();

Examples of water.MRTask

Related Classes of water.MRTask