List of parallelDo() Examples

Examples of parallelDo()

org.apache.crunch.PCollection.parallelDo()
Applies the given doFn to the elements of this {@code PCollection} andreturns a new {@code PCollection} that is the output of this processing. @param doFn The {@code DoFn} to apply @param type The {@link PType} of the resulting {@code PCollection} @return a new {@code PCollection}

Examples of org.apache.crunch.PCollection.parallelDo()

          opts.morphlineId, 
          morphlineVariables,
          settings,
          opts.inputFileFormat != null
          );
      collection = collection.parallelDo(
          "morphline",
          morphlineFn, 
          Avros.nulls() // trick to enable morphline to emit any kind of output data, including non-avro data
          );

View Full Code Here

Examples of org.apache.crunch.PCollection.parallelDo()

          "morphline",
          morphlineFn, 
          Avros.nulls() // trick to enable morphline to emit any kind of output data, including non-avro data
          );
      
      collection = collection.parallelDo(
          FilterFns.REJECT_ALL(), // aka dropRecord
          Avros.nulls() // trick to enable morphline to emit any kind of output data, including non-avro data
          );
  
      writeOutput(opts, pipeline, collection);

View Full Code Here

Examples of org.apache.crunch.PCollection.parallelDo()

      PCollection collection = pipeline.read(new NLineFileSource<String>(tmpFile, Writables.strings(), numLinesPerSplit));


      if (!randomizeFewInputFiles) {
        collection = randomize(collection); // uses a high latency MapReduce job
      }
      collection = collection.parallelDo(new HeartbeatFn(), collection.getPType());
      return collection;
    }
  }
  
  private void writeOutput(CrunchIndexerToolOptions opts, Pipeline pipeline, PCollection collection) {

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.