Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutput

Package com.google.appengine.tools.mapreduce.outputs

Examples of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutput

com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutput
An {@link Output} that writes bytes to a set of Cloud Storage files, one per shard.Produces a single file output (usually on a per-shard basis). This format does not insert any separator characters, so it by default cannot be read back with the CloudStorageLineInputReader.

    builder.setInput(input);
    builder.setMapper(new LongToBytesMapper());
    builder.setKeyMarshaller(Marshallers.getByteBufferMarshaller());
    builder.setValueMarshaller(Marshallers.getByteBufferMarshaller());
    builder.setReducer(ValueProjectionReducer.<ByteBuffer, ByteBuffer>create());
    builder.setOutput(new GoogleCloudStorageFileOutput("bucket", "fileNamePattern-%04d",
        "application/octet-stream"));
    builder.setNumReducers(2);
    runTest(builder.build(), new Verifier<GoogleCloudStorageFileSet>() {
      @Override
      public void verify(MapReduceResult<GoogleCloudStorageFileSet> result) throws Exception {

View Full Code Here

    Marshaller<Integer> intermediateValueMarshaller = Marshallers.getIntegerMarshaller();
    Reducer<Integer, Integer, ArrayList<Integer>> reducer = new CollisionFindingReducer();
    Marshaller<ArrayList<Integer>> outputMarshaller = Marshallers.getSerializationMarshaller();


    Output<ArrayList<Integer>, GoogleCloudStorageFileSet> output = new MarshallingOutput<>(
        new GoogleCloudStorageFileOutput(bucket, "CollidingSeeds-%04d", "integers"),
        outputMarshaller);
    // [START mapReduceSpec]
    MapReduceSpecification<Long, Integer, Integer, ArrayList<Integer>, GoogleCloudStorageFileSet>
        spec = new MapReduceSpecification.Builder<>(input, mapper, reducer, output)
            .setKeyMarshaller(intermediateKeyMarshaller)

View Full Code Here

TOP

Related Classes of com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutput

com.google.appengine.demos.mapreduce.randomcollisions.CollisionFindingServlet

com.google.appengine.tools.cloudstorage.GcsFilename

com.google.appengine.tools.mapreduce.EndToEndTest

com.google.appengine.tools.mapreduce.GoogleCloudStorageFileSet

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.