Examples of com.twitter.elephantbird.mapreduce.output.RCFileThriftOutputFormat

Package com.twitter.elephantbird.mapreduce.output

Examples of com.twitter.elephantbird.mapreduce.output.RCFileThriftOutputFormat

com.twitter.elephantbird.mapreduce.output.RCFileThriftOutputFormat
OutputFormat for storing Thrift objects in RCFile.
Each of the top level fields is stored in a separate column. Thrift field ids are stored in RCFile metadata.
The user can write either a {@link ThriftWritable} with the Thrift objector a {@link BytesWritable} with serialized Thrift bytes. The latterensures that all the fields are preserved even if the current Thrift definition does not match the definition represented in the serialized bytes. Any fields not recognized by current Thrift class are stored in the last column.

  private static RecordWriter<Writable, Writable>
  createThriftWriter(Class<?> thriftClass, final File file)
                    throws IOException, InterruptedException {


    OutputFormat outputFormat = (
      new RCFileThriftOutputFormat(ThriftUtils.getTypeRef(thriftClass.getName())) {
        @Override
        public Path getDefaultWorkFile(TaskAttemptContext context,
            String extension) throws IOException {
          return new Path(file.toURI().toString());
        }

View Full Code Here

    writable = new ThriftWritable(typeRef);
  }


  @Override @SuppressWarnings("unchecked")
  public OutputFormat getOutputFormat() throws IOException {
    return new RCFileThriftOutputFormat(typeRef);
  }

View Full Code Here

TOP

Related Classes of com.twitter.elephantbird.mapreduce.output.RCFileThriftOutputFormat

com.twitter.elephantbird.pig.load.TestRCFileThriftStorage

com.twitter.elephantbird.pig.store.RCFileThriftPigStorage

com.twitter.elephantbird.thrift.TStructDescriptor.Field

org.apache.hadoop.hive.serde2.columnar.BytesRefWritable

org.apache.thrift.protocol.TBinaryProtocol

org.apache.thrift.protocol.TField

org.apache.thrift.transport.TMemoryInputTransport

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.