Examples of TableInputFormat

org.apache.hadoop.hbase.mapreduce.TableInputFormat
Convert HBase tabular data into a format that is consumable by Map/Reduce.
org.apache.hadoop.zebra.mapred.TableInputFormat
the tuple object for reuse. // indices of various fields in the input Tuple. int idxName, idxSalary, idxBonusPct; @Override public void configure(JobConf job) { Schema projection = TableInputFormat.getProjection(job); // determine the field indices. idxName = projection.getColumnIndex("Name"); idxSalary = projection.getColumnIndex("Salary"); idxBonusPct = projection.getColumnIndex("BonusPct"); } @Override public void map(BytesWritable key, Tuple value, OutputCollector<K, V> output, Reporter reporter) throws IOException { try { String name = (String) value.get(idxName); int salary = (Integer) value.get(idxSalary); double bonusPct = (Double) value.get(idxBonusPct); // do something with the input data } catch (ExecException e) { e.printStackTrace(); } } @Override public void close() throws IOException { // no-op } } A little bit more explanation on the PIG {@link Tuple} objects. A Tuple is anordered list of PIG datum objects. The permitted PIG datum types can be categorized as Scalar types and Composite types.
Supported Scalar types include seven native Java types: Boolean, Byte, Integer, Long, Float, Double, String, as well as one PIG class called {@link DataByteArray} that represents type-less byte array.
Supported Composite types include:
- {@link Map} : It is the same as Java Map class, with the additionalrestriction that the key-type must be one of the scalar types PIG recognizes, and the value-type any of the scaler or composite types PIG understands.
- {@link DataBag} : A DataBag is a collection of Tuples.
- {@link Tuple} : Yes, Tuple itself can be a datum in another Tuple.
org.apache.hadoop.zebra.mapreduce.TableInputFormat
the tuple object for reuse. // indices of various fields in the input Tuple. int idxName, idxSalary, idxBonusPct; @Override public void configure(Job job) { Schema projection = TableInputFormat.getProjection(job); // determine the field indices. idxName = projection.getColumnIndex("Name"); idxSalary = projection.getColumnIndex("Salary"); idxBonusPct = projection.getColumnIndex("BonusPct"); } @Override public void map(BytesWritable key, Tuple value, OutputCollector<K, V> output, Reporter reporter) throws IOException { try { String name = (String) value.get(idxName); int salary = (Integer) value.get(idxSalary); double bonusPct = (Double) value.get(idxBonusPct); // do something with the input data } catch (ExecException e) { e.printStackTrace(); } } @Override public void close() throws IOException { // no-op } } A little bit more explanation on the PIG {@link Tuple} objects. A Tuple is anordered list of PIG datum objects. The permitted PIG datum types can be categorized as Scalar types and Composite types.
Supported Scalar types include seven native Java types: Boolean, Byte, Integer, Long, Float, Double, String, as well as one PIG class called {@link DataByteArray} that represents type-less byte array.
Supported Composite types include:
- {@link Map} : It is the same as Java Map class, with the additionalrestriction that the key-type must be one of the scalar types PIG recognizes, and the value-type any of the scaler or composite types PIG understands.
- {@link DataBag} : A DataBag is a collection of Tuples.
- {@link Tuple} : Yes, Tuple itself can be a datum in another Tuple.

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

class HBaseInputFormat implements InputFormat<ImmutableBytesWritable, Result> {


    private final TableInputFormat inputFormat;


    public HBaseInputFormat() {
        inputFormat = new TableInputFormat();
    }

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

        return null;
    }


    @Override
    public InputFormat getInputFormat() {      
        TableInputFormat inputFormat = new HBaseTableIFBuilder()
        .withLimit(limit_)
        .withGt(gt_)
        .withGte(gte_)
        .withLt(lt_)
        .withLte(lte_)

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

class HBaseInputFormat implements InputFormat<ImmutableBytesWritable, ResultWritable> {


  private final TableInputFormat inputFormat;


  public HBaseInputFormat() {
    inputFormat = new TableInputFormat();
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

        return null;
    }


    @Override
    public InputFormat getInputFormat() {
        TableInputFormat inputFormat = new HBaseTableIFBuilder()
        .withLimit(limit_)
        .withGt(gt_)
        .withGte(gte_)
        .withLt(lt_)
        .withLte(lte_)
        .withConf(m_conf)
        .build();
        inputFormat.setScan(scan);
        return inputFormat;
    }

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

  @Override
  public RecordReader<E, Void> createRecordReader(InputSplit inputSplit,
      TaskAttemptContext taskAttemptContext) throws IOException, InterruptedException {
    Configuration conf = Hadoop.TaskAttemptContext
        .getConfiguration.invoke(taskAttemptContext);
    TableInputFormat delegate = getDelegate(conf);
    RecordReader<E, Void> unfilteredRecordReader = new HBaseRecordReaderWrapper<E>(
        delegate.createRecordReader(inputSplit, taskAttemptContext), entityMapper);
    if (view != null) {
      // use the constraints to filter out entities from the reader
      return new FilteredRecordReader<E>(unfilteredRecordReader,
          view.getConstraints(), view.getAccessor());
    }

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

    }
    return unfilteredRecordReader;
  }


  private TableInputFormat getDelegate(Configuration conf) throws IOException {
    TableInputFormat delegate = new TableInputFormat();
    String tableName = HBaseMetadataProvider.getTableName(dataset.getName());
    conf.set(TableInputFormat.INPUT_TABLE, tableName);
    if (view != null) {
      Job tempJob = new Job();
      Scan scan = ((BaseEntityScanner) view.newEntityScanner()).getScan();
      TableMapReduceUtil.initTableMapperJob(tableName, scan, TableMapper.class, null,
          null, tempJob);
      Configuration tempConf = Hadoop.JobContext.getConfiguration.invoke(tempJob);
      conf.set(SCAN, tempConf.get(SCAN));
    }
    delegate.setConf(conf);
    return delegate;
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

        return null;
    }


    @Override
    public InputFormat getInputFormat() {      
        TableInputFormat inputFormat = new HBaseTableIFBuilder()
        .withLimit(limit_)
        .withGt(gt_)
        .withGte(gte_)
        .withLt(lt_)
        .withLte(lte_)

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

        return null;
    }


    @Override
    public InputFormat getInputFormat() {      
        TableInputFormat inputFormat = new HBaseTableIFBuilder()
        .withLimit(limit_)
        .withGt(gt_)
        .withGte(gte_)
        .withLt(lt_)
        .withLte(lte_)

View Full Code Here

Examples of org.apache.hadoop.hbase.mapreduce.TableInputFormat

        return null;
    }


    @Override
    public InputFormat getInputFormat() {      
        TableInputFormat inputFormat = new HBaseTableIFBuilder()
        .withLimit(limit_)
        .withGt(gt_)
        .withGte(gte_)
        .withLt(lt_)
        .withLte(lte_)

View Full Code Here

Examples of org.apache.hadoop.zebra.mapred.TableInputFormat

    private org.apache.hadoop.zebra.schema.Schema projectionSchema;
  /**
   * default constructor
   */
  public TableLoader() {
    inputFormat = new TableInputFormat();
  }

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.