Examples of co.cask.cdap.api.data.batch.BatchReadable

co.cask.cdap.api.data.batch.BatchReadable
Interface for datasets that can be input to a batch job.
In order to feed a dataset into a batch job, the dataset must be splittable into chunks so that it's possible to process every part of the dataset in parallel. Every chunk must be readable as a collection of {key,value} records.
@param < KEY> The key type. @param < VALUE> The value type.

        TextStreamInputFormat.setTimeRange(job, stream.getStartTime(), stream.getEndTime());
        job.setInputFormatClass(TextStreamInputFormat.class);


      } else {
        // We checked on validation phase that it implements BatchReadable
        BatchReadable inputDataSet = (BatchReadable) context.getDataSet(inputDataSetName);
        List<Split> inputSplits = context.getInputDataSelection();
        if (inputSplits == null) {
          inputSplits = inputDataSet.getSplits();
        }
        context.setInput(inputDataSetName, inputSplits);


        LOG.debug("Using Dataset {} as input for MapReduce Job", inputDataSetName);
        DataSetInputFormat.setInput(job, inputDataSetName);

View Full Code Here

  Configuration setInputDataset(String datasetName) {
    Configuration hConf = new Configuration(getHConf());
    Dataset dataset = basicSparkContext.getDataSet(datasetName);
    List<Split> inputSplits;
    if (dataset instanceof BatchReadable) {
      BatchReadable curDataset = (BatchReadable) dataset;
      inputSplits = curDataset.getSplits();
    } else {
      throw new IllegalArgumentException("Failed to read dataset " + datasetName + ". The dataset does not implement" +
                                           " BatchReadable");
    }
    hConf.setClass(MRJobConfig.INPUT_FORMAT_CLASS_ATTR, SparkDatasetInputFormat.class, InputFormat.class);

View Full Code Here

TOP

Related Classes of co.cask.cdap.api.data.batch.BatchReadable

co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService

co.cask.cdap.internal.app.runtime.spark.AbstractSparkContext

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.