Examples of com.cloudera.cdk.data.Dataset

com.cloudera.cdk.data.Dataset

A logical representation of a set of data entities.

Logically, all datasets have two generic properties: a name, and a descriptor that holds information such as the dataset's schema and its partitioning information. Concrete implementations of {@code Dataset} may supportadditional properties, mandatory or otherwise, as needed. {@code Dataset}s are not normally instantiated directly, but managed by a repository (also implementation-specific).

Implementations of {@link Dataset} are immutable.

@param < E> The type of entities stored in this {@code Dataset}. @see View @see DatasetRepository @see DatasetWriter @see DatasetReader @see PartitionStrategy @see DatasetDescriptor @see Schema

    if (!initialized) {
      // initialize here rather than in activateOptions to avoid initialization
      // cycle in Configuration and log4j
      try {
        DatasetRepository repo = DatasetRepositories.open(datasetRepositoryUri);
        Dataset dataset = repo.load(datasetName);
        if (dataset.getDescriptor().isPartitioned()) {
          partitionStrategy = dataset.getDescriptor().getPartitionStrategy();
        }
        URL schemaUrl = dataset.getDescriptor().getSchemaUrl();
        if (schemaUrl != null) {
          setAvroSchemaUrl(schemaUrl.toExternalForm());
        }
      } catch (Exception e) {
        throw new FlumeException(e);


  @Override
  public int run(String[] args) throws Exception {
    DatasetRepository repo = DatasetRepositories.open("repo:hdfs://localhost:8020/user/cloudera");


    Dataset movies = repo.load("movies");
    DatasetReader reader = movies.newReader();
    try {
      reader.open();
      for (Object rec : reader) {
        System.err.println("Movie: " + rec);
      }

Examples of com.cloudera.cdk.data.Dataset

Related Classes of com.cloudera.cdk.data.Dataset