Examples of com.cloudera.cdk.data.PartitionStrategy

com.cloudera.cdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

A {@code PartitionStrategy} is configured with one or more{@link FieldPartitioner}s upon creation. When a {@link Dataset} is configuredwith a partition strategy, we say that data is partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} which, in turn, produces a {@link PartitionKey}that is used by the dataset implementation to select the proper partition.

Users should use the inner {@link Builder} to create new instances.
@see FieldPartitioner @see PartitionKey @see DatasetDescriptor @see Dataset

    // Load the users dataset
    Dataset<GenericRecord> users = repo.load("users");


    // Get the partition strategy and use it to construct a partition key for
    // hash(username)=0
    PartitionStrategy partitionStrategy = users.getDescriptor().getPartitionStrategy();
    PartitionKey partitionKey = partitionStrategy.partitionKey(0);


    // Get the dataset partition for the partition key
    Dataset<GenericRecord> partition = users.getPartition(partitionKey, false);


    // Get a reader for the partition and read all the users

View Full Code Here


    // Construct a filesystem dataset repository rooted at /tmp/data
    DatasetRepository repo = DatasetRepositories.open("repo:hdfs:/tmp/data");


    // Create a partition strategy that hash partitions on username with 10 buckets
    PartitionStrategy partitionStrategy =
        new PartitionStrategy.Builder().hash("username", 10).build();


    // Create a dataset of users with the Avro schema in the repository
    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
        .schemaUri("resource:user.avsc")

View Full Code Here

    builder.set("level", "INFO");
    builder.set("component", "StagingToPersistentSerial");
    builder.set("message", "Fake log message");


    // access the partition strategy, which produces keys from records
    final PartitionStrategy partitioner = data.getDescriptor()
        .getPartitionStrategy();


    return partitioner.partitionKeyForEntity(builder.build());
  }

View Full Code Here

0 1 2

TOP

Related Classes of com.cloudera.cdk.data.PartitionStrategy

com.cloudera.cdk.data.crunch.TestCrunchDatasets

com.cloudera.cdk.data.filesystem.FileSystemDataset

com.cloudera.cdk.data.filesystem.FileSystemDatasetRepository

com.cloudera.cdk.data.filesystem.TestFileSystemDataset

com.cloudera.cdk.data.filesystem.TestFileSystemDatasetRepository

com.cloudera.cdk.data.filesystem.TestPartitionedDatasetWriter

com.cloudera.cdk.data.filesystem.TestPathConversion

com.cloudera.cdk.data.hbase.HBaseMetadataProvider

com.cloudera.cdk.data.hcatalog.HiveUtils

com.cloudera.cdk.data.partition.HashFieldPartitioner

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.