Examples of org.kitesdk.data.PartitionStrategy

org.kitesdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

When a {@link Dataset} is configuredwith a partition strategy, that data is considered partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} to determine which partition to write to.

You should use the inner {@link Builder} to create new instances.
@see DatasetDescriptor @see Dataset

    }


    for (FileStatus stat : fileStatuses) {
      Path p = fileSystem.makeQualified(stat.getPath());
      PartitionKey key = keyFromDirectory(p.getName());
      PartitionStrategy subPartitionStrategy = Accessor.getDefault()
          .getSubpartitionStrategy(partitionStrategy, 1);
      Builder<E> builder = new FileSystemDataset.Builder<E>()
          .namespace(namespace)
          .name(name)
          .fileSystem(fileSystem)

View Full Code Here

          "root directory to dataset (directory: %s).", partitionUri, directoryUri));
    }


    Iterable<String> parts = Splitter.on('/').split(relativizedUri.getPath());


    PartitionStrategy partitionStrategy = dataset.getDescriptor().getPartitionStrategy();
    List<FieldPartitioner> fieldPartitioners = partitionStrategy.getFieldPartitioners();
    if (Iterables.size(parts) > fieldPartitioners.size()) {
      throw new IllegalArgumentException(String.format("Too many partition directories " +
          "for %s (%s), expecting %s.", partitionUri, Iterables.size(parts),
          fieldPartitioners.size()));
    }

View Full Code Here

  private static final String[] colors = { "green", "blue", "pink", "brown", "yellow" };


  @Override
  public int run(String[] args) throws Exception {
    // Create a partition strategy that hash partitions on username with 10 buckets
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .identity("favoriteColor", "favorite_color")
        .build();


    // Create a dataset of users with the Avro schema
    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()

View Full Code Here

0 1 2 3 4 5

TOP

Related Classes of org.kitesdk.data.PartitionStrategy

org.kitesdk.cli.commands.TestCreatePartitionStrategyCommand

org.kitesdk.data.crunch.CrunchDatasets$GetStorageKey

org.kitesdk.data.crunch.TestCrunchDatasets

org.kitesdk.data.hbase.avro.AvroKeyEntitySchemaParserTest

org.kitesdk.data.hbase.avro.AvroUtils

org.kitesdk.data.hbase.HBaseMetadataProviderTest

org.kitesdk.data.spi.Constraints

org.kitesdk.data.spi.EntityAccessor

org.kitesdk.data.spi.filesystem.FileSystemDataset

org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.