Examples of org.kitesdk.data.PartitionStrategy

org.kitesdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

When a {@link Dataset} is configuredwith a partition strategy, that data is considered partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} to determine which partition to write to.

You should use the inner {@link Builder} to create new instances.
@see DatasetDescriptor @see Dataset

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPartitionedWriterSingle() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
      "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPartitionedWriterDouble() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", 2).hash("email", 3).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testGetPartitionReturnsNullIfNoAutoCreate() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
      "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testWriteToSubpartition() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", "username_part", 2).hash("email", 3).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testDropPartition() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

    Assert.assertNotNull(caught);
  }
  
  @Test
  public void testMerge() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
        "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPathIterator_Partition_Directory() {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .hash("username", 2).hash("email", 3).build();


    final FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

        .size());
  }


  @Test
  public void testOverridePartitionStrategy() {
    PartitionStrategy strat = new PartitionStrategy.Builder()
        .hash("keyPart1", "keyPart1", 10).build();
    AvroKeySchema avroKeySchema = parser.parseKeySchema(entitySchema, strat);
    assertEquals(Type.INT, avroKeySchema.getAvroSchema().getField("keyPart1")
        .schema().getType());
    assertEquals(1, avroKeySchema.getPartitionStrategy().getFieldPartitioners()

View Full Code Here

    repo.delete("ns", "records"); // clean up
  }


  @Test
  public void testProvidedPartitions() {
    PartitionStrategy expected = new PartitionStrategy.Builder()
        .provided("year", "int")
        .provided("month", "int")
        .provided("day", "long")
        .provided("us_state")
        .build();

View Full Code Here

    // convert the schema to Hive columns
    table.getSd().setCols(convertSchema(descriptor.getSchema()));


    // copy partitioning info
    if (descriptor.isPartitioned()) {
      PartitionStrategy ps = descriptor.getPartitionStrategy();
      table.getParameters().put(PARTITION_EXPRESSION_PROPERTY_NAME,
          Accessor.getDefault().toExpression(ps));
      table.setPartitionKeys(partitionColumns(ps, descriptor.getSchema()));
    }

View Full Code Here

0 1 2 3 4 5

TOP

Related Classes of org.kitesdk.data.PartitionStrategy

org.kitesdk.cli.commands.TestCreatePartitionStrategyCommand

org.kitesdk.data.crunch.CrunchDatasets$GetStorageKey

org.kitesdk.data.crunch.TestCrunchDatasets

org.kitesdk.data.hbase.avro.AvroKeyEntitySchemaParserTest

org.kitesdk.data.hbase.avro.AvroUtils

org.kitesdk.data.hbase.HBaseMetadataProviderTest

org.kitesdk.data.spi.Constraints

org.kitesdk.data.spi.EntityAccessor

org.kitesdk.data.spi.filesystem.FileSystemDataset

org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.