List of partitionKey() Examples

Examples of partitionKey()

com.cloudera.cdk.data.PartitionStrategy.partitionKey()

Construct a partition key with a variadic array of values corresponding to the field partitioners in this partition strategy.

It is permitted to have fewer values than field partitioners, in which case all subpartititions in the unspecified parts of the key are matched by the key.

Null values are not permitted.

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

    Assert.assertTrue("Partitioned directory 0 exists",
      fileSystem.exists(new Path(testDirectory, "username=0")));
    Assert.assertTrue("Partitioned directory 1 exists",
      fileSystem.exists(new Path(testDirectory, "username=1")));
    checkTestUsers(ds, 10);
    PartitionKey key0 = partitionStrategy.partitionKey(0);
    PartitionKey key1 = partitionStrategy.partitionKey(1);
    int total = readTestUsersInPartition(ds, key0, null)
      + readTestUsersInPartition(ds, key1, null);
    Assert.assertEquals(10, total);

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

      fileSystem.exists(new Path(testDirectory, "username=0")));
    Assert.assertTrue("Partitioned directory 1 exists",
      fileSystem.exists(new Path(testDirectory, "username=1")));
    checkTestUsers(ds, 10);
    PartitionKey key0 = partitionStrategy.partitionKey(0);
    PartitionKey key1 = partitionStrategy.partitionKey(1);
    int total = readTestUsersInPartition(ds, key0, null)
      + readTestUsersInPartition(ds, key1, null);
    Assert.assertEquals(10, total);


    testPartitionKeysAreEqual(ds, key0, key1);

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

      for (int i2 = 0; i2 < 3; i2++) {
        String part = "username=" + i1 + "/email=" + i2;
        Assert.assertTrue("Partitioned directory " + part + " exists",
          fileSystem.exists(new Path(testDirectory, part)));
        total += readTestUsersInPartition(ds,
          partitionStrategy.partitionKey(i1, i2), null);
      }
    }
    Assert.assertEquals(10, total);


    testPartitionKeysAreEqual(ds, key0, key1);

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

            .partitionStrategy(partitionStrategy)
            .build())
        .build();


    Assert
      .assertNull(ds.getPartition(partitionStrategy.partitionKey(1), false));
  }


  @Test
  @SuppressWarnings("deprecation")
  public void testWriteToSubpartition() throws IOException {

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

    Assert.assertTrue(
      fileSystem.isDirectory(new Path(testDirectory, "username=0")));
    Assert.assertTrue(
      fileSystem.isDirectory(new Path(testDirectory, "username=1")));


    ds.dropPartition(partitionStrategy.partitionKey(0));
    Assert.assertFalse(
      fileSystem.isDirectory(new Path(testDirectory, "username=0")));


    ds.dropPartition(partitionStrategy.partitionKey(1));
    Assert.assertFalse(

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()


    ds.dropPartition(partitionStrategy.partitionKey(0));
    Assert.assertFalse(
      fileSystem.isDirectory(new Path(testDirectory, "username=0")));


    ds.dropPartition(partitionStrategy.partitionKey(1));
    Assert.assertFalse(
      fileSystem.isDirectory(new Path(testDirectory, "username=1")));


    DatasetException caught = null;

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

      fileSystem.isDirectory(new Path(testDirectory, "username=1")));


    DatasetException caught = null;


    try {
      ds.dropPartition(partitionStrategy.partitionKey(0));
    } catch (DatasetException e) {
      caught = e;
    }


    Assert.assertNotNull(caught);

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

    Dataset<Record> outputDataset = repo.create("out", new DatasetDescriptor.Builder()
        .schema(USER_SCHEMA).partitionStrategy(partitionStrategy).build());


    writeTestUsers(inputDataset, 10);


    PartitionKey key = partitionStrategy.partitionKey(0);
    Dataset<Record> inputPart0 = inputDataset.getPartition(key, false);
    Dataset<Record> outputPart0 = outputDataset.getPartition(key, true);


    Pipeline pipeline = new MRPipeline(TestCrunchDatasets.class);
    PCollection<GenericData.Record> data = pipeline.read(

View Full Code Here

Examples of com.cloudera.cdk.data.PartitionStrategy.partitionKey()

    Dataset<GenericRecord> users = repo.load("users");


    // Get the partition strategy and use it to construct a partition key for
    // hash(username)=0
    PartitionStrategy partitionStrategy = users.getDescriptor().getPartitionStrategy();
    PartitionKey partitionKey = partitionStrategy.partitionKey(0);


    // Get the dataset partition for the partition key
    Dataset<GenericRecord> partition = users.getPartition(partitionKey, false);


    // Get a reader for the partition and read all the users

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.