Examples of com.urbanairship.datacube.backfill.HBaseSnapshotter

Package com.urbanairship.datacube.backfill

Examples of com.urbanairship.datacube.backfill.HBaseSnapshotter

com.urbanairship.datacube.backfill.HBaseSnapshotter
Takes a "snapshot" of an HBase column family in two steps: - Mapreduce to write all KeyValues of the source into HFiles on disk - Use LoadIncrementalHFiles to bulk-load the HFiles into the target CF. The snapshot isn't a true snapshot because writers could alter the source data while we're mapreducing over it.

        
        Configuration hadoopConf = getTestUtil().getConfiguration();
        
        // Take a snapshot of the "live" table and store it in the "snapshot" table
        boolean success;
        success = new HBaseSnapshotter(hadoopConf, LIVE_CUBE_TABLE, CF, SNAPSHOT_TABLE, 
                new Path("hdfs://localhost:" + getTestUtil().getDFSCluster().getNameNodePort() + "/snapshot_hfiles"),
                false, null, null).runWithCheckedExceptions();
        Assert.assertTrue(success);
        
        // Backfill (re-count all existing events into the backfill table using the new cube)

View Full Code Here

        
        // Get some cube data into the source table, doesn't really matter what.
        DbHarnessTests.basicTest(hbaseDbHarness);
        
        // Snapshot the source table
        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, SNAPSHOT_DEST_TABLE,
                new Path("hdfs:///test_hfiles"), false, null, null).runWithCheckedExceptions());
        // The snapshot should be equal to the source table
        assertTablesEqual(conf, CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE);
        
        // Simulate a backfill by copying the live cube
        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, BACKFILLED_TABLE,
                new Path("hdfs:///test_hfiles"), true, null, null).runWithCheckedExceptions());
        
        // Since the backfilled table is identical to the snapshot, there should be no changes to the
        // live production table
        HBaseBackfillMerger backfiller = new HBaseBackfillMerger(conf, ArrayUtils.EMPTY_BYTE_ARRAY,

View Full Code Here

        
        // Get some cube data into the source table, doesn't really matter what.
        DbHarnessTests.basicTest(hbaseDbHarness);
        
        // Copy the source table using the snapshotter
        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, SNAPSHOT_DEST_TABLE,
                new Path("hdfs:///test_hfiles"), false, null, null).runWithCheckedExceptions());
        
        // The snapshot should be equal to the source table
        assertTablesEqual(conf, CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE);

View Full Code Here

        
        // Before doing any snapshotting/backfilling, there's one value "5" in the cube.
        cubeIo.writeSync(new LongOp(5), new WriteBuilder(cube).at(onlyDimension, "coord1"));
        
        // Snapshot the source table
        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, SNAPSHOT_DEST_TABLE,
                new Path("hdfs:///test_hfiles"), false, null, null).runWithCheckedExceptions());


        // Simulate a backfill by copying the live cube
        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, BACKFILLED_TABLE,
                new Path("hdfs:///test_hfiles"), true, null, null).runWithCheckedExceptions());
        
        // Simulate two writes to the live table that wouldn't be seen by the app as it backfills.
        // This is like a client doing a write concurrently with a backfill.
        cubeIo.writeSync(new LongOp(6), new WriteBuilder(cube).at(onlyDimension, "coord1"));

View Full Code Here

TOP

Related Classes of com.urbanairship.datacube.backfill.HBaseSnapshotter

com.urbanairship.datacube.HBaseBackfillerTest

com.urbanairship.datacube.HBaseBackfillIntegrationTest

org.apache.hadoop.fs.FileStatus

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.hbase.client.HBaseAdmin

org.apache.hadoop.hbase.client.HTable

org.apache.hadoop.hbase.client.ResultScanner

org.apache.hadoop.hbase.client.Scan

org.apache.hadoop.hbase.HColumnDescriptor

org.apache.hadoop.hbase.HTableDescriptor

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.