Examples of com.urbanairship.datacube.backfill.HBaseBackfillMerger

Package com.urbanairship.datacube.backfill

Examples of com.urbanairship.datacube.backfill.HBaseBackfillMerger

com.urbanairship.datacube.backfill.HBaseBackfillMerger
Backfilling is a 3 step process: (1) snapshot the live cube data into a so-called snapshot table (2) do the application-level event processing into a cube that is stored in some non-visible table (3) Merge the snapshot, the backfill table, and the live cube, making changes to the live cube. This MR job does step 3. TODO InputSplit locality for mapreduce

        Assert.assertEquals(1L, newCubeWrapper.getHourColorCount(midnight.plusHours(2), Color.BLUE));
        Assert.assertEquals(0L, newCubeWrapper.getHourColorCount(midnight.plusHours(1), Color.RED));
        Assert.assertEquals(0L, newCubeWrapper.getHourColorCount(midnight.plusHours(2), Color.RED));
        
        // Merge the backfill table and the snapshot table into the live cube table
        success = new HBaseBackfillMerger(hadoopConf, ArrayUtils.EMPTY_BYTE_ARRAY, LIVE_CUBE_TABLE, 
                SNAPSHOT_TABLE, BACKFILL_TABLE, CF, LongOpDeserializer.class).runWithCheckedExceptions();
        Assert.assertTrue(success);


        // Now, after the backfill, all the events' colors should have been counted.
        Assert.assertEquals(2L, newCubeWrapper.getHourColorCount(midnight.plusHours(2), Color.BLUE));

View Full Code Here

        Assert.assertTrue(new HBaseSnapshotter(conf, CUBE_DATA_TABLE, CF, BACKFILLED_TABLE,
                new Path("hdfs:///test_hfiles"), true, null, null).runWithCheckedExceptions());
        
        // Since the backfilled table is identical to the snapshot, there should be no changes to the
        // live production table
        HBaseBackfillMerger backfiller = new HBaseBackfillMerger(conf, ArrayUtils.EMPTY_BYTE_ARRAY, 
                CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE, BACKFILLED_TABLE, CF, LongOp.LongOpDeserializer.class);
        Assert.assertTrue(backfiller.runWithCheckedExceptions());
        assertTablesEqual(conf, CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE);
        
    }

View Full Code Here

                new Path("hdfs:///test_hfiles"), false, null, null).runWithCheckedExceptions());
        
        // The snapshot should be equal to the source table
        assertTablesEqual(conf, CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE);
        
        HBaseBackfillMerger backfiller = new HBaseBackfillMerger(conf, ArrayUtils.EMPTY_BYTE_ARRAY, 
                CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE, BACKFILLED_TABLE, CF, 
                LongOp.LongOpDeserializer.class);
        Assert.assertTrue(backfiller.runWithCheckedExceptions());
        
        // After backfilling from an empty "backfilled" table, the live cube should have no rows
        Assert.assertTrue(!new HTable(conf, CUBE_DATA_TABLE).getScanner(CF).iterator().hasNext());
    }

View Full Code Here

        // Simulate two writes to the live table that wouldn't be seen by the app as it backfills.
        // This is like a client doing a write concurrently with a backfill.
        cubeIo.writeSync(new LongOp(6), new WriteBuilder(cube).at(onlyDimension, "coord1"));
        cubeIo.writeSync(new LongOp(7), new WriteBuilder(cube).at(onlyDimension, "coord2"));
        
        HBaseBackfillMerger backfiller = new HBaseBackfillMerger(conf, ArrayUtils.EMPTY_BYTE_ARRAY,
                CUBE_DATA_TABLE, SNAPSHOT_DEST_TABLE, BACKFILLED_TABLE, CF, 
                LongOp.LongOpDeserializer.class);
        Assert.assertTrue(backfiller.runWithCheckedExceptions());
        
        // After everyhing's done, the two writes that occurred concurrently with the backfill
        // should be seen in the live table. coord1 should be 5+6=11 and coord2 should be 7.
        Assert.assertEquals(11L, 
                cubeIo.get(new ReadBuilder(cube).at(onlyDimension, "coord1")).get().getLong());

View Full Code Here

TOP

Related Classes of com.urbanairship.datacube.backfill.HBaseBackfillMerger

com.urbanairship.datacube.HBaseBackfillerTest

com.urbanairship.datacube.HBaseBackfillIntegrationTest

org.apache.hadoop.fs.Path

org.apache.hadoop.hbase.client.HTable

org.apache.hadoop.hbase.client.ResultScanner

org.apache.hadoop.hbase.client.Scan

org.apache.hadoop.mapreduce.Job

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.