Examples of org.apache.flume.sink.hbase.HBaseSink

org.apache.flume.sink.hbase.HBaseSink
A simple sink which reads events from a channel and writes them to HBase. The Hbase configution is picked up from the first hbase-site.xml encountered in the classpath. This sink supports batch reading of events from the channel, and writing them to Hbase, to minimize the number of flushes on the hbase tables. To use this sink, it has to be configured with certain mandatory parameters:
table: The name of the table in Hbase to write to.
columnFamily: The column family in Hbase to write to.
This sink will commit each transaction if the table's write buffer size is reached or if the number of events in the current transaction reaches the batch size, whichever comes first.
Other optional parameters are:
serializer: A class implementing {@link HBaseEventSerializer}. An instance of this class will be used to write out events to hbase.
serializer.*: Passed in the configure() method to serializer as an object of {@link org.apache.flume.Context}.
batchSize: This is the batch size used by the client. This is the maximum number of events the sink will commit per transaction. The default batch size is 100 events.
Note: While this sink flushes all events in a transaction to HBase in one shot, Hbase does not guarantee atomic commits on multiple rows. So if a subset of events in a batch are written to disk by Hbase and Hbase fails, the flume transaction is rolled back, causing flume to write all the events in the transaction all over again, which will cause duplicates. The serializer is expected to take care of the handling of duplicates etc. HBase also does not support batch increments, so if multiple increments are returned by the serializer, then HBase failure will cause them to be re-written, when HBase comes back up.

      "and uncomment this annotation to run this test.")
  @Test(expected = EventDeliveryException.class)
  public void testHBaseFailure() throws Exception {
    ctx.put("batchSize", "2");
    testUtility.createTable(tableName.getBytes(), columnFamily.getBytes());
    HBaseSink sink = new HBaseSink(testUtility.getConfiguration());
    Configurables.configure(sink, ctx);
    //Reset the context to a higher batchSize
    ctx.put("batchSize", "100");
    Channel channel = new MemoryChannel();
    Configurables.configure(channel, new Context());
    sink.setChannel(channel);
    sink.start();
    Transaction tx = channel.getTransaction();
    tx.begin();
    for(int i = 0; i < 3; i++){
      Event e = EventBuilder.withBody(Bytes.toBytes(valBase + "-" + i));
      channel.put(e);
    }
    tx.commit();
    tx.close();
    sink.process();
    HTable table = new HTable(testUtility.getConfiguration(), tableName);
    byte[][] results = getResults(table, 2);
    byte[] out;
    int found = 0;
    for(int i = 0; i < 2; i++){
      for(int j = 0; j < 2; j++){
        if(Arrays.equals(results[j],Bytes.toBytes(valBase + "-" + i))){
          found++;
          break;
        }
      }
    }
    Assert.assertEquals(2, found);
    out = results[2];
    Assert.assertArrayEquals(Longs.toByteArray(2), out);
    testUtility.shutdownMiniCluster();
    sink.process();
    sink.stop();
  }

View Full Code Here

    testUtility.shutdownMiniCluster();
  }


  @Test
  public void testOneEvent() throws Exception {
    HBaseSink sink = new HBaseSink(testUtility.getConfiguration());
    Configurables.configure(sink, ctx);
    Channel channel = new MemoryChannel();
    Configurables.configure(channel, new Context());
    sink.setChannel(channel);
    sink.start();
    Transaction tx = channel.getTransaction();
    tx.begin();
    Event e = EventBuilder.withBody(
        Bytes.toBytes(valBase));
    channel.put(e);
    tx.commit();
    tx.close();


    sink.process();
    sink.stop();
    HTable table = new HTable(testUtility.getConfiguration(), tableName);
    Scan scan = new Scan();
    scan.addColumn(columnFamily.getBytes(),plCol.getBytes());
    scan.setStartRow( Bytes.toBytes("default"));
    ResultScanner rs = table.getScanner(scan);

View Full Code Here

0 1

TOP

Related Classes of org.apache.flume.sink.hbase.HBaseSink

org.apache.flume.Channel

org.apache.flume.Context

org.apache.flume.Event

org.apache.flume.instrumentation.SinkCounter

org.apache.flume.sink.hbase.TestHBaseSink

org.apache.flume.Transaction

org.apache.hadoop.hbase.client.HTable

org.apache.hadoop.hbase.client.Increment

java.lang.reflect.Method

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.