Examples of com.cloudera.flume.handlers.hdfs.SeqfileEventSink

Package com.cloudera.flume.handlers.hdfs

Examples of com.cloudera.flume.handlers.hdfs.SeqfileEventSink

com.cloudera.flume.handlers.hdfs.SeqfileEventSink
This is an event sink that dumps to a hadoop sequence file on the local file system..

    FlumeTestHarness.setupLocalWriteDir();
    File tmp = FlumeTestHarness.tmpdir;


    for (int i = 0; i < 3000; i++) {
      File path = new File(tmp, "" + i);
      EventSink snk = new SeqfileEventSink(path);
      snk.open();
      Event e = new EventImpl(("foo " + i).getBytes());
      snk.append(e);
      snk.close();
    }


  }

View Full Code Here

   * attributes and events.
   */
  private void restoreAckFramingToLoggedState(final String f,
      MemorySinkSource mem) throws IOException, InterruptedException {
    EventSink ackfixed = new AckChecksumInjector<EventSink>(
        new SeqfileEventSink(new File(loggedDir, f).getAbsoluteFile()));
    try {
      ackfixed.open();
      EventUtil.dumpAll(mem, ackfixed);
      ackfixed.close();
    } catch (IOException e) {

View Full Code Here

  synchronized public EventSink newAckWritingSink(final Tagger tagger,
      AckListener al) throws IOException {
    File dir = getDir(State.WRITING);
    final String tag = tagger.newTag();


    EventSink bareSink = new SeqfileEventSink(
        new File(dir, tag).getAbsoluteFile());
    EventSink curSink = new AckChecksumInjector<EventSink>(bareSink,
        tag.getBytes(), al);


    writingQ.add(tag);

View Full Code Here

   */
  synchronized public EventSink newWritingSink(final Tagger tagger)
      throws IOException {
    File dir = getDir(State.WRITING);
    final String tag = tagger.newTag();
    EventSink curSink = new SeqfileEventSink(
        new File(dir, tag).getAbsoluteFile());
    writingQ.add(tag);
    WALData data = new WALData(tag);
    table.put(tag, data);

View Full Code Here

   */
  synchronized public EventSink newWritingSink(final Tagger tagger)
      throws IOException {
    File dir = getDir(State.WRITING);
    final String tag = tagger.newTag();
    EventSink curSink = new SeqfileEventSink(
        new File(dir, tag).getAbsoluteFile());
    writingQ.add(tag);
    DFOData data = new DFOData(tag);
    table.put(tag, data);
    writingCount.incrementAndGet();

View Full Code Here

    wal.open(); // create dirs


    // create a seq file with no ack close.
    File f = new File(wal.writingDir,
        "writing.00000000.20100204-015814F430-0800.seq");
    SeqfileEventSink sf = new SeqfileEventSink(f);
    AckChecksumInjector<EventSink> inj = new AckChecksumInjector<EventSink>(sf);
    inj.open();
    inj.append(new EventImpl("test".getBytes()));
    // notice ack checksum inj not closed, but the subsink is.
    // no ack end event sent on purpose, which forces recover() to reframe the
    // data.
    sf.close();


    // do the low level recovery
    wal.recover();


    // check to make sure wal file is gone

View Full Code Here

    wal.open(); // create dirs


    // create a seq file with no ack close.
    File f = new File(wal.writingDir,
        "writing.00000000.20100204-015814F430-0800.seq");
    SeqfileEventSink sf = new SeqfileEventSink(f);
    AckChecksumInjector<EventSink> inj = new AckChecksumInjector<EventSink>(sf);
    inj.open();
    inj.append(new EventImpl("test".getBytes()));


    // Cause a state ack check state error
    Event e = inj.openEvent();
    sf.append(e);


    // proper close event
    inj.close();


    // do the low level recovery

View Full Code Here

    wal.open(); // create dirs


    // create a seq file with no ack close or ack open
    File f = new File(wal.writingDir,
        "writing.00000000.20100204-015814F430-0800.seq");
    SeqfileEventSink sf = new SeqfileEventSink(f);
    sf.open();
    sf.append(new EventImpl("test".getBytes()));
    sf.close();


    // do the low level recovery
    wal.recover();


    // check to make sure wal file is gone

View Full Code Here

    wal.open(); // create dirs


    // create a seq file with no ack close.
    File f = new File(wal.writingDir,
        "writing.00000000.20100204-015814F430-0800.seq");
    SeqfileEventSink sf = new SeqfileEventSink(f);
    AckChecksumInjector<EventSink> inj = new AckChecksumInjector<EventSink>(sf);
    inj.open();
    inj.append(new EventImpl("test".getBytes()));


    // need to keep the tag from the inj, but purposely mess up checksum
    Event e = inj.closeEvent();
    byte[] ref = e.get(AckChecksumInjector.ATTR_ACK_HASH);
    Arrays.fill(ref, (byte) 0);
    sf.append(e);


    // close, and do not send good ack close
    sf.close();


    // do the low level recovery
    wal.recover();


    // check to make sure wal file is gone

View Full Code Here

    BenchmarkHarness.setupLocalWriteDir();
    File tmp = BenchmarkHarness.tmpdir;


    for (int i = 0; i < 3000; i++) {
      File path = new File(tmp, "" + i);
      EventSink snk = new SeqfileEventSink(path);
      snk.open();
      Event e = new EventImpl(("foo " + i).getBytes());
      snk.append(e);
      snk.close();
    }


  }

View Full Code Here

0 1 2

TOP

Related Classes of com.cloudera.flume.handlers.hdfs.SeqfileEventSink

com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager

com.cloudera.flume.agent.durability.NaiveFileWALManager

com.cloudera.flume.agent.durability.TestNaiveFileWALHandles

com.cloudera.flume.agent.durability.TestNaiveFileWALManager

com.cloudera.flume.conf.FlumeConfiguration

com.cloudera.flume.conf.SinkFactory.SinkBuilder

com.cloudera.flume.handlers.log4j.SeqfileAppender

com.cloudera.flume.PerfDiskIO

com.cloudera.flume.PerfHdfsIO

org.apache.hadoop.conf.Configuration

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.