Examples of com.cloudera.flume.handlers.rolling.ProcessTagger

Package com.cloudera.flume.handlers.rolling

Examples of com.cloudera.flume.handlers.rolling.ProcessTagger

com.cloudera.flume.handlers.rolling.ProcessTagger
This tagger uses process name and pid as its prefix to manage log write-ahead files. It uses a file name convention and to tag on batches of events.

    String rpt = "foo";
    String snk = "{ ackChecker => [ console, counter(\"" + rpt + "\") ] }";
    Context ctx = new ReportTestingContext();
    EventSink es = FlumeBuilder.buildSink(ctx, snk);
    NaiveFileWALDeco wal = new NaiveFileWALDeco(ctx, es, node.getWalManager(),
        new TimeTrigger(new ProcessTagger(), 100), node.getAckChecker()
            .getAgentAckQueuer(), 10000);


    // open and send data.
    wal.open();
    for (int i = 0; i < count; i++) {

View Full Code Here

      };


      FlumeNode node = FlumeNode.getInstance();
      EventSinkDecorator<EventSink> deco = new NaiveFileWALDeco(
          LogicalNodeContext.testingContext(), snk, node.getWalManager(),
          new TimeTrigger(new ProcessTagger(), 1000), node.getAckChecker()
              .getAgentAckQueuer(), 1000);


      deco.open();
      deco.append(e);
      deco.close();

View Full Code Here

    // putting in large ridiculous constant
    NaiveFileWALManager wal = new NaiveFileWALManager(dir);
    wal.open();


    // get a sink write to it and then be done with it.
    Tagger t = new ProcessTagger();
    EventSink sink = wal.newWritingSink(t);
    sink.open();
    sink.append(new EventImpl("foo".getBytes()));
    assertEquals(1, wal.getWritingTags().size());

View Full Code Here

    when(msrc.next()).thenReturn(e1).thenReturn(e2).thenReturn(e3).thenReturn(
        null);


    DiskFailoverDeco snk = new DiskFailoverDeco(msnk, LogicalNodeContext
        .testingContext(), FlumeNode.getInstance().getDFOManager(),
        new TimeTrigger(new ProcessTagger(), 60000), 300000);
    snk.open();
    try {
      EventUtil.dumpAll(msrc, snk);
      snk.close();
    } catch (IOException ioe) {

View Full Code Here

    // putting in large ridiculous constant
    NaiveFileFailoverManager wal = new NaiveFileFailoverManager(dir);
    wal.open();


    // get a sink write to it and then be done with it.
    Tagger t = new ProcessTagger();
    EventSink sink = wal.newWritingSink(t);
    sink.open();
    sink.append(new EventImpl("foo".getBytes()));
    assertEquals(1, wal.getWritingTags().size());

View Full Code Here

        Preconditions.checkArgument(dfonode != null,
            "Context does not have a logical node name");
        DiskFailoverManager dfoman = node.getAddDFOManager(dfonode);


        return new DiskFailoverDeco(null, context, dfoman, new TimeTrigger(
            new ProcessTagger(), delayMillis), checkmillis);
      }
    };
  }

View Full Code Here

  // References package exposed for testing
  final RollSink roller;


  CollectorSink(Context ctx, String snkSpec, long millis, AckListener ackDest)
      throws FlumeSpecException {
    this(ctx, snkSpec, millis, new ProcessTagger(), 250, ackDest);
  }

View Full Code Here

          context = new FormatContext(context, format);
        }


        try {
          EventSink snk = new CollectorSink(context, logdir, prefix, millis,
              new ProcessTagger(), 250, FlumeNode.getInstance()
                  .getCollectorAckListener());
          return snk;
        } catch (FlumeSpecException e) {
          LOG.error("CollectorSink spec error " + e, e);
          throw new IllegalArgumentException(

View Full Code Here

      TestAckListener fakeMasterRpc = new TestAckListener();


      // massive roll millis because the test will force fast and frequent rolls
      CollectorSink cs = new CollectorSink(LogicalNodeContext.testingContext(),
          "file:///" + dir.getAbsolutePath(), "test", 1000000,
          new ProcessTagger(), 250, fakeMasterRpc) {
        @Override
        public void append(Event e) throws IOException, InterruptedException {
          LOG.info("Pre  append: "
              + e.getAttrs().get(AckChecksumInjector.ATTR_ACK_HASH));
          super.append(e);

View Full Code Here

    // putting in large ridiculous constant
    NaiveFileFailoverManager wal = new NaiveFileFailoverManager(dir);
    wal.open();


    // get a sink write to it and then be done with it.
    Tagger t = new ProcessTagger();
    EventSink sink = wal.newWritingSink(t);
    sink.open();
    sink.append(new EventImpl("foo".getBytes()));
    assertEquals(1, wal.getWritingTags().size());

View Full Code Here

0 1 2 3 4

TOP

Related Classes of com.cloudera.flume.handlers.rolling.ProcessTagger

com.cloudera.flume.agent.diskfailover.DiskFailoverDeco

com.cloudera.flume.agent.diskfailover.TestDiskFailoverDeco

com.cloudera.flume.agent.diskfailover.TestDiskFailoverManager

com.cloudera.flume.agent.durability.TestNaiveFileWALDeco

com.cloudera.flume.agent.durability.TestNaiveFileWALManager

com.cloudera.flume.agent.durability.TestNaiveFileWALManagerConcurrently

com.cloudera.flume.agent.TestEndToEndAckFailureRecovery

com.cloudera.flume.collector.CollectorSink

com.cloudera.flume.collector.TestCollectorSink

java.util.regex.Matcher

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.