Examples of com.linkedin.databus.core.Checkpoint

com.linkedin.databus.core.Checkpoint
This class represents the state of a consumer consuming events from a Databus server for a single timeline (physical partition). There are two main types of checkpoints:
- Online consumption - used for consuming events a relay
- Bootstrap - used for consuming events from a bootstrap server
The type of a checkpoint is determined by the consumptionMode parameter {@link #getConsumptionMode()}.
Online-Consumption checkpoints Online-consumption checkpoints {@link DbusClientMode#ONLINE_CONSUMPTION} represent the state ofa consumer of event stream from a relay. The main properties of the checkpoint are:
- {@code consumption_mode} - must be {@link DbusClientMode#ONLINE_CONSUMPTION}
- {@code windowScn} - the sequence number (SCN) of the current window; -1 denotes"flexible" checkpoint (see below). If the SCN is 0, and tsNescs is greater than 0 then the relay may (if capable) stream events that have timestamp greater than or equal to tsNsecs. However, the relay MUST ensure that it does not miss any events that have a timestamp greater than or equal to tsNsecs. TODO: Until we have this capability in the relays we don't have to define the exact behavior
- {@code prevScn} - the sequence number (SCN) of the window before current; -1 means"unknown".
- {@code windowOffset} - the number of events processed from the current window;-1 means the entire window has been processed including the end-of-window event and prevScn must be equal to windowScn.
- {@code tsNsecs} - optional value that is set to the timestamp of the EOP event in the window ofevents with the highest SCN that has been successfully consumed. If tsNsecs is greater than 0 then the value of windowScn must not be -1 (see discussion on flexible checkpoints below).
Flexible online-consumption checkpoints Used by consumers which do not care from where they start consuming. The relay will make the best effort to serve whatever data it has. Flexible checkpoints can be created using {@link #createFlexibleCheckpoint()} or invoking {@link #setFlexible()} on an existing Checkpoint.If a flexible checkpoint has tsNsecs set, the value of tsNsecs must be -1 (unset value).
Bootstrap checkpoints These are common fields for snapshot and catchup bootsrap checkpoints (see below).
- {@code bootstrap_since_scn} - the SCN of the last fully-consumed window or 0 for a full bootstrap. Must be >= 0.
- {@code bootstrap_start_scn} - the last SCN written to the snapshot when the snapshot started; must be setbefore any data is read, i.e. {@code bootstrap_snapshot_source_index} > 0 or{@code snapshot_offset} > 0 or {@code bootstrap_target_scn} != -1
- {@code bootstrap_target_scn} - the last SCN written to the log tables when the snapshot of the last sourcecompleted. It provides an upper bound of how much dirty data might have been read while snapshotting. It specifies the SCN up to which catch-up should be performed to guarantee consistency of the bootstrap results.
- {@code bootstrap_start_tsnsecs}- (optional) the timestamp of the EOP event of the highest window successfully processed by the client before the client fell off the relay. This value is optionally set by the bootstrap client before bootstrapping begins, and is never changed during the entire bootstrap sequence (snapshot and catchup phases).
Bootstrap snapshot checkpoints Bootstrap snapshot checkpoints ( {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}) represent a consumer in the SNAPSHOT phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}
- {@code bootstrap_snapshot_source_index} - the index of the current source being snapshotted or the indexof the next source if {@code bootstrap_snapshot_offet} == -1
- {@code snapshot_source} - the name of the current source being snapshotted or the name of the next sourceif {@code bootstrap_snapshot_offet} == -1
- {@code bootstrap_snapshot_offset} - number of rows successfully read for the current snapshot source; if -1,all rows have been read
- {@code bootstrap_snapshot_file_record_offset} - Applicable for V3 bootstrap only; refers to the offset of therecord within the AVRO block
- {@code storage_cluster_name} - Applicable for V3 bootstrap only; refers to name of espresso storage cluster
Bootstrap catchup checkpoints Bootstrap catchup checkpoints ( {@link DbusClientMode#BOOTSTRAP_CATCHUP}) represent a consumer in the CATCHUP phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_CATCHUP}
- {@code bootstrap_snapshot_source_index} - the index of the last snapshot source
- {@code bootstrap_snapshot_offset} - should always be -1
- {@code catchup_source} - the name of the current catch-up source or the name of the next sourceif {@code windowScn} == -1
- {@code windowScn} - the sequence number (SCN) of the current window being caught-up;
- {@code windowOffset} - the number of events processed from the current window;
@see CheckpointMult for multi-partition checkpoints

        CONN_FACTORY.createConnection(BOOTSTRAP_SERVER_INFO, callback, remoteExceptionHandler);
    try
    {
      log.info("initial setup");
      final List<String> sourceNamesList = Arrays.asList(SOURCE1_NAME);
      final Checkpoint cp = Checkpoint.createOnlineConsumptionCheckpoint(0);
      BootstrapCheckpointHandler cpHandler = new BootstrapCheckpointHandler(sourceNamesList);
      cpHandler.createInitialBootstrapCheckpoint(cp, 0L);
      final DummyDatabusBootstrapConnectionStateMessage bstCallback =
          new DummyDatabusBootstrapConnectionStateMessage(log);


      log.info("process a normal startSCN which should establish the connection");
      sendStartScnHappyPath(conn, cp, bstCallback, SOURCE1_NAME, 100L, log);
      Assert.assertTrue(conn.isConnected());


      //wait for the response
      TestUtil.assertWithBackoff(new ConditionCheck()
      {
        @Override
        public boolean check()
        {
          return null != bstCallback.getCheckpoint();
        }
      }, "wait for /startSCN response", 100, log);


      log.info("verify /startSCN response");
      final Checkpoint startScnCp = bstCallback.getCheckpoint();
      Assert.assertNotNull(startScnCp);
      Assert.assertEquals(100L, startScnCp.getBootstrapStartScn().longValue());


      log.info("instrument the client pipeline so that we can intercept and delay the channelClosed message");
      final Semaphore passMessage = new Semaphore(1);
      final CountDownLatch closeSent = new CountDownLatch(1);
      passMessage.acquire();

View Full Code Here

        if(_config.getPredicatePushDown())
          writer = createEventWriter(request, clientFreeBufferSize, null, enc);
        else
          writer = createEventWriter(request, clientFreeBufferSize, keyFilter, enc);


        Checkpoint cp = new Checkpoint(checkpointString);




      DbusClientMode consumptionMode = cp.getConsumptionMode();


      LOG.info("Bootstrap request received: " +
          "fetchSize=" + clientFreeBufferSize +
          ", consumptionMode=" + consumptionMode +
          ", checkpoint=" + checkpointString +
          ", predicatePushDown=" + _config.getPredicatePushDown()
          );


      try
      {
        boolean phaseCompleted = false;
        switch (consumptionMode)
        {
        case BOOTSTRAP_SNAPSHOT:
          phaseCompleted = processor.streamSnapShotRows(new Checkpoint(
              checkpointString), writer);
          break;
        case BOOTSTRAP_CATCHUP:
          phaseCompleted = processor.streamCatchupRows(new Checkpoint(
              checkpointString), writer);
          break;
        default:
          if (null != bootstrapStatsCollector)
            bootstrapStatsCollector.registerErrBootstrap();

View Full Code Here

        RequestProcessingException
    {
          initBootstrapServerInfo();


        String ckptStr = request.getParams().getProperty(CHECKPOINT_PARAM);
        Checkpoint ckpt = null;


        if ( null != ckptStr)
        {
          ckpt = new Checkpoint(ckptStr);


          String bsServerInfo = ckpt.getBootstrapServerInfo();


          if ((null != bsServerInfo) && (null != _serverInfo))
          {
            ServerInfo expServerInfo = null;
            try

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    HttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapStartScnHttpResponseProcessor  processor =
        new BootstrapStartScnHttpResponseProcessor(null, queue, stateMsg, cp, remoteExHandler,
                                                   null);
    ChannelBuffer buf = getScnResponse();
    HttpChunk httpChunk = new DefaultHttpChunk(buf);

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    HttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapStartScnHttpResponseProcessor  processor =
        new BootstrapStartScnHttpResponseProcessor(null, queue, stateMsg, cp, remoteExHandler,
                                                   null);
    ChannelBuffer buf = getScnResponse();
    HttpChunk httpChunk = new DefaultHttpChunk(buf);

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    HttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapStartScnHttpResponseProcessor  processor =
        new BootstrapStartScnHttpResponseProcessor(null, queue, stateMsg, cp, remoteExHandler,
                                                   null);
    ChannelBuffer buf = getScnResponse();
    HttpChunk httpChunk = new DefaultHttpChunk(buf);

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    HttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapStartScnHttpResponseProcessor  processor =
        new BootstrapStartScnHttpResponseProcessor(null, queue, stateMsg, cp, remoteExHandler,
                                                   null);
    ChannelBuffer buf = getScnResponse();
    HttpChunk httpChunk = new DefaultHttpChunk(buf);
    HttpChunkTrailer httpChunkTrailer = new DefaultHttpChunkTrailer();


    processor.startResponse(httpResponse);
    processor.addChunk(httpChunk);
    processor.addTrailer(httpChunkTrailer);
    processor.finishResponse();
    processor.channelException(new Exception("dummy exception"));


    Assert.assertEquals("Error Handled", true, processor._errorHandled);
    Assert.assertEquals("Processor Response State", AbstractHttpResponseProcessorDecorator.ResponseStatus.CHUNKS_EXCEPTION,processor._responseStatus);
    Assert.assertEquals("Actor Queue Size", 1, queue.getMessages().size());
    Assert.assertEquals("Expected ConnectionStateMessage","TestConnectionStateMessage", queue.getMessages().get(0).getClass().getSimpleName());
    TestConnectionStateMessage gotMsg = (TestConnectionStateMessage)(queue.getMessages().get(0));
    Assert.assertEquals("Expected ConnectionStateMessage State", TestConnectionStateMessage.State.STARTSCN_RESPONSE_SUCCESS, gotMsg._state);
    Assert.assertEquals("StartSCN Response Id Check",new Long(5678912),cp.getBootstrapStartScn());
  }

View Full Code Here

  {
    // Both Exception and ChannelClose getting triggered
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapStartScnHttpResponseProcessor  processor =
        new BootstrapStartScnHttpResponseProcessor(null, queue, stateMsg, cp, remoteExHandler,
                                                   null);
    processor.channelException(new Exception("dummy exception"));

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    DefaultHttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapTargetScnHttpResponseProcessor  processor =
        new BootstrapTargetScnHttpResponseProcessor(null, queue, stateMsg, cp,
                                                    remoteExHandler, null);
    ChannelBuffer buf = getScnResponse();
    HttpChunk httpChunk = new DefaultHttpChunk(buf);
    HttpChunkTrailer httpChunkTrailer = new DefaultHttpChunkTrailer();


    processor.startResponse(httpResponse);
    processor.addChunk(httpChunk);
    processor.addTrailer(httpChunkTrailer);
    processor.finishResponse();


    Assert.assertEquals("Error Handled", false, processor._errorHandled);
    Assert.assertEquals("Processor Response State", AbstractHttpResponseProcessorDecorator.ResponseStatus.CHUNKS_FINISHED,processor._responseStatus);
    Assert.assertEquals("Actor Queue Size", 1, queue.getMessages().size());
    Assert.assertEquals("Expected ConnectionStateMessage","TestConnectionStateMessage", queue.getMessages().get(0).getClass().getSimpleName());
    TestConnectionStateMessage gotMsg = (TestConnectionStateMessage)(queue.getMessages().get(0));
    Assert.assertEquals("Expected ConnectionStateMessage State", TestConnectionStateMessage.State.TARGETSCN_RESPONSE_SUCCESS, gotMsg._state);
    Assert.assertEquals("StartSCN Response Id Check",new Long(5678912),cp.getBootstrapTargetScn());
  }

View Full Code Here

  {
    TestAbstractQueue queue = new TestAbstractQueue();
    TestConnectionStateMessage stateMsg = new TestConnectionStateMessage();
    DefaultHttpResponse httpResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
    TestRemoteExceptionHandler remoteExHandler = new TestRemoteExceptionHandler();
    Checkpoint cp = new Checkpoint();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    BootstrapTargetScnHttpResponseProcessor  processor =
        new BootstrapTargetScnHttpResponseProcessor(null, queue, stateMsg, cp,
                                                    remoteExHandler, null);
    ChannelBuffer buf = getScnResponse();
    httpResponse.setContent(buf);
    httpResponse.setHeader(HttpHeaders.Names.CONTENT_LENGTH,"1000");
    processor.startResponse(httpResponse);
    processor.finishResponse();


    Assert.assertEquals("Error Handled", false, processor._errorHandled);
    Assert.assertEquals("Processor Response State", AbstractHttpResponseProcessorDecorator.ResponseStatus.CHUNKS_FINISHED,processor._responseStatus);
    Assert.assertEquals("Actor Queue Size", 1, queue.getMessages().size());
    Assert.assertEquals("Expected ConnectionStateMessage","TestConnectionStateMessage", queue.getMessages().get(0).getClass().getSimpleName());
    TestConnectionStateMessage gotMsg = (TestConnectionStateMessage)(queue.getMessages().get(0));
    System.out.println("Long Max is :" + Long.MAX_VALUE);
    Assert.assertEquals("Expected ConnectionStateMessage State", TestConnectionStateMessage.State.TARGETSCN_RESPONSE_SUCCESS, gotMsg._state);
    Assert.assertEquals("StartSCN Response Id Check",new Long(5678912),cp.getBootstrapTargetScn());
  }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of com.linkedin.databus.core.Checkpoint

com.linkedin.databus.bootstrap.producer.DatabusBootstrapProducer

com.linkedin.databus.bootstrap.server.BootstrapRequestProcessor

com.linkedin.databus.bootstrap.server.BootstrapRequestProcessorBase

com.linkedin.databus.bootstrap.server.StartSCNRequestProcessor

com.linkedin.databus.bootstrap.server.TestBootstrapProcessor

com.linkedin.databus.bootstrap.test.TestBootstrap

com.linkedin.databus.client.BootstrapDispatcher

com.linkedin.databus.client.BootstrapPullThread

com.linkedin.databus.client.DatabusSourcesConnection

com.linkedin.databus.client.GenericDispatcher

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.