Examples of com.linkedin.databus.core.Checkpoint

com.linkedin.databus.core.Checkpoint
This class represents the state of a consumer consuming events from a Databus server for a single timeline (physical partition). There are two main types of checkpoints:
- Online consumption - used for consuming events a relay
- Bootstrap - used for consuming events from a bootstrap server
The type of a checkpoint is determined by the consumptionMode parameter {@link #getConsumptionMode()}.
Online-Consumption checkpoints Online-consumption checkpoints {@link DbusClientMode#ONLINE_CONSUMPTION} represent the state ofa consumer of event stream from a relay. The main properties of the checkpoint are:
- {@code consumption_mode} - must be {@link DbusClientMode#ONLINE_CONSUMPTION}
- {@code windowScn} - the sequence number (SCN) of the current window; -1 denotes"flexible" checkpoint (see below). If the SCN is 0, and tsNescs is greater than 0 then the relay may (if capable) stream events that have timestamp greater than or equal to tsNsecs. However, the relay MUST ensure that it does not miss any events that have a timestamp greater than or equal to tsNsecs. TODO: Until we have this capability in the relays we don't have to define the exact behavior
- {@code prevScn} - the sequence number (SCN) of the window before current; -1 means"unknown".
- {@code windowOffset} - the number of events processed from the current window;-1 means the entire window has been processed including the end-of-window event and prevScn must be equal to windowScn.
- {@code tsNsecs} - optional value that is set to the timestamp of the EOP event in the window ofevents with the highest SCN that has been successfully consumed. If tsNsecs is greater than 0 then the value of windowScn must not be -1 (see discussion on flexible checkpoints below).
Flexible online-consumption checkpoints Used by consumers which do not care from where they start consuming. The relay will make the best effort to serve whatever data it has. Flexible checkpoints can be created using {@link #createFlexibleCheckpoint()} or invoking {@link #setFlexible()} on an existing Checkpoint.If a flexible checkpoint has tsNsecs set, the value of tsNsecs must be -1 (unset value).
Bootstrap checkpoints These are common fields for snapshot and catchup bootsrap checkpoints (see below).
- {@code bootstrap_since_scn} - the SCN of the last fully-consumed window or 0 for a full bootstrap. Must be >= 0.
- {@code bootstrap_start_scn} - the last SCN written to the snapshot when the snapshot started; must be setbefore any data is read, i.e. {@code bootstrap_snapshot_source_index} > 0 or{@code snapshot_offset} > 0 or {@code bootstrap_target_scn} != -1
- {@code bootstrap_target_scn} - the last SCN written to the log tables when the snapshot of the last sourcecompleted. It provides an upper bound of how much dirty data might have been read while snapshotting. It specifies the SCN up to which catch-up should be performed to guarantee consistency of the bootstrap results.
- {@code bootstrap_start_tsnsecs}- (optional) the timestamp of the EOP event of the highest window successfully processed by the client before the client fell off the relay. This value is optionally set by the bootstrap client before bootstrapping begins, and is never changed during the entire bootstrap sequence (snapshot and catchup phases).
Bootstrap snapshot checkpoints Bootstrap snapshot checkpoints ( {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}) represent a consumer in the SNAPSHOT phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}
- {@code bootstrap_snapshot_source_index} - the index of the current source being snapshotted or the indexof the next source if {@code bootstrap_snapshot_offet} == -1
- {@code snapshot_source} - the name of the current source being snapshotted or the name of the next sourceif {@code bootstrap_snapshot_offet} == -1
- {@code bootstrap_snapshot_offset} - number of rows successfully read for the current snapshot source; if -1,all rows have been read
- {@code bootstrap_snapshot_file_record_offset} - Applicable for V3 bootstrap only; refers to the offset of therecord within the AVRO block
- {@code storage_cluster_name} - Applicable for V3 bootstrap only; refers to name of espresso storage cluster
Bootstrap catchup checkpoints Bootstrap catchup checkpoints ( {@link DbusClientMode#BOOTSTRAP_CATCHUP}) represent a consumer in the CATCHUP phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_CATCHUP}
- {@code bootstrap_snapshot_source_index} - the index of the last snapshot source
- {@code bootstrap_snapshot_offset} - should always be -1
- {@code catchup_source} - the name of the current catch-up source or the name of the next sourceif {@code windowScn} == -1
- {@code windowScn} - the sequence number (SCN) of the current window being caught-up;
- {@code windowOffset} - the number of events processed from the current window;
@see CheckpointMult for multi-partition checkpoints

  @Test
  /** Test bootstrap transition: Request_Stream to Stream_request_success */
  public void testTransition_RequestStreamToStreamRequestSuccess() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, 12000, 1);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

  @Test
  /** Test bootstrap transition: Request_Stream to Stream_request_sent */
  public void testTransition_RequestStreamToStreamRequestSet() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, 12000, 1);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

  @Test
  /** Test bootstrap transition: Stream_Response_Success - Happy Path */
  public void testTransition_StreamResponseSuccessHappyPath() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, 12000, 1);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

  @Test
  /** Test bootstrap transition: Stream_Response_Success - readEvents returned 0 bytes */
  public void testTransition_StreamResponseSuccessEmptyResponse() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, 12000, 0);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

  /** Test bootstrap transition: Stream_Response_Success - readEvents threw Exception */
  public void testTransition_StreamResponseReadEventsException() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, true, false, null, 12000, 1, true);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

  }


  @Override
  public Checkpoint getLastPersistedCheckpoint()
  {
    Checkpoint cp =_checkpointPersistenceProvider.loadCheckpoint(_sources);
    return cp;
  }

View Full Code Here

                         sourcesResp);


    //send partial /stream
    callback.clearLastMsg();
    objCapture.clear();
    Checkpoint cp = new Checkpoint();
    cp.setFlexible();
    CheckpointMult cpm = new CheckpointMult();
    cpm.addCheckpoint(PhysicalPartition.ANY_PHYSICAL_PARTITION, cp);
    conn.requestStream("1", null, 1000, cpm, null, msg);

View Full Code Here

                         sourcesResp);


    //send partial /stream
    callback.clearLastMsg();
    objCapture.clear();
    Checkpoint cp = new Checkpoint();
    cp.setFlexible();
    CheckpointMult cpm = new CheckpointMult();
    cpm.addCheckpoint(PhysicalPartition.ANY_PHYSICAL_PARTITION, cp);
    conn.requestStream("1", null, 1000, cpm, null, msg);

View Full Code Here

                         sourcesResp);


    //send partial /stream
    callback.clearLastMsg();
    objCapture.clear();
    Checkpoint cp = new Checkpoint();
    cp.setFlexible();
    CheckpointMult cpm = new CheckpointMult();
    cpm.addCheckpoint(PhysicalPartition.ANY_PHYSICAL_PARTITION, cp);
    conn.requestStream("1", null, 1000, cpm, null, msg);

View Full Code Here

    callback.clearLastMsg();
    objCapture.clear();


    serverChannel.close();


    Checkpoint cp = new Checkpoint();
    cp.setFlexible();
    CheckpointMult cpm = new CheckpointMult();
    cpm.addCheckpoint(PhysicalPartition.ANY_PHYSICAL_PARTITION, cp);
    conn.requestStream("1", null, 1000, cpm, null, msg);


    waitForCallback(callback,

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of com.linkedin.databus.core.Checkpoint

com.linkedin.databus.bootstrap.producer.DatabusBootstrapProducer

com.linkedin.databus.bootstrap.server.BootstrapRequestProcessor

com.linkedin.databus.bootstrap.server.BootstrapRequestProcessorBase

com.linkedin.databus.bootstrap.server.StartSCNRequestProcessor

com.linkedin.databus.bootstrap.server.TestBootstrapProcessor

com.linkedin.databus.bootstrap.test.TestBootstrap

com.linkedin.databus.client.BootstrapDispatcher

com.linkedin.databus.client.BootstrapPullThread

com.linkedin.databus.client.DatabusSourcesConnection

com.linkedin.databus.client.GenericDispatcher

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.