Examples of com.linkedin.databus.core.Checkpoint

• com.linkedin.databus.core.Checkpoint
  This class represents the state of a consumer consuming events from a Databus server for a single timeline (physical partition). There are two main types of checkpoints:

  • Online consumption - used for consuming events a relay
  • Bootstrap - used for consuming events from a bootstrap server

  The type of a checkpoint is determined by the consumptionMode parameter {@link #getConsumptionMode()}.

  Online-Consumption checkpoints Online-consumption checkpoints {@link DbusClientMode#ONLINE_CONSUMPTION} represent the state ofa consumer of event stream from a relay. The main properties of the checkpoint are:

  • {@code consumption_mode} - must be {@link DbusClientMode#ONLINE_CONSUMPTION}
  • {@code windowScn} - the sequence number (SCN) of the current window; -1 denotes"flexible" checkpoint (see below). If the SCN is 0, and tsNescs is greater than 0 then the relay may (if capable) stream events that have timestamp greater than or equal to tsNsecs. However, the relay MUST ensure that it does not miss any events that have a timestamp greater than or equal to tsNsecs. TODO: Until we have this capability in the relays we don't have to define the exact behavior
  • {@code prevScn} - the sequence number (SCN) of the window before current; -1 means"unknown".
  • {@code windowOffset} - the number of events processed from the current window;-1 means the entire window has been processed including the end-of-window event and prevScn must be equal to windowScn.
  • {@code tsNsecs} - optional value that is set to the timestamp of the EOP event in the window ofevents with the highest SCN that has been successfully consumed. If tsNsecs is greater than 0 then the value of windowScn must not be -1 (see discussion on flexible checkpoints below).

  Flexible online-consumption checkpoints Used by consumers which do not care from where they start consuming. The relay will make the best effort to serve whatever data it has. Flexible checkpoints can be created using {@link #createFlexibleCheckpoint()} or invoking {@link #setFlexible()} on an existing Checkpoint.If a flexible checkpoint has tsNsecs set, the value of tsNsecs must be -1 (unset value).

  Bootstrap checkpoints These are common fields for snapshot and catchup bootsrap checkpoints (see below).

  • {@code bootstrap_since_scn} - the SCN of the last fully-consumed window or 0 for a full bootstrap. Must be >= 0.
  • {@code bootstrap_start_scn} - the last SCN written to the snapshot when the snapshot started; must be setbefore any data is read, i.e. {@code bootstrap_snapshot_source_index} > 0 or{@code snapshot_offset} > 0 or {@code bootstrap_target_scn} != -1
  • {@code bootstrap_target_scn} - the last SCN written to the log tables when the snapshot of the last sourcecompleted. It provides an upper bound of how much dirty data might have been read while snapshotting. It specifies the SCN up to which catch-up should be performed to guarantee consistency of the bootstrap results.
  • {@code bootstrap_start_tsnsecs}- (optional) the timestamp of the EOP event of the highest window successfully processed by the client before the client fell off the relay. This value is optionally set by the bootstrap client before bootstrapping begins, and is never changed during the entire bootstrap sequence (snapshot and catchup phases).

  Bootstrap snapshot checkpoints Bootstrap snapshot checkpoints ( {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}) represent a consumer in the SNAPSHOT phase of bootstrapping. The main properties of checkpoints are:

  • {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}
  • {@code bootstrap_snapshot_source_index} - the index of the current source being snapshotted or the indexof the next source if {@code bootstrap_snapshot_offet} == -1
  • {@code snapshot_source} - the name of the current source being snapshotted or the name of the next sourceif {@code bootstrap_snapshot_offet} == -1
  • {@code bootstrap_snapshot_offset} - number of rows successfully read for the current snapshot source; if -1,all rows have been read
  • {@code bootstrap_snapshot_file_record_offset} - Applicable for V3 bootstrap only; refers to the offset of therecord within the AVRO block
  • {@code storage_cluster_name} - Applicable for V3 bootstrap only; refers to name of espresso storage cluster

  Bootstrap catchup checkpoints Bootstrap catchup checkpoints ( {@link DbusClientMode#BOOTSTRAP_CATCHUP}) represent a consumer in the CATCHUP phase of bootstrapping. The main properties of checkpoints are:

  • {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_CATCHUP}
  • {@code bootstrap_snapshot_source_index} - the index of the last snapshot source
  • {@code bootstrap_snapshot_offset} - should always be -1
  • {@code catchup_source} - the name of the current catch-up source or the name of the next sourceif {@code windowScn} == -1
  • {@code windowScn} - the sequence number (SCN) of the current window being caught-up;
  • {@code windowOffset} - the number of events processed from the current window;

  @see CheckpointMult for multi-partition checkpoints

Related Classes of com.linkedin.databus.core.Checkpoint