Examples of Checkpoint

cascading.pipe.Checkpoint
The Checkpoint pipe, if supported by the current planner, will force data to be persisted at the point in the tuple stream an instance of Checkpoint is inserted into the pipe assembly.
If a checkpoint {@link cascading.tap.Tap} is added to the {@link cascading.flow.FlowDef} via the{@link cascading.flow.FlowDef#addCheckpoint(Checkpoint,cascading.tap.Tap)} method, that Tap instancewill be used to capture the intermediate result sets.
It is required that any Scheme used as a checkpoint must source {@link cascading.tuple.Fields#UNKNOWN} andsink {@link cascading.tuple.Fields#ALL}.
If used with a {@link cascading.scheme.hadoop.TextDelimited} {@link cascading.scheme.Scheme} class andthe {@code hasHeader} value is {@code true}, a header with the resolved field names will be written to the file.
This is especially useful for debugging complex flows.
For the {@link cascading.flow.hadoop.HadoopFlowConnector} and Hadoop platform, a Checkpoint will force a newMapReduce job ( {@link cascading.flow.hadoop.HadoopFlowStep} into the {@link cascading.flow.Flow} plan.
This can be important when used in conjunction with a {@link HashJoin} where all the operations upstreamfrom the HashJoin significantly filter out data allowing it to fit in memory.
com.linkedin.databus.core.Checkpoint
This class represents the state of a consumer consuming events from a Databus server for a single timeline (physical partition). There are two main types of checkpoints:
- Online consumption - used for consuming events a relay
- Bootstrap - used for consuming events from a bootstrap server
The type of a checkpoint is determined by the consumptionMode parameter {@link #getConsumptionMode()}.
Online-Consumption checkpoints Online-consumption checkpoints {@link DbusClientMode#ONLINE_CONSUMPTION} represent the state ofa consumer of event stream from a relay. The main properties of the checkpoint are:
- {@code consumption_mode} - must be {@link DbusClientMode#ONLINE_CONSUMPTION}
- {@code windowScn} - the sequence number (SCN) of the current window; -1 denotes"flexible" checkpoint (see below). If the SCN is 0, and tsNescs is greater than 0 then the relay may (if capable) stream events that have timestamp greater than or equal to tsNsecs. However, the relay MUST ensure that it does not miss any events that have a timestamp greater than or equal to tsNsecs. TODO: Until we have this capability in the relays we don't have to define the exact behavior
- {@code prevScn} - the sequence number (SCN) of the window before current; -1 means"unknown".
- {@code windowOffset} - the number of events processed from the current window;-1 means the entire window has been processed including the end-of-window event and prevScn must be equal to windowScn.
- {@code tsNsecs} - optional value that is set to the timestamp of the EOP event in the window ofevents with the highest SCN that has been successfully consumed. If tsNsecs is greater than 0 then the value of windowScn must not be -1 (see discussion on flexible checkpoints below).
Flexible online-consumption checkpoints Used by consumers which do not care from where they start consuming. The relay will make the best effort to serve whatever data it has. Flexible checkpoints can be created using {@link #createFlexibleCheckpoint()} or invoking {@link #setFlexible()} on an existing Checkpoint.If a flexible checkpoint has tsNsecs set, the value of tsNsecs must be -1 (unset value).
Bootstrap checkpoints These are common fields for snapshot and catchup bootsrap checkpoints (see below).
- {@code bootstrap_since_scn} - the SCN of the last fully-consumed window or 0 for a full bootstrap. Must be >= 0.
- {@code bootstrap_start_scn} - the last SCN written to the snapshot when the snapshot started; must be setbefore any data is read, i.e. {@code bootstrap_snapshot_source_index} > 0 or{@code snapshot_offset} > 0 or {@code bootstrap_target_scn} != -1
- {@code bootstrap_target_scn} - the last SCN written to the log tables when the snapshot of the last sourcecompleted. It provides an upper bound of how much dirty data might have been read while snapshotting. It specifies the SCN up to which catch-up should be performed to guarantee consistency of the bootstrap results.
- {@code bootstrap_start_tsnsecs}- (optional) the timestamp of the EOP event of the highest window successfully processed by the client before the client fell off the relay. This value is optionally set by the bootstrap client before bootstrapping begins, and is never changed during the entire bootstrap sequence (snapshot and catchup phases).
Bootstrap snapshot checkpoints Bootstrap snapshot checkpoints ( {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}) represent a consumer in the SNAPSHOT phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_SNAPSHOT}
- {@code bootstrap_snapshot_source_index} - the index of the current source being snapshotted or the indexof the next source if {@code bootstrap_snapshot_offet} == -1
- {@code snapshot_source} - the name of the current source being snapshotted or the name of the next sourceif {@code bootstrap_snapshot_offet} == -1
- {@code bootstrap_snapshot_offset} - number of rows successfully read for the current snapshot source; if -1,all rows have been read
- {@code bootstrap_snapshot_file_record_offset} - Applicable for V3 bootstrap only; refers to the offset of therecord within the AVRO block
- {@code storage_cluster_name} - Applicable for V3 bootstrap only; refers to name of espresso storage cluster
Bootstrap catchup checkpoints Bootstrap catchup checkpoints ( {@link DbusClientMode#BOOTSTRAP_CATCHUP}) represent a consumer in the CATCHUP phase of bootstrapping. The main properties of checkpoints are:
- {@code consumption_mode} - must be {@link DbusClientMode#BOOTSTRAP_CATCHUP}
- {@code bootstrap_snapshot_source_index} - the index of the last snapshot source
- {@code bootstrap_snapshot_offset} - should always be -1
- {@code catchup_source} - the name of the current catch-up source or the name of the next sourceif {@code windowScn} == -1
- {@code windowScn} - the sequence number (SCN) of the current window being caught-up;
- {@code windowOffset} - the number of events processed from the current window;
@see CheckpointMult for multi-partition checkpoints
com.persistit.CheckpointManager.Checkpoint
com.pointcliki.dizgruntled.logic.Checkpoint
A logic to display the level checkpoint flags @author Hugheth @since alpha 2.6
com.sun.grid.jgdi.configuration.Checkpoint
fr.imag.adele.apam.pax.test.performance.util.Checkpoint
org.agilewiki.jfile.transactions.db.Checkpoint
A checkpoint request is sent to a database to write all state to disk.
org.apache.jackrabbit.jcr2spi.operation.Checkpoint
Checkout...
org.archive.checkpointing.Checkpoint
Represents a single checkpoint, by its name and main store directory. @contributor gojomo
org.exist.storage.txn.Checkpoint
@author wolf
org.infinispan.test.fwk.CheckPoint
Behaves more or less like a map of {@link java.util.concurrent.Semaphore}s. One thread will wait for an event via {@code await(...)} or {@code awaitStrict(...)}, and one or more other threads will trigger the event via {@code trigger(...)} or {@code triggerForever(...)}. @author Dan Berindei @since 5.3
org.jpos.transaction.participant.CheckPoint
ptolemy.backtrack.Checkpoint
The class of checkpoint objects. A checkpoint object represents the smallest entity on which checkpoints are created and managed. It monitors one or more objects. When the {@link #rollback(long,boolean)} function is called, allthe monitored objects are rolled back to their previous states, defined by a timestamp. @author Thomas Feng @version $Id: Checkpoint.java,v 1.13.4.1 2008/03/25 22:11:40 cxh Exp $ @since Ptolemy II 5.1 @Pt.ProposedRating Red (tfeng) @Pt.AcceptedRating Red (tfeng)

Examples of com.linkedin.databus.core.Checkpoint

  protected boolean processSysEvent(DispatcherState curState, DbusEvent event)
  {
    boolean success = true;
    boolean debugEnabled = getLog().isDebugEnabled();


    Checkpoint ckptInEvent = null;
    int eventSrcId = event.getSourceId();


    if (event.isCheckpointMessage())
    {
      ByteBuffer eventValue = event.value();
      byte[] eventBytes = new byte[eventValue.limit()];
      eventValue.get(eventBytes);


      if (eventValue.limit() > 0)
      {
        try
        {
          String cpString = new String(eventBytes, "UTF-8");
          ckptInEvent = new Checkpoint(cpString);
          _lastCkpt = ckptInEvent;
          getLog().info("Bootstrap checkpoint received from the bootstrap server: " + ckptInEvent);


          _bootstrapMode = _lastCkpt.getConsumptionMode();


          curState.setEventsSeen(true);


          if (_bootstrapMode == DbusClientMode.ONLINE_CONSUMPTION)
          {
            getLog().info("Bootstrap is complete. Switching to relay consumption");
            Checkpoint restartCkpt = _lastCkpt.clone();
            _relayPuller.enqueueMessage(
                BootstrapResultMessage.createBootstrapCompleteMessage(restartCkpt));
          }
        }
        catch (RuntimeException e )

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, true,
                                  BootstrapDatabaseTooOldException.class.getName(), 12000, 1,
                                  false);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  public void testTransition_StreamResponseOtherException() throws Exception
  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, true, "Dummy Exception", 12000, 1,
                                  false);
    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  public void testTransition_StreamResponsePhaseCompleted() throws Exception
  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  public void testTransition_StreamResponsePhaseNotCompleted() throws Exception
  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, false);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    bsPuller.getMessageQueue().clear();


    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourceIdMap().put(1L, new IdNamePair(1L, "source1"));
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    bsPuller.getMessageQueue().clear();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setSnapshotOffset(10); // non -1 value
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_STREAM, null);
  }

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  public void testTransition_StreamResponseSnapshotCompleted() throws Exception
  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    bsPuller.getMessageQueue().clear();


    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourceIdMap().put(1L, new IdNamePair(1L, "source1"));
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    bsPuller.getMessageQueue().clear();
    cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setSnapshotOffset(-1);
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);
    Assert.assertTrue(cp.isSnapShotSourceCompleted(), "WindowSCN Check");
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT,"Consumption Mode check");
  }

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true,
                                  50L, 100L, "source1", "source2");


    Checkpoint cp = _ckptHandlerTwoSources.createInitialBootstrapCheckpoint(null, 0L);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourcesNameMap().put("source2", new IdNamePair(2L, "source2"));
    connState.getSourceIdMap().put(2L, new IdNamePair(2L, "source1"));
    connState.getSourceIdMap().put(2L, new IdNamePair(2L, "source1"));


    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    //set the startSCN
    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT);
    Assert.assertEquals(cp.getSnapshotSource(), "source1");
    Assert.assertFalse(cp.isSnapShotSourceCompleted());


    //start snapshot for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    //finish the snapshot phase for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);


    //set targetSCN after the snapshot phase for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_TARGET_SCN, StateId.TARGET_SCN_RESPONSE_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.TARGET_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_CATCHUP);
    Assert.assertFalse(cp.isCatchupSourceCompleted());
    Assert.assertEquals(cp.getWindowScn(), 50L, "WindowSCN Check");
    Assert.assertEquals(cp.getCatchupSource(), "source1", "Catchup Source check");


    //start catch-up for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    //finish the catch-up phase for source1
    cp.setWindowOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, cp);


    //start the snapshot phase for source2
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT);
    Assert.assertEquals(cp.getSnapshotSource(), "source2");
    Assert.assertFalse(cp.isSnapShotSourceCompleted());
    Assert.assertTrue(cp.isCatchupSourceCompleted());


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    //finish the snapshot phase for source2
    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    //set targetSCN after the snapshot phase for source2
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_TARGET_SCN, StateId.TARGET_SCN_RESPONSE_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.TARGET_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);


    //start catch-up for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_CATCHUP);
    Assert.assertFalse(cp.isCatchupSourceCompleted());
    Assert.assertEquals(cp.getWindowScn(), 50L, "WindowSCN Check");
    Assert.assertEquals(cp.getCatchupSource(), "source1", "Catchup Source check");


    cp.setWindowOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, cp);


    //finish the catch-up phase for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_STREAM, cp);


    //start catch-up for source2
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_CATCHUP);
    Assert.assertFalse(cp.isCatchupSourceCompleted());
    Assert.assertEquals(cp.getWindowScn(), 50L, "WindowSCN Check");
    Assert.assertEquals(cp.getCatchupSource(), "source2", "Catchup Source check");


    //finish the catch-up phase for source2
    cp.setWindowOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, cp);






    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_CATCHUP,
                        "Consumption Mode check");
  }

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  {
    BootstrapPullThread bsPuller =
        createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true,
                                  50L, 100L, "source1", "source2");


    Checkpoint cp = _ckptHandlerTwoSources.createInitialBootstrapCheckpoint(null, 0L);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourcesNameMap().put("source2", new IdNamePair(2L, "source2"));
    connState.getSourceIdMap().put(2L, new IdNamePair(2L, "source1"));
    connState.getSourceIdMap().put(2L, new IdNamePair(2L, "source1"));


    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    //set the startSCN
    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT);
    Assert.assertEquals(cp.getSnapshotSource(), "source1");
    Assert.assertFalse(cp.isSnapShotSourceCompleted());


    //start snapshot for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    //finish the snapshot phase for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);


    //set targetSCN after the snapshot phase for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_TARGET_SCN, StateId.TARGET_SCN_RESPONSE_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.TARGET_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_CATCHUP);
    Assert.assertFalse(cp.isCatchupSourceCompleted());
    Assert.assertEquals(cp.getWindowScn(), 50L, "WindowSCN Check");
    Assert.assertEquals(cp.getCatchupSource(), "source1", "Catchup Source check");


    //start catch-up for source1
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    //finish the catch-up phase for source1
    cp.setWindowOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, cp);


    //start the snapshot phase for source2
    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_STREAM, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT);
    Assert.assertEquals(cp.getSnapshotSource(), "source2");
    Assert.assertFalse(cp.isSnapShotSourceCompleted());
    Assert.assertTrue(cp.isCatchupSourceCompleted());


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    //finish the snapshot phase for source2
    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(Checkpoint.FULLY_CONSUMED_WINDOW_OFFSET);
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);
  }

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  /** Test bootstrap transition: Request_target_Scn to Target_Scn_Sent */
  public void testTransition_RequestTargetScnToTargetScnSent() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    bsPuller.getMessageQueue().clear();


    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourceIdMap().put(1L, new IdNamePair(1L, "source1"));
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(-1);
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT, "Consumption Mode check");
    Assert.assertFalse(cp.isBootstrapTargetScnSet());


    bsPuller.getMessageQueue().clear();
    MockBootstrapConnection mockConn = (MockBootstrapConnection) connState.getBootstrapConnection();
    mockConn.setMuteTransition(true);
    testTransitionCase(bsPuller, StateId.REQUEST_TARGET_SCN, StateId.TARGET_SCN_REQUEST_SENT, "", null);

View Full Code Here

Examples of com.linkedin.databus.core.Checkpoint

  /** Test bootstap transition: Request_target_Scn to Target_Scn_Success */
  public void testTransition_RequestTargetScnSuccess() throws Exception
  {
    BootstrapPullThread bsPuller = createBootstrapPullThread(false, false, false, false, false, null, 12000, 1, true);


    Checkpoint cp = _ckptHandlerSource1.createInitialBootstrapCheckpoint(null, 0L);
    //TODO remove
    //cp.setSnapshotSource("source1");
    //cp.setCatchupSource("source1");
    //cp.setConsumptionMode(DbusClientMode.BOOTSTRAP_SNAPSHOT);
    cp.setBootstrapServerInfo(_serverInfoName);


    bsPuller.getComponentStatus().start();
    ConnectionState connState = bsPuller.getConnectionState();
    connState.switchToBootstrap(cp);
    testTransitionCase(bsPuller, StateId.BOOTSTRAP, StateId.REQUEST_START_SCN, cp);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_START_SCN, StateId.START_SCN_RESPONSE_SUCCESS, null);
    bsPuller.getMessageQueue().clear();
    Map<Long, List<RegisterResponseEntry>> entries = new HashMap<Long, List<RegisterResponseEntry>>();


    entries.put(1L, new ArrayList<RegisterResponseEntry>());
    connState.setSourcesSchemas(entries);


    testTransitionCase(bsPuller, StateId.START_SCN_RESPONSE_SUCCESS, StateId.REQUEST_STREAM, null);
    bsPuller.getMessageQueue().clear();


    connState.getSourcesNameMap().put("source1", new IdNamePair(1L, "source1"));
    connState.getSourceIdMap().put(1L, new IdNamePair(1L, "source1"));
    testTransitionCase(bsPuller, StateId.REQUEST_STREAM, StateId.STREAM_REQUEST_SUCCESS, null);


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.STREAM_REQUEST_SUCCESS, StateId.STREAM_RESPONSE_DONE, null);


    bsPuller.getMessageQueue().clear();
    cp.setSnapshotOffset(-1);
    testTransitionCase(bsPuller, StateId.STREAM_RESPONSE_DONE, StateId.REQUEST_TARGET_SCN, null);
    Assert.assertEquals(cp.getConsumptionMode(), DbusClientMode.BOOTSTRAP_SNAPSHOT, "Consumption Mode check");
    Assert.assertFalse(cp.isBootstrapTargetScnSet());


    bsPuller.getMessageQueue().clear();
    testTransitionCase(bsPuller, StateId.REQUEST_TARGET_SCN, StateId.TARGET_SCN_RESPONSE_SUCCESS, null);
  }

View Full Code Here

0 1 2 3 4 5 6

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.