This is an implementation of {@code TaskService} that works on asingle node and across multiple nodes. It handles persisting tasks and keeping track of which tasks have not yet run to completion, so that in the event of a system failure the tasks can be run on re-start.
Durable tasks that have not yet run are persisted as instances of {@code PendingTask}, indexed by the owning identity. When a given identity is mapped to the local node, all tasks associated with that identity are started running on the local node. As long as an identity still has pending tasks scheduled to run locally, that identity is marked as active. To help minimize object creation, a cache of {@code PendingTask}s is created for each identity.
When an identity is moved from the local node to a new node, then all recurring tasks for that identity are cancelled, and all tasks for that identity are re-scheduled on the identity's new node. When an already-scheduled, persisted task tries to run on the old node, that task is dropped since it is already scheduled to run on the new node. After an identity has been moved, any subsequent attempts to schedule durable tasks on behalf of that identity on the old node will result in the tasks being scheduled to run on the new node. This is called task handoff.
Task handoff between nodes is done by noting the task in a node-specific entry in the data store. Each node will periodically query this entry to see if any tasks have been handed off. The default time in milliseconds for this period is {@code HANDOFF_PERIOD_DEFAULT}. The period may be changed using the property {@code HANDOFF_PERIOD_PROPERTY}. This checking will be delayed on node startup to give the system a chance to finish initializing. The default length of time in milliseconds for this delay is {@code HANDOFF_START_DEFAULT}. The delay may be changed using the property {@code HANDOFF_START_PROPERTY}.
When the final task for an identity completes, or an initial task for an identity is scheduled, the status of that identity as reported by this service changes. Rather than immediately reporting this status change, however, a delay is taken to see if the status is about to change back to its previous state. This helps avoid voting too frequently. The default time in milliseconds for delaying this vote is {@code VOTE_DELAY_DEFAULT}. The delay may be changed using the property {@code VOTE_DELAY_PROPERTY}.