* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements.  See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership.  The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License.  You may obtain a copy of the License at

*

*     http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/

* The Corona Job Tracker (CJT) can work in one of three modes

* - In-process: In this mode, the CJT performs its entire functionality in

* the same process as the JobClient

* - Forwarding: In this case, the CJT just forwards the calls to a remote CJT.

* - Standalone: This is the remote CJT that is serving the calls from the

* forwarding CJT.

* The CoronaJobTracker (CJT) is responsible for running a single map-reduce

* job in Corona. It is similar to the classic Map-Reduce JobTracker (JT) class,

* except that it deals with only one job. Unlike the JT, the CJT does not

* track/manage the nodes that run the map/reduce tasks. The CJT gets all that

* functionality from the ClusterManager (CM). It communicates the resource

* needs of its job to the CM, and uses the resources provided by the CM to

* launch tasks.

* <p/>

* Cluster Resource Flow in CJT

* <p/>

* When the CJT starts, it obtains a session ID in the constructor.

* This session ID is used to derive the job ID and that does not change during

* the lifetime of the CJT. When the job is started through

* JobSubmissionProtocol#submitJob call, the resource flow is started. First the

* job is initialized through CoronaJobInProgress#initTasks(). Then

* CoronaJobTracker#startJob() does the work to create the initial set of

* resource requests to be sent to the CM. CoronaJobTracker#updateResources is

* responsible for actually sending the resource requests to the CM, and it is

* invoked periodically to update the CM with requested and released resources.

* <p/>

* Apart from the initial set of resource requests, the CJT may send additional

* resource requests. This is needed to run speculative task attempts and to

* re-run task attempts that have failed and need to be run on a different

* machine. In these cases, the machine that ran the original attempt is

* specified as an excluded host in the resource request.

* <p/>

* The process of releasing resources back to the CM is a little involved.

* The resources given by the CM to the CJT are for the CJT to use for as long

* as needed, except if the resource is revoked by the CM through

* SessionDriverService#Iface#revokeResource. So once a task is finished on the

* granted machine, the CJT is allowed to reuse the machine to run other tasks.

* The decision of reusing a resource vs not is done in

* CoronaJobTracker#processTaskResource, which does the following:

* - if the task succeeded: reuse the resource if possible, otherwise release it

* - if the task failed: get a new request for running the task, and mark the

*   resource as bad so that it can be excluded from future requests.

* <p/>

* When the job finishes, the resources active at that point are not explicitly

* returned the CM, instead, a session-end notification is sent to the CM which

* effectively releases the resources for the job. Also a job end notification

* is sent to the task trackers that ran tasks, so that they can clean up their

* state (see CoronaJobTracker#close)

*/

  implements JobSubmissionProtocol,

  SessionDriverService.Iface,

  InterTrackerProtocol,

  ResourceTracker.ResourceProcessor,

  TaskStateChangeListener,

  StateRestorer {

  /** Threshold on number of map tasks for automatically choosing remote mode

   *  for a job. If the number of map tasks in the job is more than this,

   *  start a remote mode tracker

   */

  public static final String STANDALONE_CJT_THRESHOLD_CONF =

    "mapred.coronajobtracker.remote.threshold";

  /** Default threshold for automatically choosing remote mode for a job. */

  public static final int STANDALONE_CJT_THRESHOLD_DEFAULT = 1000;

  /** Timeout for connecting to a task tracker. */

  public static final String TT_CONNECT_TIMEOUT_MSEC_KEY =

    "corona.tasktracker.connect.timeout.msec";

  /** RPC timeout for RPCs to a task tracker. */

  public static final String TT_RPC_TIMEOUT_MSEC_KEY =

    "corona.tasktracker.rpc.timeout.msec";

  /** Interval between heartbeats to the parent corona job tracker. */

  public static final String HEART_BEAT_INTERVAL_KEY =

    "corona.jobtracker.heartbeat.interval";

  /** Number of grants processed under the global lock at a time. */

  public static final String GRANTS_PROCESS_PER_ITERATION =

    "corona.jobtracker.resources.per.iteration";

  /** Limit on number of task completion events to hand out in a single RPC. */

  public static final String TASK_COMPLETION_EVENTS_PER_RPC =

    "corona.jobtracker.tce.per.rpc";

  /** Corona system directory. */

  public static final String SYSTEM_DIR_KEY = "corona.system.dir";

  /** Default corona system directory. */

  public static final String DEFAULT_SYSTEM_DIR = "/tmp/hadoop/mapred/system";

  /** Number of handlers used by the RPC server.*/

  public static final String RPC_SERVER_HANDLER_COUNT =

    "mapred.job.tracker.handler.count";

  /** The number of times that job can be retried after CJT failures. */

  public static final String MAX_JT_FAILURES_CONF =

    "mapred.coronajobtracker.restart.count";

  /** Default for the number of times that job can be restarted. */

  public static final int MAX_JT_FAILURES_DEFAULT = 0;

  /** Default time after which we expire tasks that we haven't heard from */

  public static final String TASK_EXPIRY_INTERVAL_CONF = "mapred.task.timeout";

  /** Default time after which we expire launching task */

  public static final String LAUNCH_EXPIRY_INTERVAL_CONF =

      "mapred.task.launch.timeout";

  /** Default task expire interval */

  public static final int DEFAULT_TASK_EXPIRY_INTERVAL = 10 * 60 * 1000;

  /**

   * The number of handlers used by the RPC server in

   * standalone mode. The standalone mode is used for large jobs, so should

   * use more threads.

   */

  public static final String RPC_SERVER_HANDLER_COUNT_STANDALONE =

     "mapred.coronajobtracker.remote.thread.standalone";

  /**

   * If a remote JT is running, stop the local RPC server after this timeout

   * past the completion of the job.

   */

  public static final String RPC_SERVER_STOP_TIMEOUT =

    "mapred.coronajobtracker.rpcserver.stop.timeout";

  /** Logger. */

  private static final Log LOG = LogFactory.getLog(CoronaJobTracker.class);

  static {

    Utilities.makeProcessExitOnUncaughtException(LOG);

  }

  /** Configuration submitted in constructor. */

  private JobConf conf;

  /** Filesystem. */

  private FileSystem fs;

  /** Running "standalone" (in the cluster). */

  private final boolean isStandalone;

  /** The remote job tracker. */

  private volatile RemoteJTProxy remoteJT;

  /** * Grants to process in an iteration. */

  private final int grantsPerIteration;

  /** Limit on number of task completion events to hand out in a single RPC. */

  private final int maxEventsPerRpc;

  /** Handles the session with the cluster manager. */

  private SessionDriver sessionDriver;

  /** Session ID. */

  private String sessionId;

  /** Session End Status. */

  private SessionStatus sessionEndStatus = null;

  /** Will always be 1. */

  private AtomicInteger jobCounter = new AtomicInteger();

  /** Identifier for the current job. */

  private JobID jobId;

  /** The job. */

  private CoronaJobInProgress job;

  /** The grants to revoke. */

  private List<ResourceGrant> grantsToRevoke = new ArrayList<ResourceGrant>();

  /** The dead nodes. */

  private List<String> deadNodes = new ArrayList<String>();

  /** Is the job tracker running? */

  private volatile boolean running = true;

  /** Has @link close() been called? */

  private volatile boolean closed = false;

  /** The thread to assign tasks. */

  private Thread assignTasksThread;

  /** The resource tracker. */

  private ResourceTracker resourceTracker;

  /** The RPC server address. */

  private InetSocketAddress jobTrackerAddress;

  /** The RPC server. */

  private volatile Server interTrackerServer;

  /** The HTTP server. */

  private HttpServer infoServer;

  /** The HTTP server port. */

  private int infoPort;

  /** The task lookup table */

  private TaskLookupTable taskLookupTable = new TaskLookupTable();

  /** Task tracker status map. */

  private Map<String, TaskTrackerStatus> taskTrackerStatus =

    new ConcurrentHashMap<String, TaskTrackerStatus>();

  /** Task tracker statistics. */

  private final TrackerStats trackerStats;

  /** Cache of RPC clients to task trackers. */

  private TrackerClientCache trackerClientCache;

  /** Cache of the nodes */

  private TopologyCache topologyCache;

  /** The resource updater. */

  private ResourceUpdater resourceUpdater = new ResourceUpdater();

  /** The resource updater thread. */

  private Thread resourceUpdaterThread;

  /** The global lock. */

  private final Object lockObject = new Object();

  /** Mutex for closing. */

  private final Object closeLock = new Object();

  /** The job history. */

  private CoronaJobHistory jobHistory;

  /** Interval between heartbeats to the parent. */

  private final int heartbeatInterval;

  /** Has a full-fledged tracker started. */

  private volatile boolean fullTrackerStarted = false;

  /** The task launcher. */

  private CoronaTaskLauncher taskLauncher;

  /** This provides information about the resource needs of each task (TIP). */

  private HashMap<TaskInProgress, TaskContext> taskToContextMap =

    new HashMap<TaskInProgress, TaskContext>();

  /** Maintains the inverse of taskToContextMap. */

  private HashMap<Integer, TaskInProgress> requestToTipMap =

    new HashMap<Integer, TaskInProgress>();

  /** Keeping track of the speculated Maps. */

  private HashSet<TaskInProgress> speculatedMaps =

    new HashSet<TaskInProgress>();

  /** Keeping track of the speculated Reduces. */

  private HashSet<TaskInProgress> speculatedReduces =

    new HashSet<TaskInProgress>();

  /** The task launch expiry logic. */

  private ExpireTasks expireTasks;

  /** Remote JT url to redirect to. */

  private String remoteJTUrl;

  /** Parent JT address */

  private final InetSocketAddress parentAddr;

  /** Submitter for saving status of remote JT */

  private Submitter localJTSubmitter;

  /** Fetches state from parent JT */

  private final Fetcher stateFetcher;

  /** RPC client to parent JT (if any), used by fetcher and submitter */

  InterCoronaJobTrackerProtocol parentClient;

  /** Task attempt of this remote JT or null if local JT */

  private final TaskAttemptID jtAttemptId;

  /** Determines which task attempts can be commited */

  private final CommitPermissionClient commitPermissionClient;

  /** Queue with statuses to process task commit actions for */

  private final BlockingQueue<TaskTrackerStatus> commitTaskActions =

      new LinkedBlockingQueue<TaskTrackerStatus>();

  /** Thread that asynchronously dispatches commit actions from queue */

  private Thread commitTasksThread;

 

  /** The proxy job tracker client. */

  private CoronaProxyJobTrackerService.Client pjtClient = null;

  /** The transport for the pjtClient. */

  private TTransport pjtTransport = null;

 

  private TaskAttemptID tid = null;

  private String ttHost = null;

  private String ttHttpPort = null;

 

  private ParentHeartbeat parentHeartbeat = null;

 

  // for failure injection

  private CoronaFailureEventInjector failureInjector = null;

  private volatile boolean jtEnd1 = false;

  private volatile boolean jtEnd1Pass = false;

  private volatile boolean jtEnd2 = false;

  private volatile boolean jtEnd2Pass = false;

 

  // Flag and cached status for handling the following situations:

  // The job client call getJobStatus and the status shows job has completed,

  // afterwards, the remote job tracker failed. Job client needs to know the

  // related job counters and perhaps status, so buffer it, otherwise, the

  // session driver has been closed, we had no way to do failover again

  private Boolean  isJobCompleted= new Boolean(false);

  private JobStatus cachedCompletedStatus = null;

  private Counters cachedCompletedCounters = null;

 

  // We should make sure the submitJob re-enterable

  // Failure will happen when submitJob to RJT

  // When doing failover, there may multi-submitJob retry hits

  // the same job tracker

  private Boolean isJobSubmitted = new Boolean(false);

 

  // Flag for close() to check if we need to purge jobs in each task tracker

  // When doing RJT failover, we need to keep the finish tasks result

  private volatile boolean isPurgingJob = true;

 

  private void initializePJTClient()

      throws IOException {

    InetSocketAddress address =

        NetUtils.createSocketAddr(new CoronaConf(conf).getProxyJobTrackerThriftAddress());

    pjtTransport = new TFramedTransport(

      new TSocket(address.getHostName(), address.getPort()));

    pjtClient =

      new CoronaProxyJobTrackerService.Client(new TBinaryProtocol(pjtTransport));

    try {

      pjtTransport.open();

    } catch (TException e) {

      LOG.info("Transport Exception: ", e);

    }

  }

  private void closePJTClient() {

    if (pjtTransport != null) {

      pjtTransport.close();

      pjtTransport = null;

    }

  }

  /**

   * Returns time after which task expires if we haven;t heard from it

   * @return timeout

   */

  public long getTaskExpiryInterval() {

    return this.job.getConf()