Examples of org.apache.tez.client.TezClient.preWarm()

org.apache.tez.client.TezClient.preWarm()

API to help pre-allocate containers in session mode. In non-session mode this is ignored. The pre-allocated containers may be re-used by subsequent job DAGs to improve performance. The preWarm vertex should be configured and setup exactly like the other vertices in the job DAGs so that the pre-allocated containers may be re-used by the subsequent DAGs to improve performance. The processor for the preWarmVertex may be used to pre-warm the containers by pre-loading classes etc. It should be short-running so that pre-warming does not block real execution. Users can specify their custom processors or use the PreWarmProcessor from the runtime library. The parallelism of the preWarmVertex will determine the number of preWarmed containers. Pre-warming is best efforts and among other factors is limited by the free resources on the cluster. @param preWarmVertex @throws TezException @throws IOException

      // We are making the pre-warm and held containers to be the same and using the helper API to 
      // set up pre-warming. They can be made different and also custom initialization logic can be 
      // specified using other API's. We know that the OrderedWordCount dag uses default files and 
      // resources. Otherwise we would have to specify matching parameters in the preWarm API too.
      tezConf.setInt(TezConfiguration.TEZ_AM_SESSION_MIN_HELD_CONTAINERS, numPartitions);
      tezClient.preWarm(PreWarmVertex.createConfigBuilder(tezConf).build());
    }


    // the remaining code is the same as submitting any DAG.
    try {
      for (int i=0; i<inputPaths.length; ++i) {

              .getVertex("initialmap").getTaskResource());
          preWarmVertex.addTaskLocalFiles(dag.getVertex("initialmap").getTaskLocalFiles());
          preWarmVertex.setTaskEnvironment(dag.getVertex("initialmap").getTaskEnvironment());
          preWarmVertex.setTaskLaunchCmdOpts(dag.getVertex("initialmap").getTaskLaunchCmdOpts());
          
          tezSession.preWarm(preWarmVertex);
        }


        if (useTezSession) {
          LOG.info("Waiting for TezSession to get into ready state");
          waitForTezSessionReady(tezSession);

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.