Examples of PreWarmVertex

org.apache.tez.dag.api.PreWarmVertex
A {@link PreWarmVertex} is used to specify parameters to be used to setupprewarmed containers for Tez session mode. Sessions allow re-use of execution slots (containers) across DAG's. Pre- warming allows pre-allocation of containers so that the first DAG has some execution resources already available to re-use. In order to get re-use containers they must be setup identically. So the prewarm vertex must be setup identically to the real DAG vertex (typically the first vertex to execute in the read DAG). Identical settings include same execution resources, same task local files etc. This works best in use cases where all DAGs share the same files/jars/resource settings from a common template
The parallelism of the pre-warm vertex determines the number of containers to be pre-warmed. This would ideally ensures a viable number of containers to provide performance while sharing resources with other applications. Typically the session would also hold onto the same number of containers in-between DAGs in session mode via the {@link TezConfiguration#TEZ_AM_SESSION_MIN_HELD_CONTAINERS} property. Theprewarm vertex by default runs the PreWarmProcessor from the Tez runtime library. This processor can be overridden to get the default behavior along with any app specific customizations. Alternatively, the application can provide any {@link Processor} to prewarm the containers. Pre-warmingprocessors can be used to initialize classes etc. and setup the environment for the actual processing to reduce latency.

Examples of org.apache.tez.dag.api.PreWarmVertex

    Map<String, LocalResource> lrs = Maps.newHashMap();
    final TezClientForTest client = new TezClientForTest("test", conf, lrs, null);


    configure(client);
    client.start();
    PreWarmVertex vertex = PreWarmVertex.create("PreWarm", 1, Resource.newInstance(1, 1));
    client.preWarm(vertex);
    
    ArgumentCaptor<SubmitDAGRequestProto> captor1 = ArgumentCaptor.forClass(SubmitDAGRequestProto.class);
    verify(client.sessionAmProxy, times(1)).submitDAG((RpcController)any(), (SubmitDAGRequestProto) any());
    SubmitDAGRequestProto proto = captor1.getValue();

View Full Code Here

Examples of org.apache.tez.dag.api.PreWarmVertex

            doPreWarm = false;
          }
        }
        if (doPreWarm) {
          LOG.info("Pre-warming Session");
          PreWarmVertex preWarmVertex = PreWarmVertex.create("PreWarm", preWarmNumContainers, dag
              .getVertex("initialmap").getTaskResource());
          preWarmVertex.addTaskLocalFiles(dag.getVertex("initialmap").getTaskLocalFiles());
          preWarmVertex.setTaskEnvironment(dag.getVertex("initialmap").getTaskEnvironment());
          preWarmVertex.setTaskLaunchCmdOpts(dag.getVertex("initialmap").getTaskLaunchCmdOpts());
          
          tezSession.preWarm(preWarmVertex);
        }


        if (useTezSession) {

View Full Code Here

Examples of org.apache.tez.dag.api.PreWarmVertex


    if (HiveConf.getBoolVar(conf, ConfVars.HIVE_PREWARM_ENABLED)) {
      int n = HiveConf.getIntVar(conf, ConfVars.HIVE_PREWARM_NUM_CONTAINERS);
      LOG.info("Prewarming " + n + " containers  (id: " + sessionId
          + ", scratch dir: " + tezScratchDir + ")");
      PreWarmVertex prewarmVertex = utils.createPreWarmVertex(tezConfig, n,
          commonLocalResources);
      try {
        session.preWarm(prewarmVertex);
      } catch (IOException ie) {
        if (ie.getMessage().contains("Interrupted while waiting")) {

View Full Code Here

Examples of org.apache.tez.dag.api.PreWarmVertex

      IOException, TezException {


    ProcessorDescriptor prewarmProcDescriptor = ProcessorDescriptor.create(HivePreWarmProcessor.class.getName());
    prewarmProcDescriptor.setUserPayload(TezUtils.createUserPayloadFromConf(conf));


    PreWarmVertex prewarmVertex = PreWarmVertex.create("prewarm", prewarmProcDescriptor, numContainers,getContainerResource(conf));


    Map<String, LocalResource> combinedResources = new HashMap<String, LocalResource>();


    if (localResources != null) {
      combinedResources.putAll(localResources);
    }


    prewarmVertex.addTaskLocalFiles(localResources);
    prewarmVertex.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
    prewarmVertex.setTaskEnvironment(getContainerEnvironment(conf, false));
    return prewarmVertex;
  }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.