List of preWarm() Examples

Examples of preWarm()

org.apache.tez.client.TezClient.preWarm()
API to help pre-allocate containers in session mode. In non-session mode this is ignored. The pre-allocated containers may be re-used by subsequent job DAGs to improve performance. The preWarm vertex should be configured and setup exactly like the other vertices in the job DAGs so that the pre-allocated containers may be re-used by the subsequent DAGs to improve performance. The processor for the preWarmVertex may be used to pre-warm the containers by pre-loading classes etc. It should be short-running so that pre-warming does not block real execution. Users can specify their custom processors or use the PreWarmProcessor from the runtime library. The parallelism of the preWarmVertex will determine the number of preWarmed containers. Pre-warming is best efforts and among other factors is limited by the free resources on the cluster. @param preWarmVertex @throws TezException @throws IOException
org.apache.tez.client.TezSession.preWarm()
Inform the Session to pre-warm containers for upcoming DAGs. Can be invoked multiple times on the same session. A subsequent call will release containers that are not compatible with the new context. This function can only be invoked if there is no DAG running on the Session This function can be a no-op if the Session already holds the required number of containers. @param context Context for the pre-warm containers.
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.preWarm()

Examples of org.apache.tez.client.TezClient.preWarm()

      // We are making the pre-warm and held containers to be the same and using the helper API to 
      // set up pre-warming. They can be made different and also custom initialization logic can be 
      // specified using other API's. We know that the OrderedWordCount dag uses default files and 
      // resources. Otherwise we would have to specify matching parameters in the preWarm API too.
      tezConf.setInt(TezConfiguration.TEZ_AM_SESSION_MIN_HELD_CONTAINERS, numPartitions);
      tezClient.preWarm(PreWarmVertex.createConfigBuilder(tezConf).build());
    }


    // the remaining code is the same as submitting any DAG.
    try {
      for (int i=0; i<inputPaths.length; ++i) {

View Full Code Here

Examples of org.apache.tez.client.TezClient.preWarm()

              .getVertex("initialmap").getTaskResource());
          preWarmVertex.addTaskLocalFiles(dag.getVertex("initialmap").getTaskLocalFiles());
          preWarmVertex.setTaskEnvironment(dag.getVertex("initialmap").getTaskEnvironment());
          preWarmVertex.setTaskLaunchCmdOpts(dag.getVertex("initialmap").getTaskLaunchCmdOpts());
          
          tezSession.preWarm(preWarmVertex);
        }


        if (useTezSession) {
          LOG.info("Waiting for TezSession to get into ready state");
          waitForTezSessionReady(tezSession);

View Full Code Here

Examples of org.apache.tez.client.TezSession.preWarm()

          context
            .setLocalResources(contextLocalRsrcs)
            .setJavaOpts(contextJavaOpts)
            .setEnvironment(contextEnv);


          tezSession.preWarm(context);
        }


        if (useTezSession) {
          LOG.info("Waiting for TezSession to get into ready state");
          waitForTezSessionReady(tezSession);

View Full Code Here

Examples of org.apache.tez.client.TezSession.preWarm()

          context
            .setLocalResources(contextLocalRsrcs)
            .setJavaOpts(contextJavaOpts)
            .setEnvironment(contextEnv);


          tezSession.preWarm(context);
        }


        if (useTezSession) {
          LOG.info("Waiting for TezSession to get into ready state");
          waitForTezSessionReady(tezSession);

View Full Code Here

Examples of org.apache.tez.client.TezSession.preWarm()

          context
            .setLocalResources(contextLocalRsrcs)
            .setJavaOpts(contextJavaOpts)
            .setEnvironment(contextEnv);


          tezSession.preWarm(context);
        }


        if (useTezSession) {
          LOG.info("Waiting for TezSession to get into ready state");
          waitForTezSessionReady(tezSession);

View Full Code Here

Examples of org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.preWarm()

      DAGClientAMProtocolRPC.PreWarmRequestProto.Builder
        preWarmReqProtoBuilder =
          DAGClientAMProtocolRPC.PreWarmRequestProto.newBuilder();
      preWarmReqProtoBuilder.setPreWarmContext(
        DagTypeConverters.convertPreWarmContextToProto(context));
      proxy.preWarm(null, preWarmReqProtoBuilder.build());
      while (true) {
        try {
          Thread.sleep(1000);
          TezSessionStatus status = getSessionStatus();
          if (status.equals(TezSessionStatus.READY)) {

View Full Code Here

Examples of org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.preWarm()

      DAGClientAMProtocolRPC.PreWarmRequestProto.Builder
        preWarmReqProtoBuilder =
          DAGClientAMProtocolRPC.PreWarmRequestProto.newBuilder();
      preWarmReqProtoBuilder.setPreWarmContext(
        DagTypeConverters.convertPreWarmContextToProto(context));
      proxy.preWarm(null, preWarmReqProtoBuilder.build());
      while (true) {
        try {
          Thread.sleep(1000);
          TezSessionStatus status = getSessionStatus();
          if (status.equals(TezSessionStatus.READY)) {

View Full Code Here

Examples of org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.preWarm()

      DAGClientAMProtocolRPC.PreWarmRequestProto.Builder
        preWarmReqProtoBuilder =
          DAGClientAMProtocolRPC.PreWarmRequestProto.newBuilder();
      preWarmReqProtoBuilder.setPreWarmContext(
        DagTypeConverters.convertPreWarmContextToProto(context));
      proxy.preWarm(null, preWarmReqProtoBuilder.build());
      while (true) {
        try {
          Thread.sleep(1000);
          TezSessionStatus status = getSessionStatus();
          if (status.equals(TezSessionStatus.READY)) {

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.