Examples of ExecutorShuffleInfo

org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo
Contains all configuration necessary for locating the shuffle files of an executor.

Examples of org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo

  /**
   * Creates an ExecutorShuffleInfo object based on the given shuffle manager which targets this
   * context's directories.
   */
  public ExecutorShuffleInfo createExecutorInfo(String shuffleManager) {
    return new ExecutorShuffleInfo(localDirs, subDirsPerLocalDir, shuffleManager);
  }

View Full Code Here

Examples of org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo


  @Test
  public void testRegisterExecutor() {
    RpcResponseCallback callback = mock(RpcResponseCallback.class);


    ExecutorShuffleInfo config = new ExecutorShuffleInfo(new String[] {"/a", "/b"}, 16, "sort");
    byte[] registerMessage = new RegisterExecutor("app0", "exec1", config).toByteArray();
    handler.receive(client, registerMessage, callback);
    verify(blockManager, times(1)).registerExecutor("app0", "exec1", config);


    verify(callback, times(1)).onSuccess((byte[]) any());

View Full Code Here

Examples of org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo

    ExternalShuffleClient client =
      new ExternalShuffleClient(conf, new TestSecretKeyHolder(appId, secretKey), true);
    client.init(appId);
    // Registration either succeeds or throws an exception.
    client.registerWithShuffleServer(TestUtils.getLocalHost(), server.getPort(), "exec0",
      new ExecutorShuffleInfo(new String[0], 0, ""));
    client.close();
  }

View Full Code Here

Examples of org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo

    }
    int shuffleId = Integer.parseInt(blockIdParts[1]);
    int mapId = Integer.parseInt(blockIdParts[2]);
    int reduceId = Integer.parseInt(blockIdParts[3]);


    ExecutorShuffleInfo executor = executors.get(new AppExecId(appId, execId));
    if (executor == null) {
      throw new RuntimeException(
        String.format("Executor is not registered (appId=%s, execId=%s)", appId, execId));
    }

View Full Code Here

Examples of org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo

    logger.info("Application {} removed, cleanupLocalDirs = {}", appId, cleanupLocalDirs);
    Iterator<Map.Entry<AppExecId, ExecutorShuffleInfo>> it = executors.entrySet().iterator();
    while (it.hasNext()) {
      Map.Entry<AppExecId, ExecutorShuffleInfo> entry = it.next();
      AppExecId fullId = entry.getKey();
      final ExecutorShuffleInfo executor = entry.getValue();


      // Only touch executors associated with the appId that was removed.
      if (appId.equals(fullId.appId)) {
        it.remove();

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.