Examples of org.apache.giraph.worker.InputSplitPathOrganizer

org.apache.giraph.worker.InputSplitPathOrganizer
Utility class to extract the list of InputSplits from the ZooKeeper tree of "claimable splits" the master created, and to sort the list to favor local data blocks. This class provides an Iterator for the list the worker will claim splits from, making all sorting and data-code locality processing done here invisible to callers. The aim is to cut down on the number of ZK reads workers perform before locating an unclaimed InputSplit.

    when(zk.getChildrenExt(testListName, false, false, true)).
        thenReturn(testList);
    when(zk.getData("remote1", false, null)).thenReturn(remote1);
    when(zk.getData("remote2", false, null)).thenReturn(remote2);
    when(zk.getData("local", false, null)).thenReturn(local);
    InputSplitPathOrganizer lis =
      new InputSplitPathOrganizer(zk, testListName, localHost, true);
    final List<String> resultList = Lists.newArrayList(lis.getPathList());
    assertEquals("local", resultList.get(0));
  }

View Full Code Here

    when(zk.getChildrenExt(testListName, false, false, true)).
        thenReturn(testList);
    when(zk.getData("remote1", false, null)).thenReturn(remote1);
    when(zk.getData("remote2", false, null)).thenReturn(remote2);
    when(zk.getData("local", false, null)).thenReturn(local);
    InputSplitPathOrganizer lis =
      new InputSplitPathOrganizer(zk, testListName, localHost, true);
    final List<String> resultList = Lists.newArrayList(lis.getPathList());
    assertEquals("local", resultList.get(0));
  }

View Full Code Here

TOP

Related Classes of org.apache.giraph.worker.InputSplitPathOrganizer

org.apache.giraph.TestBspBasic

java.io.ByteArrayInputStream

java.io.DataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.