Examples of org.goldenorb.io.InputSplitAllocator

org.goldenorb.io.InputSplitAllocator
This class provides a construct and methods to evenly distribute raw splits created by Hadoop/HDFS amongst active partitions in a Job.

  private void loadVerticesLeader() {
    enterBarrier("startLoadVerticesBarrier");
    
    synchronized (leaderGroupMembers) {
      // Here InputSplits are sent to their constituent partitions for loading
      InputSplitAllocator inputSplitAllocator = new InputSplitAllocator(getOrbConf(), leaderGroupMembers);
      Map<OrbPartitionMember,List<RawSplit>> inputSplitAssignments = inputSplitAllocator.assignInputSplits();
      for (OrbPartitionMember orbPartitionMember : inputSplitAssignments.keySet()) {
        for (RawSplit rawSplit : inputSplitAssignments.get(orbPartitionMember)) {
          orbPartitionMember.loadVerticesFromInputSplit(rawSplit);
        }
      }

View Full Code Here

    orbPartitionMembers.add(opm3);
    orbPartitionMembers.add(opm4);
    orbPartitionMembers.add(opm5);
    orbPartitionMembers.add(opm6);
    OrbConfiguration orbConf = new OrbConfiguration();
    InputSplitAllocator isa = new InputSplitAllocator(orbConf, orbPartitionMembers);
    Map<OrbPartitionMember, List<RawSplit>> assignedSplits = isa.assignInputSplits(rawSplits);
    
    for(OrbPartitionMember orbPartitionMember: assignedSplits.keySet()){
      LOG.info(orbPartitionMember.getHostname() + ":" + orbPartitionMember.getPort() + " | " + assignedSplits.get(orbPartitionMember));
      assertTrue(assignedSplits.get(orbPartitionMember).size() < 2);
    }

View Full Code Here

    orbPartitionMembers.add(opm3);
    orbPartitionMembers.add(opm4);
    orbPartitionMembers.add(opm5);
    orbPartitionMembers.add(opm6);
    
    InputSplitAllocator isa = new InputSplitAllocator(orbConf, orbPartitionMembers);
    Map<OrbPartitionMember,List<RawSplit>> inputSplitAssignments = isa.assignInputSplits();
    long totalFileSize = 0;
    for (OrbPartitionMember orbPartitionMember : inputSplitAssignments.keySet()) {
      long rawSplitSize = 0;
      for (RawSplit rSplit : inputSplitAssignments.get(orbPartitionMember)) {
        rawSplitSize += rSplit.getDataLength();

View Full Code Here

TOP

Related Classes of org.goldenorb.io.InputSplitAllocator

org.apache.hadoop.io.DataOutputBuffer

org.apache.hadoop.io.serializer.SerializationFactory

org.apache.hadoop.io.serializer.Serializer

org.apache.hadoop.mapred.JobConf

org.apache.hadoop.mapreduce.JobContext

org.apache.hadoop.mapreduce.JobID

org.goldenorb.io.input.RawSplit

org.goldenorb.io.TestInputSplitAllocator

org.goldenorb.io.TestInputSplitAllocatorDFS

org.goldenorb.OrbPartition

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.