Examples of org.apache.pig.data.InternalSortedBag

org.apache.pig.data.InternalSortedBag
An ordered collection of Tuples (possibly) with multiples. Data is stored unsorted as it comes in, and only sorted when it is time to dump it to a file or when the first iterator is requested. Experementation found this to be the faster than storing it sorted to begin with. We allow a user defined comparator, but provide a default comparator in cases where the user doesn't specify one. This bag is not registered with SpillableMemoryManager. It calculates the number of tuples to hold in memory and spill pro-actively into files.

           bagType = PigMapReduce.sJobConf.get("pig.cachedbag.sort.type");             
           }          
            if (bagType != null && bagType.equalsIgnoreCase("default")) {                
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);           
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }
            
      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConf.get("pig.cachedbag.sort.type");             
           }          
            if (bagType != null && bagType.equalsIgnoreCase("default")) {                
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);           
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }
            
      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");             
           }          
            if (bagType != null && bagType.equalsIgnoreCase("default")) {                
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);           
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }
            
      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");
           }
            if (bagType != null && bagType.equalsIgnoreCase("default")) {
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }


            while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

                }
            }
      // by default, we create InternalSortedBag, unless user configures
            // explicitly to use old bag
            sortedBag = useDefaultBag ? BagFactory.getInstance().newSortedBag(mComparator)
                    : new InternalSortedBag(3, mComparator);


            while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");
          return res;

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");
           }
            if (bagType != null && bagType.equalsIgnoreCase("default")) {
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }


      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");
           }
            if (bagType != null && bagType.equalsIgnoreCase("default")) {
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }


      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");
           }
            if (bagType != null && bagType.equalsIgnoreCase("default")) {
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }


      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

           bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.sort.type");
           }
            if (bagType != null && bagType.equalsIgnoreCase("default")) {
              sortedBag = BagFactory.getInstance().newSortedBag(mComparator);
             } else {
            sortedBag = new InternalSortedBag(3, mComparator);
          }


      while (res.returnStatus != POStatus.STATUS_EOP) {
        if (res.returnStatus == POStatus.STATUS_ERR) {
          log.error("Error in reading from the inputs");

View Full Code Here

TOP

Related Classes of org.apache.pig.data.InternalSortedBag

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSort

java.io.File

java.io.DataOutputStream

java.io.FileInputStream

java.io.BufferedInputStream

java.io.DataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.