Examples of org.apache.pig.data.InternalDistinctBag

org.apache.pig.data.InternalDistinctBag
An unordered collection of Tuples with no multiples. Data is stored without duplicates as it comes in. When it is time to spill, that data is sorted and written to disk. The data is stored in a HashSet. When it is time to sort it is placed in an ArrayList and then sorted. Dispite all these machinations, this was found to be faster than storing it in a TreeSet. This bag spills pro-actively when the number of tuples in memory reaches a limit

                         useDefaultBag = true;
                     }
                 }
             }
             distinctBag = useDefaultBag ? BagFactory.getInstance().newDistinctBag()
                     : new InternalDistinctBag(3);


            Result in = processInput();
            while (in.returnStatus != POStatus.STATUS_EOP) {
                if (in.returnStatus == POStatus.STATUS_ERR) {
                    log.error("Error in reading from inputs");

View Full Code Here

             bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.distinct.type");             
             }            
            if (bagType != null && bagType.equalsIgnoreCase("default")) {                
              distinctBag = BagFactory.getInstance().newDistinctBag();          
             } else {
               distinctBag = new InternalDistinctBag(3);
          }
            
            while (in.returnStatus != POStatus.STATUS_EOP) {
                if (in.returnStatus == POStatus.STATUS_ERR) {
                    log.error("Error in reading from inputs");

View Full Code Here

         }
                      
      if (bagType != null && bagType.equalsIgnoreCase("default")) {                
          return BagFactory.getInstance().newDistinctBag();          
         } else {           
           return new InternalDistinctBag(3);
      }
    }

View Full Code Here

             bagType = PigMapReduce.sJobConfInternal.get().get("pig.cachedbag.distinct.type");             
             }            
            if (bagType != null && bagType.equalsIgnoreCase("default")) {                
              distinctBag = BagFactory.getInstance().newDistinctBag();          
             } else {
               distinctBag = new InternalDistinctBag(3);
          }
            
            while (in.returnStatus != POStatus.STATUS_EOP) {
                if (in.returnStatus == POStatus.STATUS_ERR) {
                    log.error("Error in reading from inputs");

View Full Code Here

0 1

TOP

Related Classes of org.apache.pig.data.InternalDistinctBag

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.PODistinct

org.apache.pig.builtin.Distinct

java.io.File

java.io.DataOutputStream

java.io.FileInputStream

java.io.BufferedInputStream

java.io.DataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.