Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

Package org.apache.pig.backend.hadoop.executionengine.physicalLayer

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator
This is the base class for all operators. This supports a generic way of processing inputs which can be overridden by operators extending this class. The input model assumes that it can either be taken from an operator or can be attached directly to this operator. Also it is assumed that inputs to an operator are always in the form of a tuple. For this pipeline rework, we assume a pull based model, i.e, the root operator is going to call getNext with the appropriate type which initiates a cascade of getNext calls that unroll to create input for the root operator to work on. Any operator that extends the PhysicalOperator, supports a getNext with all the different types of parameter types. The concrete implementation should use the result type of its input operator to decide the type of getNext's parameter. This is done to avoid switch/case based on the type as much as possible. The default is assumed to return an erroneus Result corresponding to an unsupported operation on that type. So the operators need to implement only those types that are supported.

                }
                processedSet.clear();
            } 
            
            int idx = processedSet.nextClearBit(0);
            PhysicalOperator leaf = myPlans.get(idx).getLeaves().get(0);
            
            res = leaf.getNext(dummyTuple);
           
            if (res.returnStatus == POStatus.STATUS_EOP)  {
                processedSet.set(idx++);        
                if (idx < myPlans.size()) {
                    continue;

View Full Code Here

    public void setAccumulative() {
        super.setAccumulative();
        for(PhysicalPlan p : inputPlans) {            
            Iterator<PhysicalOperator> iter = p.iterator();
            while(iter.hasNext()) {
                PhysicalOperator po = iter.next();
                if (po instanceof ExpressionOperator || po instanceof PODistinct) {
                    po.setAccumulative();
                }
            }
        }
    }

View Full Code Here

    public void setAccumStart() {
        super.setAccumStart();
        for(PhysicalPlan p : inputPlans) {            
            Iterator<PhysicalOperator> iter = p.iterator();
            while(iter.hasNext()) {
                PhysicalOperator po = iter.next();
                if (po instanceof ExpressionOperator || po instanceof PODistinct) {
                    po.setAccumStart();
                }
            }
        }
    }

View Full Code Here

    public void setAccumEnd() {      
        super.setAccumEnd();
        for(PhysicalPlan p : inputPlans) {            
            Iterator<PhysicalOperator> iter = p.iterator();
            while(iter.hasNext()) {
                PhysicalOperator po = iter.next();
                if (po instanceof ExpressionOperator || po instanceof PODistinct) {            
                    po.setAccumEnd();
                }
            }
        }
    }

View Full Code Here

                isToBeFlattenedArray = new boolean[inputPlans.size()];
            }
            planLeafOps = new PhysicalOperator[inputPlans.size()];
            for(PhysicalPlan p : inputPlans) {
                ++i;
                PhysicalOperator leaf = p.getLeaves().get(0); 
                planLeafOps[i] = leaf;
                if(leaf instanceof POProject &&
                        leaf.getResultType() == DataType.TUPLE &&
                        ((POProject)leaf).isStar())
                    isToBeFlattenedArray[i] = true;
            }
        }
        // we are calculating plan leaves

View Full Code Here

        logToPhyMap.put(loLoad, load);


        // Load is typically a root operator, but in the multiquery
        // case it might have a store as a predecessor.
        List<Operator> op = loLoad.getPlan().getPredecessors(loLoad);
        PhysicalOperator from;
        
        if(op != null) {
            from = logToPhyMap.get(op.get(0));
            try {
                currentPlan.connect(from, load);

View Full Code Here

        logToPhyMap.put(loNative, poNative);
        currentPlan.add(poNative);
        
        List<Operator> op = loNative.getPlan().getPredecessors(loNative);


        PhysicalOperator from;
        if(op != null) {
            from = logToPhyMap.get(op.get(0));
        } else {
            int errCode = 2051;
            String msg = "Did not find a predecessor for Native." ;

View Full Code Here

        poFilter.setPlan(currentPlan);
        currentPlan = currentPlans.pop();


        List<Operator> op = filter.getPlan().getPredecessors(filter);


        PhysicalOperator from;
        if(op != null) {
            from = logToPhyMap.get(op.get(0));
        } else {
            int errCode = 2051;
            String msg = "Did not find a predecessor for Filter." ;

View Full Code Here

        poSort.setLimit(sort.getLimit());
        // sort.setRequestedParallelism(s.getType());
        logToPhyMap.put(sort, poSort);
        currentPlan.add(poSort);
        List<Operator> op = sort.getPlan().getPredecessors(sort); 
        PhysicalOperator from;
        
        if(op != null) {
            from = logToPhyMap.get(op.get(0));
        } else {
            int errCode = 2051;

View Full Code Here

        currentPlan.add(poStream);
        logToPhyMap.put(stream, poStream);
        
        List<Operator> op = stream.getPlan().getPredecessors(stream);


        PhysicalOperator from;
        if(op != null) {
            from = logToPhyMap.get(op.get(0));
        } else {                
            int errCode = 2051;
            String msg = "Did not find a predecessor for Stream." ;

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

org.apache.pig.backend.executionengine.util.ExecTools

org.apache.pig.backend.hadoop.executionengine.HExecutionEngine

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer$AlgebraicPlanChecker

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer$DistinctPatcher

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.LimitAdjuster

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.