Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator$OriginalLocation

Package org.apache.pig.backend.hadoop.executionengine.physicalLayer

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator$OriginalLocation

org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator
This is the base class for all operators. This supports a generic way of processing inputs which can be overridden by operators extending this class. The input model assumes that it can either be taken from an operator or can be attached directly to this operator. Also it is assumed that inputs to an operator are always in the form of a tuple. For this pipeline rework, we assume a pull based model, i.e, the root operator is going to call getNext with the appropriate type which initiates a cascade of getNext calls that unroll to create input for the root operator to work on. Any operator that extends the PhysicalOperator, supports a getNext with all the different types of parameter types. The concrete implementation should use the result type of its input operator to decide the type of getNext's parameter. This is done to avoid switch/case based on the type as much as possible. The default is assumed to return an erroneus Result corresponding to an unsupported operation on that type. So the operators need to implement only those types that are supported.

        LOLoad load = (LOLoad)ld;
        LogicalSchema ls = load.getSchema();
        
        PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
        
        PhysicalOperator pFE = phyPlan.getSuccessors( phyPlan.getRoots().get(0) ).get(0);
        assertEquals( POForEach.class, pFE.getClass() );
        POForEach pForEach = (POForEach)pFE;
        PhysicalPlan inputPln = pForEach.getInputPlans().get(0);
        
        assertEquals(1, ls.getField(0).uid);
        assertEquals(2, ls.getField(1).uid);

View Full Code Here

        LOLoad load = (LOLoad)ld;
        LogicalSchema ls = load.getSchema();
        
        PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
        
        PhysicalOperator pFE = phyPlan.getSuccessors( phyPlan.getRoots().get(0) ).get(0);
        assertEquals( POForEach.class, pFE.getClass() );
        POForEach pForEach = (POForEach)pFE;
        PhysicalPlan inputPln = pForEach.getInputPlans().get(0);
        
        assertEquals(1, ls.getField(0).uid);
        assertEquals(2, ls.getField(1).uid);

View Full Code Here

        List<Operator> successors = oPlan.getSuccessors(op);
        if (successors == null) {
            return;
        }
        for (Operator lo : successors) {
            PhysicalOperator from = logToPhyMap.get(lo);
            try {
                currentPlan.connect(from, exprOp);
            } catch (PlanException e) {
                int errCode = 2015;
                String msg = "Invalid physical operators in the physical plan" ;

View Full Code Here

        List<Operator> successors = oPlan.getSuccessors(op);
        if (successors == null) {
            return;
        }
        for (Operator lo : successors) {
            PhysicalOperator from = logToPhyMap.get(lo);
            try {
                currentPlan.connect(from, exprOp);
            } catch (PlanException e) {
                int errCode = 2015;
                String msg = "Invalid physical operators in the physical plan" ;

View Full Code Here

        List<Operator> successors = oPlan.getSuccessors(op);
        if (successors == null) {
            return;
        }
        for (Operator lo : successors) {
            PhysicalOperator from = logToPhyMap.get(lo);
            try {
                currentPlan.connect(from, exprOp);
            } catch (PlanException e) {
                int errCode = 2015;
                String msg = "Invalid physical operators in the physical plan" ;

View Full Code Here

    
    @SuppressWarnings("unchecked")
    @Override
    public void visit( UserFuncExpression op ) throws FrontendException {       
        Object f = PigContext.instantiateFuncFromSpec(op.getFuncSpec());
        PhysicalOperator p;
        if (f instanceof EvalFunc) {
            p = new POUserFunc(new OperatorKey(DEFAULT_SCOPE, nodeGen
                    .getNextNodeId(DEFAULT_SCOPE)), -1,
                    null, op.getFuncSpec(), (EvalFunc) f);
        } else {
            p = new POUserComparisonFunc(new OperatorKey(DEFAULT_SCOPE, nodeGen
                    .getNextNodeId(DEFAULT_SCOPE)), -1,
                    null, op.getFuncSpec(), (ComparisonFunc) f);
        }
        p.setResultType(op.getType());
        currentPlan.add(p);
        List<LogicalExpression> fromList = op.getArguments();
        if(fromList!=null){
            for (LogicalExpression inputOperator : fromList) {
                PhysicalOperator from = logToPhyMap.get(inputOperator);
                try {
                    currentPlan.connect(from, p);
                } catch (PlanException e) {
                    int errCode = 2015;
                    String msg = "Invalid physical operators in the physical plan" ;

View Full Code Here

        exprOp.setColumns((ArrayList<Integer>)op.getBagColumns());        
        exprOp.setStar(false);
        logToPhyMap.put(op, exprOp);
        currentPlan.add(exprOp);
        
        PhysicalOperator from = logToPhyMap.get( op.getReferredExpression() );
        
        if( from != null ) {
            currentPlan.connect(from, exprOp);
        }
    }

View Full Code Here

    
    public static FileSpec checkLeafIsStore(
            PhysicalPlan plan,
            PigContext pigContext) throws ExecException {
        try {
            PhysicalOperator leaf = plan.getLeaves().get(0);
            FileSpec spec = null;
            if(!(leaf instanceof POStore)){
                String scope = leaf.getOperatorKey().getScope();
                POStore str = new POStore(new OperatorKey(scope,
                    NodeIdGenerator.getGenerator().getNextNodeId(scope)));
                spec = new FileSpec(FileLocalizer.getTemporaryPath(
                    pigContext).toString(),
                    new FuncSpec(Utils.getTmpFileCompressorName(pigContext)));

View Full Code Here

    
    public static FileSpec checkLeafIsStore(
            PhysicalPlan plan,
            PigContext pigContext) throws ExecException {
        try {
            PhysicalOperator leaf = plan.getLeaves().get(0);
            FileSpec spec = null;
            if(!(leaf instanceof POStore)){
                String scope = leaf.getOperatorKey().getScope();
                POStore str = new POStore(new OperatorKey(scope,
                    NodeIdGenerator.getGenerator().getNextNodeId(scope)));
                spec = new FileSpec(FileLocalizer.getTemporaryPath(
                    pigContext).toString(),
                    new FuncSpec(Utils.getTmpFileCompressorName(pigContext)));

View Full Code Here

            assertEquals(LOCogroup.GROUPTYPE.MERGE, ((LOCogroup)lp.getLeaves().get(0)).getGroupType());


            PigContext pc = new PigContext(ExecType.MAPREDUCE,cluster.getProperties());
            pc.connect();
            PhysicalPlan phyP = Util.buildPhysicalPlan(lp, pc);
            PhysicalOperator phyOp = phyP.getLeaves().get(0);
            assertTrue(phyOp instanceof POMergeCogroup);


            lp = lpt.buildPlan("store C into 'out';");
            MROperPlan mrPlan = Util.buildMRPlan(Util.buildPhysicalPlan(lp, pc),pc);            
            assertEquals(2,mrPlan.size());

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator$OriginalLocation

org.apache.pig.backend.executionengine.util.ExecTools

org.apache.pig.backend.hadoop.executionengine.HExecutionEngine

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer$AlgebraicPlanChecker

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer$DistinctPatcher

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.LimitAdjuster

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.