Examples of PlanOptimizer

edu.brown.optimizer.PlanOptimizer
@author pavlo @author sw47
org.apache.pig.experimental.plan.optimizer.PlanOptimizer
The core class of the optimizer. The basic design of this class is that it is provided a list of RuleSets. RuleSets represent all of the optimizer rules that can be run together. The rules in the RuleSet will be run repeatedly until either no rule in the RuleSet passes check and calls transform or until maxIter iterations (default 500) has been made over the RuleSet. Then the next RuleSet will be moved to. Once finished, a given RuleSet is never returned to. Each rule is has two parts: a pattern and and associated transformer. Transformers have two important functions: check(), and transform(). The pattern describes a pattern of node types that the optimizer will look to match. If that match is found anywhere in the plan, then check() will be called. check() allows the rule to look more in depth at the matched pattern and decide whether the rule should be run or not. For example, one might design a rule to push filters above join that would look for the pattern filter(join) (meaning a filter followed by a join). But only certain types of filters can be pushed. The check() function would need to decide whether the filter that it found was pushable or not. If check() returns true, the rule is said to have matched, and transform() is then called. This function is responsible for making changes in the logical plan. Once transform is complete PlanPatcher.patchUp will be called to do any necessary cleanup in the plan, such as resetting schemas, etc.
org.apache.pig.newplan.optimizer.PlanOptimizer
The core class of the optimizer. The basic design of this class is that it is provided a list of RuleSets. RuleSets represent all of the optimizer rules that can be run together. The rules in the RuleSet will be run repeatedly until either no rule in the RuleSet passes check and calls transform or until maxIter iterations (default 500) has been made over the RuleSet. Then the next RuleSet will be moved to. Once finished, a given RuleSet is never returned to. Each rule is has two parts: a pattern and and associated transformer. Transformers have two important functions: check(), and transform(). The pattern describes a pattern of node types that the optimizer will look to match. If that match is found anywhere in the plan, then check() will be called. check() allows the rule to look more in depth at the matched pattern and decide whether the rule should be run or not. For example, one might design a rule to push filters above join that would look for the pattern filter(join) (meaning a filter followed by a join). But only certain types of filters can be pushed. The check() function would need to decide whether the filter that it found was pushable or not. If check() returns true, the rule is said to have matched, and transform() is then called. This function is responsible for making changes in the logical plan. Once transform is complete PlanPatcher.patchUp will be called to do any necessary cleanup in the plan, such as resetting schemas, etc.

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        }
    }    


    private LogicalPlan migrateAndOptimizePlan(org.apache.pig.impl.logicalLayer.LogicalPlan plan) throws IOException {
        LogicalPlan newLogicalPlan = migratePlan( plan );
        PlanOptimizer optimizer = new MyPlanOptimizer( newLogicalPlan, 3 );
        optimizer.optimize();
        return newLogicalPlan;
    }

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        lpt.buildPlan("b = filter (load 'd.txt' as (a0, a1)) by (a0 is not null or a1 is not null) and IsEmpty(a0);"); 


        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (a0, a1)) by (a0 is not null or a1 is not null) and IsEmpty(a0);"); 
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        LogicalPlanTester lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id > 5);");
        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 2: simple or implication
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id > 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 3: constant expression eval
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3+4*2);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 11);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 4: simple NOT 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(NOT(NOT(id > 3)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 3);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 5: redundant NOT 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(NOT(id > 3));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 6: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (v1 is null);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (v1 is null);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 7: is not null
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(v1 is null);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (v1 is not null);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 8: combo I
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT((id > 1) OR ((v1 is null) AND (id > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 9: combo II: lhs <-> rhs
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(((id > 5) AND (v1 is null)) OR (id > 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 10: complementary OR
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id >= 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = load 'd.txt' as (id:int, v1, v2);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 11: OR Equality elimination 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id < 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id < 1);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 12: AND Equality elimination 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id < 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id < 1);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 13: negative case 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (v1 is null));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (v1 is NULL));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 14: combo III
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT((id > 1) OR ((v1 is null) AND (id > 1+2*2)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 15: combo III: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 < 3)) AND ((id > 4) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 < 3)) AND ((id > 4) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 15: combo III: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id > 4) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id > 4) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 16: conflicting OR
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id > 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id > 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 17: conflicting AND: negtive case for now
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id > 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id > 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 18: combo IV: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id < 8) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id < 8) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 19: negative AND
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id < 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id < 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 20: negative OR
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id < 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id < 5);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 20: combo V: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((id < 8) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((id < 8) OR (v1 > 5)));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 22: combo V: negative
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((v1 > 5) OR (id < 8)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((v1 > 5) OR (id < 8)));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 23: combo VI: extremely degenerate 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (((id > 5) OR (id > 6)) AND ((id > 7) OR (id > 8))));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 24: combo VII: extremely degenerate 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (id > 7));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 25: combo VII: extremely degenerate 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (((id > 5) AND (id > 7))));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 26: combo VIII: lhs<->rhs for case 25 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 7) AND (id > 5))) AND (((id > 4) OR (id > 3)) AND ((id > 2) OR (id > 1))));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 27: combo VII: rhs<->lhs for case 24 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 7) AND (((id > 4) OR (id > 3)) AND ((id > 2) OR (id > 1))));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 28: complex equality 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (id > 3)) AND ((id > 3) OR (id > 4)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 3);");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 29: complex equality 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (v1 > 3)) AND ((v1 > 3) OR (id > 4)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 4) OR (v1 > 3));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 30: complex equality 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (v1 > 3)) OR ((v1 > 3) OR (id > 4)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 4) OR (v1 > 3));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        LogicalPlanTester lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (name, age, gpa)) by age >= 50 or name > 'fred' and gpa <= 3.0 or name >= 'bob';");
        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (name, age, gpa)) by age >= 50 or name >= 'bob';");
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));
        
        // Regex filtering
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (name:chararray, age:int, registration, contributions:double)) by (name matches '^fred.*' and (chararray)registration matches '^dem.*');");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (name:chararray, age:int, registration, contributions:double)) by (name matches '^fred.*' and (chararray)registration matches '^dem.*');");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));
        
        // NOT Regex filtering 
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt') by (not $0 matches '^fred.*');");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt') by (not $0 matches '^fred.*');");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));   
        
        // naiive filtering
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt') by 1==1;");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = load 'd.txt';");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        LogicalPlanTester lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((v1 is not null) AND (id == 1) AND (1 == 1));");
        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((v1 is not null) AND (id == 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // boolean constant elimination: OR
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by (((v1 is not null) AND (id == 1)) OR (1 == 0));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((v1 is not null) AND (id == 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));
        
        // the mirror case of the above
        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((1 == 0) OR ((v1 is not null) AND (id == 1)));");
        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (id:int, v1, v2)) by ((v1 is not null) AND (id == 1));");
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        lpt.buildPlan("b = filter (load 'd.txt' as (a:chararray, b:long, c:map[], d:chararray, e:chararray)) by a == 'v' and b == 117L and c#'p1' == 'h' and c#'p2' == 'to' and ((d is not null and d != '') or (e is not null and e != ''));"); 


        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (a:chararray, b:long, c:map[], d:chararray, e:chararray)) by a == 'v' and b == 117L and c#'p1' == 'h' and c#'p2' == 'to' and ((d is not null and d != '') or (e is not null and e != ''));"); 
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));


        // mirror of the above
        lpt.buildPlan("b = filter (load 'd.txt' as (a:chararray, b:long, c:map[], d:chararray, e:chararray)) by ((d is not null and d != '') or (e is not null and e != '')) and a == 'v' and b == 117L and c#'p1' == 'h' and c#'p2' == 'to';"); 


        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (a:chararray, b:long, c:map[], d:chararray, e:chararray)) by ((d is not null and d != '') or (e is not null and e != '')) and a == 'v' and b == 117L and c#'p1' == 'h' and c#'p2' == 'to';"); 
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        lpt.buildPlan(query); 


        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan(query); 
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        lpt.buildPlan(query); 


        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan(query); 
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        lpt.buildPlan("b = filter (load 'd.txt' as (k1, k2, k3, v1, v2, v3)) by k2#'f1'#'f' is not null and (v2#'f'#'f1' is not null or v2#'f'#'f2' is not null);"); 


        org.apache.pig.impl.logicalLayer.LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan newLogicalPlan = migratePlan(plan);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (k1, k2, k3, v1, v2, v3)) by k2#'f1'#'f' is not null and (v2#'f'#'f1' is not null or v2#'f'#'f2' is not null);"); 
        plan = lpt.buildPlan("store b into 'empty';");
        LogicalPlan expected = migratePlan(plan);


        assertTrue(expected.isEqual(newLogicalPlan));
        
        lpt.buildPlan("b = filter (load 'd.txt' as (k1, k2, k3, v1, v2, v3)) by k2#'f1'#'f' is not null and (v2#'f1'#'f' is not null or v2#'f2'#'f' is not null);"); 


        plan = lpt.buildPlan("store b into 'empty';");
        newLogicalPlan = migratePlan(plan);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        lpt = new LogicalPlanTester(pc);
        lpt.buildPlan("b = filter (load 'd.txt' as (k1, k2, k3, v1, v2, v3)) by k2#'f1'#'f' is not null and (v2#'f1'#'f' is not null or v2#'f2'#'f' is not null);"); 
        plan = lpt.buildPlan("store b into 'empty';");
        expected = migratePlan(plan);

View Full Code Here

Examples of org.apache.pig.newplan.optimizer.PlanOptimizer

        // case 1: simple and implication
        String query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id > 5);" + 
                       "store b into 'empty';";
        LogicalPlan newLogicalPlan = Util.buildLp(pigServer, query);


        PlanOptimizer optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 5);" +
                "store b into 'empty';";
        LogicalPlan expected = Util.buildLp(pigServer, query);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 2: simple or implication
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id > 5);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 3: constant expression eval
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3+4*2);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 11);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 4: simple NOT 
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(NOT(NOT(id > 3)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 3);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 5: redundant NOT 
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(NOT(id > 3));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 6: negative
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (v1 is null);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (v1 is null);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 7: is not null
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(v1 is null);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (v1 is not null);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 8: combo I
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT((id > 1) OR ((v1 is null) AND (id > 5)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 9: combo II: lhs <-> rhs
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT(((id > 5) AND (v1 is null)) OR (id > 1));"+
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 10: complementary OR
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id >= 1));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = load 'd.txt' as (id:int, v1, v2);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 11: OR Equality elimination 
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id < 1));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id < 1);" +
        "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 12: AND Equality elimination 
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id < 1));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id < 1);" + "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 13: negative case 
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (v1 is null));" + "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (v1 is NULL));" + "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 14: combo III
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by NOT((id > 1) OR ((v1 is null) AND (id > 1+2*2)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id <= 1);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 15: combo III: negative
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 < 3)) AND ((id > 4) OR (v1 > 5)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 < 3)) AND ((id > 4) OR (v1 > 5)));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 15: combo III: negative
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id > 4) OR (v1 > 5)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id > 4) OR (v1 > 5)));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 16: conflicting OR
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id > 1));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) OR (id > 1));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 17: conflicting AND: negtive case for now
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id > 1));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by ((id < 1) AND (id > 1));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 18: combo IV: negative
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id < 8) OR (v1 > 5)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 5) OR (v1 > 3)) AND ((id < 8) OR (v1 > 5)));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 19: negative AND
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id < 5);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) AND (id < 5);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 20: negative OR
        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id < 5);" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1, v2)) by (id > 3) OR (id < 5);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 20: combo V: negative
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((id < 8) OR (v1 > 5)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((id < 8) OR (v1 > 5)));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 22: combo V: negative
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((v1 > 5) OR (id < 8)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((v1 > 3) OR (id > 5)) AND ((v1 > 5) OR (id < 8)));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 23: combo VI: extremely degenerate 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (((id > 5) OR (id > 6)) AND ((id > 7) OR (id > 8))));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 24: combo VII: extremely degenerate 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (id > 7));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 25: combo VII: extremely degenerate 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 1) OR (id > 2)) AND ((id > 3) OR (id > 4))) AND (((id > 5) AND (id > 7))));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 26: combo VIII: lhs<->rhs for case 25 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((((id > 7) AND (id > 5))) AND (((id > 4) OR (id > 3)) AND ((id > 2) OR (id > 1))));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 27: combo VII: rhs<->lhs for case 24 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 7) AND (((id > 4) OR (id > 3)) AND ((id > 2) OR (id > 1))));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 7);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 28: complex equality 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (id > 3)) AND ((id > 3) OR (id > 4)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (id > 3);" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 29: complex equality 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (v1 > 3)) AND ((v1 > 3) OR (id > 4)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 4) OR (v1 > 3));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));


        // case 30: complex equality 
        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by (((id > 4) OR (v1 > 3)) OR ((v1 > 3) OR (id > 4)));" +
                "store b into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        query = "b = filter (load 'd.txt' as (id:int, v1:int, v2)) by ((id > 4) OR (v1 > 3));" +
                "store b into 'empty';";
        expected = Util.buildLp(pigServer, query);;


        assertTrue(expected.isEqual(newLogicalPlan));
        
        // case 31: See PIG-2067
        query = "A = load 'a.dat' as (cookie);" +
                "B = load 'b.dat' as (cookie);" +
                "C = cogroup A by cookie, B by cookie;" +
                "E = filter C by COUNT(B)>0 AND COUNT(A)>0;" +
                "store E into 'empty';";
        newLogicalPlan = Util.buildLp(pigServer, query);;


        optimizer = new MyPlanOptimizer(newLogicalPlan, 10);
        optimizer.optimize();


        // Make sure in this case, we don't optimize
        query = "A = load 'a.dat' as (cookie);" +
            "B = load 'b.dat' as (cookie);" +
            "C = cogroup A by cookie, B by cookie;" +

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.