Examples of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler

Package org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Examples of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
The compiler that compiles a given physical plan into a DAG of MapReduce operators which can then be converted into the JobControl structure. Is implemented as a visitor of the PhysicalPlan it is compiling. Currently supports all operators except the MR Sort operator Uses a predecessor based depth first traversal. To compile an operator, first compiles the predecessors into MapReduce Operators and tries to merge the current operator into one of them. The goal being to keep the number of MROpers to a minimum. It also merges multiple Map jobs, created by compiling the inputs individually, into a single job. Here a new map job is created and then the contents of the previous map plans are added. However, any other state that was in the previous map plans, should be manually moved over. So, if you are adding something new take care about this. Ex of this is in requestedParallelism Only in case of blocking operators and splits, a new MapReduce operator is started using a store-load combination to connect the two operators. Whenever this happens care is taken to add the MROper into the MRPlan and connect it appropriately.

      visitor.visit();
      return visitor.getPhysicalPlan();
    }
    
    public static MROperPlan buildMRPlan(PhysicalPlan pp, PigContext pc) throws Exception{
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();
        return comp.getMRPlan();  
    }

View Full Code Here

        visitor.visit();
        return visitor.getPhysicalPlan();
    }


    public static MROperPlan buildMRPlan(PhysicalPlan pp, PigContext pc) throws Exception{
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();
        return comp.getMRPlan();
    }

View Full Code Here

    }


    private void run(PhysicalPlan pp, String expectedFile) throws Exception {
        String compiledPlan, goldenPlan = null;
        int MAX_SIZE = 100000;
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();


        MROperPlan mrp = comp.getMRPlan();
        PlanPrinter ppp = new PlanPrinter(mrp);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ppp.print(baos);
        compiledPlan = baos.toString();

View Full Code Here

        visitor.visit();
        return visitor.getPhysicalPlan();
    }


    public static MROperPlan buildMRPlan(PhysicalPlan pp, PigContext pc) throws Exception{
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();
        comp.aggregateScalarsFiles();
        comp.connectSoftLink();
        return comp.getMRPlan();
    }

View Full Code Here

    }


    private void run(PhysicalPlan pp, String expectedFile) throws Exception {
        String compiledPlan, goldenPlan = null;
        int MAX_SIZE = 100000;
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();


        MROperPlan mrp = comp.getMRPlan();
        PlanPrinter ppp = new PlanPrinter(mrp);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ppp.print(baos);
        compiledPlan = baos.toString();

View Full Code Here

      visitor.visit();
      return visitor.getPhysicalPlan();
    }
    
    public static MROperPlan buildMRPlan(PhysicalPlan pp, PigContext pc) throws Exception{
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();
        return comp.getMRPlan();  
    }

View Full Code Here


    private MROperPlan getMROperPlan(LipstickPigServer lps) throws Exception {
        HExecutionEngine he = new HExecutionEngine(lps.getPigContext());
        PhysicalPlan pp = he.compile(getLogicalPlan(lps), null);


        MRCompiler mrc = new MRCompiler(pp, lps.getPigContext());
        mrc.compile();
        return mrc.getMRPlan();
    }

View Full Code Here

    }


    private void run(PhysicalPlan pp, String expectedFile) throws Exception {
        String compiledPlan, goldenPlan = null;
        int MAX_SIZE = 100000;
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();


        MROperPlan mrp = comp.getMRPlan();
        PlanPrinter ppp = new PlanPrinter(mrp);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ppp.print(baos);
        compiledPlan = baos.toString();

View Full Code Here

    }


    private void run(PhysicalPlan pp, String expectedFile) throws Exception {
        String compiledPlan, goldenPlan = null;
        int MAX_SIZE = 100000;
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();


        MROperPlan mrp = comp.getMRPlan();
        PlanPrinter ppp = new PlanPrinter(mrp);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ppp.print(baos);
        compiledPlan = baos.toString();

View Full Code Here

        "b = load '/tmp/input2';" +
        "c = join a by $0, b by $0 using 'merge';" +
        "store c into '/tmp/output1';";


        PhysicalPlan pp = Util.buildPp(pigServer, query);
        MRCompiler comp = new MRCompiler(pp, pc);
        comp.compile();
        MROperPlan mrp = comp.getMRPlan();
        assertTrue(mrp.size()==2);


        MapReduceOper mrOp0 = mrp.getRoots().get(0);
        assertTrue(mrOp0.mapPlan.size()==2);
        PhysicalOperator load0 = mrOp0.mapPlan.getRoots().get(0);

View Full Code Here

0 1 2

TOP

Related Classes of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler

com.netflix.lipstick.MRPlanCalculatorTest

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.Path

org.apache.hadoop.mapreduce.InputFormat

org.apache.hadoop.mapreduce.Job

org.apache.hadoop.mapreduce.JobContext

org.apache.pig.backend.executionengine.ExecutionEngine

org.apache.pig.backend.hadoop.executionengine.HExecutionEngine

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.plans.MROperPlan

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.