Examples of eu.stratosphere.compiler.PactCompiler

eu.stratosphere.compiler.PactCompiler
The optimizer that takes the user specified program plan and creates an optimized plan that contains exact descriptions about how the physical execution will take place. It first translates the user program into an internal optimizer representation and then chooses between different alternatives for shipping strategies and local strategies.
The basic principle is taken from optimizer works in systems such as Volcano/Cascades and Selinger/System-R/DB2. The optimizer walks from the sinks down, generating interesting properties, and ascends from the sources generating alternative plans, pruning against the interesting properties.
The optimizer also assigns the memory to the individual tasks. This is currently done in a very simple fashion: All sub-tasks that need memory (e.g. reduce or join) are given an equal share of memory.

    if (port < 0) {
      throw new CompilerException("Cannot find port to job manager's RPC service in the global configuration.");
    }


    final InetSocketAddress jobManagerAddress = new InetSocketAddress(address, port);
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator(), jobManagerAddress);
    
    //  Disable Local Execution when using a Client
    ContextEnvironment.disableLocalExecution();
  }

View Full Code Here

        // we use the existing session
        shutDownAtEnd = false;
      }


      try {
        PactCompiler pc = new PactCompiler(new DataStatistics());
        OptimizedPlan op = pc.compile(plan);
        
        NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
        JobGraph jobGraph = jgg.compileJobGraph(op);
        
        JobClient jobClient = this.nephele.getJobClient(jobGraph);

View Full Code Here

        // we use the existing session
        shutDownAtEnd = false;
      }


      try {
        PactCompiler pc = new PactCompiler(new DataStatistics());
        OptimizedPlan op = pc.compile(plan);
        PlanJSONDumpGenerator gen = new PlanJSONDumpGenerator();
    
        return gen.getOptimizerPlanAsJSON(op);
      }
      finally {

View Full Code Here

   */
  public static String optimizerPlanAsJSON(Plan plan) throws Exception {
    LocalExecutor exec = new LocalExecutor();
    try {
      exec.start();
      PactCompiler pc = new PactCompiler(new DataStatistics());
      OptimizedPlan op = pc.compile(plan);
      PlanJSONDumpGenerator gen = new PlanJSONDumpGenerator();


      return gen.getOptimizerPlanAsJSON(op);
    } finally {
      exec.stop();

View Full Code Here

0 1

TOP

Related Classes of eu.stratosphere.compiler.PactCompiler

eu.stratosphere.client.LocalExecutor

eu.stratosphere.client.program.Client

eu.stratosphere.compiler.costs.DefaultCostEstimator

eu.stratosphere.compiler.dag.BinaryUnionNode

eu.stratosphere.compiler.dag.BulkIterationNode

eu.stratosphere.compiler.dag.BulkPartialSolutionNode

eu.stratosphere.compiler.dag.CoGroupNode

eu.stratosphere.compiler.dag.CollectorMapNode

eu.stratosphere.compiler.dag.CrossNode

eu.stratosphere.compiler.dag.DataSinkNode

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.