Examples of PactCompiler

eu.stratosphere.compiler.PactCompiler
The optimizer that takes the user specified program plan and creates an optimized plan that contains exact descriptions about how the physical execution will take place. It first translates the user program into an internal optimizer representation and then chooses between different alternatives for shipping strategies and local strategies.
The basic principle is taken from optimizer works in systems such as Volcano/Cascades and Selinger/System-R/DB2. The optimizer walks from the sinks down, generating interesting properties, and ascends from the sources generating alternative plans, pruning against the interesting properties.
The optimizer also assigns the memory to the individual tasks. This is currently done in a very simple fashion: All sub-tasks that need memory (e.g. reduce or join) are given an equal share of memory.
org.apache.flink.compiler.PactCompiler
The optimizer that takes the user specified program plan and creates an optimized plan that contains exact descriptions about how the physical execution will take place. It first translates the user program into an internal optimizer representation and then chooses between different alternatives for shipping strategies and local strategies.
The basic principle is taken from optimizer works in systems such as Volcano/Cascades and Selinger/System-R/DB2. The optimizer walks from the sinks down, generating interesting properties, and ascends from the sources generating alternative plans, pruning against the interesting properties.
The optimizer also assigns the memory to the individual tasks. This is currently done in a very simple fashion: All sub-tasks that need memory (e.g. reduce or join) are given an equal share of memory.

Examples of eu.stratosphere.compiler.PactCompiler

    if (port < 0) {
      throw new CompilerException("Cannot find port to job manager's RPC service in the global configuration.");
    }


    final InetSocketAddress jobManagerAddress = new InetSocketAddress(address, port);
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator(), jobManagerAddress);
    
    //  Disable Local Execution when using a Client
    ContextEnvironment.disableLocalExecution();
  }

View Full Code Here

Examples of eu.stratosphere.compiler.PactCompiler

        // we use the existing session
        shutDownAtEnd = false;
      }


      try {
        PactCompiler pc = new PactCompiler(new DataStatistics());
        OptimizedPlan op = pc.compile(plan);
        
        NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
        JobGraph jobGraph = jgg.compileJobGraph(op);
        
        JobClient jobClient = this.nephele.getJobClient(jobGraph);

View Full Code Here

Examples of eu.stratosphere.compiler.PactCompiler

        // we use the existing session
        shutDownAtEnd = false;
      }


      try {
        PactCompiler pc = new PactCompiler(new DataStatistics());
        OptimizedPlan op = pc.compile(plan);
        PlanJSONDumpGenerator gen = new PlanJSONDumpGenerator();
    
        return gen.getOptimizerPlanAsJSON(op);
      }
      finally {

View Full Code Here

Examples of eu.stratosphere.compiler.PactCompiler

   */
  public static String optimizerPlanAsJSON(Plan plan) throws Exception {
    LocalExecutor exec = new LocalExecutor();
    try {
      exec.start();
      PactCompiler pc = new PactCompiler(new DataStatistics());
      OptimizedPlan op = pc.compile(plan);
      PlanJSONDumpGenerator gen = new PlanJSONDumpGenerator();


      return gen.getOptimizerPlanAsJSON(op);
    } finally {
      exec.stop();

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

    this.configuration = config;
    configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, jobManagerAddress.getAddress().getHostAddress());
    configuration.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerAddress.getPort());
    
    this.userCodeClassLoader = userCodeClassLoader;
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator());
  }

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

    if (port < 0) {
      throw new CompilerException("Cannot find port to job manager's RPC service in the global configuration.");
    }


    this.userCodeClassLoader = userCodeClassLoader;
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator());
  }

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(DOP);


    // optimize and compile plan 
    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);
    
    // return job graph of failing job
    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(4);


    // optimize and compile plan
    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);


    // return job graph of working job
    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

    output.addInput(testMapper2);
    
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(DOP);


    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);


    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

Examples of org.apache.flink.compiler.PactCompiler

      return;
    }
  }


  private JobGraph getJobGraph(final Plan plan) throws Exception {
    final PactCompiler pc = new PactCompiler(new DataStatistics());
    final OptimizedPlan op = pc.compile(plan);
    final NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.