Examples of org.apache.flink.compiler.PactCompiler

org.apache.flink.compiler.PactCompiler
The optimizer that takes the user specified program plan and creates an optimized plan that contains exact descriptions about how the physical execution will take place. It first translates the user program into an internal optimizer representation and then chooses between different alternatives for shipping strategies and local strategies.
The basic principle is taken from optimizer works in systems such as Volcano/Cascades and Selinger/System-R/DB2. The optimizer walks from the sinks down, generating interesting properties, and ascends from the sources generating alternative plans, pruning against the interesting properties.
The optimizer also assigns the memory to the individual tasks. This is currently done in a very simple fashion: All sub-tasks that need memory (e.g. reduce or join) are given an equal share of memory.

    this.configuration = config;
    configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, jobManagerAddress.getAddress().getHostAddress());
    configuration.setInteger(ConfigConstants.JOB_MANAGER_IPC_PORT_KEY, jobManagerAddress.getPort());
    
    this.userCodeClassLoader = userCodeClassLoader;
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator());
  }

View Full Code Here

    if (port < 0) {
      throw new CompilerException("Cannot find port to job manager's RPC service in the global configuration.");
    }


    this.userCodeClassLoader = userCodeClassLoader;
    this.compiler = new PactCompiler(new DataStatistics(), new DefaultCostEstimator());
  }

View Full Code Here

    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(DOP);


    // optimize and compile plan 
    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);
    
    // return job graph of failing job
    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

    // generate plan
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(4);


    // optimize and compile plan
    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);


    // return job graph of working job
    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

    output.addInput(testMapper2);
    
    Plan plan = new Plan(output);
    plan.setDefaultParallelism(DOP);


    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);


    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

      return;
    }
  }


  private JobGraph getJobGraph(final Plan plan) throws Exception {
    final PactCompiler pc = new PactCompiler(new DataStatistics());
    final OptimizedPlan op = pc.compile(plan);
    final NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);
  }

View Full Code Here

    output.setInput(testReducer);
    testReducer.setInput(input);


    Plan plan = new Plan(output);


    PactCompiler pc = new PactCompiler(new DataStatistics());
    OptimizedPlan op = pc.compile(plan);


    NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
    return jgg.compileJobGraph(op);


  }

View Full Code Here

  // ------------------------------------------------------------------------  
  
  @Before
  public void setup() {
    this.dataStats = new DataStatistics();
    this.withStatsCompiler = new PactCompiler(this.dataStats, new DefaultCostEstimator());
    this.withStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
    
    this.noStatsCompiler = new PactCompiler(null, new DefaultCostEstimator());
    this.noStatsCompiler.setDefaultDegreeOfParallelism(DEFAULT_PARALLELISM);
  }

View Full Code Here

    
    
    private OptimizedPlan compileProgram(String jobName) {
      Plan p = createProgramPlan(jobName);
      
      PactCompiler pc = new PactCompiler(new DataStatistics());
      return pc.compile(p);
    }

View Full Code Here

        // we use the existing session
        shutDownAtEnd = false;
      }


      try {
        PactCompiler pc = new PactCompiler(new DataStatistics());
        OptimizedPlan op = pc.compile(plan);
        
        NepheleJobGraphGenerator jgg = new NepheleJobGraphGenerator();
        JobGraph jobGraph = jgg.compileJobGraph(op);
        
        JobClient jobClient = this.nephele.getJobClient(jobGraph);

View Full Code Here

0 1

TOP

Related Classes of org.apache.flink.compiler.PactCompiler

org.apache.flink.client.LocalExecutor

org.apache.flink.client.program.Client

org.apache.flink.compiler.costs.DefaultCostEstimator

org.apache.flink.compiler.dag.BinaryUnionNode

org.apache.flink.compiler.dag.BulkIterationNode

org.apache.flink.compiler.dag.BulkPartialSolutionNode

org.apache.flink.compiler.dag.CoGroupNode

org.apache.flink.compiler.dag.CollectorMapNode

org.apache.flink.compiler.dag.CrossNode

org.apache.flink.compiler.dag.DataSinkNode

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.