Examples of eu.stratosphere.compiler.costs.Costs

eu.stratosphere.compiler.costs.Costs
Simple class to represent the costs of an operation. The costs are currently tracking, network, I/O and CPU costs. Costs are composed of two parts of cost contributors:
1. Quantifiable costs. Those costs are used when estimates are available and track a quantifiable measure, such as the number of bytes for network or I/O
2. Heuristic costs. Those costs are used when no estimates are available. They can be used to track that an operator used a special operation which is heuristically considered more expensive than another operation.
The quantifiable costs may frequently be unknown, which is represented by a {@code -1} as a value for the unknowncomponents of the cost. In that case, all operations' costs are unknown and hence it is not decidable which operation to favor during pruning. In that case, the heuristic costs should contain a value to make sure that operators with different strategies are comparable, even in the absence of estimates. The heuristic costs are hence the system's mechanism of realizing pruning heuristics that favor some operations over others.


  public Costs getCumulativeCostsShare() {
    if (this.cumulativeCosts == null){
      return null;
    } else {
      Costs result = cumulativeCosts.clone();
      if (this.template != null && this.template.getOutgoingConnections() != null) {
        int outDegree = this.template.getOutgoingConnections().size();
        if (outDegree > 0) {
          result.divideBy(outDegree);
        }
      }


      return result;
    }

View Full Code Here

    // the cumulative costs are the node costs plus the costs of all inputs
    this.cumulativeCosts = nodeCosts.clone();
    
    // add all the normal inputs
    for (Iterator<PlanNode> preds = getPredecessors(); preds.hasNext();) {
      Costs parentCosts = preds.next().getCumulativeCostsShare();
      if (parentCosts != null) {
        this.cumulativeCosts.addCosts(parentCosts);
      } else {
        throw new CompilerException("Trying to set the costs of an operator before the predecessor costs are computed.");
      }
    }
    
    // add all broadcast variable inputs
    if (this.broadcastInputs != null) {
      for (NamedChannel nc : this.broadcastInputs) {
        Costs bcInputCost = nc.getSource().getCumulativeCostsShare();
        if (bcInputCost != null) {
          this.cumulativeCosts.addCosts(bcInputCost);
        } else {
          throw new CompilerException("Trying to set the costs of an operator before the broadcast input costs are computed.");
        }

View Full Code Here

    // the plan enumeration logic works as for regular two-input-operators, which is important
    // because of the branch handling logic. it does pick redistributing network channels
    // between the sink and the sink joiner, because sinks joiner has a different DOP than the sink.
    // we discard any cost and simply use the sum of the costs from the two children.
    
    Costs totalCosts = getInput1().getSource().getCumulativeCosts().clone();
    totalCosts.addCosts(getInput2().getSource().getCumulativeCosts());
    super.setCosts(totalCosts);
  }

View Full Code Here

      GroupReduceNode combinerNode = ((GroupReduceNode) node).getCombinerUtilityNode();
      combinerNode.setDegreeOfParallelism(in.getSource().getDegreeOfParallelism());
      combinerNode.setSubtasksPerInstance(in.getSource().getSubtasksPerInstance());
      
      SingleInputPlanNode combiner = new SingleInputPlanNode(combinerNode, "Combine ("+node.getPactContract().getName()+")", toCombiner, DriverStrategy.ALL_GROUP_COMBINE);
      combiner.setCosts(new Costs(0, 0));
      combiner.initProperties(toCombiner.getGlobalProperties(), toCombiner.getLocalProperties());
      
      Channel toReducer = new Channel(combiner);
      toReducer.setShipStrategy(in.getShipStrategy(), in.getShipStrategyKeys(), in.getShipStrategySortOrder());
      toReducer.setLocalStrategy(in.getLocalStrategy(), in.getLocalStrategyKeys(), in.getLocalStrategySortOrder());

View Full Code Here

      ReduceNode combinerNode = ((ReduceNode) node).getCombinerUtilityNode();
      combinerNode.setDegreeOfParallelism(in.getSource().getDegreeOfParallelism());
      combinerNode.setSubtasksPerInstance(in.getSource().getSubtasksPerInstance());
      
      SingleInputPlanNode combiner = new SingleInputPlanNode(combinerNode, "Combine ("+node.getPactContract().getName()+")", toCombiner, DriverStrategy.ALL_REDUCE);
      combiner.setCosts(new Costs(0, 0));
      combiner.initProperties(toCombiner.getGlobalProperties(), toCombiner.getLocalProperties());
      
      Channel toReducer = new Channel(combiner);
      toReducer.setShipStrategy(in.getShipStrategy(), in.getShipStrategyKeys(), in.getShipStrategySortOrder());
      toReducer.setLocalStrategy(in.getLocalStrategy(), in.getLocalStrategyKeys(), in.getLocalStrategySortOrder());

View Full Code Here

    }
    
    SourcePlanNode candidate = new SourcePlanNode(this, "DataSource("+this.getPactContract().getName()+")");
    candidate.updatePropertiesWithUniqueSets(getUniqueFields());
    
    final Costs costs = new Costs();
    if (FileInputFormat.class.isAssignableFrom(getPactContract().getFormatWrapper().getUserCodeClass()) &&
        this.estimatedOutputSize >= 0)
    {
      estimator.addFileInputCost(this.estimatedOutputSize, costs);
    }

View Full Code Here

      GroupReduceNode combinerNode = ((GroupReduceNode) node).getCombinerUtilityNode();
      combinerNode.setDegreeOfParallelism(in.getSource().getDegreeOfParallelism());
      combinerNode.setSubtasksPerInstance(in.getSource().getSubtasksPerInstance());
      
      SingleInputPlanNode combiner = new SingleInputPlanNode(combinerNode, "Combine ("+node.getPactContract().getName()+")", toCombiner, DriverStrategy.SORTED_GROUP_COMBINE, this.keyList);
      combiner.setCosts(new Costs(0, 0));
      combiner.initProperties(toCombiner.getGlobalProperties(), toCombiner.getLocalProperties());
      
      Channel toReducer = new Channel(combiner);
      toReducer.setShipStrategy(in.getShipStrategy(), in.getShipStrategyKeys(), in.getShipStrategySortOrder());
      toReducer.setLocalStrategy(LocalStrategy.COMBININGSORT, in.getLocalStrategyKeys(), in.getLocalStrategySortOrder());

View Full Code Here

      ReduceNode combinerNode = ((ReduceNode) node).getCombinerUtilityNode();
      combinerNode.setDegreeOfParallelism(in.getSource().getDegreeOfParallelism());
      combinerNode.setSubtasksPerInstance(in.getSource().getSubtasksPerInstance());
      
      SingleInputPlanNode combiner = new SingleInputPlanNode(combinerNode, "Combine ("+node.getPactContract().getName()+")", toCombiner, DriverStrategy.SORTED_PARTIAL_REDUCE, this.keyList);
      combiner.setCosts(new Costs(0, 0));
      combiner.initProperties(toCombiner.getGlobalProperties(), toCombiner.getLocalProperties());
      
      Channel toReducer = new Channel(combiner);
      toReducer.setShipStrategy(in.getShipStrategy(), in.getShipStrategyKeys(), in.getShipStrategySortOrder());
      toReducer.setLocalStrategy(LocalStrategy.SORT, in.getLocalStrategyKeys(), in.getLocalStrategySortOrder());

View Full Code Here

TOP

Related Classes of eu.stratosphere.compiler.costs.Costs

eu.stratosphere.compiler.dag.DataSourceNode

eu.stratosphere.compiler.operators.AllGroupWithPartialPreGroupProperties

eu.stratosphere.compiler.operators.AllReduceProperties

eu.stratosphere.compiler.operators.GroupReduceWithCombineProperties

eu.stratosphere.compiler.operators.ReduceProperties

eu.stratosphere.compiler.plan.PlanNode

eu.stratosphere.compiler.plan.SinkJoinerPlanNode

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.