Examples of Costs

eu.stratosphere.compiler.costs.Costs
Simple class to represent the costs of an operation. The costs are currently tracking, network, I/O and CPU costs. Costs are composed of two parts of cost contributors:
1. Quantifiable costs. Those costs are used when estimates are available and track a quantifiable measure, such as the number of bytes for network or I/O
2. Heuristic costs. Those costs are used when no estimates are available. They can be used to track that an operator used a special operation which is heuristically considered more expensive than another operation.
The quantifiable costs may frequently be unknown, which is represented by a {@code -1} as a value for the unknowncomponents of the cost. In that case, all operations' costs are unknown and hence it is not decidable which operation to favor during pruning. In that case, the heuristic costs should contain a value to make sure that operators with different strategies are comparable, even in the absence of estimates. The heuristic costs are hence the system's mechanism of realizing pruning heuristics that favor some operations over others.
mage.abilities.costs.Costs
org.apache.flink.compiler.costs.Costs
Simple class to represent the costs of an operation. The costs are currently tracking, network, I/O and CPU costs. Costs are composed of two parts of cost contributors:
1. Quantifiable costs. Those costs are used when estimates are available and track a quantifiable measure, such as the number of bytes for network or I/O
2. Heuristic costs. Those costs are used when no estimates are available. They can be used to track that an operator used a special operation which is heuristically considered more expensive than another operation.
The quantifiable costs may frequently be unknown, which is represented by a {@code -1} as a value for the unknowncomponents of the cost. In that case, all operations' costs are unknown and hence it is not decidable which operation to favor during pruning. In that case, the heuristic costs should contain a value to make sure that operators with different strategies are comparable, even in the absence of estimates. The heuristic costs are hence the system's mechanism of realizing pruning heuristics that favor some operations over others.

Examples of org.apache.flink.compiler.costs.Costs

    // the plan enumeration logic works as for regular two-input-operators, which is important
    // because of the branch handling logic. it does pick redistributing network channels
    // between the sink and the sink joiner, because sinks joiner has a different DOP than the sink.
    // we discard any cost and simply use the sum of the costs from the two children.
    
    Costs totalCosts = getInput1().getSource().getCumulativeCosts().clone();
    totalCosts.addCosts(getInput2().getSource().getCumulativeCosts());
    super.setCosts(totalCosts);
  }

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

      // create an input node for combine with same DOP as input node
      GroupReduceNode combinerNode = ((GroupReduceNode) node).getCombinerUtilityNode();
      combinerNode.setDegreeOfParallelism(in.getSource().getDegreeOfParallelism());


      SingleInputPlanNode combiner = new SingleInputPlanNode(combinerNode, "Combine ("+node.getPactContract().getName()+")", toCombiner, DriverStrategy.ALL_GROUP_COMBINE);
      combiner.setCosts(new Costs(0, 0));
      combiner.initProperties(toCombiner.getGlobalProperties(), toCombiner.getLocalProperties());
      
      Channel toReducer = new Channel(combiner);
      toReducer.setShipStrategy(in.getShipStrategy(), in.getShipStrategyKeys(), in.getShipStrategySortOrder());
      toReducer.setLocalStrategy(in.getLocalStrategy(), in.getLocalStrategyKeys(), in.getLocalStrategySortOrder());

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

    testShipStrategiesIsolated(BIG_ESTIMATES, 1);
    testShipStrategiesIsolated(BIG_ESTIMATES, 10);
  }
  
  private void testShipStrategiesIsolated(EstimateProvider estimates, int targetParallelism) {
    Costs random = new Costs();
    costEstimator.addRandomPartitioningCost(estimates, random);
    
    Costs hash = new Costs();
    costEstimator.addHashPartitioningCost(estimates, hash);
    
    Costs range = new Costs();
    costEstimator.addRangePartitionCost(estimates, range);
    
    Costs broadcast = new Costs();
    costEstimator.addBroadcastCost(estimates, targetParallelism, broadcast);
    
    int randomVsHash = random.compareTo(hash);
    int hashVsRange = hash.compareTo(range);
    int hashVsBroadcast = hash.compareTo(broadcast);

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

  
  // --------------------------------------------------------------------------------------------
  
  @Test
  public void testShipStrategyCombinationsPlain() {
    Costs hashBothSmall = new Costs();
    Costs hashSmallAndLarge = new Costs();
    Costs hashBothLarge = new Costs();
    
    Costs hashSmallBcLarge10 = new Costs();
    Costs hashLargeBcSmall10 = new Costs();
    
    Costs hashSmallBcLarge1000 = new Costs();
    Costs hashLargeBcSmall1000 = new Costs();
    
    Costs forwardSmallBcLarge10 = new Costs();
    Costs forwardLargeBcSmall10 = new Costs();
    
    Costs forwardSmallBcLarge1000 = new Costs();
    Costs forwardLargeBcSmall1000 = new Costs();
    
    costEstimator.addHashPartitioningCost(MEDIUM_ESTIMATES, hashBothSmall);
    costEstimator.addHashPartitioningCost(MEDIUM_ESTIMATES, hashBothSmall);
    
    costEstimator.addHashPartitioningCost(MEDIUM_ESTIMATES, hashSmallAndLarge);
    costEstimator.addHashPartitioningCost(BIG_ESTIMATES, hashSmallAndLarge);
    
    costEstimator.addHashPartitioningCost(BIG_ESTIMATES, hashBothLarge);
    costEstimator.addHashPartitioningCost(BIG_ESTIMATES, hashBothLarge);
    
    costEstimator.addHashPartitioningCost(MEDIUM_ESTIMATES, hashSmallBcLarge10);
    costEstimator.addBroadcastCost(BIG_ESTIMATES, 10, hashSmallBcLarge10);
    
    costEstimator.addHashPartitioningCost(BIG_ESTIMATES, hashLargeBcSmall10);
    costEstimator.addBroadcastCost(MEDIUM_ESTIMATES, 10, hashLargeBcSmall10);
    
    costEstimator.addHashPartitioningCost(MEDIUM_ESTIMATES, hashSmallBcLarge1000);
    costEstimator.addBroadcastCost(BIG_ESTIMATES, 1000, hashSmallBcLarge1000);
    
    costEstimator.addHashPartitioningCost(BIG_ESTIMATES, hashLargeBcSmall1000);
    costEstimator.addBroadcastCost(MEDIUM_ESTIMATES, 1000, hashLargeBcSmall1000);
    
    costEstimator.addBroadcastCost(BIG_ESTIMATES, 10, forwardSmallBcLarge10);
    
    costEstimator.addBroadcastCost(MEDIUM_ESTIMATES, 10, forwardLargeBcSmall10);
    
    costEstimator.addBroadcastCost(BIG_ESTIMATES, 1000, forwardSmallBcLarge1000);
    
    costEstimator.addBroadcastCost(MEDIUM_ESTIMATES, 1000, forwardLargeBcSmall1000);
    
    // hash cost is roughly monotonous
    assertTrue(hashBothSmall.compareTo(hashSmallAndLarge) < 0);
    assertTrue(hashSmallAndLarge.compareTo(hashBothLarge) < 0);
    
    // broadcast the smaller is better
    assertTrue(hashLargeBcSmall10.compareTo(hashSmallBcLarge10) < 0);
    assertTrue(forwardLargeBcSmall10.compareTo(forwardSmallBcLarge10) < 0);
    assertTrue(hashLargeBcSmall1000.compareTo(hashSmallBcLarge1000) < 0);
    assertTrue(forwardLargeBcSmall1000.compareTo(forwardSmallBcLarge1000) < 0);
    
    // broadcasting small and forwarding large is better than partition both, given size difference
    assertTrue(forwardLargeBcSmall10.compareTo(hashSmallAndLarge) < 0);
    
    // broadcasting too far is expensive again
    assertTrue(forwardLargeBcSmall1000.compareTo(hashSmallAndLarge) > 0);
    
    // assert weight is respected
    assertTrue(hashSmallBcLarge10.compareTo(hashSmallBcLarge1000) < 0);
    assertTrue(hashLargeBcSmall10.compareTo(hashLargeBcSmall1000) < 0);
    assertTrue(forwardSmallBcLarge10.compareTo(forwardSmallBcLarge1000) < 0);
    assertTrue(forwardLargeBcSmall10.compareTo(forwardLargeBcSmall1000) < 0);
    
    // forward versus hash
    assertTrue(forwardSmallBcLarge10.compareTo(hashSmallBcLarge10) < 0);
    assertTrue(forwardSmallBcLarge1000.compareTo(hashSmallBcLarge1000) < 0);
    assertTrue(forwardLargeBcSmall10.compareTo(hashLargeBcSmall10) < 0);
    assertTrue(forwardLargeBcSmall1000.compareTo(hashLargeBcSmall1000) < 0);
  }

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

    testShipStrategyCombinationsWithUnknowns(MEDIUM_ESTIMATES);
    testShipStrategyCombinationsWithUnknowns(BIG_ESTIMATES);
  }
  
  private void testShipStrategyCombinationsWithUnknowns(EstimateProvider knownEstimates) {
    Costs hashBoth = new Costs();
    Costs bcKnown10 = new Costs();
    Costs bcUnknown10 = new Costs();
    Costs bcKnown1000 = new Costs();
    Costs bcUnknown1000 = new Costs();
    
    costEstimator.addHashPartitioningCost(knownEstimates, hashBoth);
    costEstimator.addHashPartitioningCost(UNKNOWN_ESTIMATES, hashBoth);
    
    costEstimator.addBroadcastCost(knownEstimates, 10, bcKnown10);

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

  @Test
  public void testJoinCostFormulasPlain() {
    
    // hash join costs
    
    Costs hashBothSmall = new Costs();
    Costs hashBothLarge = new Costs();
    Costs hashSmallBuild = new Costs();
    Costs hashLargeBuild = new Costs();
    
    costEstimator.addHybridHashCosts(SMALL_ESTIMATES, BIG_ESTIMATES, hashSmallBuild, 1);
    costEstimator.addHybridHashCosts(BIG_ESTIMATES, SMALL_ESTIMATES, hashLargeBuild, 1);
    costEstimator.addHybridHashCosts(SMALL_ESTIMATES, SMALL_ESTIMATES, hashBothSmall, 1);
    costEstimator.addHybridHashCosts(BIG_ESTIMATES, BIG_ESTIMATES, hashBothLarge, 1);


    assertTrue(hashBothSmall.compareTo(hashSmallBuild) < 0);
    assertTrue(hashSmallBuild.compareTo(hashLargeBuild) < 0);
    assertTrue(hashLargeBuild.compareTo(hashBothLarge) < 0);
    
    // merge join costs
    
    Costs mergeBothSmall = new Costs();
    Costs mergeBothLarge = new Costs();
    Costs mergeSmallFirst = new Costs();
    Costs mergeSmallSecond = new Costs();
    
    costEstimator.addLocalSortCost(SMALL_ESTIMATES, mergeSmallFirst);
    costEstimator.addLocalSortCost(BIG_ESTIMATES, mergeSmallFirst);
    costEstimator.addLocalMergeCost(SMALL_ESTIMATES, BIG_ESTIMATES, mergeSmallFirst, 1);
    
    costEstimator.addLocalSortCost(BIG_ESTIMATES, mergeSmallSecond);
    costEstimator.addLocalSortCost(SMALL_ESTIMATES, mergeSmallSecond);
    costEstimator.addLocalMergeCost(BIG_ESTIMATES, SMALL_ESTIMATES, mergeSmallSecond, 1);
    
    costEstimator.addLocalSortCost(SMALL_ESTIMATES, mergeBothSmall);
    costEstimator.addLocalSortCost(SMALL_ESTIMATES, mergeBothSmall);
    costEstimator.addLocalMergeCost(SMALL_ESTIMATES, SMALL_ESTIMATES, mergeBothSmall, 1);
    
    costEstimator.addLocalSortCost(BIG_ESTIMATES, mergeBothLarge);
    costEstimator.addLocalSortCost(BIG_ESTIMATES, mergeBothLarge);
    costEstimator.addLocalMergeCost(BIG_ESTIMATES, BIG_ESTIMATES, mergeBothLarge, 1);
    
    
    assertTrue(mergeBothSmall.compareTo(mergeSmallFirst) < 0);
    assertTrue(mergeBothSmall.compareTo(mergeSmallSecond) < 0);
    assertTrue(mergeSmallFirst.compareTo(mergeSmallSecond) == 0);
    assertTrue(mergeSmallFirst.compareTo(mergeBothLarge) < 0);
    assertTrue(mergeSmallSecond.compareTo(mergeBothLarge) < 0);
    
    // compare merge join and hash join costs
    
    assertTrue(hashBothSmall.compareTo(mergeBothSmall) < 0);
    assertTrue(hashBothLarge.compareTo(mergeBothLarge) < 0);

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

    testJoinCostFormulasWithWeights(BIG_ESTIMATES, MEDIUM_ESTIMATES);
    testJoinCostFormulasWithWeights(MEDIUM_ESTIMATES, BIG_ESTIMATES);
  }
  
  private void testJoinCostFormulasWithWeights(EstimateProvider e1, EstimateProvider e2) {
    Costs hf1 = new Costs();
    Costs hf5 = new Costs();
    Costs hs1 = new Costs();
    Costs hs5 = new Costs();
    Costs mm1 = new Costs();
    Costs mm5 = new Costs();
    
    costEstimator.addHybridHashCosts(e1, e2, hf1, 1);
    costEstimator.addHybridHashCosts(e1, e2, hf5, 5);
    costEstimator.addHybridHashCosts(e2, e1, hs1, 1);
    costEstimator.addHybridHashCosts(e2, e1, hs5, 5);
    
    costEstimator.addLocalSortCost(e1, mm1);
    costEstimator.addLocalSortCost(e2, mm1);
    costEstimator.addLocalMergeCost(e1, e2, mm1, 1);
    
    costEstimator.addLocalSortCost(e1, mm5);
    costEstimator.addLocalSortCost(e2, mm5);
    mm5.multiplyWith(5);
    costEstimator.addLocalMergeCost(e1, e2, mm5, 5);
    
    // weight 1 versus weight 5
    assertTrue(hf1.compareTo(hf5) < 0);
    assertTrue(hs1.compareTo(hs5) < 0);

View Full Code Here

Examples of org.apache.flink.compiler.costs.Costs

  // --------------------------------------------------------------------------------------------
  
  @Test
  public void testHashJoinCostFormulasWithCaches() {
    
    Costs hashBothUnknown10 = new Costs();
    Costs hashBothUnknownCached10 = new Costs();
    
    Costs hashBothSmall10 = new Costs();
    Costs hashBothSmallCached10 = new Costs();
    
    Costs hashSmallLarge10 = new Costs();
    Costs hashSmallLargeCached10 = new Costs();
    
    Costs hashLargeSmall10 = new Costs();
    Costs hashLargeSmallCached10 = new Costs();
    
    Costs hashLargeSmall1 = new Costs();
    Costs hashLargeSmallCached1 = new Costs();
    
    costEstimator.addHybridHashCosts(UNKNOWN_ESTIMATES, UNKNOWN_ESTIMATES, hashBothUnknown10, 10);
    costEstimator.addCachedHybridHashCosts(UNKNOWN_ESTIMATES, UNKNOWN_ESTIMATES, hashBothUnknownCached10, 10);
    
    costEstimator.addHybridHashCosts(MEDIUM_ESTIMATES, MEDIUM_ESTIMATES, hashBothSmall10, 10);

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.