Examples of org.apache.flink.compiler.deadlockdetect.DeadlockPreventer

org.apache.flink.compiler.deadlockdetect.DeadlockPreventer

Certain pipelined flows may lead to deadlocks, in which case we need to make sure the pipelines are broken or made elastic enough to prevent that. This is only relevat to pipelined data flows where one operator has more than one consumers (successors in the flow). Most cases are caught by the general logic that deals with branching/joining flows. The following cases need additional checks: (source1) ------ (join) \ / \ / \/ /\ / \ / \ (source2) ------(join) Since both sources pipeline their data into a build and a probe side, they get stalled by the back pressure from the probe side (which waits for the build side to complete) and never finish the build side. We can model dependencies of pipelined / materialized connections and do a reguar cyclic dependencies check to detect such situations. Pipelined connections have a dependency from sender to receiver, non-pipelined (fully dammed) connections have a dependency from receiver to sender.

      bestPlanSinks.add((SinkPlanNode) bestPlanRoot);
    } else if (bestPlanRoot instanceof SinkJoinerPlanNode) {
      ((SinkJoinerPlanNode) bestPlanRoot).getDataSinks(bestPlanSinks);
    }
    
    DeadlockPreventer dp = new DeadlockPreventer();
    dp.resolveDeadlocks(bestPlanSinks);


    // finalize the plan
    OptimizedPlan plan = new PlanFinalizer().createFinalPlan(bestPlanSinks, program.getJobName(), program);


    // swap the binary unions for n-ary unions. this changes no strategies or memory consumers whatsoever, so

Examples of org.apache.flink.compiler.deadlockdetect.DeadlockPreventer

Related Classes of org.apache.flink.compiler.deadlockdetect.DeadlockPreventer