Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.UnionOptimizer

org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.UnionOptimizer

Optimizes union by removing the intermediate union vertex and making the successor get input from the predecessor vertices directly using VertexGroup. This should be run after MultiQueryOptimizer so that it handles cases like union followed by split and then store. For eg: 1) Union followed by store Vertex 1 (Load), Vertex 2 (Load) -> Vertex 3 (Union + Store) will be optimized to Vertex 1 (Load + Store), Vertex 2 (Load + Store). Both the vertices will be writing output to same store location directly which is supported by Tez. 2) Union followed by groupby Vertex 1 (Load), Vertex 2 (Load) -> Vertex 3 (Union + POLocalRearrange) -> Vertex 4 (Group by) will be optimized to Vertex 1 (Load + POLR), Vertex 2 (Load + POLR) -> Vertex 4 (Group by)

        }


        // Use VertexGroup in Tez
        boolean isUnionOpt = conf.getBoolean(PigConfiguration.TEZ_OPT_UNION, true);
        if (isUnionOpt) {
            UnionOptimizer uo = new UnionOptimizer(tezPlan);
            uo.visit();
        }


    }

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.UnionOptimizer

Related Classes of org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.UnionOptimizer