Examples of LateralViewJoinOperator

org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator

The lateral view join operator is used to implement the lateral view functionality. This operator was implemented with the following operator DAG in mind. For a query such as SELECT pageid, adid.* FROM example_table LATERAL VIEW explode(adid_list) AS adid The top of the operator tree will look similar to [Table Scan] / \ [Select](*) [Select](adid_list) | | | [UDTF] (explode) \ / [Lateral View Join] | | [Select] (pageid, adid.*) | .... Rows from the table scan operator are first sent to two select operators. The select operator on the left picks all the columns while the select operator on the right picks only the columns needed by the UDTF. The output of select in the left branch and output of the UDTF in the right branch are then sent to the lateral view join (LVJ). In most cases, the UDTF will generate > 1 row for every row received from the TS, while the left select operator will generate only one. For each row output from the TS, the LVJ outputs all possible rows that can be created by joining the row from the left select and one of the rows output from the UDTF. Additional lateral views can be supported by adding a similar DAG after the previous LVJ operator.

      // be called for leafs.
      assert(!stack.isEmpty());


      // LineageCtx
      LineageCtx lCtx = (LineageCtx) procCtx;
      LateralViewJoinOperator op = (LateralViewJoinOperator)nd;
      boolean isUdtfPath = true;
      Operator<? extends Serializable> inpOp = getParent(stack);
      ArrayList<ColumnInfo> cols = inpOp.getSchema().getSignature();


      if (inpOp instanceof SelectOperator) {
        isUdtfPath = false;
      }


      // Dirty hack!!
      // For the select path the columns are the ones at the end of the
      // current operators schema and for the udtf path the columns are
      // at the beginning of the operator schema.
      ArrayList<ColumnInfo> out_cols = op.getSchema().getSignature();
      int out_cols_size = out_cols.size();
      int cols_size = cols.size();
      if (isUdtfPath) {
        int cnt = 0;
        while (cnt < cols_size) {

   * The Node Processor for Column Pruning on Lateral View Join Operators.
   */
  public static class ColumnPrunerLateralViewJoinProc implements NodeProcessor {
    public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx ctx,
        Object... nodeOutputs) throws SemanticException {
      LateralViewJoinOperator op = (LateralViewJoinOperator) nd;
      ColumnPrunerProcCtx cppCtx = (ColumnPrunerProcCtx) ctx;
      List<String> cols = new ArrayList<String>();


      cols = cppCtx.genColLists(op);
      Map<String, ExprNodeDesc> colExprMap = op.getColumnExprMap();


      // As columns go down the DAG, the LVJ will transform internal column
      // names from something like 'key' to '_col0'. Because of this, we need
      // to undo this transformation using the column expression map as the
      // column names propagate up the DAG.

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of LateralViewJoinOperator

Examples of org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator

Examples of org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator