Package org.apache.crunch.impl.spark.fn

Examples of org.apache.crunch.impl.spark.fn.ReduceGroupingFunction

122
123
124
125
126
127
128
129
130
131
132
      SparkComparator scmp = new SparkComparator(groupingOptions, ptype, runtime.getRuntimeContext());
      groupedRDD = groupedRDD.sortByKey(scmp);
    }
    if (groupingOptions.getGroupingComparatorClass() != null) {
      groupedRDD = groupedRDD.mapPartitionsToPair(
          new ReduceGroupingFunction(groupingOptions, ptype, runtime.getRuntimeContext()));
    }

    return groupedRDD
        .map(new ReduceInputFunction(keySerde, valueSerde))
        .mapToPair(new PairMapIterableFunction(ptype.getInputMapFn(), runtime.getRuntimeContext()));
View Full Code Here
110
111
112
113
114
115
116
117
118
119
120
      SparkComparator scmp = new SparkComparator(groupingOptions, ptype, runtime.getRuntimeContext());
      groupedRDD = groupedRDD.sortByKey(scmp);
    }
    if (groupingOptions.getGroupingComparatorClass() != null) {
      groupedRDD = groupedRDD.mapPartitions(
          new ReduceGroupingFunction(groupingOptions, ptype, runtime.getRuntimeContext()));
    }

    return groupedRDD
        .map(new ReduceInputFunction(keySerde, valueSerde))
        .map(new PairMapIterableFunction(ptype.getInputMapFn(), runtime.getRuntimeContext()));
View Full Code Here
120
121
122
123
124
125
126
127
128
129
130
      SparkComparator scmp = new SparkComparator(groupingOptions, ptype, runtime.getRuntimeContext());
      groupedRDD = groupedRDD.sortByKey(scmp);
    }
    if (groupingOptions.getGroupingComparatorClass() != null) {
      groupedRDD = groupedRDD.mapPartitions(
          new ReduceGroupingFunction(groupingOptions, ptype, runtime.getRuntimeContext()));
    }

    return groupedRDD
        .map(new ReduceInputFunction(keySerde, valueSerde))
        .map(new PairMapIterableFunction(ptype.getInputMapFn(), runtime.getRuntimeContext()));
View Full Code Here
TOP

Related Classes of org.apache.crunch.impl.spark.fn.ReduceGroupingFunction

  • org.apache.crunch.impl.spark.ByteArray
  • org.apache.crunch.impl.spark.collect.PGroupedTableImpl
  • org.apache.hadoop.mapreduce.Job
  • org.apache.crunch.CrunchRuntimeException
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.