Examples of eu.stratosphere.pact.runtime.sort.CombiningUnilateralSortMerger

eu.stratosphere.pact.runtime.sort.CombiningUnilateralSortMerger

The {@link CombiningUnilateralSortMerger} is part of a merge-sort implementation.The {@link ReduceTask} requires a grouping of the incoming key-value pairs by key. For a {@link GroupReduceFunction} grouping is achieved bydetermining a total order for the given set of pairs (sorting). Thereafter an iteration over the ordered set is performed and each time the key changes the consecutive objects are united into a new group. Reducers have a combining feature can reduce the data before it is written to disk. In order to implement a combining Reducer, the {@link eu.stratosphere.pact.ReduceFunction.stub.ReduceStub#combine(Key,Iterator,Collector)} method must be implemented and the ReduceFunction must be annotated with the {@link eu.stratosphere.api.java.record.operators.ReduceOperator.Combinable} annotation.For a {@link ReduceFuntion} the combine method can automatically call the reduce function without a combine annotation.Conceptually, a merge sort with combining works as follows: (1) Divide the unsorted list into n sublists of about 1/n the size. (2) Sort each sublist recursively by re-applying merge sort. (3) Combine all tuples with the same key within a sublist (4) Merge the two sublists back into one sorted list. Internally, the {@link CombiningUnilateralSortMerger} logic is factored into four threads (read, sort, combine,spill) which communicate through a set of blocking queues (forming a closed loop). Memory is allocated using the {@link MemoryManager} interface. Thus the component will most likely not exceed theuser-provided memory limits.

        if (!(localStub instanceof GenericCombine)) {
          throw new IllegalStateException("Performing combining sort outside a reduce task!");
        }


        @SuppressWarnings({ "rawtypes", "unchecked" })
        CombiningUnilateralSortMerger<?> cSorter = new CombiningUnilateralSortMerger(
          (GenericCombine) localStub, getMemoryManager(), getIOManager(), this.inputIterators[inputNum], 
          this, this.inputSerializers[inputNum], getLocalStrategyComparator(inputNum),
          this.config.getMemoryInput(inputNum), this.config.getFilehandlesInput(inputNum),
          this.config.getSpillingThresholdInput(inputNum));
        cSorter.setUdfConfiguration(this.config.getStubParameters());


        // set the input to null such that it will be lazily fetched from the input strategy
        this.inputs[inputNum] = null;
        this.localStrategies[inputNum] = cSorter;
        break;

Examples of eu.stratosphere.pact.runtime.sort.CombiningUnilateralSortMerger

Related Classes of eu.stratosphere.pact.runtime.sort.CombiningUnilateralSortMerger