Examples of ChannelWriterOutputView

eu.stratosphere.nephele.services.iomanager.ChannelWriterOutputView
A {@link DataOutputView} that is backed by a {@link BlockChannelWriter}, making it effectively a data output stream. The view writes it data in blocks to the underlying channel, adding a minimal header to each block. The data can be re-read by a {@link ChannelReaderInputView}, if it uses the same block size.
org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView
A {@link org.apache.flink.core.memory.DataOutputView} that is backed by a{@link BlockChannelWriter}, making it effectively a data output stream. The view writes it data in blocks to the underlying channel, adding a minimal header to each block. The data can be re-read by a {@link ChannelReaderInputView}, if it uses the same block size.

Examples of org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

      final Channel.ID mergedChannelID = this.ioManager.createChannel();
      registerChannelToBeRemovedAtShudown(mergedChannelID);
      final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                              mergedChannelID, this.numWriteBuffersToCluster);
      registerOpenChannelToBeRemovedAtShudown(writer);
      final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, writeBuffers, 
                                      this.memManager.getPageSize());
      
      final WriterCollector<E> collector = new WriterCollector<E>(output, this.serializer);
      final FlatCombineFunction<E> combineStub = CombiningUnilateralSortMerger.this.combineStub;


      // combine and write to disk
      try {
        while (groupedIter.nextKey()) {
          combineStub.combine(groupedIter.getValues(), collector);
        }
      }
      catch (Exception e) {
        throw new IOException("An error occurred in the combiner user code.");
      }
      output.close(); //IS VERY IMPORTANT!!!!
      
      final int numBlocksWritten = output.getBlockCount();
      
      // register merged result to be removed at shutdown
      unregisterOpenChannelToBeRemovedAtShudown(writer);
      
      // remove the merged channel readers from the clear-at-shutdown list

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView


        // create writer
        final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                                channel, this.numWriteBuffersToCluster);
        registerOpenChannelToBeRemovedAtShudown(writer);
        final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, this.writeMemory,
                                      this.memManager.getPageSize());


        // write sort-buffer to channel
        if (LOG.isDebugEnabled()) {
          LOG.debug("Spilling buffer " + element.id + ".");
        }
        element.buffer.writeToOutput(output);
        if (LOG.isDebugEnabled()) {
          LOG.debug("Spilled buffer " + element.id + ".");
        }


        output.close();
        unregisterOpenChannelToBeRemovedAtShudown(writer);
        
        channelIDs.add(new ChannelWithBlockCount(channel, output.getBlockCount()));


        // pass empty sort-buffer to reading thread
        element.buffer.reset();
        this.queues.empty.add(element);
      }

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

      final Channel.ID mergedChannelID = this.ioManager.createChannel();
      registerChannelToBeRemovedAtShudown(mergedChannelID);
      final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                              mergedChannelID, this.numWriteBuffersToCluster);
      registerOpenChannelToBeRemovedAtShudown(writer);
      final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, writeBuffers, 
                                      this.memManager.getPageSize());


      // read the merged stream and write the data back
      final TypeSerializer<E> serializer = this.serializer;
      E rec = serializer.createInstance();
      while ((rec = mergeIterator.next(rec)) != null) {
        serializer.serialize(rec, output);
      }
      output.close();
      final int numBlocksWritten = output.getBlockCount();
      
      // register merged result to be removed at shutdown
      unregisterOpenChannelToBeRemovedAtShudown(writer);
      
      // remove the merged channel readers from the clear-at-shutdown list

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

      
      // create the channel for the probe side and claim one buffer for it
      this.probeSideChannel = ioAccess.createBlockChannelWriter(probeChannelEnumerator.next(), bufferReturnQueue);
      // creating the ChannelWriterOutputView without memory will cause it to draw one segment from the
      // write behind queue, which is the spare segment we had above.
      this.probeSideBuffer = new ChannelWriterOutputView(this.probeSideChannel, this.memorySegmentSize);
    }
  }

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

    if (isInMemory()) {
      return;
    }
    // ATTENTION: The following lines are duplicated code from finalizeBuildPhase
    this.probeSideChannel = ioAccess.createBlockChannelWriter(probeChannelEnumerator.next(), bufferReturnQueue);
    this.probeSideBuffer = new ChannelWriterOutputView(this.probeSideChannel, this.memorySegmentSize);
  }

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.