Examples of eu.stratosphere.nephele.services.iomanager.BlockChannelWriter

Package eu.stratosphere.nephele.services.iomanager

Examples of eu.stratosphere.nephele.services.iomanager.BlockChannelWriter

eu.stratosphere.nephele.services.iomanager.BlockChannelWriter
A writer that writes data in blocks to a file channel. The writer receives the data blocks in the form of {@link eu.stratosphere.core.memory.MemorySegment}, which it writes entirely to the channel, regardless of how space in the segment is used. The writing happens in an asynchronous fashion. That is, a write request is not processed by the thread that issues it, but by an asynchronous writer thread. Once the request is done, the asynchronous writer adds the MemorySegment to a return queue where it can be popped by the worker thread, to be reused. The return queue is in this case a {@link java.util.concurrent.LinkedBlockingQueue}, such that the working thread blocks until the request has been served, if the request is still pending when the it requires the segment back.
Typical write behind is realized, by having a small set of segments in the return queue at all times. When a memory segment must be written, the request is issued to the writer and a new segment is immediately popped from the return queue. Once too many requests have been issued and the I/O thread cannot keep up, the working thread naturally blocks until another segment is available again.

        // open next channel
        Channel.ID channel = enumerator.next();
        registerChannelToBeRemovedAtShudown(channel);


        // create writer
        final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                                channel, this.numWriteBuffersToCluster);
        registerOpenChannelToBeRemovedAtShudown(writer);
        final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, this.writeMemory,
                                      this.memManager.getPageSize());

View Full Code Here

      final MergeIterator<E> mergeIterator = getMergingIterator(channelIDs, readBuffers, channelAccesses);


      // create a new channel writer
      final Channel.ID mergedChannelID = this.ioManager.createChannel();
      registerChannelToBeRemovedAtShudown(mergedChannelID);
      final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                              mergedChannelID, this.numWriteBuffersToCluster);
      registerOpenChannelToBeRemovedAtShudown(writer);
      final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, writeBuffers, 
                                      this.memManager.getPageSize());

View Full Code Here

        if (LOG.isDebugEnabled()) {
          LOG.debug("Creating temp file " + channel.toString() + '.');
        }


        // create writer
        final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                                channel, this.numWriteBuffersToCluster);
        registerOpenChannelToBeRemovedAtShudown(writer);
        final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, this.writeMemory,
                                      this.memManager.getPageSize());

View Full Code Here

      final KeyGroupedIterator<E> groupedIter = new KeyGroupedIterator<E>(mergeIterator, this.serializer, this.comparator2);


      // create a new channel writer
      final Channel.ID mergedChannelID = this.ioManager.createChannel();
      registerChannelToBeRemovedAtShudown(mergedChannelID);
      final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(
                              mergedChannelID, this.numWriteBuffersToCluster);
      registerOpenChannelToBeRemovedAtShudown(writer);
      final ChannelWriterOutputView output = new ChannelWriterOutputView(writer, writeBuffers, 
                                      this.memManager.getPageSize());

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_SHORT_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, NUM_MEMORY_SEGMENTS);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_SHORT; i++) {

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_LONG_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, NUM_MEMORY_SEGMENTS);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_LONG; i++) {

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_SHORT_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, NUM_MEMORY_SEGMENTS);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_SHORT; i++) {

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_SHORT_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, NUM_MEMORY_SEGMENTS);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_SHORT; i++) {

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_SHORT_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, 1);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_SHORT; i++) {

View Full Code Here

    final TestData.Generator generator = new TestData.Generator(SEED, KEY_MAX, VALUE_SHORT_LENGTH, KeyMode.RANDOM, ValueMode.RANDOM_LENGTH);
    final Channel.ID channel = this.ioManager.createChannel();
    
    // create the writer output view
    List<MemorySegment> memory = this.memoryManager.allocatePages(this.parentTask, NUM_MEMORY_SEGMENTS);
    final BlockChannelWriter writer = this.ioManager.createBlockChannelWriter(channel);
    final ChannelWriterOutputView outView = new ChannelWriterOutputView(writer, memory, MEMORY_PAGE_SIZE);
    
    // write a number of pairs
    final Record rec = new Record();
    for (int i = 0; i < NUM_PAIRS_SHORT; i++) {

View Full Code Here

TOP

Related Classes of eu.stratosphere.nephele.services.iomanager.BlockChannelWriter

eu.stratosphere.core.memory.MemorySegment

eu.stratosphere.pact.runtime.io.ChannelViewsTest

eu.stratosphere.pact.runtime.sort.CombiningUnilateralSortMerger$CombiningSpillingThread

eu.stratosphere.pact.runtime.sort.UnilateralSortMerger$SpillingThread

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.