Examples of BlockChannelReader

eu.stratosphere.nephele.services.iomanager.BlockChannelReader
A reader that reads data in blocks from a file channel. The reader reads the blocks into a {@link eu.stratosphere.core.memory.MemorySegment} in an asynchronous fashion. That is, a readrequest is not processed by the thread that issues it, but by an asynchronous reader thread. Once the read request is done, the asynchronous reader adds the full MemorySegment to a return queue where it can be popped by the worker thread, once it needs the data. The return queue is in this case a {@link java.util.concurrent.LinkedBlockingQueue}, such that the working thread blocks until the request has been served, if the request is still pending when the it requires the data.
Typical pre-fetching reads are done by issuing the read requests early and popping the return queue once the data is actually needed.
The reader has no notion whether the size of the memory segments is actually the size of the blocks on disk, or even whether the file was written in blocks of the same size, or in blocks at all. Ensuring that the writing and reading is consistent with each other (same blocks sizes) is up to the programmer.
org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader
A reader that reads data in blocks from a file channel. The reader reads the blocks into a {@link org.apache.flink.core.memory.MemorySegment} in an asynchronous fashion. That is, a readrequest is not processed by the thread that issues it, but by an asynchronous reader thread. Once the read request is done, the asynchronous reader adds the full MemorySegment to a return queue where it can be popped by the worker thread, once it needs the data. The return queue is in this case a {@link java.util.concurrent.LinkedBlockingQueue}, such that the working thread blocks until the request has been served, if the request is still pending when the it requires the data.
Typical pre-fetching reads are done by issuing the read requests early and popping the return queue once the data is actually needed.
The reader has no notion whether the size of the memory segments is actually the size of the blocks on disk, or even whether the file was written in blocks of the same size, or in blocks at all. Ensuring that the writing and reading is consistent with each other (same blocks sizes) is up to the programmer.

Examples of org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader

        numBuffersSpilled += toSpill;
      }


      // now close the writer and create the reader
      currentWriter.close();
      final BlockChannelReader reader = ioManager.createBlockChannelReader(currentWriter.getChannelID());


      // gather some memory segments to circulate while reading back the data
      final ArrayList<MemorySegment> readSegments = new ArrayList<MemorySegment>();
      try {
        while (readSegments.size() < minBuffersForSpilledReadEnd) {
          readSegments.add(emptyBuffers.take());
        }


        // read the first segment
        MemorySegment firstSeg = readSegments.remove(readSegments.size() - 1);
        reader.readBlock(firstSeg);
        firstSeg = reader.getReturnQueue().take();


        // create the read end reading one less buffer, because the first buffer is already read back
        readEnd = new ReadEnd(firstSeg, emptyBuffers, fullBuffers, reader, readSegments, segmentSize,
          numBuffersSpilled - 1);
      } catch (InterruptedException e) {

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader

      // recollect memory from a previous view
      if (this.externalInView != null) {
        this.externalInView.close();
      }
      
      final BlockChannelReader reader = this.ioManager.createBlockChannelReader(this.writer.getChannelID());
      this.externalInView = new HeaderlessChannelReaderInputView(reader, this.fullSegments, this.blockCount, this.numBytesInLastSegment, false);
      return this.externalInView;
    }
  }

View Full Code Here

Examples of org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader

      // first set up the reader with some memory.
      final List<MemorySegment> segments = new ArrayList<MemorySegment>(2);
      segments.add(getNextBuffer());
      segments.add(getNextBuffer());
      
      final BlockChannelReader inReader = this.ioManager.createBlockChannelReader(p.getBuildSideChannel().getChannelID());
      final ChannelReaderInputView inView = new HeaderlessChannelReaderInputView(inReader, segments,
            p.getBuildSideBlockCount(), p.getLastSegmentLimit(), false);
      final ChannelReaderInputViewIterator<BT> inIter = new ChannelReaderInputViewIterator<BT>(inView, 
          this.availableMemory, this.buildSideSerializer);
      final TypeComparator<BT> btComparator = this.buildSideComparator;

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.