Examples of org.apache.flink.core.memory.MemorySegment

org.apache.flink.core.memory.MemorySegment
This class represents a piece of memory allocated from the memory manager. The segment is backed by a byte array and features random put and get methods for the basic types that are stored in a byte-wise fashion in the memory.
Comments on the implementation: We make heavy use of operations that are supported by native instructions, to achieve a high efficiency. Multi byte types (int, long, float, double, ...) are read and written with "unsafe" native commands. Little-endian to big-endian conversion and vice versa are done using the static reverseBytes methods in the boxing data types (for example {@link Integer#reverseBytes(int)}). On x86/amd64, these are translated by the jit compiler to bswap intrinsic commands. Below is an example of the code generated for the {@link MemorySegment#putLongBigEndian(int,long)}function by the just-in-time compiler. The code is grabbed from an oracle jvm 7 using the hotspot disassembler library (hsdis32.dll) and the jvm command -XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,*UnsafeMemorySegment.putLongBigEndian. Note that this code realizes both the byte order swapping and the reinterpret cast access to get a long from the byte array.
```
 [Verified Entry Point] 0x00007fc403e19920: sub    $0x18,%rsp 0x00007fc403e19927: mov    %rbp,0x10(%rsp)    ;*synchronization entry ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@-1 (line 652) 0x00007fc403e1992c: mov    0xc(%rsi),%r10d    ;*getfield memory ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLong@4 (line 611) ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@12 (line 653) 0x00007fc403e19930: bswap  %rcx 0x00007fc403e19933: shl    $0x3,%r10 0x00007fc403e19937: movslq %edx,%r11 0x00007fc403e1993a: mov    %rcx,0x10(%r10,%r11,1)  ;*invokevirtual putLong ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLong@14 (line 611) ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@12 (line 653) 0x00007fc403e1993f: add    $0x10,%rsp 0x00007fc403e19943: pop    %rbp 0x00007fc403e19944: test   %eax,0x5ba76b6(%rip)        # 0x00007fc4099c1000 ;   {poll_return} 0x00007fc403e1994a: retq  
```

    @Override
    protected MemorySegment nextSegment(MemorySegment current, int bytesUsed) throws IOException
    {
      finalizeSegment(current, bytesUsed);
      
      final MemorySegment next;
      if (this.writer == null) {
        this.targetList.add(current);
        next = this.memSource.nextSegment();
      } else {
        this.writer.writeBlock(current);

View Full Code Here

    this.writer.close();
    
    // re-collect all memory segments
    ArrayList<MemorySegment> list = new ArrayList<MemorySegment>(this.numSegments);  
    for (int i = 0; i < this.numSegments; i++) {
      final MemorySegment m = queue.poll();
      if (m == null) {
        // we get null if the queue is empty. that should not be the case if the reader was properly closed.
        throw new RuntimeException("ChannelWriterOutputView: MemorySegments have been taken from return queue by different actor.");
      }
      list.add(m);

View Full Code Here

  {
    if (current != null) {
      writeSegment(current, posInSegment, false);
    }
    
    final MemorySegment next = this.writer.getNextReturnedSegment();
    this.blockCount++;
    return next;
  }

View Full Code Here

      return numSegments;
    }
    
    MemorySegment[] close() throws IOException
    {
      final MemorySegment current = getCurrentSegment();
      if (current == null) {
        throw new IllegalStateException("Illegal State in HashPartition: No current buffer when finilizing build side.");
      }
      finalizeSegment(current, getCurrentPositionInSegment());
      clear();

View Full Code Here

   */
  public MemorySegment getNextReturnedSegment() throws IOException
  {
    try {
      while (true) {
        final MemorySegment next = this.returnBuffers.poll(2000, TimeUnit.MILLISECONDS);
        if (next != null) {
          return next;
        } else {
          if (this.closed) {
            throw new IOException("The writer has been closed.");

View Full Code Here

            final int posHashCode = hashCode % initialBucketCount;
            final long pointer = pIter.getPointer();
            // get the bucket for the given hash code
            final int bucketArrayPos = posHashCode >> this.bucketsPerSegmentBits;
            final int bucketInSegmentPos = (posHashCode & this.bucketsPerSegmentMask) << NUM_INTRA_BUCKET_BITS;
            final MemorySegment bucket = this.buckets[bucketArrayPos];
            insertBucketEntry(part, bucket, bucketInSegmentPos, hashCode, pointer);
          }
        } else {
          this.writeBehindBuffersAvailable--; // we are not in-memory, thus the probe side buffer will grab one wbb.
          if (this.writeBehindBuffers.size() == 0) { // prepareProbePhase always requires one buffer in the writeBehindBuffers-Queue.

View Full Code Here

  
  @SuppressWarnings("unchecked")
  private <T extends Comparable<T>> void assertNormalizableKey(NormalizableKey<T> key1, NormalizableKey<T> key2, int len) {
    
    byte[] normalizedKeys = new byte[2*len];
    MemorySegment wrapper = new MemorySegment(normalizedKeys);
    
    key1.copyNormalizedKey(wrapper, 0, len);
    key2.copyNormalizedKey(wrapper, len, len);
    
    for (int i = 0; i < len; i++) {

View Full Code Here

      final int posHashCode = hash % this.numBuckets;
      
      // get the bucket for the given hash code
      final int bucketArrayPos = posHashCode >> this.bucketsPerSegmentBits;
      final int bucketInSegmentOffset = (posHashCode & this.bucketsPerSegmentMask) << NUM_INTRA_BUCKET_BITS;
      final MemorySegment bucket = this.buckets[bucketArrayPos];
      
      // get the basic characteristics of the bucket
      final int partitionNumber = bucket.get(bucketInSegmentOffset + HEADER_PARTITION_OFFSET);
      final HashPartition<BT, PT> p = this.partitionsBeingBuilt.get(partitionNumber);
      
      // for an in-memory partition, process set the return iterators, else spill the probe records
      if (p.isInMemory()) {
        this.recordComparator.setReference(next);

View Full Code Here

    final int posHashCode = hash % this.numBuckets;
    
    // get the bucket for the given hash code
    final int bucketArrayPos = posHashCode >> this.bucketsPerSegmentBits;
    final int bucketInSegmentOffset = (posHashCode & this.bucketsPerSegmentMask) << NUM_INTRA_BUCKET_BITS;
    final MemorySegment bucket = this.buckets[bucketArrayPos];
    
    // get the basic characteristics of the bucket
    final int partitionNumber = bucket.get(bucketInSegmentOffset + HEADER_PARTITION_OFFSET);
    final HashPartition<BT, PT> p = this.partitionsBeingBuilt.get(partitionNumber);
    
    // for an in-memory partition, process set the return iterators, else spill the probe records
    if (p.isInMemory()) {
      this.recordComparator.setReference(record);

View Full Code Here

        final long pointer = pIter.getPointer();
        
        // get the bucket for the given hash code
        final int bucketArrayPos = posHashCode >> this.bucketsPerSegmentBits;
        final int bucketInSegmentPos = (posHashCode & this.bucketsPerSegmentMask) << NUM_INTRA_BUCKET_BITS;
        final MemorySegment bucket = this.buckets[bucketArrayPos];
        
        insertBucketEntry(newPart, bucket, bucketInSegmentPos, hashCode, pointer);
      }
    }
    else {

View Full Code Here

0 1 2 3 4 5 6

TOP

Related Classes of org.apache.flink.core.memory.MemorySegment

org.apache.flink.api.common.typeutils.ComparatorTestBase

org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader

org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter

org.apache.flink.runtime.io.disk.iomanager.ChannelReaderInputView

org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

org.apache.flink.runtime.io.disk.iomanager.IOManagerTest

org.apache.flink.runtime.io.disk.SimpleCollectingOutputView

org.apache.flink.runtime.io.disk.SpillingBuffer

org.apache.flink.runtime.io.network.bufferprovider.GlobalBufferPool

org.apache.flink.runtime.io.network.bufferprovider.LocalBufferPool

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.