Examples of org.apache.flink.core.memory.MemorySegment

org.apache.flink.core.memory.MemorySegment
This class represents a piece of memory allocated from the memory manager. The segment is backed by a byte array and features random put and get methods for the basic types that are stored in a byte-wise fashion in the memory.
Comments on the implementation: We make heavy use of operations that are supported by native instructions, to achieve a high efficiency. Multi byte types (int, long, float, double, ...) are read and written with "unsafe" native commands. Little-endian to big-endian conversion and vice versa are done using the static reverseBytes methods in the boxing data types (for example {@link Integer#reverseBytes(int)}). On x86/amd64, these are translated by the jit compiler to bswap intrinsic commands. Below is an example of the code generated for the {@link MemorySegment#putLongBigEndian(int,long)}function by the just-in-time compiler. The code is grabbed from an oracle jvm 7 using the hotspot disassembler library (hsdis32.dll) and the jvm command -XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,*UnsafeMemorySegment.putLongBigEndian. Note that this code realizes both the byte order swapping and the reinterpret cast access to get a long from the byte array.
```
 [Verified Entry Point] 0x00007fc403e19920: sub    $0x18,%rsp 0x00007fc403e19927: mov    %rbp,0x10(%rsp)    ;*synchronization entry ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@-1 (line 652) 0x00007fc403e1992c: mov    0xc(%rsi),%r10d    ;*getfield memory ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLong@4 (line 611) ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@12 (line 653) 0x00007fc403e19930: bswap  %rcx 0x00007fc403e19933: shl    $0x3,%r10 0x00007fc403e19937: movslq %edx,%r11 0x00007fc403e1993a: mov    %rcx,0x10(%r10,%r11,1)  ;*invokevirtual putLong ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLong@14 (line 611) ; - org.apache.flink.runtime.memory.UnsafeMemorySegment::putLongBigEndian@12 (line 653) 0x00007fc403e1993f: add    $0x10,%rsp 0x00007fc403e19943: pop    %rbp 0x00007fc403e19944: test   %eax,0x5ba76b6(%rip)        # 0x00007fc4099c1000 ;   {poll_return} 0x00007fc403e1994a: retq  
```

      T[] data = getSortedData();
      // Get the normKeyLen on which we are testing
      int normKeyLen = getNormKeyLen(halfLength, data, comparator);


      // Write the data into different 2 memory segements
      MemorySegment memSegLow = setupNormalizedKeysMemSegment(data, normKeyLen, comparator);
      MemorySegment memSegHigh = setupNormalizedKeysMemSegment(data, normKeyLen, comparator);


      boolean fullyDetermines = !comparator.isNormalizedKeyPrefixOnly(normKeyLen);
      
      // Compare every element with every bigger element
      for (int l = 0; l < data.length - 1; l++) {

View Full Code Here

    final int segmentOffsetI = (i % this.recordsPerSegment) * this.recordSize;
    
    final int bufferNumJ = j / this.recordsPerSegment;
    final int segmentOffsetJ = (j % this.recordsPerSegment) * this.recordSize;
    
    final MemorySegment segI = this.sortBuffer.get(bufferNumI);
    final MemorySegment segJ = this.sortBuffer.get(bufferNumJ);
    
    int val = MemorySegment.compare(segI, segJ, segmentOffsetI, segmentOffsetJ, this.numKeyBytes);
    return this.useNormKeyUninverted ? val : -val;
  }

View Full Code Here

    final int segmentOffsetI = (i % this.recordsPerSegment) * this.recordSize;
    
    final int bufferNumJ = j / this.recordsPerSegment;
    final int segmentOffsetJ = (j % this.recordsPerSegment) * this.recordSize;
    
    final MemorySegment segI = this.sortBuffer.get(bufferNumI);
    final MemorySegment segJ = this.sortBuffer.get(bufferNumJ);
    
    MemorySegment.swapBytes(segI, segJ, this.swapBuffer, segmentOffsetI, segmentOffsetJ, this.recordSize);
  }

View Full Code Here

    final int recordsPerSegment = this.recordsPerSegment;
    int recordsLeft = this.numRecords;
    int currentMemSeg = 0;
    
    while (recordsLeft > 0) {
      final MemorySegment currentIndexSegment = this.sortBuffer.get(currentMemSeg++);
      inView.set(currentIndexSegment, 0);
      
      // check whether we have a full or partially full segment
      if (recordsLeft >= recordsPerSegment) {
        // full segment

View Full Code Here

    final int recordsPerSegment = this.recordsPerSegment;
    int currentMemSeg = start / recordsPerSegment;
    int offset = (start % recordsPerSegment) * this.recordSize;
    
    while (num > 0) {
      final MemorySegment currentIndexSegment = this.sortBuffer.get(currentMemSeg++);
      inView.set(currentIndexSegment, offset);
      
      // check whether we have a full or partially full segment
      if (num >= recordsPerSegment && offset == 0) {
        // full segment

View Full Code Here

    final int segmentOffsetI = (i % this.indexEntriesPerSegment) * this.indexEntrySize;
    
    final int bufferNumJ = j / this.indexEntriesPerSegment;
    final int segmentOffsetJ = (j % this.indexEntriesPerSegment) * this.indexEntrySize;
    
    final MemorySegment segI = this.sortIndex.get(bufferNumI);
    final MemorySegment segJ = this.sortIndex.get(bufferNumJ);
    
    int val = MemorySegment.compare(segI, segJ, segmentOffsetI + OFFSET_LEN, segmentOffsetJ + OFFSET_LEN, this.numKeyBytes);
    
    if (val != 0 || this.normalizedKeyFullyDetermines) {
      return this.useNormKeyUninverted ? val : -val;
    }
    
    final long pointerI = segI.getLong(segmentOffsetI);
    final long pointerJ = segJ.getLong(segmentOffsetJ);
    
    return compareRecords(pointerI, pointerJ);
  }

View Full Code Here

    final int segmentOffsetI = (i % this.indexEntriesPerSegment) * this.indexEntrySize;
    
    final int bufferNumJ = j / this.indexEntriesPerSegment;
    final int segmentOffsetJ = (j % this.indexEntriesPerSegment) * this.indexEntrySize;
    
    final MemorySegment segI = this.sortIndex.get(bufferNumI);
    final MemorySegment segJ = this.sortIndex.get(bufferNumJ);
    
    MemorySegment.swapBytes(segI, segJ, this.swapBuffer, segmentOffsetI, segmentOffsetJ, this.indexEntrySize);
  }

View Full Code Here

  public void writeToOutput(final ChannelWriterOutputView output) throws IOException {
    int recordsLeft = this.numRecords;
    int currentMemSeg = 0;
    while (recordsLeft > 0)
    {
      final MemorySegment currentIndexSegment = this.sortIndex.get(currentMemSeg++);
      int offset = 0;
      // check whether we have a full or partially full segment
      if (recordsLeft >= this.indexEntriesPerSegment) {
        // full segment
        for (;offset <= this.lastIndexEntryOffset; offset += this.indexEntrySize) {
          final long pointer = currentIndexSegment.getLong(offset);
          this.recordBuffer.setReadPosition(pointer);
          this.serializer.copy(this.recordBuffer, output);
          
        }
        recordsLeft -= this.indexEntriesPerSegment;
      } else {
        // partially filled segment
        for (; recordsLeft > 0; recordsLeft--, offset += this.indexEntrySize)
        {
          final long pointer = currentIndexSegment.getLong(offset);
          this.recordBuffer.setReadPosition(pointer);
          this.serializer.copy(this.recordBuffer, output);
        }
      }
    }

View Full Code Here

    int currentMemSeg = start / this.indexEntriesPerSegment;
    int offset = (start % this.indexEntriesPerSegment) * this.indexEntrySize;
    
    while (num > 0)
    {
      final MemorySegment currentIndexSegment = this.sortIndex.get(currentMemSeg++);
      // check whether we have a full or partially full segment
      if (num >= this.indexEntriesPerSegment && offset == 0) {
        // full segment
        for (;offset <= this.lastIndexEntryOffset; offset += this.indexEntrySize) {
          final long pointer = currentIndexSegment.getLong(offset);
          this.recordBuffer.setReadPosition(pointer);
          this.serializer.copy(this.recordBuffer, output);
        }
        num -= this.indexEntriesPerSegment;
      } else {
        // partially filled segment
        for (; num > 0 && offset <= this.lastIndexEntryOffset; num--, offset += this.indexEntrySize)
        {
          final long pointer = currentIndexSegment.getLong(offset);
          this.recordBuffer.setReadPosition(pointer);
          this.serializer.copy(this.recordBuffer, output);
        }
      }
      offset = 0;

View Full Code Here

    final int numSegs = (numBuckets >>> this.bucketsPerSegmentBits) + ( (numBuckets & this.bucketsPerSegmentMask) == 0 ? 0 : 1);
    final MemorySegment[] table = new MemorySegment[numSegs];
    
    // go over all segments that are part of the table
    for (int i = 0, bucket = 0; i < numSegs && bucket < numBuckets; i++) {
      final MemorySegment seg = getNextBuffer();
      
      // go over all buckets in the segment
      for (int k = 0; k < bucketsPerSegment && bucket < numBuckets; k++, bucket++) {
        final int bucketOffset = k * HASH_BUCKET_SIZE;  
        
        // compute the partition that the bucket corresponds to
        final byte partition = assignPartition(bucket, numPartitions);
        
        // initialize the header fields
        seg.put(bucketOffset + HEADER_PARTITION_OFFSET, partition);
        seg.putInt(bucketOffset + HEADER_COUNT_OFFSET, 0);
        seg.putLong(bucketOffset + HEADER_FORWARD_OFFSET, BUCKET_FORWARD_POINTER_NOT_SET);
      }
      
      table[i] = seg;
    }
    this.buckets = table;

View Full Code Here

0 1 2 3 4 5 6

TOP

Related Classes of org.apache.flink.core.memory.MemorySegment

org.apache.flink.api.common.typeutils.ComparatorTestBase

org.apache.flink.runtime.io.disk.iomanager.BlockChannelReader

org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter

org.apache.flink.runtime.io.disk.iomanager.ChannelReaderInputView

org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView

org.apache.flink.runtime.io.disk.iomanager.IOManagerTest

org.apache.flink.runtime.io.disk.SimpleCollectingOutputView

org.apache.flink.runtime.io.disk.SpillingBuffer

org.apache.flink.runtime.io.network.bufferprovider.GlobalBufferPool

org.apache.flink.runtime.io.network.bufferprovider.LocalBufferPool

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.