Examples of org.apache.accumulo.core.file.blockfile.cache.LruBlockCache

Package org.apache.accumulo.core.file.blockfile.cache

Examples of org.apache.accumulo.core.file.blockfile.cache.LruBlockCache

org.apache.accumulo.core.file.blockfile.cache.LruBlockCache
A block cache implementation that is memory-aware using {@link HeapSize}, memory-bound using an LRU eviction algorithm, and concurrent: backed by a {@link ConcurrentHashMap} and with a non-blocking eviction thread giving constant-time {@link #cacheBlock} and {@link #getBlock} operations.
Contains three levels of block priority to allow for scan-resistance and in-memory families. A block is added with an inMemory flag if necessary, otherwise a block becomes a single access priority. Once a blocked is accessed again, it changes to multiple access. This is used to prevent scans from thrashing the cache, adding a least-frequently-used element to the eviction algorithm.
Each priority is given its own chunk of the total cache to ensure fairness during eviction. Each priority will retain close to its maximum size, however, if any priority is not using its entire chunk the others are able to grow beyond their chunk size.
Instantiated at a minimum with the total size and average block size. All sizes are in bytes. The block size is not especially important as this cache is fully dynamic in its sizing of blocks. It is only used for pre-allocating data structures and in initial heap estimation of the map.
The detailed constructor defines the sizes for the three priorities (they should total to the maximum size defined). It also sets the levels that trigger and control the eviction thread.
The acceptable size is the cache size level which triggers the eviction process to start. It evicts enough blocks to get the size below the minimum size specified.
Eviction happens in a separate thread and involves a single full-scan of the map. It determines how many bytes must be freed to reach the minimum size, and then while scanning determines the fewest least-recently-used blocks necessary from each of the three priorities (would be 3 times bytes to free). It then uses the priority chunk sizes to evict fairly according to the relative sizes and usage.

    
    long blockSize = acuConf.getMemoryInBytes(Property.TSERV_DEFAULT_BLOCKSIZE);
    long dCacheSize = acuConf.getMemoryInBytes(Property.TSERV_DATACACHE_SIZE);
    long iCacheSize = acuConf.getMemoryInBytes(Property.TSERV_INDEXCACHE_SIZE);
    
    _iCache = new LruBlockCache(iCacheSize, blockSize);
    _dCache = new LruBlockCache(dCacheSize, blockSize);
    
    Runtime runtime = Runtime.getRuntime();
    if (!usingNativeMap && maxMemory + dCacheSize + iCacheSize > runtime.maxMemory()) {
      throw new IllegalArgumentException(String.format(
          "Maximum tablet server map memory %,d and block cache sizes %,d is too large for this JVM configuration %,d", maxMemory, dCacheSize + iCacheSize,

View Full Code Here

    
    long blockSize = acuConf.getMemoryInBytes(Property.TSERV_DEFAULT_BLOCKSIZE);
    long dCacheSize = acuConf.getMemoryInBytes(Property.TSERV_DATACACHE_SIZE);
    long iCacheSize = acuConf.getMemoryInBytes(Property.TSERV_INDEXCACHE_SIZE);
    
    _iCache = new LruBlockCache(iCacheSize, blockSize);
    _dCache = new LruBlockCache(dCacheSize, blockSize);
    
    Runtime runtime = Runtime.getRuntime();
    if (!usingNativeMap && maxMemory + dCacheSize + iCacheSize > runtime.maxMemory()) {
      throw new IllegalArgumentException(String.format(
          "Maximum tablet server map memory %,d and block cache sizes %,d is too large for this JVM configuration %,d", maxMemory, dCacheSize + iCacheSize,

View Full Code Here

    
    long blockSize = acuConf.getMemoryInBytes(Property.TSERV_DEFAULT_BLOCKSIZE);
    long dCacheSize = acuConf.getMemoryInBytes(Property.TSERV_DATACACHE_SIZE);
    long iCacheSize = acuConf.getMemoryInBytes(Property.TSERV_INDEXCACHE_SIZE);
    
    _iCache = new LruBlockCache(iCacheSize, blockSize);
    _dCache = new LruBlockCache(dCacheSize, blockSize);
    
    Runtime runtime = Runtime.getRuntime();
    if (!usingNativeMap && maxMemory + dCacheSize + iCacheSize > runtime.maxMemory()) {
      throw new IllegalArgumentException(String.format(
          "Maximum tablet server map memory %,d and block cache sizes %,d is too large for this JVM configuration %,d", maxMemory, dCacheSize + iCacheSize,

View Full Code Here

    
    long blockSize = acuConf.getMemoryInBytes(Property.TSERV_DEFAULT_BLOCKSIZE);
    long dCacheSize = acuConf.getMemoryInBytes(Property.TSERV_DATACACHE_SIZE);
    long iCacheSize = acuConf.getMemoryInBytes(Property.TSERV_INDEXCACHE_SIZE);
    
    _iCache = new LruBlockCache(iCacheSize, blockSize);
    _dCache = new LruBlockCache(dCacheSize, blockSize);
    
    Runtime runtime = Runtime.getRuntime();
    if (!usingNativeMap && maxMemory + dCacheSize + iCacheSize > runtime.maxMemory()) {
      throw new IllegalArgumentException(String.format(
          "Maximum tablet server map memory %,d and block cache sizes %,d is too large for this JVM configuration %,d", maxMemory, dCacheSize + iCacheSize,

View Full Code Here

    public void openReader() throws IOException {
      byte[] data = baos.toByteArray();
      bais = new SeekableByteArrayInputStream(data);
      in = new FSDataInputStream(bais);
      
      LruBlockCache indexCache = new LruBlockCache(100000000, 100000);
      LruBlockCache dataCache = new LruBlockCache(100000000, 100000);
      
      CachableBlockFile.Reader _cbr = new CachableBlockFile.Reader(in, data.length, conf, dataCache, indexCache);
      reader = new RFile.Reader(_cbr);
      iter = new ColumnFamilySkippingIterator(reader);

View Full Code Here


      bais = new SeekableByteArrayInputStream(data);
      in = new FSDataInputStream(bais);
      fileLength = data.length;


      LruBlockCache indexCache = new LruBlockCache(100000000, 100000);
      LruBlockCache dataCache = new LruBlockCache(100000000, 100000);


      CachableBlockFile.Reader _cbr = new CachableBlockFile.Reader(in, fileLength, conf, dataCache, indexCache, AccumuloConfiguration.getDefaultConfiguration());
      reader = new RFile.Reader(_cbr);
      iter = new ColumnFamilySkippingIterator(reader);

View Full Code Here


    long blockSize = acuConf.getMemoryInBytes(Property.TSERV_DEFAULT_BLOCKSIZE);
    long dCacheSize = acuConf.getMemoryInBytes(Property.TSERV_DATACACHE_SIZE);
    long iCacheSize = acuConf.getMemoryInBytes(Property.TSERV_INDEXCACHE_SIZE);


    _iCache = new LruBlockCache(iCacheSize, blockSize);
    _dCache = new LruBlockCache(dCacheSize, blockSize);


    Runtime runtime = Runtime.getRuntime();
    if (!usingNativeMap && maxMemory + dCacheSize + iCacheSize > runtime.maxMemory()) {
      throw new IllegalArgumentException(String.format(
          "Maximum tablet server map memory %,d and block cache sizes %,d is too large for this JVM configuration %,d", maxMemory, dCacheSize + iCacheSize,

View Full Code Here

  public void testBackgroundEvictionThread() throws Exception {
    
    long maxSize = 100000;
    long blockSize = calculateBlockSizeDefault(maxSize, 9); // room for 9, will evict
    
    LruBlockCache cache = new LruBlockCache(maxSize, blockSize);
    
    Block[] blocks = generateFixedBlocks(10, blockSize, "block");
    
    // Add all the blocks
    for (Block block : blocks) {
      cache.cacheBlock(block.blockName, block.buf);
    }
    
    // Let the eviction run
    int n = 0;
    while (cache.getEvictionCount() == 0) {
      Thread.sleep(1000);
      assertTrue(n++ < 1);
    }
    // A single eviction run should have occurred
    assertEquals(cache.getEvictionCount(), 1);
  }

View Full Code Here

  public void testCacheSimple() throws Exception {
    
    long maxSize = 1000000;
    long blockSize = calculateBlockSizeDefault(maxSize, 101);
    
    LruBlockCache cache = new LruBlockCache(maxSize, blockSize);
    
    Block[] blocks = generateRandomBlocks(100, blockSize);
    
    long expectedCacheSize = cache.heapSize();
    
    // Confirm empty
    for (Block block : blocks) {
      assertTrue(cache.getBlock(block.blockName) == null);
    }
    
    // Add blocks
    for (Block block : blocks) {
      cache.cacheBlock(block.blockName, block.buf);
      expectedCacheSize += block.heapSize();
    }
    
    // Verify correctly calculated cache heap size
    assertEquals(expectedCacheSize, cache.heapSize());
    
    // Check if all blocks are properly cached and retrieved
    for (Block block : blocks) {
      byte buf1[] = cache.getBlock(block.blockName);
      assertTrue(buf1 != null);
      assertEquals(buf1.length, block.buf.length);
    }
    
    // Verify correctly calculated cache heap size
    assertEquals(expectedCacheSize, cache.heapSize());
    
    // Check if all blocks are properly cached and retrieved
    for (Block block : blocks) {
      byte buf1[] = cache.getBlock(block.blockName);
      assertTrue(buf1 != null);
      assertEquals(buf1.length, block.buf.length);
    }
    
    // Expect no evictions
    assertEquals(0, cache.getEvictionCount());
    // Thread t = new LruBlockCache.StatisticsThread(cache);
    // t.start();
    // t.join();
  }

View Full Code Here

  public void testCacheEvictionSimple() throws Exception {
    
    long maxSize = 100000;
    long blockSize = calculateBlockSizeDefault(maxSize, 10);
    
    LruBlockCache cache = new LruBlockCache(maxSize, blockSize, false);
    
    Block[] blocks = generateFixedBlocks(10, blockSize, "block");
    
    long expectedCacheSize = cache.heapSize();
    
    // Add all the blocks
    for (Block block : blocks) {
      cache.cacheBlock(block.blockName, block.buf);
      expectedCacheSize += block.heapSize();
    }
    
    // A single eviction run should have occurred
    assertEquals(1, cache.getEvictionCount());
    
    // Our expected size overruns acceptable limit
    assertTrue(expectedCacheSize > (maxSize * LruBlockCache.DEFAULT_ACCEPTABLE_FACTOR));
    
    // But the cache did not grow beyond max
    assertTrue(cache.heapSize() < maxSize);
    
    // And is still below the acceptable limit
    assertTrue(cache.heapSize() < (maxSize * LruBlockCache.DEFAULT_ACCEPTABLE_FACTOR));
    
    // All blocks except block 0 and 1 should be in the cache
    assertTrue(cache.getBlock(blocks[0].blockName) == null);
    assertTrue(cache.getBlock(blocks[1].blockName) == null);
    for (int i = 2; i < blocks.length; i++) {
      assertEquals(cache.getBlock(blocks[i].blockName), blocks[i].buf);
    }
  }

View Full Code Here

0 1 2

TOP

Related Classes of org.apache.accumulo.core.file.blockfile.cache.LruBlockCache

org.apache.accumulo.core.file.blockfile.cache.TestLruBlockCache

org.apache.accumulo.core.file.rfile.RFileTest$TestRFile

org.apache.accumulo.server.tabletserver.TabletServerResourceManager

org.apache.accumulo.tserver.TabletServerResourceManager

java.util.concurrent.atomic.AtomicLong

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.