Examples of HLog

org.apache.hadoop.hbase.HLog
HLog stores all the edits to the HStore. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
A single HLog is used by several HRegions simultaneously.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the {@link #append(Text,Text,Text,TreeMap,long)} or {@link #completeCacheFlush(Text,Text,long)} calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
TODO: Vuk Ercegovac also pointed out that keeping HBase HRegion edit logs in HDFS is currently flawed. HBase writes edits to logs and to a memcache. The 'atomic' write to the log is meant to serve as insurance against abnormal RegionServer exit: on startup, the log is rerun to reconstruct an HRegion's last wholesome state. But files in HDFS do not 'exist' until they are cleanly closed -- something that will not happen if RegionServer exits without running its 'close'.
org.apache.hadoop.hbase.regionserver.HLog
HLog stores all the edits to the HStore. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
A single HLog is used by several HRegions simultaneously.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the append or completeCacheFlush calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
Synchronized methods can never execute in parallel. However, between the start of a cache flush and the completion point, appends are allowed but log rolling is not. To prevent log rolling taking place during this period, a separate reentrant lock is used.
org.apache.hadoop.hbase.regionserver.wal.HLog
HLog stores all the edits to the HStore. Its the hbase write-ahead-log implementation. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
There is one HLog per RegionServer. All edits for all Regions carried by a particular RegionServer are entered first in the HLog.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the append or completeCacheFlush calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
Synchronized methods can never execute in parallel. However, between the start of a cache flush and the completion point, appends are allowed but log rolling is not. To prevent log rolling taking place during this period, a separate reentrant lock is used.
To read an HLog, call {@link #getReader(org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,org.apache.hadoop.conf.Configuration)}.

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

  private static final byte [] CF = HConstants.CATALOG_FAMILY;
  
  @Before public void setup() throws IOException {
    this.fs = FileSystem.get(TEST_UTIL.getConfiguration());
    this.fs.delete(this.testdir, true);
    this.wal = new HLog(fs, new Path(this.testdir, "logs"),
      new Path(this.testdir, "archive"),
      TEST_UTIL.getConfiguration());
    this.parent = createRegion(this.testdir, this.wal);
    TEST_UTIL.getConfiguration().setBoolean("hbase.testing.nocluster", true);
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    Path regionDir = HRegion.getRegionDir(tableDir, info.getEncodedName());
    FileSystem fs = FileSystem.get(conf);
    HBaseFileSystem.makeDirOnFileSystem(fs, regionDir);
    // Write HRI to a file in case we need to recover .META.
    writeRegioninfoOnFilesystem(info, regionDir, fs, conf);
    HLog effectiveHLog = hlog;
    if (hlog == null && !ignoreHLog) {
      effectiveHLog = new HLog(fs, new Path(regionDir, HConstants.HREGION_LOGDIR_NAME),
          new Path(regionDir, HConstants.HREGION_OLDLOGDIR_NAME), conf);
    }
    HRegion region = HRegion.newHRegion(tableDir,
        effectiveHLog, fs, conf, info, hTableDescriptor, null);
    if (initialize) {

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

      listPaths(fs, b.getRegionDir());
    }


    Configuration conf = a.getBaseConf();
    HTableDescriptor tabledesc = a.getTableDesc();
    HLog log = a.getLog();
    Path tableDir = a.getTableDir();
    // Presume both are of same region type -- i.e. both user or catalog
    // table regions.  This way can use comparator.
    final byte[] startKey =
      (a.comparator.matchingRows(a.getStartKey(), 0, a.getStartKey().length,

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    final Path logdir = new Path(c.get("hbase.tmp.dir"),
        "hlog" + tableDir.getName()
        + EnvironmentEdgeManager.currentTimeMillis());
    final Path oldLogDir = new Path(c.get("hbase.tmp.dir"),
        HConstants.HREGION_OLDLOGDIR_NAME);
    final HLog log = new HLog(fs, logdir, oldLogDir, c);
    try {
      processTable(fs, tableDir, log, c, majorCompact);
    } finally {
       log.close();
       // TODO: is this still right?
       BlockCache bc = new CacheConfig(c).getBlockCache();
       if (bc != null) bc.shutdown();
    }
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    WALEdit edit = new WALEdit();
    edit.add(kv);


    List<WALActionsListener> listeners = new ArrayList<WALActionsListener>();
    listeners.add(replication);
    HLog hlog = new HLog(fs, logDir, oldLogDir, conf, listeners,
      URLEncoder.encode("regionserver:60020", "UTF8"));


    manager.init();
    HTableDescriptor htd = new HTableDescriptor();
    htd.addFamily(new HColumnDescriptor(f1));
    // Testing normal log rolling every 20
    for(long i = 1; i < 101; i++) {
      if(i > 1 && i % 20 == 0) {
        hlog.rollWriter();
      }
      LOG.info(i);
      HLogKey key = new HLogKey(hri.getRegionName(), test, seq++,
          System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
      hlog.append(hri, key, edit, htd, true);
    }


    // Simulate a rapid insert that's followed
    // by a report that's still not totally complete (missing last one)
    LOG.info(baseline + " and " + time);
    baseline += 101;
    time = baseline;
    LOG.info(baseline + " and " + time);


    for (int i = 0; i < 3; i++) {
      HLogKey key = new HLogKey(hri.getRegionName(), test, seq++,
          System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
      hlog.append(hri, key, edit, htd, true);
    }


    assertEquals(6, manager.getHLogs().get(slaveId).size());


    hlog.rollWriter();


    manager.logPositionAndCleanOldLogs(manager.getSources().get(0).getCurrentPath(),
        "1", 0, false, false);


    HLogKey key = new HLogKey(hri.getRegionName(), test, seq++,
        System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
    hlog.append(hri, key, edit, htd, true);


    assertEquals(1, manager.getHLogs().size());




    // TODO Need a case with only 2 HLogs and we only want to delete the first one

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    fs.delete(logdir, true);


    HTableDescriptor htd = new HTableDescriptor(table);
    htd.addFamily(hcd);
    HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false);
    HLog hlog = new HLog(fs, logdir, oldLogDir, conf);
    HRegion region = new HRegion(basedir, hlog, fs, conf, info, htd, null);


    store = new Store(basedir, region, hcd, fs, conf);
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

      = HLog.getHLogDirectoryName(this.serverNameFromMasterPOV.toString());


      Path logdir = new Path(rootDir, logName);
      final Path oldLogDir = new Path(rootDir, HConstants.HREGION_OLDLOGDIR_NAME);
      if (LOG.isDebugEnabled()) LOG.debug("logdir=" + logdir);
      this.hlogForMeta = new HLog(this.fs.getBackingFs(), logdir, oldLogDir, this.conf,
          getMetaWALActionListeners(), false, this.serverNameFromMasterPOV.toString(), true);
    }
    return this.hlogForMeta;
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

   * @param oldLogDir
   * @return WAL instance.
   * @throws IOException
   */
  protected HLog instantiateHLog(Path logdir, Path oldLogDir) throws IOException {
    return new HLog(this.fs.getBackingFs(), logdir, oldLogDir, this.conf,
      getWALActionListeners(), this.serverNameFromMasterPOV.toString());
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    return c.getBlockCacheColumnFamilySummaries(this.conf);
  }


  @Override
  public byte[][] rollHLogWriter() throws IOException, FailedLogCloseException {
    HLog wal = this.getWAL();
    return wal.rollWriter(true);
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    }
    numRows -= 2;
  }


  protected void tearDown() throws Exception {
    HLog hlog = region.getLog();
    region.close();
    hlog.closeAndDelete();
    super.tearDown();
  }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.