Examples of HLog

org.apache.hadoop.hbase.HLog
HLog stores all the edits to the HStore. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
A single HLog is used by several HRegions simultaneously.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the {@link #append(Text,Text,Text,TreeMap,long)} or {@link #completeCacheFlush(Text,Text,long)} calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
TODO: Vuk Ercegovac also pointed out that keeping HBase HRegion edit logs in HDFS is currently flawed. HBase writes edits to logs and to a memcache. The 'atomic' write to the log is meant to serve as insurance against abnormal RegionServer exit: on startup, the log is rerun to reconstruct an HRegion's last wholesome state. But files in HDFS do not 'exist' until they are cleanly closed -- something that will not happen if RegionServer exits without running its 'close'.
org.apache.hadoop.hbase.regionserver.HLog
HLog stores all the edits to the HStore. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
A single HLog is used by several HRegions simultaneously.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the append or completeCacheFlush calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
Synchronized methods can never execute in parallel. However, between the start of a cache flush and the completion point, appends are allowed but log rolling is not. To prevent log rolling taking place during this period, a separate reentrant lock is used.
org.apache.hadoop.hbase.regionserver.wal.HLog
HLog stores all the edits to the HStore. Its the hbase write-ahead-log implementation. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.
There is one HLog per RegionServer. All edits for all Regions carried by a particular RegionServer are entered first in the HLog.
Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the append or completeCacheFlush calls.
An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.
It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.
Synchronized methods can never execute in parallel. However, between the start of a cache flush and the completion point, appends are allowed but log rolling is not. To prevent log rolling taking place during this period, a separate reentrant lock is used.
To read an HLog, call {@link #getReader(org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,org.apache.hadoop.conf.Configuration)}.

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    }




    ServerName fakeServer = new ServerName("nsupgrade",96,123);
    String metaLogName = HLogUtil.getHLogDirectoryName(fakeServer.toString());
    HLog metaHLog = HLogFactory.createMetaHLog(fs, rootDir,
        metaLogName, conf, null,
        fakeServer.toString());
    HRegion meta = HRegion.openHRegion(rootDir, HRegionInfo.FIRST_META_REGIONINFO,
        HTableDescriptor.META_TABLEDESC, metaHLog, conf);
    HRegion region = null;
    try {
      for(Path regionDir : FSUtils.getRegionDirs(fs, oldTablePath)) {
        LOG.info("Migrating ACL region "+regionDir.getName());
        HRegionInfo oldRegionInfo = HRegionFileSystem.loadRegionInfoFileContent(fs, regionDir);
        HRegionInfo newRegionInfo =
            new HRegionInfo(newTableName,
                oldRegionInfo.getStartKey(),
                oldRegionInfo.getEndKey(),
                oldRegionInfo.isSplit(),
                oldRegionInfo.getRegionId());
        newRegionInfo.setOffline(oldRegionInfo.isOffline());
        region =
            new HRegion(
                HRegionFileSystem.openRegionFromFileSystem(conf, fs, oldTablePath,
                    oldRegionInfo, false),
                metaHLog,
                conf,
                oldDesc,
                null);
        region.initialize();
        //Run major compaction to archive old stores
        //to keep any snapshots to _acl_ unbroken
        region.compactStores(true);
        region.waitForFlushesAndCompactions();
        region.close();


        //Create new region dir
        Path newRegionDir = new Path(newTablePath, newRegionInfo.getEncodedName());
        if(!fs.exists(newRegionDir)) {
          if(!fs.mkdirs(newRegionDir)) {
            throw new IllegalStateException("Failed to create new region dir: " + newRegionDir);
          }
        }


        //create new region info file, delete in case one exists
        HRegionFileSystem.openRegionFromFileSystem(conf, fs, newTablePath, newRegionInfo, false);


        //migrate region contents
        for(FileStatus file : fs.listStatus(regionDir, new FSUtils.UserTableDirFilter(fs)))  {
          if(file.getPath().getName().equals(HRegionFileSystem.REGION_INFO_FILE))
            continue;
          if(!fs.rename(file.getPath(), newRegionDir))  {
            throw new IllegalStateException("Failed to move file "+file.getPath()+" to " +
                newRegionDir);
          }
        }
        meta.put(MetaEditor.makePutFromRegionInfo(newRegionInfo));
        meta.delete(MetaEditor.makeDeleteFromRegionInfo(oldRegionInfo));
      }
    } finally {
      meta.flushcache();
      meta.waitForFlushesAndCompactions();
      meta.close();
      metaHLog.closeAndDelete();
      if(region != null) {
        region.close();
      }
    }
    if(!fs.rename(oldTablePath, backupDir)) {

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

  @Override
  public boolean removeFromOnlineRegions(final HRegion r, ServerName destination) {
    HRegion toReturn = this.onlineRegions.remove(r.getRegionInfo().getEncodedName());


    if (destination != null) {
      HLog wal = getWAL();
      long closeSeqNum = wal.getEarliestMemstoreSeqNum(r.getRegionInfo().getEncodedNameAsBytes());
      if (closeSeqNum == HConstants.NO_SEQNUM) {
        // No edits in WAL for this region; get the sequence number when the region was opened.
        closeSeqNum = r.getOpenSeqNum();
        if (closeSeqNum == HConstants.NO_SEQNUM) {
          closeSeqNum = 0;

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

  @Override
  public RollWALWriterResponse rollWALWriter(final RpcController controller,
      final RollWALWriterRequest request) throws ServiceException {
    try {
      requestCount.increment();
      HLog wal = this.getWAL();
      byte[][] regionsToFlush = wal.rollWriter(true);
      RollWALWriterResponse.Builder builder = RollWALWriterResponse.newBuilder();
      if (regionsToFlush != null) {
        for (byte[] region: regionsToFlush) {
          builder.addRegionToFlush(ByteString.copyFrom(region));
        }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    numRows -= 2;
  }


  @After
  public void tearDown() throws Exception {
    HLog hlog = region.getLog();
    region.close();
    hlog.closeAndDelete();
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

      assertTrue(CellUtil.matchingRow(results.get(0), Bytes.toBytes("row" + i)));
      assertEquals(Bytes.toInt(CellUtil.cloneValue(results.get(0))), i%2);
      results.clear();
    }
    assertFalse(scanner.next(results));
    HLog hlog = testRegion.getLog();
    testRegion.close();
    hlog.closeAndDelete();
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    WALEdit edit = new WALEdit();
    edit.add(kv);


    List<WALActionsListener> listeners = new ArrayList<WALActionsListener>();
    listeners.add(replication);
    HLog hlog = HLogFactory.createHLog(fs, utility.getDataTestDir(), logName,
        conf, listeners, URLEncoder.encode("regionserver:60020", "UTF8"));


    manager.init();
    HTableDescriptor htd = new HTableDescriptor();
    htd.addFamily(new HColumnDescriptor(f1));
    // Testing normal log rolling every 20
    for(long i = 1; i < 101; i++) {
      if(i > 1 && i % 20 == 0) {
        hlog.rollWriter();
      }
      LOG.info(i);
      HLogKey key = new HLogKey(hri.getRegionName(), test, seq++,
          System.currentTimeMillis(), HConstants.DEFAULT_CLUSTER_ID);
      hlog.append(hri, test, edit, System.currentTimeMillis(), htd);
    }


    // Simulate a rapid insert that's followed
    // by a report that's still not totally complete (missing last one)
    LOG.info(baseline + " and " + time);
    baseline += 101;
    time = baseline;
    LOG.info(baseline + " and " + time);


    for (int i = 0; i < 3; i++) {
      hlog.append(hri, test, edit, System.currentTimeMillis(), htd);
    }


    assertEquals(6, manager.getHLogs().get(slaveId).size());


    hlog.rollWriter();


    manager.logPositionAndCleanOldLogs(manager.getSources().get(0).getCurrentPath(),
        "1", 0, false, false);


    hlog.append(hri, test, edit, System.currentTimeMillis(), htd);


    assertEquals(1, manager.getHLogs().size());




    // TODO Need a case with only 2 HLogs and we only want to delete the first one

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    TEST_UTIL.createTable(tableName, Bytes.toBytes("cf"));
    try {
      HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName);
      try {
        HRegionServer server = TEST_UTIL.getRSForFirstRegionInTable(tableName);
        HLog log = server.getWAL();
        checkMinLogRolls(log, 5);
      } finally {
        table.close();
      }
    } finally {

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    final byte[] family = Bytes.toBytes("cf");


    TEST_UTIL.createTable(tableName, family);
    try {
      HRegionServer server = TEST_UTIL.getRSForFirstRegionInTable(tableName);
      HLog log = server.getWAL();
      final HTable table = new HTable(TEST_UTIL.getConfiguration(), tableName);


      Thread writerThread = new Thread("writer") {
        @Override
        public void run() {

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog

    Path basedir = new Path(DIR);
    Path logdir = new Path(DIR+"/logs");
    Path oldLogDir = new Path(basedir, HConstants.HREGION_OLDLOGDIR_NAME);
    fs.delete(logdir, true);
    HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false);
    HLog hlog = new HLog(fs, logdir, oldLogDir, conf);
    HRegion region = new HRegion(basedir, hlog, fs, conf, info, htd, null);
    store = new Store(basedir, region, hcd, fs, conf);
  }

View Full Code Here

Examples of org.apache.hadoop.hbase.regionserver.wal.HLog


    HTableDescriptor htd = new HTableDescriptor(Bytes.toBytes("table"));
    htd.addFamily(hcd);
    HRegionInfo info = new HRegionInfo(htd.getName(), null, null, false);


    HLog hlog = new HLog(fs, logdir, oldLogDir, conf);
    HRegion region = HRegion.createHRegion(info, basedir, conf, htd);
    HRegion.closeHRegion(region);
    Path tableDir = new Path(basedir, Bytes.toString(htd.getName()));
    region = new HRegion(tableDir, hlog, fs, conf, info, htd, null);

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.