* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements.  See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership.  The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License.  You may obtain a copy of the License at

*

*     http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/

* Implementation of {@link HLog} to go against {@link FileSystem}; i.e. keep WALs in HDFS.

* Only one HLog/WAL is ever being written at a time.  When a WAL hits a configured maximum size,

* it is rolled.  This is done internal to the implementation.

*

* <p>As data is flushed from the MemStore to other on-disk structures (files sorted by

* key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given

* HRegion-sequence id.  A bunch of work in the below is done keeping account of these region

* sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.

*

* <p>It is only practical to delete entire files. Thus, we delete an entire on-disk file

* <code>F</code> when all of the edits in <code>F</code> have a log-sequence-id that's older

* (smaller) than the most-recent flush.

*

* <p>To read an HLog, call {@link HLogFactory#createReader(org.apache.hadoop.fs.FileSystem,

* org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration)}.

*/

  // IMPLEMENTATION NOTES:

  //

  // At the core is a ring buffer.  Our ring buffer is the LMAX Disruptor.  It tries to

  // minimize synchronizations and volatile writes when multiple contending threads as is the case

  // here appending and syncing on a single WAL.  The Disruptor is configured to handle multiple

  // producers but it has one consumer only (the producers in HBase are IPC Handlers calling append

  // and then sync).  The single consumer/writer pulls the appends and syncs off the ring buffer.

  // When a handler calls sync, it is given back a future. The producer 'blocks' on the future so

  // it does not return until the sync completes.  The future is passed over the ring buffer from

  // the producer/handler to the consumer thread where it does its best to batch up the producer

  // syncs so one WAL sync actually spans multiple producer sync invocations.  How well the

  // batching works depends on the write rate; i.e. we tend to batch more in times of

  // high writes/syncs.

  //

  // Calls to append now also wait until the append has been done on the consumer side of the

  // disruptor.  We used to not wait but it makes the implemenation easier to grok if we have

  // the region edit/sequence id after the append returns.

  //

  // TODO: Handlers need to coordinate appending AND syncing.  Can we have the threads contend

  // once only?  Probably hard given syncs take way longer than an append.

  //

  // The consumer threads pass the syncs off to multiple syncing threads in a round robin fashion

  // to ensure we keep up back-to-back FS sync calls (FS sync calls are the long poll writing the

  // WAL).  The consumer thread passes the futures to the sync threads for it to complete

  // the futures when done.

  //

  // The 'sequence' in the below is the sequence of the append/sync on the ringbuffer.  It

  // acts as a sort-of transaction id.  It is always incrementing.

  //

  // The RingBufferEventHandler class hosts the ring buffer consuming code.  The threads that

  // do the actual FS sync are implementations of SyncRunner.  SafePointZigZagLatch is a

  // synchronization class used to halt the consumer at a safe point --  just after all outstanding

  // syncs and appends have completed -- so the log roller can swap the WAL out under it.

  static final Log LOG = LogFactory.getLog(FSHLog.class);

  private static final int DEFAULT_SLOW_SYNC_TIME_MS = 100; // in ms

 

  /**

   * The nexus at which all incoming handlers meet.  Does appends and sync with an ordering.

   * Appends and syncs are each put on the ring which means handlers need to

   * smash up against the ring twice (can we make it once only? ... maybe not since time to append

   * is so different from time to sync and sometimes we don't want to sync or we want to async

   * the sync).  The ring is where we make sure of our ordering and it is also where we do

   * batching up of handler sync calls.

   */

  private final Disruptor<RingBufferTruck> disruptor;

  /**

   * An executorservice that runs the disrutpor AppendEventHandler append executor.

   */

  private final ExecutorService appendExecutor;

  /**

   * This fellow is run by the above appendExecutor service but it is all about batching up appends

   * and syncs; it may shutdown without cleaning out the last few appends or syncs.  To guard

   * against this, keep a reference to this handler and do explicit close on way out to make sure

   * all flushed out before we exit.

   */

  private final RingBufferEventHandler ringBufferEventHandler;

  /**

   * Map of {@link SyncFuture}s keyed by Handler objects.  Used so we reuse SyncFutures.

   * TODO: Reus FSWALEntry's rather than create them anew each time as we do SyncFutures here.

   * TODO: Add a FSWalEntry and SyncFuture as thread locals on handlers rather than have them

   * get them from this Map?

   */

  private final Map<Thread, SyncFuture> syncFuturesByHandler;

  private final FileSystem fs;

  private final Path fullPathLogDir;

  private final Path fullPathOldLogDir;

  private final Configuration conf;

  private final String logFilePrefix;

  /**

   * The highest known outstanding unsync'd WALEdit sequence number where sequence number is the

   * ring buffer sequence.  Maintained by the ring buffer consumer.

   */

  private volatile long highestUnsyncedSequence = -1;

  /**

   * Updated to the ring buffer sequence of the last successful sync call.  This can be less than

   * {@link #highestUnsyncedSequence} for case where we have an append where a sync has not yet

   * come in for it.  Maintained by the syncing threads.

   */

  private final AtomicLong highestSyncedSequence = new AtomicLong(0);

  private WALCoprocessorHost coprocessorHost;

  /**

   * FSDataOutputStream associated with the current SequenceFile.writer

   */

  private FSDataOutputStream hdfs_out;

  // All about log rolling if not enough replicas outstanding.

  // Minimum tolerable replicas, if the actual value is lower than it, rollWriter will be triggered

  private final int minTolerableReplication;

  // DFSOutputStream.getNumCurrentReplicas method instance gotten via reflection.

  private final Method getNumCurrentReplicas;

  private final Method getPipeLine; // refers to DFSOutputStream.getPipeLine

  private final int slowSyncNs;

  private final static Object [] NO_ARGS = new Object []{};

  // If live datanode count is lower than the default replicas value,

  // RollWriter will be triggered in each sync(So the RollWriter will be

  // triggered one by one in a short time). Using it as a workaround to slow

  // down the roll frequency triggered by checkLowReplication().

  private final AtomicInteger consecutiveLogRolls = new AtomicInteger(0);

  private final int lowReplicationRollLimit;

  // If consecutiveLogRolls is larger than lowReplicationRollLimit,

  // then disable the rolling in checkLowReplication().

  // Enable it if the replications recover.

  private volatile boolean lowReplicationRollEnabled = true;

  /**

   * Current log file.

   */

  volatile Writer writer;

  /** The barrier used to ensure that close() waits for all log rolls and flushes to finish. */

  private final DrainBarrier closeBarrier = new DrainBarrier();

  /**

   * This lock makes sure only one log roll runs at a time. Should not be taken while any other

   * lock is held. We don't just use synchronized because that results in bogus and tedious

   * findbugs warning when it thinks synchronized controls writer thread safety.  It is held when

   * we are actually rolling the log.  It is checked when we are looking to see if we should roll

   * the log or not.

   */

  private final ReentrantLock rollWriterLock = new ReentrantLock(true);

  // Listeners that are called on WAL events.

  private final List<WALActionsListener> listeners =

    new CopyOnWriteArrayList<WALActionsListener>();

  private volatile boolean closed = false;

  /**

   * Set when this WAL is for meta only (we run a WAL for all regions except meta -- it has its

   * own dedicated WAL).

   */

  private final boolean forMeta;

  // The timestamp (in ms) when the log file was created.

  private final AtomicLong filenum = new AtomicLong(-1);

  // Number of transactions in the current Hlog.

  private final AtomicInteger numEntries = new AtomicInteger(0);

  // If > than this size, roll the log.

  private final long logrollsize;

  /**

   * The total size of hlog

   */

  private AtomicLong totalLogSize = new AtomicLong(0);

  /**

   * If WAL is enabled.

   */

  private final boolean enabled;

  /*

   * If more than this many logs, force flush of oldest region to oldest edit

   * goes to disk.  If too many and we crash, then will take forever replaying.

   * Keep the number of logs tidy.

   */

  private final int maxLogs;

  /** Number of log close errors tolerated before we abort */

  private final int closeErrorsTolerated;

  private final AtomicInteger closeErrorCount = new AtomicInteger();

  private final MetricsWAL metrics;

  // Region sequence id accounting across flushes and for knowing when we can GC a WAL.  These

  // sequence id numbers are by region and unrelated to the ring buffer sequence number accounting

  // done above in failedSequence, highest sequence, etc.

  /**

   * This lock ties all operations on oldestFlushingRegionSequenceIds and

   * oldestFlushedRegionSequenceIds Maps with the exception of append's putIfAbsent call into

   * oldestUnflushedSeqNums. We use these Maps to find out the low bound regions sequence id, or

   * to find regions  with old sequence ids to force flush; we are interested in old stuff not the

   * new additions (TODO: IS THIS SAFE?  CHECK!).

   */

  private final Object regionSequenceIdLock = new Object();

  /**

   * Map of encoded region names to their OLDEST -- i.e. their first, the longest-lived --

   * sequence id in memstore. Note that this sequence id is the region sequence id.  This is not

   * related to the id we use above for {@link #highestSyncedSequence} and

   * {@link #highestUnsyncedSequence} which is the sequence from the disruptor ring buffer.

   */

  private final ConcurrentSkipListMap<byte [], Long> oldestUnflushedRegionSequenceIds =

    new ConcurrentSkipListMap<byte [], Long>(Bytes.BYTES_COMPARATOR);

  /**

   * Map of encoded region names to their lowest or OLDEST sequence/edit id in memstore currently

   * being flushed out to hfiles. Entries are moved here from

   * {@link #oldestUnflushedRegionSequenceIds} while the lock {@link #regionSequenceIdLock} is held

   * (so movement between the Maps is atomic). This is not related to the id we use above for

   * {@link #highestSyncedSequence} and {@link #highestUnsyncedSequence} which is the sequence from

   * the disruptor ring buffer, an internal detail.

   */

  private final Map<byte[], Long> lowestFlushingRegionSequenceIds =

    new TreeMap<byte[], Long>(Bytes.BYTES_COMPARATOR);

/**

  * Map of region encoded names to the latest region sequence id.  Updated on each append of

  * WALEdits to the WAL. We create one map for each WAL file at the time it is rolled.

  * <p>When deciding whether to archive a WAL file, we compare the sequence IDs in this map to

  * {@link #lowestFlushingRegionSequenceIds} and {@link #oldestUnflushedRegionSequenceIds}.

  * See {@link FSHLog#areAllRegionsFlushed(Map, Map, Map)} for more info.

  * <p>

  * This map uses byte[] as the key, and uses reference equality. It works in our use case as we

  * use {@link HRegionInfo#getEncodedNameAsBytes()} as keys. For a given region, it always returns

  * the same array.

  */

  private Map<byte[], Long> highestRegionSequenceIds = new HashMap<byte[], Long>();

  /**

   * WAL Comparator; it compares the timestamp (log filenum), present in the log file name.

   */

  public final Comparator<Path> LOG_NAME_COMPARATOR = new Comparator<Path>() {

    @Override

    public int compare(Path o1, Path o2) {

      long t1 = getFileNumFromFileName(o1);

      long t2 = getFileNumFromFileName(o2);

      if (t1 == t2) return 0;

      return (t1 > t2) ? 1 : -1;

    }

  };

  /**

   * Map of wal log file to the latest sequence ids of all regions it has entries of.

   * The map is sorted by the log file creation timestamp (contained in the log file name).

   */

  private NavigableMap<Path, Map<byte[], Long>> byWalRegionSequenceIds =

    new ConcurrentSkipListMap<Path, Map<byte[], Long>>(LOG_NAME_COMPARATOR);

  /**

   * Exception handler to pass the disruptor ringbuffer.  Same as native implementation only it

   * logs using our logger instead of java native logger.

   */

  static class RingBufferExceptionHandler implements ExceptionHandler {

    @Override

    public void handleEventException(Throwable ex, long sequence, Object event) {

      LOG.error("Sequence=" + sequence + ", event=" + event, ex);

      throw new RuntimeException(ex);

    }

    @Override

    public void handleOnStartException(Throwable ex) {

      LOG.error(ex);

      throw new RuntimeException(ex);

    }

    @Override

    public void handleOnShutdownException(Throwable ex) {

      LOG.error(ex);

      throw new RuntimeException(ex);

    }

  }

  /**

   * Constructor.

   *

   * @param fs filesystem handle

   * @param root path for stored and archived hlogs

   * @param logDir dir where hlogs are stored

   * @param conf configuration to use

   * @throws IOException

   */

  public FSHLog(final FileSystem fs, final Path root, final String logDir,

    final Configuration conf)

  throws IOException {

    this(fs, root, logDir, HConstants.HREGION_OLDLOGDIR_NAME, conf, null, true, null, false);

  }

  /**

   * Create an edit log at the given <code>dir</code> location.

   *

   * You should never have to load an existing log. If there is a log at

   * startup, it should have already been processed and deleted by the time the

   * HLog object is started up.

   *

   * @param fs filesystem handle

   * @param rootDir path to where logs and oldlogs

   * @param logDir dir where hlogs are stored

   * @param oldLogDir dir where hlogs are archived

   * @param conf configuration to use

   * @param listeners Listeners on WAL events. Listeners passed here will

   * be registered before we do anything else; e.g. the

   * Constructor {@link #rollWriter()}.

   * @param failIfLogDirExists If true IOException will be thrown if dir already exists.

   * @param prefix should always be hostname and port in distributed env and

   *        it will be URL encoded before being used.

   *        If prefix is null, "hlog" will be used

   * @param forMeta if this hlog is meant for meta updates

   * @throws IOException

   */

  public FSHLog(final FileSystem fs, final Path rootDir, final String logDir,

      final String oldLogDir, final Configuration conf,

      final List<WALActionsListener> listeners,

      final boolean failIfLogDirExists, final String prefix, boolean forMeta)

  throws IOException {

    super();

    this.fs = fs;

    this.fullPathLogDir = new Path(rootDir, logDir);

    this.fullPathOldLogDir = new Path(rootDir, oldLogDir);

    this.forMeta = forMeta;

    this.conf = conf;

    // Register listeners.  TODO: Should this exist anymore?  We have CPs?

    if (listeners != null) {

      for (WALActionsListener i: listeners) {

        registerWALActionsListener(i);

      }

    }

    // Get size to roll log at. Roll at 95% of HDFS block size so we avoid crossing HDFS blocks

    // (it costs a little x'ing bocks)

    long blocksize = this.conf.getLong("hbase.regionserver.hlog.blocksize",

      FSUtils.getDefaultBlockSize(this.fs, this.fullPathLogDir));

    this.logrollsize =

      (long)(blocksize * conf.getFloat("hbase.regionserver.logroll.multiplier", 0.95f));

    this.maxLogs = conf.getInt("hbase.regionserver.maxlogs", 32);

    this.minTolerableReplication = conf.getInt( "hbase.regionserver.hlog.tolerable.lowreplication",

        FSUtils.getDefaultReplication(fs, this.fullPathLogDir));

    this.lowReplicationRollLimit =

      conf.getInt("hbase.regionserver.hlog.lowreplication.rolllimit", 5);

    this.enabled = conf.getBoolean("hbase.regionserver.hlog.enabled", true);

    this.closeErrorsTolerated = conf.getInt("hbase.regionserver.logroll.errors.tolerated", 0);

    // If prefix is null||empty then just name it hlog

    this.logFilePrefix =

      prefix == null || prefix.isEmpty() ? "hlog" : URLEncoder.encode(prefix, "UTF8");

    int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT, 200);

    LOG.info("WAL configuration: blocksize=" + StringUtils.byteDesc(blocksize) +

      ", rollsize=" + StringUtils.byteDesc(this.logrollsize) +

      ", enabled=" + this.enabled + ", prefix=" + this.logFilePrefix + ", logDir=" +

      this.fullPathLogDir + ", oldLogDir=" + this.fullPathOldLogDir);

    boolean dirExists = false;

    if (failIfLogDirExists && (dirExists = this.fs.exists(fullPathLogDir))) {

      throw new IOException("Target HLog directory already exists: " + fullPathLogDir);

    }

    if (!dirExists && !fs.mkdirs(fullPathLogDir)) {

      throw new IOException("Unable to mkdir " + fullPathLogDir);

    }

    if (!fs.exists(this.fullPathOldLogDir)) {

      if (!fs.mkdirs(this.fullPathOldLogDir)) {

        throw new IOException("Unable to mkdir " + this.fullPathOldLogDir);

      }

    }

    // rollWriter sets this.hdfs_out if it can.

    rollWriter();

    this.slowSyncNs =

        1000000 * conf.getInt("hbase.regionserver.hlog.slowsync.ms",

          DEFAULT_SLOW_SYNC_TIME_MS);

    // handle the reflection necessary to call getNumCurrentReplicas(). TODO: Replace with

    // HdfsDataOutputStream#getCurrentBlockReplication() and go without reflection.

    this.getNumCurrentReplicas = getGetNumCurrentReplicas(this.hdfs_out);

    this.getPipeLine = getGetPipeline(this.hdfs_out);

    this.coprocessorHost = new WALCoprocessorHost(this, conf);

    this.metrics = new MetricsWAL();

    // This is the 'writer' -- a single threaded executor.  This single thread 'consumes' what is

    // put on the ring buffer.

    String hostingThreadName = Thread.currentThread().getName();

    this.appendExecutor = Executors.

      newSingleThreadExecutor(Threads.getNamedThreadFactory(hostingThreadName + ".append"));

    // Preallocate objects to use on the ring buffer.  The way that appends and syncs work, we will

    // be stuck and make no progress if the buffer is filled with appends only and there is no

    // sync. If no sync, then the handlers will be outstanding just waiting on sync completion

    // before they return.

    final int preallocatedEventCount =

      this.conf.getInt("hbase.regionserver.wal.disruptor.event.count", 1024 * 16);

    // Using BlockingWaitStrategy.  Stuff that is going on here takes so long it makes no sense

    // spinning as other strategies do.

    this.disruptor =

      new Disruptor<RingBufferTruck>(RingBufferTruck.EVENT_FACTORY, preallocatedEventCount,

        this.appendExecutor, ProducerType.MULTI, new BlockingWaitStrategy());

    // Advance the ring buffer sequence so that it starts from 1 instead of 0,

    // because SyncFuture.NOT_DONE = 0.

    this.disruptor.getRingBuffer().next();

    this.ringBufferEventHandler =

      new RingBufferEventHandler(conf.getInt("hbase.regionserver.hlog.syncer.count", 5),

        maxHandlersCount);

    this.disruptor.handleExceptionsWith(new RingBufferExceptionHandler());

    this.disruptor.handleEventsWith(new RingBufferEventHandler [] {this.ringBufferEventHandler});

    // Presize our map of SyncFutures by handler objects.

    this.syncFuturesByHandler = new ConcurrentHashMap<Thread, SyncFuture>(maxHandlersCount);

    // Starting up threads in constructor is a no no; Interface should have an init call.

    this.disruptor.start();

  }

  /**

   * Find the 'getNumCurrentReplicas' on the passed <code>os</code> stream.

   * @return Method or null.

   */

  private static Method getGetNumCurrentReplicas(final FSDataOutputStream os) {

    // TODO: Remove all this and use the now publically available

    // HdfsDataOutputStream#getCurrentBlockReplication()

    Method m = null;

    if (os != null) {

      Class<? extends OutputStream> wrappedStreamClass = os.getWrappedStream().getClass();

      try {

        m = wrappedStreamClass.getDeclaredMethod("getNumCurrentReplicas", new Class<?>[] {});

        m.setAccessible(true);

      } catch (NoSuchMethodException e) {

        LOG.info("FileSystem's output stream doesn't support getNumCurrentReplicas; " +

         "HDFS-826 not available; fsOut=" + wrappedStreamClass.getName());

      } catch (SecurityException e) {

        LOG.info("No access to getNumCurrentReplicas on FileSystems's output stream; HDFS-826 " +

          "not available; fsOut=" + wrappedStreamClass.getName(), e);

        m = null; // could happen on setAccessible()

      }

    }

    if (m != null) {

      if (LOG.isTraceEnabled()) LOG.trace("Using getNumCurrentReplicas");

    }

    return m;

  }

  @Override

  public void registerWALActionsListener(final WALActionsListener listener) {

    this.listeners.add(listener);

  }

  @Override

  public boolean unregisterWALActionsListener(final WALActionsListener listener) {

    return this.listeners.remove(listener);

  }

  @Override

  public long getFilenum() {

    return this.filenum.get();

  }

  /**

   * Method used internal to this class and for tests only.

   * @return The wrapped stream our writer is using; its not the

   * writer's 'out' FSDatoOutputStream but the stream that this 'out' wraps

   * (In hdfs its an instance of DFSDataOutputStream).

   *

   * usage: see TestLogRolling.java

   */

  OutputStream getOutputStream() {

    return this.hdfs_out.getWrappedStream();

  }

  @Override

  public byte [][] rollWriter() throws FailedLogCloseException, IOException {

    return rollWriter(false);

  }

  private Path getNewPath() throws IOException {

    this.filenum.set(System.currentTimeMillis());

    Path newPath = computeFilename();

    while (fs.exists(newPath)) {

      this.filenum.incrementAndGet();

      newPath = computeFilename();

    }

    return newPath;

  }

  Path getOldPath() {

    long currentFilenum = this.filenum.get();

    Path oldPath = null;

    if (currentFilenum > 0) {

      // ComputeFilename  will take care of meta hlog filename

      oldPath = computeFilename(currentFilenum);

    } // I presume if currentFilenum is <= 0, this is first file and null for oldPath if fine?

    return oldPath;

  }

  /**

   * Tell listeners about pre log roll.

   * @throws IOException

   */

  private void tellListenersAboutPreLogRoll(final Path oldPath, final Path newPath)

  throws IOException {

    if (!this.listeners.isEmpty()) {

      for (WALActionsListener i : this.listeners) {

        i.preLogRoll(oldPath, newPath);

      }

    }

  }

  /**

   * Tell listeners about post log roll.

   * @throws IOException

   */

  private void tellListenersAboutPostLogRoll(final Path oldPath, final Path newPath)

  throws IOException {

    if (!this.listeners.isEmpty()) {

      for (WALActionsListener i : this.listeners) {

        i.postLogRoll(oldPath, newPath);

      }

    }

  }

  /**

   * Run a sync after opening to set up the pipeline.

   * @param nextWriter

   * @param startTimeNanos

   */

  private void preemptiveSync(final ProtobufLogWriter nextWriter) {

    long startTimeNanos = System.nanoTime();

    try {

      nextWriter.sync();

      postSync(System.nanoTime() - startTimeNanos, 0);

    } catch (IOException e) {

      // optimization failed, no need to abort here.

      LOG.warn("pre-sync failed but an optimization so keep going", e);

    }

  }

  @Override

  public byte [][] rollWriter(boolean force) throws FailedLogCloseException, IOException {

    rollWriterLock.lock();

    try {

      // Return if nothing to flush.

      if (!force && (this.writer != null && this.numEntries.get() <= 0)) return null;

      byte [][] regionsToFlush = null;

      if (this.closed) {

        LOG.debug("HLog closed. Skipping rolling of writer");

        return regionsToFlush;

      }

      if (!closeBarrier.beginOp()) {

        LOG.debug("HLog closing. Skipping rolling of writer");

        return regionsToFlush;

      }

      TraceScope scope = Trace.startSpan("FSHLog.rollWriter");

      try {

        Path oldPath = getOldPath();

        Path newPath = getNewPath();

        // Any exception from here on is catastrophic, non-recoverable so we currently abort.

        FSHLog.Writer nextWriter = this.createWriterInstance(fs, newPath, conf);

        FSDataOutputStream nextHdfsOut = null;

        if (nextWriter instanceof ProtobufLogWriter) {

          nextHdfsOut = ((ProtobufLogWriter)nextWriter).getStream();

          // If a ProtobufLogWriter, go ahead and try and sync to force setup of pipeline.

          // If this fails, we just keep going.... it is an optimization, not the end of the world.

          preemptiveSync((ProtobufLogWriter)nextWriter);

        }

        tellListenersAboutPreLogRoll(oldPath, newPath);

        // NewPath could be equal to oldPath if replaceWriter fails.

        newPath = replaceWriter(oldPath, newPath, nextWriter, nextHdfsOut);

        tellListenersAboutPostLogRoll(oldPath, newPath);

        // Can we delete any of the old log files?

        if (getNumRolledLogFiles() > 0) {

          cleanOldLogs();

          regionsToFlush = findRegionsToForceFlush();

        }

      } finally {

        closeBarrier.endOp();

        assert scope == NullScope.INSTANCE || !scope.isDetached();

        scope.close();

      }

      return regionsToFlush;

    } finally {

      rollWriterLock.unlock();

    }

  }

  /**

   * This method allows subclasses to inject different writers without having to

   * extend other methods like rollWriter().

   *

   * @param fs

   * @param path

   * @param conf

   * @return Writer instance

   * @throws IOException

   */

  protected Writer createWriterInstance(final FileSystem fs, final Path path,

      final Configuration conf) throws IOException {

    if (forMeta) {

      //TODO: set a higher replication for the hlog files (HBASE-6773)

    }

    return HLogFactory.createWALWriter(fs, path, conf);

  }

  /**

   * Archive old logs that could be archived: a log is eligible for archiving if all its WALEdits

   * have been flushed to hfiles.

   * <p>

   * For each log file, it compares its region to sequenceId map

   * (@link {@link FSHLog#highestRegionSequenceIds} with corresponding region entries in

   * {@link FSHLog#lowestFlushingRegionSequenceIds} and

   * {@link FSHLog#oldestUnflushedRegionSequenceIds}. If all the regions in the map are flushed

   * past of their value, then the wal is eligible for archiving.

   * @throws IOException

   */

  private void cleanOldLogs() throws IOException {

    Map<byte[], Long> oldestFlushingSeqNumsLocal = null;

    Map<byte[], Long> oldestUnflushedSeqNumsLocal = null;

    List<Path> logsToArchive = new ArrayList<Path>();

    // make a local copy so as to avoid locking when we iterate over these maps.

    synchronized (regionSequenceIdLock) {

      oldestFlushingSeqNumsLocal = new HashMap<byte[], Long>(this.lowestFlushingRegionSequenceIds);

      oldestUnflushedSeqNumsLocal =

        new HashMap<byte[], Long>(this.oldestUnflushedRegionSequenceIds);

    }

    for (Map.Entry<Path, Map<byte[], Long>> e : byWalRegionSequenceIds.entrySet()) {

      // iterate over the log file.

      Path log = e.getKey();

      Map<byte[], Long> sequenceNums = e.getValue();

      // iterate over the map for this log file, and tell whether it should be archive or not.

      if (areAllRegionsFlushed(sequenceNums, oldestFlushingSeqNumsLocal,

          oldestUnflushedSeqNumsLocal)) {

        logsToArchive.add(log);

        LOG.debug("WAL file ready for archiving " + log);

      }

    }

    for (Path p : logsToArchive) {

      this.totalLogSize.addAndGet(-this.fs.getFileStatus(p).getLen());

      archiveLogFile(p);

      this.byWalRegionSequenceIds.remove(p);

    }

  }

  /**

   * Takes a region:sequenceId map for a WAL file, and checks whether the file can be archived.

   * It compares the region entries present in the passed sequenceNums map with the local copy of

   * {@link #oldestUnflushedRegionSequenceIds} and {@link #lowestFlushingRegionSequenceIds}. If,

   * for all regions, the value is lesser than the minimum of values present in the

   * oldestFlushing/UnflushedSeqNums, then the wal file is eligible for archiving.

   * @param sequenceNums for a HLog, at the time when it was rolled.

   * @param oldestFlushingMap

   * @param oldestUnflushedMap

   * @return true if wal is eligible for archiving, false otherwise.

   */

   static boolean areAllRegionsFlushed(Map<byte[], Long> sequenceNums,

      Map<byte[], Long> oldestFlushingMap, Map<byte[], Long> oldestUnflushedMap) {

    for (Map.Entry<byte[], Long> regionSeqIdEntry : sequenceNums.entrySet()) {

      // find region entries in the flushing/unflushed map. If there is no entry, it meansj

      // a region doesn't have any unflushed entry.

      long oldestFlushing = oldestFlushingMap.containsKey(regionSeqIdEntry.getKey()) ?

          oldestFlushingMap.get(regionSeqIdEntry.getKey()) : Long.MAX_VALUE;

      long oldestUnFlushed = oldestUnflushedMap.containsKey(regionSeqIdEntry.getKey()) ?

          oldestUnflushedMap.get(regionSeqIdEntry.getKey()) : Long.MAX_VALUE;

          // do a minimum to be sure to contain oldest sequence Id

      long minSeqNum = Math.min(oldestFlushing, oldestUnFlushed);

      if (minSeqNum <= regionSeqIdEntry.getValue()) return false;// can't archive

    }

    return true;

  }

  /**

   * Iterates over the given map of regions, and compares their sequence numbers with corresponding

   * entries in {@link #oldestUnflushedRegionSequenceIds}. If the sequence number is greater or

   * equal, the region is eligible to flush, otherwise, there is no benefit to flush (from the

   * perspective of passed regionsSequenceNums map), because the region has already flushed the

   * entries present in the WAL file for which this method is called for (typically, the oldest

   * wal file).

   * @param regionsSequenceNums

   * @return regions which should be flushed (whose sequence numbers are larger than their

   * corresponding un-flushed entries.

   */

  private byte[][] findEligibleMemstoresToFlush(Map<byte[], Long> regionsSequenceNums) {

    List<byte[]> regionsToFlush = null;

    // Keeping the old behavior of iterating unflushedSeqNums under oldestSeqNumsLock.

    synchronized (regionSequenceIdLock) {

      for (Map.Entry<byte[], Long> e : regionsSequenceNums.entrySet()) {

        Long unFlushedVal = this.oldestUnflushedRegionSequenceIds.get(e.getKey());

        if (unFlushedVal != null && unFlushedVal <= e.getValue()) {

          if (regionsToFlush == null) regionsToFlush = new ArrayList<byte[]>();

          regionsToFlush.add(e.getKey());

        }

      }

    }

    return regionsToFlush == null ? null : regionsToFlush

        .toArray(new byte[][] { HConstants.EMPTY_BYTE_ARRAY });

  }

  /**

   * If the number of un-archived WAL files is greater than maximum allowed, it checks

   * the first (oldest) WAL file, and returns the regions which should be flushed so that it could

   * be archived.

   * @return regions to flush in order to archive oldest wal file.

   * @throws IOException

   */

  byte[][] findRegionsToForceFlush() throws IOException {

    byte [][] regions = null;

    int logCount = getNumRolledLogFiles();

    if (logCount > this.maxLogs && logCount > 0) {

      Map.Entry<Path, Map<byte[], Long>> firstWALEntry =

        this.byWalRegionSequenceIds.firstEntry();

      regions = findEligibleMemstoresToFlush(firstWALEntry.getValue());

    }

    if (regions != null) {

      StringBuilder sb = new StringBuilder();

      for (int i = 0; i < regions.length; i++) {

        if (i > 0) sb.append(", ");

        sb.append(Bytes.toStringBinary(regions[i]));

      }

      LOG.info("Too many hlogs: logs=" + logCount + ", maxlogs=" +

         this.maxLogs + "; forcing flush of " + regions.length + " regions(s): " +

         sb.toString());

    }

    return regions;

  }

  /**

   * Cleans up current writer closing it and then puts in place the passed in

   * <code>nextWriter</code>

   *

   * @param oldPath

   * @param newPath

   * @param nextWriter

   * @param nextHdfsOut

   * @return <code>newPath</code>

   * @throws IOException

   */

  Path replaceWriter(final Path oldPath, final Path newPath, FSHLog.Writer nextWriter,

      final FSDataOutputStream nextHdfsOut)

  throws IOException {

    // Ask the ring buffer writer to pause at a safe point.  Once we do this, the writer

    // thread will eventually pause. An error hereafter needs to release the writer thread

    // regardless -- hence the finally block below.  Note, this method is called from the FSHLog

    // constructor BEFORE the ring buffer is set running so it is null on first time through

    // here; allow for that.

    SyncFuture syncFuture = null;

    SafePointZigZagLatch zigzagLatch = (this.ringBufferEventHandler == null)?

      null: this.ringBufferEventHandler.attainSafePoint();

    TraceScope scope = Trace.startSpan("FSHFile.replaceWriter");

    try {

      // Wait on the safe point to be achieved.  Send in a sync in case nothing has hit the

      // ring buffer between the above notification of writer that we want it to go to

      // 'safe point' and then here where we are waiting on it to attain safe point.  Use

      // 'sendSync' instead of 'sync' because we do not want this thread to block waiting on it

      // to come back.  Cleanup this syncFuture down below after we are ready to run again.

      try {

        if (zigzagLatch != null) {

          Trace.addTimelineAnnotation("awaiting safepoint");

          syncFuture = zigzagLatch.waitSafePoint(publishSyncOnRingBuffer());

        }

      } catch (FailedSyncBeforeLogCloseException e) {

        if (isUnflushedEntries()) throw e;

        // Else, let is pass through to the close.

        LOG.warn("Failed last sync but no outstanding unsync edits so falling through to close; " +

          e.getMessage());

      }

      // It is at the safe point.  Swap out writer from under the blocked writer thread.

      // TODO: This is close is inline with critical section.  Should happen in background?

      try {

        if (this.writer != null) {

          Trace.addTimelineAnnotation("closing writer");

          this.writer.close();

          Trace.addTimelineAnnotation("writer closed");

        }

        this.closeErrorCount.set(0);

      } catch (IOException ioe) {

        int errors = closeErrorCount.incrementAndGet();

        if (!isUnflushedEntries() && (errors <= this.closeErrorsTolerated)) {

          LOG.warn("Riding over failed WAL close of " + oldPath + ", cause=\"" +

            ioe.getMessage() + "\", errors=" + errors +

            "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK");

        } else {

          throw ioe;

        }

      }

      this.writer = nextWriter;

      this.hdfs_out = nextHdfsOut;

      int oldNumEntries = this.numEntries.get();

      this.numEntries.set(0);

      if (oldPath != null) {

        this.byWalRegionSequenceIds.put(oldPath, this.highestRegionSequenceIds);

        this.highestRegionSequenceIds = new HashMap<byte[], Long>();

        long oldFileLen = this.fs.getFileStatus(oldPath).getLen();

        this.totalLogSize.addAndGet(oldFileLen);

        LOG.info("Rolled WAL " + FSUtils.getPath(oldPath) + " with entries=" + oldNumEntries +

          ", filesize=" + StringUtils.byteDesc(oldFileLen) + "; new WAL " +

          FSUtils.getPath(newPath));

      } else {

        LOG.info("New WAL " + FSUtils.getPath(newPath));

      }

    } catch (InterruptedException ie) {

      // Perpetuate the interrupt

      Thread.currentThread().interrupt();

    } catch (IOException e) {

      long count = getUnflushedEntriesCount();

      LOG.error("Failed close of HLog writer " + oldPath + ", unflushedEntries=" + count, e);

      throw new FailedLogCloseException(oldPath + ", unflushedEntries=" + count, e);

    } finally {

      try {

        // Let the writer thread go regardless, whether error or not.

        if (zigzagLatch != null) {

          zigzagLatch.releaseSafePoint();

          // It will be null if we failed our wait on safe point above.

          if (syncFuture != null) blockOnSync(syncFuture);

        }

      } finally {

        scope.close();

      }

    }

    return newPath;

  }

  long getUnflushedEntriesCount() {

    long highestSynced = this.highestSyncedSequence.get();

    return highestSynced > this.highestUnsyncedSequence?

      0: this.highestUnsyncedSequence - highestSynced;

  }

  boolean isUnflushedEntries() {

    return getUnflushedEntriesCount() > 0;

  }

  private void archiveLogFile(final Path p) throws IOException {

    Path newPath = getHLogArchivePath(this.fullPathOldLogDir, p);

    // Tell our listeners that a log is going to be archived.

    if (!this.listeners.isEmpty()) {

      for (WALActionsListener i : this.listeners) {

        i.preLogArchive(p, newPath);

      }

    }

    if (!FSUtils.renameAndSetModifyTime(this.fs, p, newPath)) {

      throw new IOException("Unable to rename " + p + " to " + newPath);

    }

    // Tell our listeners that a log has been archived.

    if (!this.listeners.isEmpty()) {

      for (WALActionsListener i : this.listeners) {

        i.postLogArchive(p, newPath);

      }

    }

  }

  /**

   * This is a convenience method that computes a new filename with a given

   * file-number.

   * @param filenum to use

   * @return Path

   */

  protected Path computeFilename(final long filenum) {

    this.filenum.set(filenum);

    return computeFilename();

  }

  /**

   * This is a convenience method that computes a new filename with a given

   * using the current HLog file-number

   * @return Path

   */

  protected Path computeFilename() {

    if (this.filenum.get() < 0) {

      throw new RuntimeException("hlog file number can't be < 0");

    }

    String child = logFilePrefix + "." + filenum;

    if (forMeta) {

      child += HLog.META_HLOG_FILE_EXTN;

    }

    return new Path(fullPathLogDir, child);

  }

* A log file has a creation timestamp (in ms) in its file name ({@link #filenum}.

* This helper method returns the creation timestamp from a given log file.

* It extracts the timestamp assuming the filename is created with the

* {@link #computeFilename(long filenum)} method.

* @param fileName

* @return timestamp, as in the log file name.

*/

  protected long getFileNumFromFileName(Path fileName) {

    if (fileName == null) throw new IllegalArgumentException("file name can't be null");

    // The path should start with dir/<prefix>.

    String prefixPathStr = new Path(fullPathLogDir, logFilePrefix + ".").toString();

    if (!fileName.toString().startsWith(prefixPathStr)) {

      throw new IllegalArgumentException("The log file " + fileName + " doesn't belong to" +

          " this regionserver " + prefixPathStr);

    }

    String chompedPath = fileName.toString().substring(prefixPathStr.length());

    if (forMeta) chompedPath = chompedPath.substring(0, chompedPath.indexOf(META_HLOG_FILE_EXTN));

    return Long.parseLong(chompedPath);

  }

  @Override

  public void closeAndDelete() throws IOException {

    close();

    if (!fs.exists(this.fullPathLogDir)) return;

    FileStatus[] files = fs.listStatus(this.fullPathLogDir);

    if (files != null) {

      for(FileStatus file : files) {

        Path p = getHLogArchivePath(this.fullPathOldLogDir, file.getPath());

        // Tell our listeners that a log is going to be archived.

        if (!this.listeners.isEmpty()) {

          for (WALActionsListener i : this.listeners) {

            i.preLogArchive(file.getPath(), p);

          }

        }

        if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {

          throw new IOException("Unable to rename " + file.getPath() + " to " + p);

        }

        // Tell our listeners that a log was archived.

        if (!this.listeners.isEmpty()) {

          for (WALActionsListener i : this.listeners) {

            i.postLogArchive(file.getPath(), p);

          }

        }

      }

      LOG.debug("Moved " + files.length + " WAL file(s) to " +

        FSUtils.getPath(this.fullPathOldLogDir));

    }

    if (!fs.delete(fullPathLogDir, true)) {

      LOG.info("Unable to delete " + fullPathLogDir);

    }

  }

  @Override

  public void close() throws IOException {

    if (this.closed) return;

    try {

      // Prevent all further flushing and rolling.

      closeBarrier.stopAndDrainOps();

    } catch (InterruptedException e) {

      LOG.error("Exception while waiting for cache flushes and log rolls", e);

      Thread.currentThread().interrupt();

    }

    // Shutdown the disruptor.  Will stop after all entries have been processed.  Make sure we have

    // stopped incoming appends before calling this else it will not shutdown.  We are

    // conservative below waiting a long time and if not elapsed, then halting.

    if (this.disruptor != null) {

      long timeoutms = conf.getLong("hbase.wal.disruptor.shutdown.timeout.ms", 60000);