Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode$ShutdownAvatarThread

Package org.apache.hadoop.hdfs.server.namenode

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode$ShutdownAvatarThread

org.apache.hadoop.hdfs.server.namenode.AvatarNode
This is an implementation of the AvatarNode, a hot standby for the NameNode. This is really cool, believe me! The AvatarNode has two avatars.. the Standby avatar and the Active avatar. In the Standby avatar, the AvatarNode is consuming transaction logs generated by the primary (via a transaction log stored in a shared device). Typically, the primary Namenode is writing transactions to a NFS filesystem and the Standby is reading the log from the same NFS filesystem. The Standby is also making periodic checkpoints to the primary namenode. A manual command can switch the AvatarNode from the Standby avatar to the Active avatar. In the Active avatar, the AvatarNode performs precisely the same functionality as a real usual Namenode. The switching from Standby avatar to the Active avatar is fast and can typically occur within seconds. Typically, an adminstrator will run require two shared mount points for transaction logs. It has to be set in fs.name.dir.shared0 and fs.name.dir.shared1 (similarly for edits). Then the adminstrator starts the AvatarNode on two different machines as follows: bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -zero -active bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -one -standby The first AvatarNode uses fs.name.dir.shared0 while the second AvatarNode uses fs.name.dir.shared1 to write its transaction logs. Also, at startup, the first instance is the primary Namenode and the second instance is the Standby After a while, the adminstrator decides to change the avatar of the second instance to Active. In this case, he/she has to first ensure that the first instance is really really dead. This code does not handle the split-brain scenario where there are two active namenodes in one cluster.

    DFSTestUtil.createFile(fs, new Path("/testFailoverWithRestart"), 1024,
        (short) 1, System.currentTimeMillis());
    FailoverTestUtilHandler handler = new FailoverTestUtilHandler();
    handler.simulateShutdownCrash = false;
    InjectionHandler.set(handler);
    AvatarNode node = cluster.getStandbyAvatar(0).avatar;
    cluster.killPrimary();
    cluster.getStandbyAvatar(0).avatar.quiesceForFailover(false);
    while (!handler.waitForRestartTrigger) {
      LOG.info("Waiting for restart");
      Thread.sleep(1000);

View Full Code Here

    handler.simulateShutdownCrash = false;
    InjectionHandler.set(handler);
    // Txid will now mismatch.
    cluster.getPrimaryAvatar(0).avatar.namesystem.getEditLog()
        .setLastWrittenTxId(0);
    AvatarNode node = cluster.getStandbyAvatar(0).avatar;
    try {
      cluster.failOver();
      fail("Did not throw exception");
    } catch (IOException ie) {
      while (!handler.waitForRestartTrigger) {

View Full Code Here

    DFSTestUtil.createFile(fs, new Path("/testFailoverRetryBlocksMisMatch"),
        (long) totalBlocks * 1024, (short) 3, System.currentTimeMillis());
    // This sets the primary number of blocks to 0.
    cluster.getPrimaryAvatar(0).avatar.namesystem.close();
    cluster.getPrimaryAvatar(0).avatar.namesystem.blocksMap.close();
    AvatarNode node = cluster.getStandbyAvatar(0).avatar;
    try {
      cluster.failOver();
      fail("Did not throw exception");
    } catch (Exception e) {
      while (!handler.waitForRestartTrigger) {

View Full Code Here

      fs.create(new Path("/" + random.nextInt()));
    }
  }


  protected ZookeeperTxId getLastTxid() throws Exception {
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(0).avatar;
    String address = primaryAvatar
        .getStartupConf().get(NameNode.DFS_NAMENODE_RPC_ADDRESS_KEY);
    return zkClient.getPrimaryLastTxId(address, false);
  }

View Full Code Here

        .getStartupConf().get(NameNode.DFS_NAMENODE_RPC_ADDRESS_KEY);
    return zkClient.getPrimaryLastTxId(address, false);
  }


  protected long getSessionId() throws Exception {
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(0).avatar;
    String address = primaryAvatar
        .getStartupConf().get(NameNode.DFS_NAMENODE_RPC_ADDRESS_KEY);
    return zkClient.getPrimarySsId(address, false);
  }

View Full Code Here

    InjectionHandler.set(h);
    setUp("testOngoingLeaseRecoveryDuringShutdown", discardLastBlock);


    // set cgi to avoid NPE at NN, since we talk to it directly
    UserGroupInformation.setCurrentUser(UnixUserGroupInformation.login(conf));
    AvatarNode nn = cluster.getPrimaryAvatar(0).avatar;
    FsPermission perm = new FsPermission((short) 0264);
    String clientName = ((DistributedAvatarFileSystem) fs)
        .getClient().getClientName();
    
    // create a file directly and add one block
    nn.create("/test", perm, clientName, true, true, (short) 3, (long) 1024);
    nn.addBlock("/test", clientName);


    assertEquals(1,
        cluster.getPrimaryAvatar(0).avatar.namesystem.getBlocksTotal());
    
    // set lease period to something short

View Full Code Here

      throws Exception {
    TestAvatarQuiesceHandler h = new TestAvatarQuiesceHandler(event);
    InjectionHandler.set(h);
    setUp("testQuiesceWhenSavingNamespace: " + event, false);
    // fail once
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    
    createEdits(20);


    // wait to the savers are up for some checkpoint
    // and waiting until we interrupt
    h.waitReachedSynchronizationPoint();
      
    standby.quiesceStandby(getCurrentTxId(primary) - 1);
    // SLS + ELS + SLS + 20 edits
    assertEquals(23, getCurrentTxId(primary));
    assertEquals(getCurrentTxId(primary), getCurrentTxId(standby));
    if(expectException)
      assertTrue(h.exceptionEvent);
    
    //assert that the cancelled checkpoint was removed
    long ckptTxId = h.cancelledCkptTxid;
    for (Iterator<StorageDirectory> it = standby.getFSImage().dirIterator(); it
        .hasNext();) {
      StorageDirectory sd = it.next();
      File image = new File(sd.getRoot(), NNStorage.getImageFileName(ckptTxId));
      assertFalse(image.exists());
      assertFalse(new File(sd.getRoot(),

View Full Code Here

    TestAvatarShellInjectionHandler h = new TestAvatarShellInjectionHandler();
    InjectionHandler.set(h);
    setUp(false, "testPrepareFailover");
    try {
      AvatarShell shell = new AvatarShell(conf);
      AvatarNode standbyAvatar = cluster.getStandbyAvatar(0).avatar;
      assertEquals(0, shell.run(new String[] { "-prepfailover" }));   
      assertTrue(standbyAvatar.isInSafeMode());
      while(h.prepareFailover.size() < cluster.getDataNodes().size()) {
        DFSTestUtil.waitNMilliSecond(100);
      }
    } finally {
      InjectionHandler.clear();

View Full Code Here

  public void testAvatarShellLeaveSafeMode() throws Exception {
    setUp(false, "testAvatarShellLeaveSafeMode");
    int blocksBefore = blocksInFile();


    AvatarShell shell = new AvatarShell(conf);
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(0).avatar;
    primaryAvatar.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
    assertTrue(primaryAvatar.isInSafeMode());
    assertEquals(0, shell.run(new String[] { "-zero", "-safemode", "leave" }));
    assertFalse(primaryAvatar.isInSafeMode());
    assertFalse(cluster.getPrimaryAvatar(0).avatar.isInSafeMode());
    int blocksAfter = blocksInFile();
    assertTrue(blocksBefore == blocksAfter);


  }

View Full Code Here

    int blocksBefore = blocksInFile();
    cluster.failOver();
    cluster.restartStandby();


    AvatarShell shell = new AvatarShell(conf);
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(0).avatar;
    primaryAvatar.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
    assertTrue(primaryAvatar.isInSafeMode());
    assertEquals(0, shell.run(new String[] { "-one", "-safemode", "leave" }));
    assertFalse(primaryAvatar.isInSafeMode());
    assertFalse(cluster.getPrimaryAvatar(0).avatar.isInSafeMode());
    int blocksAfter = blocksInFile();
    assertTrue(blocksBefore == blocksAfter);
  }

View Full Code Here

0 1 2 3 4 5 6 7

TOP

Related Classes of org.apache.hadoop.hdfs.server.namenode.AvatarNode$ShutdownAvatarThread

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.Path

org.apache.hadoop.hdfs.AvatarFailoverSnapshot

org.apache.hadoop.hdfs.AvatarZooKeeperClient

org.apache.hadoop.hdfs.FailoverTestUtil

org.apache.hadoop.hdfs.FileStatusExtended

org.apache.hadoop.hdfs.MiniAvatarCluster

org.apache.hadoop.hdfs.protocol.AvatarConstants.InstanceId

org.apache.hadoop.hdfs.protocol.AvatarConstants.StartupOption

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.