Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode$InvalidatesCleaner

Package org.apache.hadoop.hdfs.server.namenode

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode$InvalidatesCleaner

org.apache.hadoop.hdfs.server.namenode.AvatarNode
This is an implementation of the AvatarNode, a hot standby for the NameNode. This is really cool, believe me! The AvatarNode has two avatars.. the Standby avatar and the Active avatar. In the Standby avatar, the AvatarNode is consuming transaction logs generated by the primary (via a transaction log stored in a shared device). Typically, the primary Namenode is writing transactions to a NFS filesystem and the Standby is reading the log from the same NFS filesystem. The Standby is also making periodic checkpoints to the primary namenode. A manual command can switch the AvatarNode from the Standby avatar to the Active avatar. In the Active avatar, the AvatarNode performs precisely the same functionality as a real usual Namenode. The switching from Standby avatar to the Active avatar is fast and can typically occur within seconds. Typically, an adminstrator will run require two shared mount points for transaction logs. It has to be set in fs.name.dir.shared0 and fs.name.dir.shared1 (similarly for edits). Then the adminstrator starts the AvatarNode on two different machines as follows: bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -zero -active bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -one -standby The first AvatarNode uses fs.name.dir.shared0 while the second AvatarNode uses fs.name.dir.shared1 to write its transaction logs. Also, at startup, the first instance is the primary Namenode and the second instance is the Standby After a while, the adminstrator decides to change the avatar of the second instance to Active. In this case, he/she has to first ensure that the first instance is really really dead. This code does not handle the split-brain scenario where there are two active namenodes in one cluster.

      assertTrue(locs.length != 0);
    }
  }


  private boolean blocksReceived(int nBlocks, String fileName) throws IOException {
    AvatarNode avatar = cluster.getPrimaryAvatar(0).avatar;
    LocatedBlocks lbks = avatar.namesystem.getBlockLocations(fileName, 0,
        Long.MAX_VALUE);
    int blocks = lbks.locatedBlockCount();
    if (blocks != nBlocks)
      return false;

View Full Code Here

        argList.add(StartupOption.SERVICE.getName());
        argList.add(nni.nameserviceId);
      }
      a0Args = new String[argList.size()];
      argList.toArray(a0Args);
      AvatarNode a0 = instantiateAvatarNode(a0Args, 
                         getServerConf(AvatarConstants.
                                       StartupOption.
                                       NODEZERO.
                                       getName(), nni));


      avatars.add(new AvatarInfo(a0,
                                 AvatarState.ACTIVE,
                                 nni.nn0Port, nni.nnDn0Port, nni.http0Port, nni.rpc0Port,
                                 AvatarConstants.StartupOption.NODEZERO.
                                 getName()));
      
      // wait for up to 10 seconds until the ACTIVE is initialized
      for (int i = 0; i < 10; i++) {
        if (a0.isInitDone())
          break;
        LOG.info("Waiting for the ACTIVE to be initialized...");
        sleep(1000);
      }
      if (!a0.isInitDone()) {
        throw new IOException("The ACTIVE cannot be initialized");
      }
    }


    {

View Full Code Here

  @SuppressWarnings("unchecked")
  public void doGet(HttpServletRequest request,
                    HttpServletResponse response
                    ) throws ServletException, IOException {
    ServletContext context = getServletContext();
    AvatarNode nn = (AvatarNode)context.getAttribute("avatar.node");
    FSNamesystem fsn = nn.getNamesystem();
    boolean isStandby = false;
    boolean isFailover = false;
    Set <DatanodeID> heartbeats = null;
    Set <DatanodeID> reports = null;
    String msg = null;
    fsn.writeLock();
    // We need to capture all information under a namesystem lock, since we
    // might leave safemode and the safemode object might be set to null.
    try {
      isStandby = (nn.reportAvatar() == Avatar.STANDBY);
      StandbySafeMode sm = nn.getStandbySafeMode();
      if (sm == null && isStandby) {
        msg = standbyNull;
      } else if (sm == null) {
        msg = notStandby;
      } else {

View Full Code Here

    TestAvatarIngestingHandler h = new TestAvatarIngestingHandler(
        InjectionEvent.FSEDIT_LOG_WRITE_END_LOG_SEGMENT);
    InjectionHandler.set(h);
    setUp(3, "testCheckpointWithRestarts", true, true);
    // simulate interruption, no ckpt failure
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    createEdits(20);
    h.doCheckpoint();
    // 2 for initial checkpoint + 2 for this checkpoint + SLS + 20 edits
    h.disabled = false;
    assertEquals(25, getCurrentTxId(primary));

View Full Code Here

    TestAvatarIngestingHandler h = new TestAvatarIngestingHandler(
        InjectionEvent.STANDBY_RECOVER_STATE);
    InjectionHandler.set(h);
    setUp(3, "testRecoverState", true, true);
    // simulate interruption, no ckpt failure
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    h.setDisabled(true);


    createEdits(20);
    h.doCheckpoint();
    // 2 for initial checkpoint + 2 for this checkpoint + SLS + 20 edits
    assertEquals(25, getCurrentTxId(primary));
    createEdits(20);
    h.setDisabled(false);


    // sleep to see if the state recovers correctly
    DFSTestUtil.waitNSecond(5);
    h.doCheckpoint();
    standby.quiesceStandby(getCurrentTxId(primary) - 1);


    assertEquals(47, getCurrentTxId(primary));
    assertEquals(getCurrentTxId(primary), getCurrentTxId(standby));
  }

View Full Code Here

  public void testBlocksMisMatch() throws Exception {
    setUp("testBlocksMisMatch", false);
    int totalBlocks = 50;
    DFSTestUtil.createFile(fs, new Path("/testBlocksMisMatch"),
        (long) totalBlocks * 1024, (short) 3, System.currentTimeMillis());
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    standby.quiesceStandby(cluster
        .getPrimaryAvatar(0).avatar.getLastWrittenTxId());
    // This sets the standby number of blocks to 0.
    standby.namesystem.close();
    try {
      cluster.failOver();

View Full Code Here

  @Test
  public void testStandbyTailingJNFailure() throws Exception {
    setUp("testStandbyTailingJNFailure");


    DFSTestUtil.createFile(fs, new Path("/test"), 1024, (short) 1, 0);
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    handler.doCheckpoint();


    // Find the journal node the Standby is tailing from.
    JournalNode jn = getStandbyTailingJN();
    assertNotNull(jn);


    jn.stopAndJoin(0);
    handler.instantiatedIngest = false;


    LOG.info("Stopped JournalNode with port "
        + jn.getBoundHttpAddress().getPort());
    LOG.info("current journal port : " + handler.currentJournalHttpPort);


    DFSTestUtil.createFile(fs, new Path("/test1"), 1024, (short) 1, 0);


    // Wait for ingest.
    long start = System.currentTimeMillis();
    while (!handler.instantiatedIngest
        && System.currentTimeMillis() - start < 10000) {
      DFSTestUtil.waitSecond();
    }


    assertTrue(handler.instantiatedIngest);


    LOG.info("Ingest failures : "
        + standby.getAvatarNodeMetrics().numIngestFailures.get());
    assertTrue(standby.getAvatarNodeMetrics().numIngestFailures
        .get() <= 1);
  }

View Full Code Here

  protected void doTestDelayedCheckpoint(boolean enableQJM) throws Exception {
    TestAvatarCheckpointingHandler h = new TestAvatarCheckpointingHandler();
    InjectionHandler.set(h);
    setUp(3600, "testDelayedCheckpoint", enableQJM);


    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    createEdits(40);


    long start = System.currentTimeMillis();
    while (System.currentTimeMillis() - start < 10000) {
      if (h.receivedEvents.contains(InjectionEvent.STANDBY_DELAY_CHECKPOINT)) {
        // checkpoint has been delayed at least once
        break;
      }
    }
    // checkpoint was delayed
    assertTrue(h.receivedEvents
        .contains(InjectionEvent.STANDBY_DELAY_CHECKPOINT));
    // no actual checkpoint happened at startup
    assertFalse(h.receivedEvents
        .contains(InjectionEvent.STANDBY_BEFORE_ROLL_EDIT));


    if (!enableQJM) {
      standby.quiesceStandby(DFSAvatarTestUtil.getCurrentTxId(primary) - 1);
      // edits + SLS
      assertEquals(41, DFSAvatarTestUtil.getCurrentTxId(primary));
    }
  }

View Full Code Here

      throws IOException, ConfigException, InterruptedException{
    MiniAvatarCluster cluster = new MiniAvatarCluster(conf, 1, true, null, null, 2, true); 
    try {
      NameNodeInfo nn1 = cluster.getNameNode(0);
      NameNodeInfo nn2 = cluster.getNameNode(1);
      AvatarNode nn1zero = nn1.avatars.get(0).avatar;
      AvatarNode nn1one = nn1.avatars.get(1).avatar;
      AvatarNode nn2zero = nn2.avatars.get(0).avatar;
      AvatarNode nn2one = nn2.avatars.get(1).avatar;
      assertNotNull("cannot create nn1 avatar 0", nn1zero);
      assertNotNull("cannot create nn1 avatar 1", nn1one);
      assertNotNull("cannot create nn2 avatar 0", nn2zero);
      assertNotNull("cannot create nn2 avatar 1", nn2one);
      
      int ns1zero = nn1zero.getNamespaceID();
      int ns1one = nn1one.getNamespaceID();
      assertEquals("namespace ids for namenode 1 should be the same",
          ns1zero, ns1one);
      int ns2zero = nn2zero.getNamespaceID();
      int ns2one = nn2one.getNamespaceID();
      assertEquals("namespace ids for namenode 2 should be the same",
          ns2zero, ns2one);
      int lv1zero = nn1zero.getFSImage().getLayoutVersion();
      int lv1one = nn1one.getFSImage().getLayoutVersion();
      int lv2zero = nn2zero.getFSImage().getLayoutVersion();
      int lv2one = nn2one.getFSImage().getLayoutVersion();
      assertNotSame("namespace ids should be different", ns1zero, ns2zero);
      LOG.info("nn1zero: lv=" + lv1zero + ";uri=" + nn1zero.getNameNodeAddress());
      LOG.info("nn1one: lv=" + lv1one + ";uri=" + nn1one.getNameNodeAddress());
      LOG.info("nn2zero: lv=" + lv2zero + ";uri=" + nn2zero.getNameNodeAddress());
      LOG.info("nn2one: lv=" + lv2one + ";uri=" + nn2one.getNameNodeAddress());


      // check number of volumes in fsdataset
      AvatarDataNode dn = cluster.getDataNodes().get(0);
      
      for (NamespaceService nsos : dn.getAllNamespaceServices()) {

View Full Code Here

  public void testFedSingleNN() 
      throws IOException, ConfigException, InterruptedException {
    MiniAvatarCluster cluster = new MiniAvatarCluster(conf, 1, true, null, null, 1, true); 
    try {
      NameNodeInfo nn1 = cluster.getNameNode(0);
      AvatarNode nn1zero = nn1.avatars.get(0).avatar;
      AvatarNode nn1one = nn1.avatars.get(1).avatar;
      assertNotNull("cannot create nn1 zero", nn1zero);
      assertNotNull("cannot create nn1 one", nn1one);


      int nsid1zero = nn1zero.getNamespaceID();
      int nsid1one = nn1one.getNamespaceID();
      assertEquals("namespace ids for namenode 1 should be the same",
          nsid1zero, nsid1one);
      int lv1zero = nn1zero.getFSImage().getLayoutVersion();
      int lv1one = nn1one.getFSImage().getLayoutVersion();
      LOG.info("nn1: lv=" + lv1zero + ";nsid=" + nsid1zero + ";uri="
          + nn1zero.getNameNodeAddress());
      LOG.info("nn1: lv=" + lv1one + ";nsid=" + nsid1one + ";uri="
          + nn1one.getNameNodeAddress());


      // check number of vlumes in fsdataset
      AvatarDataNode dn = cluster.getDataNodes().get(0);
      for (NamespaceService nsos : dn.getAllNamespaceServices()) {
        LOG.info("reg: nsid =" + nsos.getNamespaceId() + "; name="

View Full Code Here

0 1 2 3 4 5 6 7

TOP

Related Classes of org.apache.hadoop.hdfs.server.namenode.AvatarNode$InvalidatesCleaner

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.Path

org.apache.hadoop.hdfs.AvatarFailoverSnapshot

org.apache.hadoop.hdfs.AvatarZooKeeperClient

org.apache.hadoop.hdfs.FailoverTestUtil

org.apache.hadoop.hdfs.FileStatusExtended

org.apache.hadoop.hdfs.MiniAvatarCluster

org.apache.hadoop.hdfs.protocol.AvatarConstants.InstanceId

org.apache.hadoop.hdfs.protocol.AvatarConstants.StartupOption

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.