List of startDataNodes() Examples

Examples of startDataNodes()

org.apache.hadoop.dfs.MiniDFSCluster.startDataNodes()
Modify the config and start up the DataNodes. The info port for DataNodes is guaranteed to use a free port. @param conf the base configuration to use in starting the DataNodes. Thiswill be modified as necessary. @param numDataNodes Number of DataNodes to start; may be zero @param manageDfsDirs if true, the data directories for DataNodes will becreated and dfs.data.dir will be set in the conf @param operation the operation with which to start the DataNodes. If nullor StartupOption.FORMAT, then StartupOption.REGULAR will be used. @param racks array of strings indicating the rack that each DataNode is on @throws IllegalStateException if NameNode has been shutdown
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()
Modify the config and start up additional DataNodes. The info port for DataNodes is guaranteed to use a free port. Data nodes can run with the name node in the mini cluster or a real name node. For example, running with a real name node is useful when running simulated data nodes with a real name node. If minicluster's name node is null assume that the conf has been set with the right address:port of the name node. @param conf the base configuration to use in starting the DataNodes. Thiswill be modified as necessary. @param numDataNodes Number of DataNodes to start; may be zero @param manageDfsDirs if true, the data directories for DataNodes will becreated and dfs.data.dir will be set in the conf @param operation the operation with which to start the DataNodes. If nullor StartupOption.FORMAT, then StartupOption.REGULAR will be used. @param racks array of strings indicating the rack that each DataNode is on @param hosts array of strings indicating the hostnames for each DataNode @param simulatedCapacities array of capacities of the simulated data nodes @throws IllegalStateException if NameNode has been shutdown
org.apache.hadoop.hdfs.MiniDFSClusterWithNodeGroup.startDataNodes()

Examples of org.apache.hadoop.dfs.MiniDFSCluster.startDataNodes()

          fileName.toString(), 0, (long)fileLen);
      assertEquals(blocks.locatedBlockCount(), 1);
      LocatedBlock block = blocks.get(0);
      
      // bring up a second datanode
      cluster.startDataNodes(conf, 1, true, null, null);
      cluster.waitActive();
      final int sndNode = 1;
      DataNode datanode = cluster.getDataNodes().get(sndNode);
      
      // replicate the block to the second datanode

View Full Code Here

Examples of org.apache.hadoop.dfs.MiniDFSCluster.startDataNodes()

      // keep a copy of all datanode descriptor
      DatanodeDescriptor[] datanodes = (DatanodeDescriptor[])
         namesystem.heartbeats.toArray(new DatanodeDescriptor[REPLICATION_FACTOR]);
      
      // start two new nodes
      cluster.startDataNodes(conf, 2, true, null, null);
      cluster.waitActive();
      
      // bring down first datanode
      DatanodeDescriptor datanode = datanodes[0];
      DataNodeProperties dnprop = cluster.stopDataNode(datanode.getName());

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()


      // Write the input file
      UtilsForTests.writeFile(dfs.getNameNode(), conf, 
                              new Path(inDir + "/file"), (short)1);


      dfs.startDataNodes(conf, 1, true, null, null, null, null);
      dfs.waitActive();


      namenode = (dfs.getFileSystem()).getUri().getHost() + ":" 
                 + (dfs.getFileSystem()).getUri().getPort();

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

      // keep a copy of all datanode descriptor
      DatanodeDescriptor[] datanodes = (DatanodeDescriptor[])
         namesystem.heartbeats.toArray(new DatanodeDescriptor[REPLICATION_FACTOR]);
      
      // start two new nodes
      cluster.startDataNodes(conf, 2, true, null, null);
      cluster.waitActive(false);
      
      LOG.info("Bringing down first DN");
      // bring down first datanode
      DatanodeDescriptor datanode = datanodes[0];

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()


      // Write the input file
      UtilsForTests.writeFile(dfs.getNameNode(), conf, 
                              new Path(inDir + "/file"), (short)1);


      dfs.startDataNodes(conf, 1, true, null, null, null, null);
      dfs.waitActive();


      namenode = (dfs.getFileSystem()).getUri().getHost() + ":" 
                 + (dfs.getFileSystem()).getUri().getPort();

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

          fileName.toString(), 0, (long)fileLen);
      assertEquals(blocks.locatedBlockCount(), 1);
      LocatedBlock block = blocks.get(0);
      
      // bring up a second datanode
      cluster.startDataNodes(conf, 1, true, null, null);
      cluster.waitActive();
      final int sndNode = 1;
      DataNode datanode = cluster.getDataNodes().get(sndNode);
      
      // replicate the block to the second datanode

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

      if (!fileSys.mkdirs(inDir)) {
        throw new IOException("Mkdirs failed to create " + inDir.toString());
      }
      Path file1 = new Path(dir1 + "/file1");
      writeFile(conf, file1, (short)1, 1);
      dfs.startDataNodes(conf, 1, true, null, rack2, hosts2, null);
      dfs.waitActive();


      // create file on two datanodes.
      Path file2 = new Path(dir2 + "/file2");
      writeFile(conf, file2, (short)2, 2);

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

      assertEquals(fileSplit.getOffset(0), 0);
      assertEquals(fileSplit.getLength(0), BLOCKSIZE);
      assertEquals(fileSplit.getLocations()[0], hosts1[0]); // should be on r1


      // create another file on 3 datanodes and 3 racks.
      dfs.startDataNodes(conf, 1, true, null, rack3, hosts3, null);
      dfs.waitActive();
      Path file3 = new Path(dir3 + "/file3");
      writeFile(conf, new Path(dir3 + "/file3"), (short)3, 3);
      inFormat = new DummyInputFormat();
      FileInputFormat.setInputPaths(job, dir1 + "," + dir2 + "," + dir3);

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

      // expect the result is corrupt
      assertTrue(outStr.contains(NamenodeFsck.CORRUPT_STATUS));
      System.out.println(outStr);
      
      // bring up data nodes & cleanup cluster
      cluster.startDataNodes(conf, 4, true, null, null);
      cluster.waitActive();
      cluster.waitClusterUp();
      fs = cluster.getFileSystem();
      util.cleanup(fs, "/srcdat");
    } finally {

View Full Code Here

Examples of org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes()

      
      // manually set safemode.
      fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
      
      // now bring up the datanode and wait for it to be active.
      cluster.startDataNodes(conf, 1, true, null, null);
      cluster.waitActive();
      
      LOG.info("Datanode is started.");


      // wait longer than dfs.safemode.extension

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.