Examples of org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$Packet

org.apache.hadoop.hdfs.DFSClient
DFSOutputStream creates files from a stream of bytes. The client application writes data that is cached internally by this stream. Data is broken up into packets, each packet is typically 64K in size. A packet comprises of chunks. Each chunk is typically 512 bytes and has an associated checksum with it. When a client application fills up the currentPacket, it is enqueued into dataQueue. The DataStreamer thread picks up packets from the dataQueue, sends it to the first datanode in the pipeline and moves it from the dataQueue to the ackQueue. The ResponseProcessor receives acks from the datanodes. When an successful ack for a packet is received from all datanodes, the ResponseProcessor removes the corresponding packet from the ackQueue. In case of error, all outstanding packets and moved from ackQueue. A new pipeline is setup by eliminating the bad datanode from the original pipeline. The DataStreamer now starts sending packets from the dataQueue.


  @Test
  public void testDecommissionStatus() throws IOException, InterruptedException {
    InetSocketAddress addr = new InetSocketAddress("localhost", cluster
        .getNameNodePort());
    DFSClient client = new DFSClient(addr, conf);
    DatanodeInfo[] info = client.datanodeReport(DatanodeReportType.LIVE);
    assertEquals("Number of Datanodes ", 2, info.length);
    DistributedFileSystem fileSys = cluster.getFileSystem();
    DFSAdmin admin = new DFSAdmin(cluster.getConfiguration(0));


    short replicas = 2;

View Full Code Here

            DFSConfigKeys.DFS_CLIENT_FAILOVER_SLEEPTIME_MAX_DEFAULT));
    ClientProtocol proxy = (ClientProtocol) Proxy.newProxyInstance(
        failoverProxyProvider.getInterface().getClassLoader(),
        new Class[] { ClientProtocol.class }, dummyHandler);
    
    DFSClient client = new DFSClient(null, proxy, conf, null);
    return client;
  }

View Full Code Here

    }
  }


  @Test (timeout=60000)
  public void testCreateSnapshot() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new CreateSnapshotOp(client, "/test", "s1");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testDeleteSnapshot() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new DeleteSnapshotOp(client, "/test", "s1");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testRenameSnapshot() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new RenameSnapshotOp(client, "/test", "s1", "s2");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testCreate() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new CreateOp(client, "/testfile");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testAppend() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new AppendOp(client, "/testfile");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testRename() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new RenameOp(client, "/file1", "/file2");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testRename2() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new Rename2Op(client, "/file1", "/file2");
    testClientRetryWithFailover(op);
  }

View Full Code Here

    testClientRetryWithFailover(op);
  }
  
  @Test (timeout=60000)
  public void testConcat() throws Exception {
    final DFSClient client = genClientWithDummyHandler();
    AtMostOnceOp op = new ConcatOp(client, new Path("/test/file"), 5);
    testClientRetryWithFailover(op);
  }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$Packet

org.apache.hadoop.fs.FileStatus

org.apache.hadoop.fs.FSDataOutputStream

org.apache.hadoop.fs.FsServerDefaults

org.apache.hadoop.fs.FsStatus

org.apache.hadoop.fs.Hdfs

org.apache.hadoop.fs.HdfsBlockLocation

org.apache.hadoop.fs.LocatedFileStatus

org.apache.hadoop.fs.MD5MD5CRC32CastagnoliFileChecksum

org.apache.hadoop.fs.MD5MD5CRC32FileChecksum

org.apache.hadoop.fs.MD5MD5CRC32GzipFileChecksum

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.