Examples of TransferManager

com.amazonaws.services.s3.transfer.TransferManager
can poll your transfer's status to check its progress if (myUpload.isDone() == false) { System.out.println("Transfer: " + myUpload.getDescription()); System.out.println(" - State: " + myUpload.getState()); System.out.println(" - Progress: " + myUpload.getProgress().getBytesTransferred()); } // Transfers also allow you to set a ProgressListener to receive // asynchronous notifications about your transfer's progress. myUpload.addProgressListener(myProgressListener); // Or you can block the current thread and wait for your transfer to // to complete. If the transfer fails, this method will throw an // AmazonClientException or AmazonServiceException detailing the reason. myUpload.waitForCompletion();
Note: Transfers are stored in memory. If the JVM is restarted, previous transfers are no longer accessible. If needed, clean up any multipart uploads that are incomplete.
org.syncany.plugins.transfer.TransferManager
The transfer manager synchronously connects to the remote storage. It is responsible for file upload, download and deletion.
All its operations are strictly synchronous and throw a {@code StorageException} if they fail. The implementations have to make surethat
- the repository is not corrupted, e.g. duplicate files or corrupt files
- files matching the specified file format are complete, i.e. fully uploaded
- methods that need an established connections re-connect if necessary
A transfer manager may organize files according to their type or name as it is optimal for the given storage. {@link RemoteFile}s can be classified by their sub-type. For network-transfer optimization reasons, it might be useful to place {@link MultichunkRemoteFile}s and {@link DatabaseRemoteFile}s in a separate sub-folder on the remote storage. @author Philipp C. Heckel

Examples of com.amazonaws.services.s3.transfer.TransferManager


        String bucketId = fileName.getBucketId();


        this.awsCredentials = awsCredentials;
        this.service = service;
        this.transferManager = new TransferManager(service);
        this.serverSideEncryption = S3FileSystemConfigBuilder
            .getInstance().getServerSideEncryption(fileSystemOptions);


        Region region = S3FileSystemConfigBuilder.getInstance().getRegion(
            fileSystemOptions);

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

  public void execute() throws MojoExecutionException, MojoFailureException {


    // Setup AWS S3 client
    AWSCredentials credentials = new BasicAWSCredentials(awsAccessKey, awsSecretKey);
    AmazonS3Client uploadClient = new AmazonS3Client(credentials);
    TransferManager transfers = new TransferManager(credentials);


    // Make sure key prefix does not start with a slash but has one at the
    // end
    if (keyPrefix.startsWith("/"))
      keyPrefix = keyPrefix.substring(1);
    if (!keyPrefix.endsWith("/"))
      keyPrefix = keyPrefix + "/";


    // Keep track of how much data has been transferred
    long totalBytesTransferred = 0L;
    int items = 0;
    Queue<Upload> uploads = new LinkedBlockingQueue<Upload>();


    try {
      // Check if S3 bucket exists
      getLog().debug("Checking whether bucket " + bucket + " exists");
      if (!uploadClient.doesBucketExist(bucket)) {
        getLog().error("Desired bucket '" + bucket + "' does not exist!");
        return;
      }


      getLog().debug("Collecting files to transfer from " + resources.getDirectory());
      List<File> res = getResources();
      for (File file : res) {
        // Make path of resource relative to resources directory
        String filename = file.getName();
        String extension = FilenameUtils.getExtension(filename);
        String path = file.getPath().substring(resources.getDirectory().length());
        String key = concat("/", keyPrefix, path).substring(1);


        // Delete old file version in bucket
        getLog().debug("Removing existing object at " + key);
        uploadClient.deleteObject(bucket, key);


        // Setup meta data
        ObjectMetadata meta = new ObjectMetadata();
        meta.setCacheControl("public, max-age=" + String.valueOf(valid * 3600));


        FileInputStream fis = null;
        GZIPOutputStream gzipos = null;
        final File fileToUpload;


        if (gzip && ("js".equals(extension) || "css".equals(extension))) {
          try {
            fis = new FileInputStream(file);
            File gzFile = File.createTempFile(file.getName(), null);
            gzipos = new GZIPOutputStream(new FileOutputStream(gzFile));
            IOUtils.copy(fis, gzipos);
            fileToUpload = gzFile;
            meta.setContentEncoding("gzip");
            if ("js".equals(extension))
              meta.setContentType("text/javascript");
            if ("css".equals(extension))
              meta.setContentType("text/css");
          } catch (FileNotFoundException e) {
            getLog().error(e);
            continue;
          } catch (IOException e) {
            getLog().error(e);
            continue;
          } finally {
            IOUtils.closeQuietly(fis);
            IOUtils.closeQuietly(gzipos);
          }
        } else {
          fileToUpload = file;
        }


        // Do a random check for existing errors before starting the next upload
        if (erroneousUpload != null)
          break;


        // Create put object request
        long bytesToTransfer = fileToUpload.length();
        totalBytesTransferred += bytesToTransfer;
        PutObjectRequest request = new PutObjectRequest(bucket, key, fileToUpload);
        request.setProgressListener(new UploadListener(credentials, bucket, key, bytesToTransfer));
        request.setMetadata(meta);


        // Schedule put object request
        getLog().info("Uploading " + key + " (" + FileUtils.byteCountToDisplaySize((int) bytesToTransfer) + ")");
        Upload upload = transfers.upload(request);
        uploads.add(upload);
        items ++;
      }
    } catch (AmazonServiceException e) {
      getLog().error("Uploading resources failed: " + e.getMessage());

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    catch (StorageException e) {
      cleanupFailed = true;
    }


    assertTrue(cleanupFailed);
    TransferManager transferManager = new TransactionAwareTransferManager(
        new UnreliableLocalTransferPlugin().createTransferManager(testConnection, null), null);
    assertEquals(2, transferManager.list(MultichunkRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "multichunks").list().length);
    assertEquals(2, transferManager.list(DatabaseRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "databases").list().length);
    assertEquals(1, transferManager.list(TransactionRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "transactions").list().length);
    assertEquals(1, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "actions").list().length);


    clientA.cleanup(cleanupOptions);
    assertEquals(1, transferManager.list(MultichunkRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "multichunks").list().length);
    assertEquals(3, transferManager.list(DatabaseRemoteFile.class).size());
    assertEquals(3, new File(testConnection.getPath(), "databases").list().length);
    assertEquals(0, transferManager.list(TransactionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "transactions").list().length);
    assertEquals(0, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "actions").list().length);


    clientA.deleteTestData();
  }

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    catch (StorageException e) {
      cleanupFailed = true;
    }


    assertTrue(cleanupFailed);
    TransferManager transferManager = new TransactionAwareTransferManager(
        new UnreliableLocalTransferPlugin().createTransferManager(testConnection, null), null);
    assertEquals(2, transferManager.list(MultichunkRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "multichunks").list().length);


    // Note that the list here differs from the actual files, because the transaction fails
    // while deletions have been done
    assertEquals(2, transferManager.list(DatabaseRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "databases").list().length);


    assertEquals(1, transferManager.list(TransactionRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "transactions").list().length);


    assertEquals(1, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "actions").list().length);


    File tempDir = TestFileUtil.createTempDirectoryInSystemTemp();
    File tempFile = File.createTempFile("multichunk", "", tempDir);
    for (RemoteFile remoteFile : transferManager.list(DatabaseRemoteFile.class).values()) {
      transferManager.download(remoteFile, tempFile);
      assertTrue(tempFile.exists());
      tempFile.delete();
    }


    clientA.cleanup(cleanupOptions);


    assertEquals(2, transferManager.list(MultichunkRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "multichunks").list().length);


    assertEquals(1, transferManager.list(DatabaseRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "databases").list().length);


    assertEquals(0, transferManager.list(TransactionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "transactions").list().length);


    assertEquals(0, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "actions").list().length);
  }

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    }
    catch (StorageException e) {
      cleanupFailed = true;
    }


    TransferManager transferManager = new TransactionAwareTransferManager(
        new UnreliableLocalTransferPlugin().createTransferManager(testConnection, null), null);


    assertTrue(cleanupFailed);
    assertEquals(2, transferManager.list(MultichunkRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "multichunks").list().length);


    assertEquals(1, transferManager.list(DatabaseRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "databases").list().length);


    assertEquals(0, transferManager.list(TransactionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "transactions").list().length);


    assertEquals(1, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(1, new File(testConnection.getPath(), "actions").list().length);


    // Two temporary files left (first deletion failed)
    assertEquals(2, transferManager.list(TempRemoteFile.class).size());
    assertEquals(2, new File(testConnection.getPath(), "temporary").list().length);


    clientA.cleanup(cleanupOptions);


    // Functional cleanup results in removal of action file and unreferenced files
    assertEquals(0, transferManager.list(ActionRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "actions").list().length);


    assertEquals(0, transferManager.list(TempRemoteFile.class).size());
    assertEquals(0, new File(testConnection.getPath(), "temporary").list().length);
  }

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    assertTrue(transactionTO.getActions().get(0).getRemoteFile().getName().contains("multichunk-"));
    assertTrue(transactionTO.getActions().get(1).getRemoteFile().getName().contains("multichunk-"));
    assertTrue(transactionTO.getActions().get(2).getRemoteFile().getName().contains("database-"));


    // 2. Double check if list() does not return the multichunk
    TransferManager transferManager = new TransactionAwareTransferManager(
        new UnreliableLocalTransferPlugin().createTransferManager(testConnection, null), null);
    Map<String, MultichunkRemoteFile> multiChunkList = transferManager.list(MultichunkRemoteFile.class);
    assertEquals(0, multiChunkList.size());


    // 3. Second try fails in the beginning, to see if cleanTransactions was successful
    boolean secondUpFailed = false;

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    assertTrue(transactionTO.getActions().get(0).getRemoteFile().getName().contains("multichunk-"));
    assertTrue(transactionTO.getActions().get(1).getRemoteFile().getName().contains("multichunk-"));
    assertTrue(transactionTO.getActions().get(2).getRemoteFile().getName().contains("database-"));


    // 2. Double check if list() does not return the multichunk
    TransferManager transferManager = new TransactionAwareTransferManager(
        new UnreliableLocalTransferPlugin().createTransferManager(testConnection, null), null);
    Map<String, MultichunkRemoteFile> multiChunkList = transferManager.list(MultichunkRemoteFile.class);
    assertEquals(0, multiChunkList.size());


    // 3. Second try fails in the beginning, to see if cleanTransactions was successful
    boolean secondUpFailed = false;

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    TransferPlugin plugin = Plugins.get(getPluginId(), TransferPlugin.class);


    TransferSettings connection = plugin.createEmptySettings();


    // This should cause a Storage exception, because the path does not exist
    TransferManager transferManager = plugin.createTransferManager(connection, null);


    transferManager.connect();
  }

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    // Setup
    File tempFromDir = TestFileUtil.createTempDirectoryInSystemTemp();
    File tempToDir = TestFileUtil.createTempDirectoryInSystemTemp();


    // Create connection, upload, list, download
    TransferManager transferManager = loadPluginAndCreateTransferManager();


    transferManager.init(true);
    transferManager.connect();


    // Clear up previous test (if test location is reused)
    cleanTestLocation(transferManager);


    // Run!

View Full Code Here

Examples of org.syncany.plugins.transfer.TransferManager

    }
  }


  @Test
  public void testDeleteNonExistentFile() throws StorageException {
    TransferManager transferManager = loadPluginAndCreateTransferManager();
    transferManager.connect();


    boolean deleteSuccess = transferManager.delete(new MultichunkRemoteFile("multichunk-dddddddddddddddddddddddddddddddddddddddd")); // does not
                                                                      // exist
    assertTrue(deleteSuccess);
  }

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.