Upload on S3 with the jclouds Library

Eugen Paraschiv

Apr. 18, 13 · Interview

Likes (1)

Comment

Save

8.6K Views

There are several good ways to upload content to an S3 bucket in the Java world – in this article we’ll look at what the jclouds library provides for this purpose.

To use jclouds – specifically the APIs discussed in this article, this simple Maven dependency should be added to the pom of the project:

<dependency>
   <groupId>org.jclouds</groupId>
   <artifactId>jclouds-allblobstore</artifactId>
   <version>1.5.9</version>
</dependency>

1. Uploading to Amazon S3

The first step, in order to access any of these APIs, is to create a BlobStoreContext:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(BlobStoreContext.class);

This represents the entry-point to a general key-value storage service, such as Amazon S3 – but not limited to it.

For the more specific S3 only implementation, the context can be created similarly:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(S3BlobStoreContext.class);

And even more specifically:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(AWSS3BlobStoreContext.class);

When the authenticated context is no longer needed, closing it is required to release all resources – threads and connections – associated to it.

2. The four S3 APIs of jclouds

The jclouds library provides four different APIs to upload content to S3 bucket, ranging from simple but inflexible to complex and powerful, all obtained via the BlobStoreContext. Let’s start with the simplest.

**2.1. Upload via the Map API**

The easiest way jclouds can be used to interact with an S3 bucket is by representing that bucket as a Map. The API is obtained from the context:

InputStreamMap bucket = context.createInputStreamMap("bucketName");

Then, to upload a simple HTML file:

bucket.putString("index1.html", "<html><body>hello world1</body></html>");

The InputStreamMap API exposes several other types of PUT operations – files, raw bytes – both for single and bulk.

A simple integration test can be used as an example:

@Test
public void whenFileIsUploadedToS3WithMapApi_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   InputStreamMap bucket = context.createInputStreamMap("bucketName");
   bucket.putString("index1.html", "<html><body>hello world1</body></html>");
   context.close();
}

2.2. Upload via BlobMap

Using the simple Map API is straightforward but ultimately limited – for example, there is no way to pass in metadata about the content being uploaded. When more flexibility and customization is necessary, this simplified approach to uploading data to S3 via a Map is no longer enough.

The next API we’ll look at is the Blob Map API – this is obtained from the context:

BlobMap bucket = context.createBlobMap("bucketName");

The API allows the client to access more lower level details, such as Content-Length, Content-Type, Content-Encoding, eTag hash and others; to upload new content in the bucket:

Blob blob = bucket.blobBuilder().name("index2.html").
   payload("<html><body>hello world2</body></html>").
      contentType("text/html").calculateMD5().build();

The API also allows setting a variety of payloads on the create request.

A simple integration test for uploading a basic HTML file to S3 via the Blob Map API:

@Test
public void whenFileIsUploadedToS3WithBlobMap_thenNoExceptions() throws IOException {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobMap bucket = context.createBlobMap("bucketName");
   Blob blob = bucket.blobBuilder().name("index2.html").
      payload("<html><body>hello world2</body></html>").
         contentType("text/html").calculateMD5().build();
   bucket.put(blob.getMetadata().getName(), blob);
   context.close();
}

2.3. Upload via BlobStore

The previous APIs had no way to upload content using multipart upload – this makes them ill suited when working with large files. This limitation is addressed by the next API we’re going to look at – the synchronous BlobStore API.

This is obtained from the context:

BlobStore blobStore = context.getBlobStore();

To use the multipart support and upload a file to S3:

Blob blob = blobStore.blobBuilder("index3.html").
   payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());

The payload builder is the same one that was being used by the BlobMap API, so the same flexibility in specifying lower level metadata information about the blob is available here. The difference is the PutOptions supported by the PUT operation of the API – namely the multipart support.

The previous integration test now has multipart enabled:

@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobStore blobStore = context.getBlobStore();
   Blob blob = blobStore.blobBuilder("index3.html").
      payload("<html><body>hello world3</body></html>").contentType("text/html").build();
   blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
   context.close();
}

2.4. Upload via AsyncBlobStore

While the previous BlobStore API was synchronous, there is also an asynchronous API for BlobStore – AsyncBlobStore. The API is similarly obtained from the context:

AsyncBlobStore blobStore = context.getAsyncBlobStore();

The only difference between the two is that the async API is returning ListenableFuture for the PUT asynchronous operation:

Blob blob = blobStore.blobBuilder("index4.html").
   .payload("<html><body>hello world4</body></html>").build();
blobStore.putBlob("bucketName", blob)<strong>.get()</strong>;

The integration test displaying this operation is similar to the synchronous one:

@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobStore blobStore = context.getBlobStore();
   Blob blob = blobStore.blobBuilder("index4.html").
      payload("<html><body>hello world4</body></html>").contentType("text/html").build();
   Future<String> putOp = blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
   putOp.get();
   context.close();
}

3. Conclusion

In this article, we analysed the four APIs that the jclouds library provides to upload content to Amazon S3. These four APIs are generic and they work with other key-value storage services as well – such as Microsoft Azure Storage for example.

In the next article we’ll look at the Amazon specific S3 API available in jclouds – the AWSS3Client. We’ll implement the operation of uploading a large file, dynamically calculate the optimal number of parts for any given file, and perform the upload of all parts in parallel.

P.S. You might dig following me on Twitter.

AWS Upload API Library integration test

Published at DZone with permission of Eugen Paraschiv, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

Trending