DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Using the OpenAPI Generator To Build Out an Integration Test Client
  • 4 Email Validation and Verification Techniques for App Developers
  • Cloud Backends for Frontend Developers
  • Implement OpenAPI for AWS Lambda in Quarkus Using API Gateway Integrator

Trending

  • The API-Centric Revolution: Decoding Data Integration in the Age of Microservices and Cloud Computing
  • AI for Web Devs: Project Introduction and Setup
  • Understanding Europe's Cyber Resilience Act and What It Means for You
  • Five Tools for Data Scientists to 10X their Productivity
  1. DZone
  2. Data Engineering
  3. Databases
  4. Upload on S3 with the jclouds Library

Upload on S3 with the jclouds Library

Eugen Paraschiv user avatar by
Eugen Paraschiv
·
Apr. 18, 13 · Interview
Like (1)
Save
Tweet
Share
7.86K Views

Join the DZone community and get the full member experience.

Join For Free

There are several good ways to upload content to an S3 bucket in the Java world – in this article we’ll look at what the jclouds library provides for this purpose.

To use jclouds – specifically the APIs discussed in this article, this simple Maven dependency should be added to the pom of the project:

<dependency>
   <groupId>org.jclouds</groupId>
   <artifactId>jclouds-allblobstore</artifactId>
   <version>1.5.9</version>
</dependency>

1. Uploading to Amazon S3

The first step, in order to access any of these APIs, is to create a BlobStoreContext:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(BlobStoreContext.class);

This represents the entry-point to a general key-value storage service, such as Amazon S3 – but not limited to it.

For the more specific S3 only implementation, the context can be created similarly:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(S3BlobStoreContext.class);

And even more specifically:

BlobStoreContext context =
  ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
    .buildView(AWSS3BlobStoreContext.class);

When the authenticated context is no longer needed, closing it is required to release all resources – threads and connections – associated to it.

2. The four S3 APIs of jclouds

The jclouds library provides four different APIs to upload content to S3 bucket, ranging from simple but inflexible to complex and powerful, all obtained via the BlobStoreContext. Let’s start with the simplest.

2.1. Upload via the Map API

The easiest way jclouds can be used to interact with an S3 bucket is by representing that bucket as a Map. The API is obtained from the context:

InputStreamMap bucket = context.createInputStreamMap("bucketName");

Then, to upload a simple HTML file:

bucket.putString("index1.html", "<html><body>hello world1</body></html>");

The InputStreamMap API exposes several other types of PUT operations – files, raw bytes – both for single and bulk.

A simple integration test can be used as an example:

@Test
public void whenFileIsUploadedToS3WithMapApi_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   InputStreamMap bucket = context.createInputStreamMap("bucketName");
   bucket.putString("index1.html", "<html><body>hello world1</body></html>");
   context.close();
}

2.2. Upload via BlobMap

Using the simple Map API is straightforward but ultimately limited – for example, there is no way to pass in metadata about the content being uploaded. When more flexibility and customization is necessary, this simplified approach to uploading data to S3 via a Map is no longer enough.

The next API we’ll look at is the Blob Map API – this is obtained from the context:

BlobMap bucket = context.createBlobMap("bucketName");

The API allows the client to access more lower level details, such as Content-Length, Content-Type, Content-Encoding, eTag hash and others; to upload new content in the bucket:

Blob blob = bucket.blobBuilder().name("index2.html").
   payload("<html><body>hello world2</body></html>").
      contentType("text/html").calculateMD5().build();

The API also allows setting a variety of payloads on the create request.

A simple integration test for uploading a basic HTML file to S3 via the Blob Map API:

@Test
public void whenFileIsUploadedToS3WithBlobMap_thenNoExceptions() throws IOException {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobMap bucket = context.createBlobMap("bucketName");
   Blob blob = bucket.blobBuilder().name("index2.html").
      payload("<html><body>hello world2</body></html>").
         contentType("text/html").calculateMD5().build();
   bucket.put(blob.getMetadata().getName(), blob);
   context.close();
}

2.3. Upload via BlobStore

The previous APIs had no way to upload content using multipart upload – this makes them ill suited when working with large files. This limitation is addressed by the next API we’re going to look at – the synchronous BlobStore API.

This is obtained from the context:

BlobStore blobStore = context.getBlobStore();

To use the multipart support and upload a file to S3:

Blob blob = blobStore.blobBuilder("index3.html").
   payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());

The payload builder is the same one that was being used by the BlobMap API, so the same flexibility in specifying lower level metadata information about the blob is available here. The difference is the PutOptions supported by the PUT operation of the API – namely the multipart support.

The previous integration test now has multipart enabled:

@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobStore blobStore = context.getBlobStore();
   Blob blob = blobStore.blobBuilder("index3.html").
      payload("<html><body>hello world3</body></html>").contentType("text/html").build();
   blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
   context.close();
}

2.4. Upload via AsyncBlobStore

While the previous BlobStore API was synchronous, there is also an asynchronous API for BlobStore – AsyncBlobStore. The API is similarly obtained from the context:

AsyncBlobStore blobStore = context.getAsyncBlobStore();

The only difference between the two is that the async API is returning ListenableFuture for the PUT asynchronous operation:

Blob blob = blobStore.blobBuilder("index4.html").
   .payload("<html><body>hello world4</body></html>").build();
blobStore.putBlob("bucketName", blob)<strong>.get()</strong>;

The integration test displaying this operation is similar to the synchronous one:

@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
   BlobStoreContext context =
      ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
         .buildView(AWSS3BlobStoreContext.class);
   BlobStore blobStore = context.getBlobStore();
   Blob blob = blobStore.blobBuilder("index4.html").
      payload("<html><body>hello world4</body></html>").contentType("text/html").build();
   Future<String> putOp = blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
   putOp.get();
   context.close();
}

3. Conclusion

In this article, we analysed the four APIs that the jclouds library provides to upload content to Amazon S3. These four APIs are generic and they work with other key-value storage services as well – such as Microsoft Azure Storage for example.

In the next article we’ll look at the Amazon specific S3 API available in jclouds – the AWSS3Client. We’ll implement the operation of uploading a large file, dynamically calculate the optimal number of parts for any given file, and perform the upload of all parts in parallel.

P.S. You might dig following me on Twitter.





 

AWS Upload API Library integration test

Published at DZone with permission of Eugen Paraschiv, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Using the OpenAPI Generator To Build Out an Integration Test Client
  • 4 Email Validation and Verification Techniques for App Developers
  • Cloud Backends for Frontend Developers
  • Implement OpenAPI for AWS Lambda in Quarkus Using API Gateway Integrator

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: