.NET on AWS – Persisting Application Data to S3 (Part 1)

Amazon S3 is the most commonly used managed storage solution in AWS. It provides object storage in a highly scalable and secure way. AWS guarantees 11 9s for its durability. Objects stored in S3 are shared access objects; shared meaning they can be accessed by different clients at the same time.

S3 provides low latency for data, and it has high throughput (able to move data in and out of S3 quickly). S3 is highly available, durable, and can encrypt data. S3 provides access management, lifecycle management, and the ability to query-in-place (no need to move to a data lake). Static website hosting is another very popular feature of S3.

The following storage tiers are offered by S3:

• Standard
• Intelligent Tiering (save cost)
• Standard IA
  • storage cost is reduced.
  • Cost for read.
• 1Zone IA
  • Lowest price but does not offer much availability/durability.

For more information, please visit the official website at this link.

Application Requirements

Persisting application data is a common requirement in software/services development. There are all kinds of data storage solutions available. RDBMS and NoSQL solutions are very popular for some applications, and same time storing data on files is also very common.

For our application, let's assume we are asked to store application data on files, and we decided to use AWS S3 storage service for this purpose.

Technical Stack

We will be building a web API using .NET6 tech stack. The application will allow users to perform typical CRUD operations for the Note entity. A very simple application with the bare minimum to keep our focus on data storage on AWS S3 from a .NET application.

The source code for the application is available on this GitHub repository.

We will be performing various actions on AWS cloud and our .NET application as follows

• AWS Resources Setup
  • Create S3 bucket
  • Create an IAM user and access keys.
  • Create a Policy and attach it with the User.
• .NET6 Application Wiring
  • Create a .NET6 application.
  • Wire .NET6 application with AWS resources.
• .NET6 Application Code
  • Write Application code for CRUD operations for S3 objects.

Let's start with these steps next:

Creating an S3 Bucket

S3 buckets are created in a region. The S3 bucket name has to be globally unique. The reason for that is that the DNS record is going to be assigned to each bucket we create.

For all other fields, we can use the defaults for now.

Once the bucket is created, the next step is to set up access to this bucket for our application.

Creating a User for Application Access

In order for our application to access the S3 bucket, we will need to create a new user in AWS IAM Service and give this user access to our S3 bucket (you can create a role instead if you like).

In the following screenshot, I created a user for our application:

For other fields, use the default settings. We’ll also need to set up programmatic access for this user (access keys), but we will do that a little bit later.

Next, let's create a policy that will allow permissions for the bucket.

Creating a Policy

From the IAM policies screen, we can create a custom policy as follows:

This policy is allowing some actions on the bucket we created earlier.

Next, we can attach this policy to the application user.

Attach Policy to User

From the IAM dashboard, we can edit the user to attach the policy as shown below:

Create Access Keys for User

We also need to create access keys for our users, which we will use in our .NET application later.

The following screen from the IAM user shows this step:

For now, I selected Local code as I will be running the application from my development machine. However, you can try other options if you like.

Here is the screen showing access keys created for the user. Note down these keys somewhere for later reference.

So, until this point,

• We have created an S3 bucket to store our application data files.
• We then create a user for our application.
• We created a custom policy to allow certain action permissions on the S3 bucket and attach this policy to our application user.
• We also created the access keys for this user, which we will use in our .NET application later.

The AWS resource setup part is done, and we can now start working on .NET6 Application part.

Create .NET6 Web API Application

To start, I created a .NET6 application using a visual studio built-in template, as shown below

.NET6 Application Wiring With S3

To wire up the application, the following configurations are setup in appsettings.json file and the corresponding C# class to read these configs (note these configs are pointing to the bucket we created earlier and AWS AccessKeys from the user setup earlier)

And in the Program.cs class read the configs and populate the S3Settings class as shown below:

At this point, we have wired up the configuration needed by our application to connect/access S3.

With all this done, we can now focus on writing the application code for the REST API, and we will do that in the next post.

Summary

In this post, we started with a basic introduction to S3 and discussed our application data storage requirement, and S3 can be used for this purpose. We created an S3 bucket to store our application data and also created a user policy and attached the policy to the user. We then created access keys for this user to use in our .NET application later. Next, we created a very basic .NET6 application and did some configuration wiring to make it ready to access our S3 bucket.

In the next post, we will write our application code to allow users to perform CRUD operations, which will result in creating various files in the S3 bucket, and we will be able to access those files in our application.

The source code is available on this GitHub repository.

Let me know if you have some comments or questions.