Useful Tools for Local Development With AWS Services
Over the last 2.5 years, I've been working with AWS and a wide range of its services. In this post, I'll describe some of the tools that I use.
Join the DZone community and get the full member experience.Join For Free
Over the last 2.5 years, I've been working with AWS and a wide range of its services. During this time, I noticed that for most projects, it's useful to be able to test your application against AWS services without having to deploy or move your code into the cloud. There are several free solutions available for you to use depending on the services required by your project. In this post, I'll describe some of the tools that I use.
At one of my previous projects, we made extensive use of the combination of DynamoDB and Elasticsearch for storing and querying data. The fact that DynamoDB is a managed database service with immense scale and performance benefits makes DynamoDB a great fit for high traffic applications.
As a user, it's quite simple to use as it's a key-value store. Most of the other AWS databases are managed instances of existing services. However, DynamoDB is an AWS specific service that you can't really download and install locally. Luckily, back in 2018, AWS introduced a simpler way to work with DynamoDB utilizing DynamoDB local, a dockerized version of DynamoDB. You can simply run as a docker container to develop and test against.
Running DynamoDB local is as simple as executing:
$ docker run -p 8000:8000 amazon/dynamodb-local
Or, if it's part of a bigger set of dependencies, you could leverage
With that, it's a matter of running:
$ docker-compose up
And you should see something like:
With the AWS CLI, you can easily query for available tables:
$ aws dynamodb list-tables --endpoint-url http://localhost:8000
Which should result in something like:
And of course, you can use the AWS SDK with your preferred language as well.
I hear you thinking: are there no limitations? Yes, of course, there are some limitations with using DynamoDB local compared to the managed service. For instance, parallel scans are not supported (they will happen sequentially). Most limitations are nicely outlined in the DynamoDB Developer Guide.
Amazon also provides docker images for other services as well, like AWS Step functions Local and OpenDistro for Elasticsearch. Be sure to check out the Amazon repo on DockerHub for other useful images.
When you're developing a simple service that only depends on DynamoDB, DynamoDB local is a good choice. However, once you start to leverage more and more services, it might be worthwhile to look for other options as not all services are available as single docker images.
When you're building services that are part of a microservices architecture, you're probably using other AWS services like SNS, SQS, and perhaps S3. This is where a tool like LocalStack can add a lot of value. So what is LocalStack?
LocalStack is a project open-sourced by Atlassian that provides an easy way to develop AWS cloud applications directly from your localhost. It spins up a testing environment on your local machine that provides almost the same feature parity and APIs as the real AWS cloud environment, minus the scaling and robustness, of course.
Localstack focuses primarily on providing a local AWS cloud environment that adheres to the AWS APIs and offers a free and pro version, which you can leverage depending on your requirements. In my experience, the free/community version offers a lot of value and supports a whole range of services.
You can install LocalStack via
pip if you're familiar with python and its package system, or you can use it via docker(compose). On my Mac, I found that installing LocalStack as a python package was a bit of a hassle, so I always prefer to use it via docker-compose.
Using LocalStack with docker-compose is as simple as creating a docker-compose.yml file with the content:
If you're running on a Mac, be sure to prepend
TMPDIR=/private$TMPDIR before running:
$ TMPDIR=/private$TMPDIR docker-compose up
Afterward, you should see something similar to the following output:
As you can see, it starts a whole bunch of services out of the box. If you don't use all those services, you can also provide a list of services required when starting
localstackby providing a
$ TMPDIR=/private$TMPDIR SERVICES=s3,sqs docker-compose up
Now you should see in the startup output that it only started S3 and SQS:
Update: I just learned that homebrew also supports installing LocalStack. I've not used it, so I can't say if it's any good, but it looks pretty simple :-)
$ brew install localstack
If you don't want to manually start LocalStack via docker-compose but want to start it, for instance, during your build/test phase, you can also leverage testcontainers and add a
localstackrule to your Unit test:
If you're already using LocalStack, it's worthwhile to also install
awslocal. It's a CLI that proxies the AWS CLI and adds the
--endpoint-url http://localhost:4566/ after every command, so you don't have to.
You can install it by running:
$ pip install awscli-local
Localstack used to come with a Web UI, which is now marked as deprecated. As an alternative, I would recommend using Commandeer. It's a very useful tool and also supports working with LocalStack (next to a whole bunch of other services). It can give a nice overview of the services started with LocalStack and offers dashboards and UIs, for example, DynamoDB, in which you can explore data, create tables, etc.
If you know other useful tools that help you in your day to day work working with and developing on the AWS platform, feel free to leave a comment!
Published at DZone with permission of Jeroen Reijn. See the original article here.
Opinions expressed by DZone contributors are their own.