Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

CloudFormation: Backing Up DynamoDB the Easy Way

DZone's Guide to

CloudFormation: Backing Up DynamoDB the Easy Way

Looking for a solution to back up your DyanmoDB tables? Come check out a proposed template and its architecture, which uses CloudFormation.

· Cloud Zone ·
Free Resource

Discover a centralized approach to monitor your virtual infrastructure, on-premise IT environment, and cloud infrastructure – all on a single platform.

DynamoDB is an easy-to-use NoSQL database available only on AWS. It comes with many useful features such:

  • Time-to-live for items
  • Secondary indexes
  • Scalable read and write throughput
  • Streams that contain all changes of a table

Unfortunately, there is no backup feature. You might say:

But DynamoDB replicates my data. The durability guarantees are super high. I will never lose my data. I can even replicate my data cross region

But what if your application has a bug and updates many items in a way that corrupts your data, or what if your cleanup script deletes the wrong items? You can not revert changes easily in DynamoDB. You need a backup in that case!

One of the solutions is to run a job on an EMR cluster that does the backup. But this is a multi-step process:

  1. Create an EMR cluster
  2. Wait until the cluster is ready
  3. Run the backup job
  4. Wait until the job is completed, retry if necessary
  5. Terminate the EMR cluster
  6. Wait until the cluster is terminated

Luckily, Data Pipeline is a service that can do the orchestration work for you. The Data Pipeline documentation has examples to back up and restore a DynmoDB table.

The only unsolved problem is: How to setup the Data Pipeline? My answer is: CloudFormation. I authored a template to backup a DynamoDB table. The template contains the Data Pipeline, S3 buckets for logs and the backups, and also the glue to alert you if the backup fails. The following figure shows the pipeline definition:

Data Pipeline to backup a DynamoDB table

If you want to set up a CloudFormation stack to back up a DynamoDB table, just follow the steps in the Free Template for AWS CloudFormation documentation.

One downside of this solution is that you have to spin up an EMR cluster for each table that you back up. A cluster consists of one master and one core node, e.g. of type m4.large. In us-east-1, this means two $0.10 charges (assuming the backup completes within one hour). Additionally, you pay, for EMR, two $0.03 charges (also assuming the backup finishes in under an hour). In total, you spend $0.26 per table per backup.

Learn how to auto-discover your containers and monitor their performance, capture Docker host and container metrics to allocate host resources, and provision containers.

Topics:
backup ,dynamodb ,cloud ,cloudformation ,template

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}