Real-Time Cross-Region DynamoDB Replication Using AWS Lambda

With the increasing amount of data that's available to be processed these days, people are looking to the cloud to resolve their scalability issues. One of the challenges is keeping the data in sync across various data centers/geographical locations, which is required for failure fallback and disaster recovery.

AWS Dynamo DB has emerged as a leader in NoSQL cloud solutions.

Recently, AWS launched a DynamoDB cross-region replication application, which runs on a dedicated EC2 instance.

In this post, we will discuss the serverless approach to achieving this using DynamoDB streams, triggers, and AWS Lambda.

To achieve this, we will write a lambda function and configure it with a DynamoDB trigger that will invoke the lambda function on any updates to a particular DynamoDB table.

1. Create 2 Similar Schemas/Tables in 2 Regions

For pre-existing tables, we might have to do a one-time export of that table to the other regions using AWS Data Pipeline.

For example:

Region:eg. Mumbai and Sydney 
Table Employee:
ID (Partition Key)
Name (Sort Key)
Job Title

2. Create a Lambda Function to Process Incoming DynamoDB Stream Events

/*Override the handle request method which serves as an 
entry point for the lambda function*/

@Override
public Integer handleRequest(DynamodbEvent event, Context context) {
 context.getLogger().log("Received event: " + event);

 DynamoDBProcessEvents dbProcessEvents = new DynamoDBProcessEvents();
 List < DynamodbStreamRecord > records = event.getRecords();
 try {
  for (DynamodbStreamRecord record: records) {
   //logic to process the incomng events
   switch (record.getEventName()) {
    case "INSERT":
     insertDynamoDBItem(record);
     break;
    case "MODIFY":
     updateDynamoDBItem(record);
     break;
    case "REMOVE":
     deleteDynamoDBItem(record);
     break;
    default:
     break;
   }
  }
 }
}

//Insert in to backup Table
private void insertDynamoDBItem(DynamodbStreamRecord record) {
 Map < String, AttributeValue > item = record.getDynamodb().getNewImage();
 PutItemRequest itemIn = new PutItemRequest( < backup_dbtable_name > , item);
 dynamoDBClient.putItem(itemIn);
}

// Delete from backup table
private void deleteDynamoDBItem(DynamodbStreamRecord record) {
 Map < String, AttributeValue > itemToBeRemoved = record.getDynamodb().getNewImage();;
 PrimaryKey keyDelete = getPrimaryKey(itemToBeRemoved);
 table.deleteItem(keyDelete);
}


// Update item in backup table. In this case we are deleting the existing record 
// and inserting the updated record. UpdateItemSpec or UpdateItemRequest can also
//be used
private void updateDynamoDBItem(DynamodbStreamRecord record) {
 deleteDynamoDBItem(record);
 insertDynamoDBItem(record);
}
private PrimaryKey getPrimaryKey(Map < String, AttributeValue > item) {
 AttributeValue valueHash = item.get(HASH_KEY);
 AttributeValue valueSort = item.get(SORT_KEY);
 PrimaryKey key = new PrimaryKey(HASH_KEY, Integer.parseInt(valueHash.getN()), SORT_KEY, valueSort.getS());
 return key;
}

3. Upload the Lambda Function

Create a role in IAM and assign it to the lambda function. Make sure to add policies to the role that allow read/write access to DynamoDB. 

4. Create Event Source Mapping in AWS Lambda

This event source mapping associates the DynamoDB stream with your lambda function. After you create this event source mapping, AWS lambda starts polling the stream.