Build a Twitter Leaderboard App with Redis and AWS Lambda (Part 1)

Hello and welcome to this two-part blog series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first part (this one) covers the application - by the end of this blog, you should have the solution deployed, played around with it and in the process, have a good overview of the solution.

The second part is about the Infrastructure (IaaC to be specific) aspects - it's mostly centered around AWS CDK along with some code walkthrough.

Over the course of this blog series, you will learn:

• How to use Lambda and Redis together - including VPC and other configurations to make things work
• How to use Lambda Function URL
• How to deploy your Lambda function as a Docker container (not a zip file)
• Use AWS CDK to create all the components of the solution - Infrastructure (VPC, subnets, etc.), database as well as Lambda functions (this includes multiple stacks in the context of a single CDK app)

I have used Go for the Lambda functions (AWS-lambda-go) and infrastructure (with CKD Go support), but you should be easily able to apply the concepts to the programming language of your choice.

As always, the code is available on Github

Here is a quick overview of the services involved in the solution:

• Amazon MemoryDB for Redis - It is a durable, in-memory database service that is compatible with Redis, thus empowering you to build applications using the same flexible and friendly Redis data structures, APIs, and commands that they already use today.
• Lambda Function URL is a relatively new feature (at the time of writing this blog) that provides dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g. to serve as a webhook) and don't want to setup and configure an API Gateway.
• AWS Cloud Development Kit (CDK) is all about IaaC (Infrastructure-as-code). It is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. You can choose from a list of supported programming languages (at the time of writing - TypeScript, JavaScript, Python, Java, C#/.NET, and Go (in developer preview)) to define your infrastructure components as code, just like you would with any other application!

The Twitter Hashtag Leaderboard App

Don't worry, it's simpler than it sounds! Here is the high-level architecture:

The solution can be divided into two logical parts:

• The first part handles tweet ingestion: A Lambda function fetches tweets (from Twitter), extracts hashtags for each tweet, and stores them in MemoryDB (in a Redis Sorted Set). This function gets invoked based on a schedule based on a rule in CloudWatch trigger
• The second part provides the leaderboard functionality: This is yet another Lambda function that provides an HTTP(s) endpoint (thanks to Lambda Function URL) to query the sorted set and extract the top 10 hashtags (leaderboard)

I told you, it's quite simple!

Alright, with the intro out of the way, we can move on to the fun part - which is deploying the application. Before that, make sure you have the following ready:

Requirements

• Create an AWS account (if you do not already have one) and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
• Install and configure AWS CLI
• Install and bootstrap AWS CDK
• Setup Docker
• Install Go
• Get your Twitter API credentials

Deploy the Application - One CDK Stack at a Time

We will cover the details in the second part. For now, just know that the infrastructure part of this solution is comprised of three (CDK) Stacks (in the context of a single CDK App). Although it's possible to deploy all of them together (with cdk deploy --all), we will do it one by one. This way, you can review what's happening at each stage, introspect the components that have been created, and better understand how everything is wired together.

First, clone the GitHub repo:

git clone https://github.com/abhirockzz/twitter-leaderboard-app

Starting with the Infrastructure...

The first stack deploys a VPC (and also subnets, NAT gateway, etc.), a MemoryDB for the Redis cluster, and a few security groups.

Choose a password of your choice for MemoryDB and export it as an environment variable (this is just for demonstration purposes - for production, you will have specific processes in place to handle sensitive info)

Be mindful of the password requirements. From the documentation:

"In particular, be aware of these user password constraints when using ACLs for MemoryDB:

• Passwords must be 16–128 printable characters.
• The following non-alphanumeric characters are not allowed: , "" / @."

Change to the correct folder and kick off the stack deployment:

export MEMORYDB_PASSWORD=<enter a password e.g. P@ssw0rd12345678>

cd cdk

# stack1 is the name of the stack - used for simplicity
cdk deploy stack1

There is a lot to be done. While CDK is hard at work for us, you need to wait patiently :) This is probably a good time to navigate to the CloudFormation in AWS console and see what's going on behind the scenes.

Once the stack creation is complete, go to the AWS console and check out your freshly minted VPC, MemoryDB cluster, and other components!

Here is the stack output for your reference:

The next two stacks deploy separate Lambda functions. Before moving on though, make sure you build the Go binaries for both these functions.

Go, build!

Move to the respective folders and just invoke go build for each function:

cd tweet-ingest && GOOS=linux go build -o app
cd leaderboard-function && GOOS=linux go build -o app

To package the Lambda function as a Docker container, I used the Go:1.x base image. But, you can explore other options as well. During deployment (via cdk deploy), the Docker image is built locally, pushed to a private ECR registry and finally the Lambda function is created - all this, with a few lines of code!

The Second Stack - For the First Lambda Function

The function requires your Twitter API credentials (along with the MemoryDB password) - seeds them as environment variables. Then, initiate stack creation:

export MEMORYDB_PASSWORD=<enter the password you had previously chosen e.g. P@ssw0rd12345678>
export TWITTER_API_KEY=<enter twitter API key>
export TWITTER_API_SECRET=<enter twitter API secret>
export TWITTER_ACCESS_TOKEN=<enter twitter access token>
export TWITTER_ACCESS_TOKEN_SECRET=<enter twitter API access token secret>

# note the name of the stack is stack2
cdk deploy stack2

This one will be faster (compared to stack1), I promise!

Once the stack creation is complete, to make things work, there is one manual step required. Go to the AWS console, open the Lambda function (named tweet-ingest-function), that was just created, click Add Trigger and manually add the CloudWatch trigger configuration.

Now your function will be automatically triggered once every minute!

To check how things are going, check out the logs for your Lambda function (AWS console > Lambda > Monitor)

I would also encourage you to check the VPC configuration for your Lambda function:

And Finally, Deploy the Leaderboard Function

export MEMORYDB_PASSWORD=<enter the password you had previously chosen e.g. P@ssw0rd12345678>

# note the name of the stack is stack3
cdk deploy stack3

After successful deployment, you should have a Lambda Function URL ready for you to access - you can simply copy it from the stack output!

Just access the endpoint (I have used curl CLI, but a browser should work just fine):

curl -i <enter lambda function URL from the stack output>

You will get back a JSON payload with info about the top 10 hashtags along with their names and the number of times they were mentioned - something similar to this:

[
  {
    "Score": 121,
    "Member": "AWS"
  },
  {
    "Score": 56,
    "Member": "gaming"
  },
  {
    "Score": 56,
    "Member": "RESTOCK"
  },
  {
    "Score": 56,
    "Member": "ALERT"
  },
  {
    "Score": 35,
    "Member": "aws"
  },
  {
    "Score": 26,
    "Member": "rtx3080"
  },
  {
    "Score": 26,
    "Member": "geforce3080"
  },
  {
    "Score": 24,
    "Member": "Infographic"
  },
  {
    "Score": 23,
    "Member": "箱マスク"
  },
  {
    "Score": 23,
    "Member": "startups"
  }
]

This concludes the first part. In the second one, we dive into the CDK aspects and look at some Go code - see you there!