Build a Twitter Leaderboard App With Redis and AWS Lambda (Part 2)

This is the second blog post of a two-part series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first post was about the solution overview and deployment. Hopefully, you were able to try it out end to end. As promised, part two will cover the infrastructure aspects (IaC to be specific) which are comprised of three (CDK) stacks (in the context of a single CDK App).

I will provide a walk-through of the CDK code which is written in Go, thanks to the CDK Go support. AWS Cloud Development Kit (CDK) is all about IaC (Infrastructure-as-code).

Architecture of the Solution

Just as a refresher, here is the high-level architecture of the solution:

Division of Architecture

The architecture is divided into two logical parts:

1. The first part handles tweet ingestion: A Lambda function fetches tweets (from Twitter), extracts hashtags for each tweet, and stores them in MemoryDB (in a Redis Sorted Set). This function gets invoked based on a schedule based on a rule in the CloudWatch trigger.
2. The second part provides the leaderboard functionality: This is yet another Lambda function that provides an HTTP(s) endpoint (thanks to Lambda Function URL) to query the sorted set and extract the top 10 hashtags (leaderboard).

Services Overview

Here is a quick overview of the services involved in the solution:

• Amazon MemoryDB for Redis: It is a durable, in-memory database service that is compatible with Redis, thus empowering you to build applications using the same flexible and friendly Redis data structures, APIs, and commands that they already use today.
• Lambda Function URL is a relatively new feature (at the time of writing this blog) that provides a dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g., to serve as a webhook) and don't want to set up and configure an API Gateway.
• As stated earlier, AWS Cloud Development Kit (CDK) is all about IaC (Infrastructure-as-code). It is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. You can choose from a list of supported programming languages (at the time of writing, this list includes TypeScript, JavaScript, Python, Java, C#/.Net, and Go, in developer preview) to define your infrastructure components as code, just like you would with any other application!

CDK Code Walk Through

The solution is comprised of three (CDK) Stacks (in the context of a single CDK App).

• The first stack deploys a VPC (and also subnets, NAT gateway, etc.), a MemoryDB for the Redis cluster, and a few security groups.
• The second stack deploys the first Lambda function which is responsible for ingesting tweet data into Redis.
• Finally, the third stack deploys the leaderboard Lambda function. 

Note that we could have included all the above as part of a single stack since this is a demo application. However, I wanted to demonstrate how you can leverage multiple stacks within a single CDK application. For a more complex production infrastructure, it makes sense to use multiple stacks in order to keep your resources decoupled and makes it much easier to handle and reason about your CDK logic. 

Let's walk through the code, one stack at a time:

Please note that some of the code has been redacted/omitted for brevity - you can always refer to complete code in the GitHub repo.

1.  Start with the infrastructure stack.

stack := awscdk.NewStack(scope, &id, &sprops)
vpc = awsec2.NewVpc(stack, jsii.String("demo-vpc"), nil)

authInfo := map[string]interface{}{"Type": "password", "Passwords": []string{memorydbPassword}}
user = awsmemorydb.NewCfnUser(stack, jsii.String("demo-memorydb-user"), &awsmemorydb.CfnUserProps{UserName: jsii.String("demo-user"), AccessString: jsii.String(accessString), AuthenticationMode: authInfo})
acl := awsmemorydb.NewCfnACL(stack, jsii.String("demo-memorydb-acl"), &awsmemorydb.CfnACLProps{AclName: jsii.String("demo-memorydb-acl"), UserNames: &[]*string{user.UserName()}})

//snip .....

subnetGroup := awsmemorydb.NewCfnSubnetGroup(stack, jsii.String("demo-memorydb-subnetgroup"), &awsmemorydb.CfnSubnetGroupProps{SubnetGroupName: jsii.String("demo-memorydb-subnetgroup"), SubnetIds: &subnetIDsForSubnetGroup})

memorydbSecurityGroup = awsec2.NewSecurityGroup(stack, jsii.String("memorydb-demo-sg"), &awsec2.SecurityGroupProps{Vpc: vpc, SecurityGroupName: jsii.String("memorydb-demo-sg"), AllowAllOutbound: jsii.Bool(true)})

memorydbCluster = awsmemorydb.NewCfnCluster(//... details omitted)

//...snip

twitterIngestFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)
twitterLeaderboardFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)

memorydbSecurityGroup.AddIngressRule(//... details omitted)
memorydbSecurityGroup.AddIngressRule(//... details omitted)

To summarize:

• awsec2.NewVpc:  A single line of code is all it takes to create VPC and related components such as public and private subnets, NAT gateways, and more. Compare that against a standard CloudFormation template that you would need to write to get this done!
• We created a User (along with a password for authentication), ACL (Access Control List for authorization), and Subnet group for the MemoryDB cluster. Refer to them during cluster creation with awsmemorydb.NewCfnCluster. 
• We also created the required security groups. Their main role is to allow Lambda functions to access MemoryDB. We specify explicit Inbound rules to make that possible.
  • One for MemoryDB cluster
  • One each for both the Lambda functions

Note: We are using an L1 construct for MemoryDB for Redis.

2.  The next stack deploys the tweets' ingestion Lambda Function.

//....

memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USER": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword()), "TWITTER_API_KEY": jsii.String(getTwitterAPIKey()), "TWITTER_API_SECRET": jsii.String(getTwitterAPISecret()), "TWITTER_ACCESS_TOKEN": jsii.String(getTwitterAccessToken()), "TWITTER_ACCESS_TOKEN_SECRET": jsii.String(getTwitterAccessTokenSecret())}

awslambda.NewDockerImageFunction(stack, jsii.String("lambda-memorydb-func"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(tweetIngestionFunctionName), Environment: lambdaEnvVars, Timeout: awscdk.Duration_Seconds(jsii.Number(20)), Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(tweetIngestionFunctionPath), nil), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterIngestFunctionSecurityGroup}})

//....

It's quite simple compared to the previous stack. We start by defining the environment variables required by our Lambda function (including Twitter API credentials) and then deploy it as a Docker image.

For the function to be packaged as a Docker image, I used the Go:1.x base image, but you can explore other options as well. During deployment, the Docker image is built locally, pushed to a private ECR registry, and finally the Lambda function is created: all this, with a few lines of code!

It's worth checking the L2 construct (in alpha state at the time of writing) which makes it even simpler to deploy Go functions using CDK. You can refer to the documentation.

Notice that the MemoryDB cluster and security group are automatically referred/looked-up from the previous stack (not re-created!).

3.  Finally, the third stack takes care of the leaderboard Lambda function. It's quite similar to the previous one, except for the addition of the Lambda Function URL (awslambda.NewFunctionUrl) which we use as the output for the stack:

//....
memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USERNAME": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword())}

function := awslambda.NewDockerImageFunction(stack, jsii.String("twitter-hashtag-leaderboard"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(hashtagLeaderboardFunctionName), Environment: lambdaEnvVars, Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(hashtagLeaderboardFunctionPath), nil), Timeout: awscdk.Duration_Seconds(jsii.Number(5)), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterLeaderboardFunctionSecurityGroup}})

funcURL := awslambda.NewFunctionUrl(stack, jsii.String("func-url"), &awslambda.FunctionUrlProps{AuthType: awslambda.FunctionUrlAuthType_NONE, Function: function})

awscdk.NewCfnOutput(stack, jsii.String("Function URL"), &awscdk.CfnOutputProps{Value: funcURL.Url()})

That's all for this blog post. I'll close out with links to AWS Go CDK v2 references:

• MemoryDB
• Lambda  
• VPC, etc. 
• CDK Go v2 

This concludes the two-part series. Stay tuned for more and as always, happy coding!