DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • The Technology Stack Needed To Build a Web3 Application
  • Use Lambda Function URL To Write a Serverless App Backed by DynamoDB
  • How to Build a Concurrent Chat App With Go and WebSockets
  • Building Scalable Data Lake Using AWS

Trending

  • IoT and Cybersecurity: Addressing Data Privacy and Security Challenges
  • System Coexistence: Bridging Legacy and Modern Architecture
  • How to Convert XLS to XLSX in Java
  • After 9 Years, Microsoft Fulfills This Windows Feature Request
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Build a Twitter Leaderboard App With Redis and AWS Lambda (Part 2)

Build a Twitter Leaderboard App With Redis and AWS Lambda (Part 2)

This is the second blog post of a two-part series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda.

By 
Abhishek Gupta user avatar
Abhishek Gupta
DZone Core CORE ·
Jun. 02, 22 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
38.8K Views

Join the DZone community and get the full member experience.

Join For Free

This is the second blog post of a two-part series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first post was about the solution overview and deployment. Hopefully, you were able to try it out end to end. As promised, part two will cover the infrastructure aspects (IaC to be specific) which are comprised of three (CDK) stacks (in the context of a single CDK App).

I will provide a walk-through of the CDK code which is written in Go, thanks to the CDK Go support. AWS Cloud Development Kit (CDK) is all about IaC (Infrastructure-as-code).

Architecture of the Solution

Just as a refresher, here is the high-level architecture of the solution:

High-level architecture

High-level architecture

Division of Architecture

The architecture is divided into two logical parts:

  1. The first part handles tweet ingestion: A Lambda function fetches tweets (from Twitter), extracts hashtags for each tweet, and stores them in MemoryDB (in a Redis Sorted Set). This function gets invoked based on a schedule based on a rule in the CloudWatch trigger.
  2. The second part provides the leaderboard functionality: This is yet another Lambda function that provides an HTTP(s) endpoint (thanks to Lambda Function URL) to query the sorted set and extract the top 10 hashtags (leaderboard).

Services Overview

Here is a quick overview of the services involved in the solution:

  • Amazon MemoryDB for Redis: It is a durable, in-memory database service that is compatible with Redis, thus empowering you to build applications using the same flexible and friendly Redis data structures, APIs, and commands that they already use today.
  • Lambda Function URL is a relatively new feature (at the time of writing this blog) that provides a dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g., to serve as a webhook) and don't want to set up and configure an API Gateway.
  • As stated earlier, AWS Cloud Development Kit (CDK) is all about IaC (Infrastructure-as-code). It is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. You can choose from a list of supported programming languages (at the time of writing, this list includes TypeScript, JavaScript, Python, Java, C#/.Net, and Go, in developer preview) to define your infrastructure components as code, just like you would with any other application!

CDK Code Walk Through

The solution is comprised of three (CDK) Stacks (in the context of a single CDK App).

  • The first stack deploys a VPC (and also subnets, NAT gateway, etc.), a MemoryDB for the Redis cluster, and a few security groups.
  • The second stack deploys the first Lambda function which is responsible for ingesting tweet data into Redis.
  • Finally, the third stack deploys the leaderboard Lambda function. 

Note that we could have included all the above as part of a single stack since this is a demo application. However, I wanted to demonstrate how you can leverage multiple stacks within a single CDK application. For a more complex production infrastructure, it makes sense to use multiple stacks in order to keep your resources decoupled and makes it much easier to handle and reason about your CDK logic. 

Let's walk through the code, one stack at a time:

Please note that some of the code has been redacted/omitted for brevity - you can always refer to complete code in the GitHub repo.

1.  Start with the infrastructure stack.

stack := awscdk.NewStack(scope, &id, &sprops)
vpc = awsec2.NewVpc(stack, jsii.String("demo-vpc"), nil)

authInfo := map[string]interface{}{"Type": "password", "Passwords": []string{memorydbPassword}}
user = awsmemorydb.NewCfnUser(stack, jsii.String("demo-memorydb-user"), &awsmemorydb.CfnUserProps{UserName: jsii.String("demo-user"), AccessString: jsii.String(accessString), AuthenticationMode: authInfo})
acl := awsmemorydb.NewCfnACL(stack, jsii.String("demo-memorydb-acl"), &awsmemorydb.CfnACLProps{AclName: jsii.String("demo-memorydb-acl"), UserNames: &[]*string{user.UserName()}})

//snip .....

subnetGroup := awsmemorydb.NewCfnSubnetGroup(stack, jsii.String("demo-memorydb-subnetgroup"), &awsmemorydb.CfnSubnetGroupProps{SubnetGroupName: jsii.String("demo-memorydb-subnetgroup"), SubnetIds: &subnetIDsForSubnetGroup})

memorydbSecurityGroup = awsec2.NewSecurityGroup(stack, jsii.String("memorydb-demo-sg"), &awsec2.SecurityGroupProps{Vpc: vpc, SecurityGroupName: jsii.String("memorydb-demo-sg"), AllowAllOutbound: jsii.Bool(true)})

memorydbCluster = awsmemorydb.NewCfnCluster(//... details omitted)

//...snip

twitterIngestFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)
twitterLeaderboardFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)

memorydbSecurityGroup.AddIngressRule(//... details omitted)
memorydbSecurityGroup.AddIngressRule(//... details omitted)


To summarize:

  • awsec2.NewVpc:  A single line of code is all it takes to create VPC and related components such as public and private subnets, NAT gateways, and more. Compare that against a standard CloudFormation template that you would need to write to get this done!
  • We created a User (along with a password for authentication), ACL (Access Control List for authorization), and Subnet group for the MemoryDB cluster. Refer to them during cluster creation with awsmemorydb.NewCfnCluster. 
  • We also created the required security groups. Their main role is to allow Lambda functions to access MemoryDB. We specify explicit Inbound rules to make that possible.
    • One for MemoryDB cluster
    • One each for both the Lambda functions

Note: We are using an L1 construct for MemoryDB for Redis.

2.  The next stack deploys the tweets' ingestion Lambda Function.

//....

memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USER": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword()), "TWITTER_API_KEY": jsii.String(getTwitterAPIKey()), "TWITTER_API_SECRET": jsii.String(getTwitterAPISecret()), "TWITTER_ACCESS_TOKEN": jsii.String(getTwitterAccessToken()), "TWITTER_ACCESS_TOKEN_SECRET": jsii.String(getTwitterAccessTokenSecret())}

awslambda.NewDockerImageFunction(stack, jsii.String("lambda-memorydb-func"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(tweetIngestionFunctionName), Environment: lambdaEnvVars, Timeout: awscdk.Duration_Seconds(jsii.Number(20)), Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(tweetIngestionFunctionPath), nil), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterIngestFunctionSecurityGroup}})

//....


It's quite simple compared to the previous stack. We start by defining the environment variables required by our Lambda function (including Twitter API credentials) and then deploy it as a Docker image.

For the function to be packaged as a Docker image, I used the Go:1.x base image, but you can explore other options as well. During deployment, the Docker image is built locally, pushed to a private ECR registry, and finally the Lambda function is created: all this, with a few lines of code!

It's worth checking the L2 construct (in alpha state at the time of writing) which makes it even simpler to deploy Go functions using CDK. You can refer to the documentation.

Notice that the MemoryDB cluster and security group are automatically referred/looked-up from the previous stack (not re-created!).

3.  Finally, the third stack takes care of the leaderboard Lambda function. It's quite similar to the previous one, except for the addition of the Lambda Function URL (awslambda.NewFunctionUrl) which we use as the output for the stack:

//....
memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USERNAME": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword())}

function := awslambda.NewDockerImageFunction(stack, jsii.String("twitter-hashtag-leaderboard"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(hashtagLeaderboardFunctionName), Environment: lambdaEnvVars, Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(hashtagLeaderboardFunctionPath), nil), Timeout: awscdk.Duration_Seconds(jsii.Number(5)), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterLeaderboardFunctionSecurityGroup}})

funcURL := awslambda.NewFunctionUrl(stack, jsii.String("func-url"), &awslambda.FunctionUrlProps{AuthType: awslambda.FunctionUrlAuthType_NONE, Function: function})

awscdk.NewCfnOutput(stack, jsii.String("Function URL"), &awscdk.CfnOutputProps{Value: funcURL.Url()})


That's all for this blog post. I'll close out with links to AWS Go CDK v2 references:

  • MemoryDB
  • Lambda  
  • VPC, etc. 
  • CDK Go v2 

This concludes the two-part series. Stay tuned for more and as always, happy coding!

AWS AWS Lambda app Build (game engine) Redis (company) twitter

Opinions expressed by DZone contributors are their own.

Related

  • The Technology Stack Needed To Build a Web3 Application
  • Use Lambda Function URL To Write a Serverless App Backed by DynamoDB
  • How to Build a Concurrent Chat App With Go and WebSockets
  • Building Scalable Data Lake Using AWS

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!