DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Dynamic Data Processing Using Serverless Java With Quarkus on AWS Lambda by Enabling SnapStart (Part 2)
  • From Zero to Scale With AWS Serverless
  • Building a Serverless Application on AWS With AWS SAM
  • Dynamic Data Processing Using Serverless Java With Quarkus on AWS Lambda (Part 1)

Trending

  • Enhancing Security With ZTNA in Hybrid and Multi-Cloud Deployments
  • Understanding and Mitigating IP Spoofing Attacks
  • Using Python Libraries in Java
  • Customer 360: Fraud Detection in Fintech With PySpark and ML
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Use Golang for Data Processing With Amazon SNS and AWS Lambda

Use Golang for Data Processing With Amazon SNS and AWS Lambda

In this blog, you saw an example of how to use Lambda to process messages sent to SNS and store them in DynamoDB, thanks to the SNS and Lamdba integration.

By 
Abhishek Gupta user avatar
Abhishek Gupta
DZone Core CORE ·
May. 01, 23 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
16.2K Views

Join the DZone community and get the full member experience.

Join For Free

In this blog post, you will be using the aws-lambda-go library along with the AWS Go SDK v2 for an application that will process records from an Amazon SNS topic and store them in a DynamoDB table.

You will also learn how to use Go bindings for AWS CDK to implement “Infrastructure-as-code” for the entire solution and deploy it with the AWS CDK CLI.

The code is available on GitHub.

AWS CLI

Introduction

Amazon Simple Notification Service (SNS) is a highly available, durable, and scalable messaging service that enables the exchange of messages between applications or microservices. It uses a publish/subscribe model where publishers send messages to topics, and subscribers receive messages from topics they are interested in. Clients can subscribe to the SNS topic and receive published messages using a supported endpoint type, such as Amazon Kinesis Data Firehose, Amazon SQS, AWS Lambda, HTTP, email, mobile push notifications, and mobile text messages (SMS).

AWS Lambda and Amazon SNS integration enable developers to build event-driven architectures that can scale automatically and respond to changes in real time. When a new message is published to an SNS topic, it can trigger a Lambda function (Amazon SNS invokes your function asynchronously with an event that contains a message and metadata) which can perform a set of actions, such as processing the message, storing data in a database, sending emails or SMS messages, or invoking other AWS services.

Prerequisites

Before you proceed, make sure you have the Go programming language (v1.18 or higher) and AWS CDK installed.

Clone the project and change it to the right directory:

Shell
 
git clone https://github.com/abhirockzz/sns-lambda-events-golang

cd sns-lambda-events-golang


Use CDK To Deploy the Solution

To start the deployment, simply invoke cdk deploy and wait for a bit. You will see a list of resources that will be created and will need to provide your confirmation to proceed.

Shell
 
cd cdk

cdk deploy

# output

Bundling asset SNSLambdaGolangStack/sns-function/Code/Stage...

✨  Synthesis time: 5.94s

This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

//.... omitted

Do you wish to deploy these changes (y/n)? y


This will start creating the AWS resources required for our application.

If you want to see the AWS CloudFormation template which will be used behind the scenes, run cdk synth and check the cdk.out folder.

You can keep track of the progress in the terminal or navigate to AWS console: CloudFormation > Stacks > SNSLambdaGolangStack

Once all the resources are created, you can try out the application. You should have:
  • A Lambda function
  • A SNS topic
  • A DynamoDB table
  • Along with a few other components (like IAM roles etc.)

Verify the Solution

You can check the table and SNS info in the stack output (in the terminal or the Outputs tab in the AWS CloudFormation console for your Stack):

SNSLambdaGolangStack

Send few messages to the SNS topic. For the purposes of this demo, you can use the AWS CLI:
Shell
 
export SNS_TOPIC_ARN=<enter the queue url from cloudformation output>

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user1@foo.com" --message-attributes 'name={DataType=String, StringValue="user1"}, city={DataType=String,StringValue="seattle"}'

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user2@foo.com" --message-attributes 'name={DataType=String, StringValue="user2"}, city={DataType=String,StringValue="new delhi"}'

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user3@foo.com" --message-attributes 'name={DataType=String, StringValue="user3"}, city={DataType=String,StringValue="new york"}'


You can also use the AWS console to send SQS messages.

Check the DynamoDB table to confirm that the file metadata has been stored. You can use the AWS console or the AWS CLI aws dynamodb scan --table-name <enter the table name from cloudformation output>

Don’t Forget To Clean Up

Once you’re done, to delete all the services, simply use:

Shell
 
cdk destroy

#output prompt (choose 'y' to continue)

Are you sure you want to delete: SQSLambdaGolangStack (y/n)?


You were able to setup and try the complete solution. Before we wrap up, let’s quickly walk through some of important parts of the code to get a better understanding of what’s going the behind the scenes.

Code Walk Through

Some of the code (error handling, logging etc.) has been omitted for brevity since we only want to focus on the important parts.

CDK

You can refer to the CDK code here.

We start by creating a DynamoDB table:

Shell
 
table := awsdynamodb.NewTable(stack, jsii.String("dynamodb-table"),
        &awsdynamodb.TableProps{
            PartitionKey: &awsdynamodb.Attribute{
                Name: jsii.String("email"),
                Type: awsdynamodb.AttributeType_STRING},
        })

    table.ApplyRemovalPolicy(awscdk.RemovalPolicy_DESTROY)


Then, we handle the Lambda function (CDK will take care of building and deploying the function) and make sure we provide it appropriate permissions to write to the DynamoDB table.

Shell
 
function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("sns-function"),
        &awscdklambdagoalpha.GoFunctionProps{
            Runtime:     awslambda.Runtime_GO_1_X(),
            Environment: &map[string]*string{"TABLE_NAME": table.TableName()},
            Entry:       jsii.String(functionDir),
        })

    table.GrantWriteData(function)


Then, we create the SNS topic and add that as an event source to the Lambda function.

Shell
 
snsTopic := awssns.NewTopic(stack, jsii.String("sns-topic"), nil)
    function.AddEventSource(awslambdaeventsources.NewSnsEventSource(snsTopic, nil))


Finally, we export the SNS topic and DynamoDB table name as CloudFormation outputs.

Shell
 
awscdk.NewCfnOutput(stack, jsii.String("sns-topic-name"),
        &awscdk.CfnOutputProps{
            ExportName: jsii.String("sns-topic-name"),
            Value:      snsTopic.TopicName()})

    awscdk.NewCfnOutput(stack, jsii.String("dynamodb-table-name"),
        &awscdk.CfnOutputProps{
            ExportName: jsii.String("dynamodb-table-name"),
            Value:      table.TableName()})


Lambda Function

You can refer to the Lambda Function code here.

The Lambda function handler iterates over each SNS topic, and for each of them:

  • Stores the message body in the primary key attribute (email) of the DynamoDB table
  • Rest of the message attributes are stored as is.
Shell
 
func handler(ctx context.Context, snsEvent events.SNSEvent) {
    for _, record := range snsEvent.Records {

        snsRecord := record.SNS

        item := make(map[string]types.AttributeValue)
        item["email"] = &types.AttributeValueMemberS{Value: snsRecord.Message}

        for attrName, attrVal := range snsRecord.MessageAttributes {
            fmt.Println(attrName, "=", attrVal)
            attrValMap := attrVal.(map[string]interface{})

            dataType := attrValMap["Type"]

            val := attrValMap["Value"]

            switch dataType.(string) {
            case "String":
                item[attrName] = &types.AttributeValueMemberS{Value: val.(string)}
            }
        }

        _, err := client.PutItem(context.Background(), &dynamodb.PutItemInput{
            TableName: aws.String(table),
            Item:      item,
        })
    }
}


Wrap Up

In this blog, you saw an example of how to use Lambda to process messages sent to SNS and store them in DynamoDB, thanks to the SNS and Lamdba integration. The entire infrastructure life-cycle was automated using AWS CDK.

All this was done using the Go programming language, which is well-supported in DynamoDB, AWS Lambda, and AWS CDK.

Happy building!

AWS Lambda Amazon DynamoDB Command-line interface Data processing Golang

Published at DZone with permission of Abhishek Gupta, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Dynamic Data Processing Using Serverless Java With Quarkus on AWS Lambda by Enabling SnapStart (Part 2)
  • From Zero to Scale With AWS Serverless
  • Building a Serverless Application on AWS With AWS SAM
  • Dynamic Data Processing Using Serverless Java With Quarkus on AWS Lambda (Part 1)

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!