System to System Messaging – Event-Driven Architecture
System to System Messaging – Event-Driven Architecture
Take a look at how you can use these AWS tools to create a system-to-system messaging stack.
Join the DZone community and get the full member experience.Join For Free
Learn how to migrate and modernize stateless applications and run them in a Kubernetes cluster.
In the current cloud ecosystem, Reactive Programming is the new buzzword for developing any business use case. We will demonstrate an architecture which shows the system-to-system messages or email when we have any event-driven invocation (an upload any object to S3, in this case).
We will be familiarized with three integral components in this post. Simple Notification Service (SNS), Simple Queue Service (SQS), and Simple Email Service (SES).
Other components which are used in the architecture include:
- Cognito User Pool
- API Gateway
Let’s look at the high-level architecture. When a client uploads an object to the configured S3 bucket, an S3 event notification will fire towards SNS and a trigger for a Lambda Function is executed. SNS publishing the event inside the respective topic. There will be a subscriber for that topic: an SQS queue. The triggered Lambda Function stores the signed downloadable URL in the DynamoDB.
The SQS queue stores the event for asynchronous processing, e.g. thumbnail generation or image classification. A different Lambda Function is configured for the SQS queue which gets invoked when we have an incoming message. Within the scope of this blog post, we are not going to discuss the asynchronous processing part. Due to the decoupling of publishing and subscribing with SNS, we are free to add more consumers for the events later.
In our case, we are going to send the events to SNS and then allow interested applications to subscribe. This is referred to as the messaging fanout pattern. Instead of sending events directly to all parties, by using SNS as an intermediate broker we decouple publishing and subscription.
SNS is a simple pub/sub service which organizes around topics. A topic groups together messages of the same type which might be of interest to a set of subscribers. In case of a new message being published to a topic, SNS will notify all subscribers. You can configure delivery policies including configuration of maximum receive rates and retry delays.
We will also subscribe to an SQS queue to the topic, storing the events for asynchronous processing by, e.g., another Lambda function or a long-running polling service. The next section explains how to implement the architecture.
Most of the coding is done in Node.js which is very lightweight and configurable.
We will create an S3 Bucket which will configure to send notifications using Lambda to SNS Topic and also store the data to DynamoDB.
The above implementation shows sends the details to SNS topic and also storing the URL in DynamoDB table. There are other SNS topics which it currently subscribes but those are not part of this post.
The above components are also configured by reactive architecture. This pattern is also called fanning out architecture. All the below configurations are part of this architecture which needs to done only one time and will be scalable automatically.
S3 Object Notification
We need to add the above mentioned Lambda function in the Properties tab and Events settings. All the other tabs and security can also be configured but not in scope for this post.
SNS is a fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients. Amazon SNS makes it simple and cost effective to send push notifications to mobile device users, email recipients or even send messages to other distributed services.
We need to create an SNS topic and add an SQS subscription to the topic. Since this is an SQS topic which we will be publishing, so there will be no need for verification of the endpoint.
Save the ARN of the topic and add the same in the above mentioned Lambda function. The protocol of the subscription should be SQS and in the next part, I will demonstrate how to configure the same. We need to be in the Same Region as the Bucket to get the triggered notification.
SQS refers to a message queuing service for reliably communicating among distributed software components and microservices – at any scale.
We have created a simple Queuein SQS which consumes the messages and a different lambda gets triggered based on the incoming message.
We need to save the ARN of the topic and add it in the above SNS configurations. There are also other parameters which can be used to customize and orchestrate the queue.
The Lambda function will be configured in the Triggers tab and we need to confirm the queue to receive the SNS Messages in the Queue Actions (not shown) drop down.
The picture aboce shows that Lambda is configured for getting the messages in the queue and send the same to the email using SES for dedicated users.
The above lambda shows the implementation of getting the contents from SQS queue and sending emails. The emails are being fetched from a dedicated user pool in Cognito Service. We are getting the details of the users by making an AWS SDK invocation inside the lambda asynchronously. The scope of configuring the Cognito Service is not part of this post. We will show the configuration of Simple Email Service (SES) used in the above implementation. The open source NodeMailer.js library is used to send the email by using simple SMTP protocol.
Amazon Simple Email Service (Amazon SES) is a cost-effective email service built on the reliable and scalable infrastructure that Amazon.com developed to serve its own customer base.
We will verify an email Address by adding the same and confirming the same through the auto-generated email send by AWS as part of the managed service. This Email Address will be now added in the TO field of the Lambda function for sending emails.
We need to configure the SMTP credentials in order to send the email from the functions. The credentails can be obtained in the following tab.
Server Name, Port and Authentication Credentials (which will be one time generated) needs to be added in the lambda functions.
Note: Amazon doesn’t grant unlimited SES usage to new users; instead. they are placed in Amazon sandbox mode. In the sandbox, the users have full access to all Amazon SES sending methods, however, the From address and To addresses must be verified once and there is a limit of 200 messages per 24 hour period.
To remove the restriction on recipient addresses and increase your sending limits, you can opt out from sandbox mode to do this — follow the step in this link.
Finally, we are done!
We can also add other subscribers to the SNS topic such as automated notification in a Slack channel or saving the details to another storage. But overall, this architecture should serve the purpose for having more customization as per the business implementation.
We have seen how it is possible to implement an event processing pipeline with potentially multiple consumers and producers using fully managed building blocks like SNS, SQS, and Lambda. SNS provides pub/sub functionality to decouple producers and consumers, while SQS gives us the ability to process events asynchronously.
Opinions expressed by DZone contributors are their own.