DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Because the DevOps movement has redefined engineering responsibilities, SREs now have to become stewards of observability strategy.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Related

  • From Zero to Scale With AWS Serverless
  • Idempotency and Reliability in Event-Driven Systems: A Practical Guide
  • AWS CDK: Infrastructure as Abstract Data Types, Part 3
  • AWS CDK: Infrastructure as Abstract Data Types, Part 2

Trending

  • Rust, WASM, and Edge: Next-Level Performance
  • Endpoint Security Controls: Designing a Secure Endpoint Architecture, Part 2
  • Efficient API Communication With Spring WebClient
  • IoT and Cybersecurity: Addressing Data Privacy and Security Challenges
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

In this article, learn how to deploy the end-to-end application using CDK using components that were written using the Go programming language!

By 
Abhishek Gupta user avatar
Abhishek Gupta
DZone Core CORE ·
Jan. 09, 23 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
11.1K Views

Join the DZone community and get the full member experience.

Join For Free

A previous post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster.

This blog will take it a notch further.

  • The solution consists of an MSK Serverless cluster, a producer application on AWS App Runner, and a consumer application in Kubernetes (EKS) persisting data to DynamoDB.
  • The core components (MSK cluster, EKS, and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK.
  • Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles.

High level architecture

All the components in this solution have been written in Go.

  • The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication).
  • The CDK stacks have been written using CDK Go library.

Prerequisites

You will need the following:

  • An AWS account
  • Install AWS CDK, AWS CLI, Docker, eksctl and curl.

Use CDK to Provision MSK, EKS, and DynamoDB

AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.

All the code and config are present in this GitHub repo. Clone the GitHub repo and change it to the right directory:

git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb
cd msk-cdk-apprunner-eks-dynamodb/cdk


Deploy the first CDK stack:

cdk deploy MSKDynamoDBEKSInfraStack


Wait for all the components to get provisioned, including the MSK Serverless cluster, EKS cluster and DynamoDB.

You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code here.

Deploy MSK Producer Application to App Runner Using CDK

Deploy the second CDK stack.

Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository.

Make sure to enter the MSK Serverless broker endpoint URL.

export MSK_BROKER=<enter endpoint>
export MSK_TOPIC=test-topic

cdk deploy AppRunnerServiceStack


Wait for the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code and the producer application.

Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such:

Outputs:
AppRunnerServiceStack.AppURL = <app URL>
AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image>
....


Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body.

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL


Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB.

Configure IRSA for Consumer Application

Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your pods to use the service account.

Exit the cdk directory and change to the root of the project:

cd ..


Create an IAM OIDC Identity Provider for Your Cluster With eksctl

export EKS_CLUSTER_NAME=<EKS cluster name>

oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

aws iam list-open-id-connect-providers | grep $oidc_id

eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve


Define IAM Roles for the Application

Configure IAM Roles for Service Accounts (also known as IRSA). Refer to the documentation for details.

Start by creating a Kubernetes Service Account:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-app-sa
EOF


To verify:

 
kubectl get serviceaccount/eks-app-sa -o yaml


Set your AWS Account ID and OIDC Identity provider as environment variables:

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")


Create a JSON file with Trusted Entities for the role:

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json


To verify:

 
cat trust.json


Now, create the IAM role:

export ROLE_NAME=msk-consumer-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS"


You will need to create and attach the policy to the role, since we only want the consumer application to consume data from the MSK cluster and put data to DynamoDB table. This needs to be fine-grained.

In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created:

aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy


Finally, associate the IAM role with the Kubernetes Service Account that you created earlier:

kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

#confirm
kubectl get serviceaccount/eks-app-sa -o yaml


Deploy MSK Consumer Application to EKS

You can refer to the consumer application code here.

Make sure to update the consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output).

kubectl apply -f msk-consumer/app-iam.yaml

# verify Pods
kubectl get pods -l=app=msk-iam-consumer-app


Verify End-To-End Solution

Continue to send records using the App Runner producer application:

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL


Check consumer app logs on EKS to verify:

kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}')


Scale-Out Consumer App

The MSK topic created by the producer application has three topic partitions, so we can have a maximum of three consumer instances. 

Scale-out to three replicas:

kubectl scale deployment/msk-iam-consumer-app --replicas=3


Verify the number of Pods and check logs for each of them. Notice how the data consumption is balanced across the three instances.

kubectl get pods -l=app=msk-iam-consumer-app


Conclusion

You were able to deploy the end-to-end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!

AWS Amazon DynamoDB app CDK (programming library) AWS Cloud

Published at DZone with permission of Abhishek Gupta, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • From Zero to Scale With AWS Serverless
  • Idempotency and Reliability in Event-Driven Systems: A Practical Guide
  • AWS CDK: Infrastructure as Abstract Data Types, Part 3
  • AWS CDK: Infrastructure as Abstract Data Types, Part 2

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!