DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
The Latest "Software Integration: The Intersection of APIs, Microservices, and Cloud-Based Systems" Trend Report
Get the report
  1. DZone
  2. Partner Zones
  3. AWS Cloud
  4. Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

In this article, learn how to deploy the end-to-end application using CDK using components that were written using the Go programming language!

Abhishek Gupta user avatar by
Abhishek Gupta
CORE ·
Jan. 09, 23 · Tutorial
Like (2)
Save
Tweet
Share
10.50K Views

Join the DZone community and get the full member experience.

Join For Free

A previous post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster.

This blog will take it a notch further.

  • The solution consists of an MSK Serverless cluster, a producer application on AWS App Runner, and a consumer application in Kubernetes (EKS) persisting data to DynamoDB.
  • The core components (MSK cluster, EKS, and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK.
  • Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles.

High level architecture

All the components in this solution have been written in Go.

  • The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication).
  • The CDK stacks have been written using CDK Go library.

Prerequisites

You will need the following:

  • An AWS account
  • Install AWS CDK, AWS CLI, Docker, eksctl and curl.

Use CDK to Provision MSK, EKS, and DynamoDB

AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.

All the code and config are present in this GitHub repo. Clone the GitHub repo and change it to the right directory:

git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb
cd msk-cdk-apprunner-eks-dynamodb/cdk


Deploy the first CDK stack:

cdk deploy MSKDynamoDBEKSInfraStack


Wait for all the components to get provisioned, including the MSK Serverless cluster, EKS cluster and DynamoDB.

You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code here.

Deploy MSK Producer Application to App Runner Using CDK

Deploy the second CDK stack.

Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository.

Make sure to enter the MSK Serverless broker endpoint URL.

export MSK_BROKER=<enter endpoint>
export MSK_TOPIC=test-topic

cdk deploy AppRunnerServiceStack


Wait for the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code and the producer application.

Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such:

Outputs:
AppRunnerServiceStack.AppURL = <app URL>
AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image>
....


Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body.

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL


Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB.

Configure IRSA for Consumer Application

Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your pods to use the service account.

Exit the cdk directory and change to the root of the project:

cd ..


Create an IAM OIDC Identity Provider for Your Cluster With eksctl

export EKS_CLUSTER_NAME=<EKS cluster name>

oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

aws iam list-open-id-connect-providers | grep $oidc_id

eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve


Define IAM Roles for the Application

Configure IAM Roles for Service Accounts (also known as IRSA). Refer to the documentation for details.

Start by creating a Kubernetes Service Account:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-app-sa
EOF


To verify:

 
kubectl get serviceaccount/eks-app-sa -o yaml


Set your AWS Account ID and OIDC Identity provider as environment variables:

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")


Create a JSON file with Trusted Entities for the role:

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json


To verify:

 
cat trust.json


Now, create the IAM role:

export ROLE_NAME=msk-consumer-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS"


You will need to create and attach the policy to the role, since we only want the consumer application to consume data from the MSK cluster and put data to DynamoDB table. This needs to be fine-grained.

In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created:

aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy


Finally, associate the IAM role with the Kubernetes Service Account that you created earlier:

kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

#confirm
kubectl get serviceaccount/eks-app-sa -o yaml


Deploy MSK Consumer Application to EKS

You can refer to the consumer application code here.

Make sure to update the consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output).

kubectl apply -f msk-consumer/app-iam.yaml

# verify Pods
kubectl get pods -l=app=msk-iam-consumer-app


Verify End-To-End Solution

Continue to send records using the App Runner producer application:

export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL


Check consumer app logs on EKS to verify:

kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}')


Scale-Out Consumer App

The MSK topic created by the producer application has three topic partitions, so we can have a maximum of three consumer instances. 

Scale-out to three replicas:

kubectl scale deployment/msk-iam-consumer-app --replicas=3


Verify the number of Pods and check logs for each of them. Notice how the data consumption is balanced across the three instances.

kubectl get pods -l=app=msk-iam-consumer-app


Conclusion

You were able to deploy the end-to-end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!

AWS Amazon DynamoDB app CDK (programming library) AWS Cloud

Published at DZone with permission of Abhishek Gupta, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • HTTP vs Messaging for Microservices Communications
  • Application Architecture Design Principles
  • NoSQL vs SQL: What, Where, and How
  • Spring Boot, Quarkus, or Micronaut?

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: