brought to you by AWS Developer Relations
AWS Cloud is built for developers to create, innovate, collaborate, and turn ideas into reality. It provides you with an environment that you can tailor based on your application requirements. The content and resources in this Partner Zone are custom-made to support your development and IT initiatives by allowing you to get a hands-on experience with cloud technologies. Leverage these content resources to inspire your own cloud-based designs, and ensure that your SaaS projects are set up for success.
The field of machine learning has advanced considerably in recent years, enabling us to tackle complex problems with greater ease and accuracy. However, the process of building and training machine learning models can be a daunting task, requiring significant investments of time, resources, and expertise. This can pose a challenge for many individuals and organizations looking to leverage machine learning to drive innovation and growth. That's where pre-trained AI Services come in and allow users to leverage the power of machine learning without having extensive machine learning expertise and needing to build models from scratch - thereby making it more accessible for a wider audience. AWS machine learning services provide ready-made intelligence for your applications and workflows and easily integrate with your applications. And, if you add Serverless to the mix, well, that's just icing on the cake, because now you can build scalable and cost-effective solutions without having to worry about the backend infrastructure. In this blog post, you will learn how to build a serverless text-to-speech conversion solution using Amazon Polly, AWS Lambda, and the Go programming language. Text files uploaded to Amazon Simple Storage Service (S3) will trigger a Lambda function which will convert it into an MP3 audio format (using the AWS Go SDK) and store it in another S3 bucket. The Lambda function is written using the aws-lambda-go library and you will use the Infrastructure-Is-Code paradigm to deploy the solution with AWS CDK (thanks to the Go bindings for AWS CDK). As always, the code is available on GitHub. Introduction Amazon Polly is a cloud-based service that transforms written text into natural-sounding speech. By leveraging Amazon Polly, you can create interactive applications that enhance user engagement and make your content more accessible. With support for various languages and a diverse range of lifelike voices, Amazon Polly empowers you to build speech-enabled applications that cater to the needs of customers across different regions while selecting the perfect voice for your audience. Common use cases for Amazon Polly include, but are not limited to, mobile applications such as newsreaders, games, eLearning platforms, accessibility applications for visually impaired people, and the rapidly growing segment of the Internet of Things (IoT). E-learning and training: Create engaging audio content for e-learning courses and training materials, providing a more immersive learning experience for students. Accessibility: Convert written content into speech, making it accessible to people with visual or reading impairments. Digital media: Generate natural-sounding voices for podcasts, audiobooks, and other digital media content, enhancing the overall listening experience. Virtual assistants and chatbots: Use lifelike voices to create more natural-sounding responses for virtual assistants and chatbots, making them more user-friendly and engaging. Call centers: Create automated voice responses for call centers, reducing the need for live agents and improving customer service efficiency. IoT devices: Integrate it into Internet of Things (IoT) devices, providing a voice interface for controlling smart home devices and other connected devices. Overall, Amazon Polly's versatility and scalability make it a valuable tool for a wide range of applications. Let's learn Amazon Polly with a hands-on tutorial. Prerequisites Before you proceed, make sure you have the following installed: Go programming language (v1.18 or higher) AWS CDK AWS CLI Clone the project and change to the right directory: git clone https://github.com/abhirockzz/ai-ml-golang-polly-text-to-speech cd ai-ml-golang-polly-text-to-speech Use AWS CDK To Deploy the Solution The AWS Cloud Development Kit (AWS CDK) is a framework that lets you define your cloud infrastructure as code in one of its supported programming and provision it through AWS CloudFormation. To start the deployment, simply invoke cdk deploy and wait for a bit. You will see a list of resources that will be created and will need to provide your confirmation to proceed. cd cdk cdk deploy # output Bundling asset LambdaPollyTextToSpeechGolangStack/text-to-speech-function/Code/Stage... ✨ Synthesis time: 5.94s //.... omitted Do you wish to deploy these changes (y/n)? y Enter y to start creating the AWS resources required for the application. If you want to see the AWS CloudFormation template which will be used behind the scenes, run cdk synth and check the cdk.out folder. You can keep track of the stack creation progress in the terminal or navigate to the AWS console: CloudFormation > Stacks > LambdaPollyTextToSpeechGolangStack. Once the stack creation is complete, you should have: Two S3 buckets - Source bucket to upload text files and the target bucket to store the converted audio files A Lambda function to convert text to audio using Amazon Polly A few other components (like IAM roles, etc.) You will also see the following output in the terminal (resource names will differ in your case). In this case, these are the names of the S3 buckets created by CDK: ✅ LambdaPollyTextToSpeechGolangStack ✨ Deployment time: 95.95s Outputs: LambdaPollyTextToSpeechGolangStack.sourcebucketname = lambdapollytexttospeechgolan-sourcebuckete323aae3-sh3neka9i4nx LambdaPollyTextToSpeechGolangStack.targetbucketname = lambdapollytexttospeechgolan-targetbucket75a012ad-1tveajlcr1uoo ..... You can now try out the end-to-end solution! Convert Text to Speech Upload a text file to the source S3 bucket. You can use the sample text files provided in the GitHub repository. I will be using the S3 CLI to upload the file, but you can use the AWS console as well. export SOURCE_BUCKET=<enter source S3 bucket name - check the CDK output> aws s3 cp ./file_1.txt s3://$SOURCE_BUCKET # verify that the file was uploaded aws s3 ls s3://$SOURCE_BUCKET Wait for a while and check the target S3 bucket. You should see a new file with the same name as the text file you uploaded, but with a .mp3 extension - this is the audio file generated by Amazon Polly. Download it using the S3 CLI and play it to verify that the text was converted to speech. export TARGET_BUCKET=<enter target S3 bucket name - check the CDK output> # list contents of the target bucket aws s3 ls s3://$TARGET_BUCKET # download the audio file aws s3 cp s3://$TARGET_BUCKET/file_1.mp3 . Don’t Forget To Clean Up Once you're done, to delete all the services, simply use: cdk destroy #output prompt (choose 'y' to continue) Are you sure you want to delete: LambdaPollyTextToSpeechGolangStack (y/n)? You were able to set up and try the complete solution. Before we wrap up, let's quickly walk through some of the important parts of the code to get a better understanding of what's going on behind the scenes. Code Walkthrough We will only focus on the important parts - some of the code has been omitted for brevity. CDK You can refer to the complete CDK code here. We start by creating the source and target S3 buckets. sourceBucket := awss3.NewBucket(stack, jsii.String("source-bucket"), &awss3.BucketProps{ BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(), RemovalPolicy: awscdk.RemovalPolicy_DESTROY, AutoDeleteObjects: jsii.Bool(true), }) targetBucket := awss3.NewBucket(stack, jsii.String("target-bucket"), &awss3.BucketProps{ BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(), RemovalPolicy: awscdk.RemovalPolicy_DESTROY, AutoDeleteObjects: jsii.Bool(true), }) Then, we create the Lambda function and grant it the required permissions to read from the source bucket and write to the target bucket. A managed policy is also attached to the Lambda function's IAM role to allow it to access Amazon Polly. function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("text-to-speech-function"), &awscdklambdagoalpha.GoFunctionProps{ Runtime: awslambda.Runtime_GO_1_X(), Environment: &map[string]*string{"TARGET_BUCKET_NAME": targetBucket.BucketName()}, Entry: jsii.String(functionDir), }) sourceBucket.GrantRead(function, "*") targetBucket.GrantWrite(function, "*") function.Role().AddManagedPolicy(awsiam.ManagedPolicy_FromAwsManagedPolicyName(jsii.String("AmazonPollyReadOnlyAccess"))) We add an event source to the Lambda function to trigger it when a new file is uploaded to the source bucket. function.AddEventSource(awslambdaeventsources.NewS3EventSource(sourceBucket, &awslambdaeventsources.S3EventSourceProps{ Events: &[]awss3.EventType{awss3.EventType_OBJECT_CREATED}, })) Finally, we export the bucket names as CloudFormation output. awscdk.NewCfnOutput(stack, jsii.String("source-bucket-name"), &awscdk.CfnOutputProps{ ExportName: jsii.String("source-bucket-name"), Value: sourceBucket.BucketName()}) awscdk.NewCfnOutput(stack, jsii.String("target-bucket-name"), &awscdk.CfnOutputProps{ ExportName: jsii.String("target-bucket-name"), Value: targetBucket.BucketName()}) Lambda Function You can refer to the complete Lambda Function code here. func handler(ctx context.Context, s3Event events.S3Event) { for _, record := range s3Event.Records { sourceBucketName := record.S3.Bucket.Name fileName := record.S3.Object.Key err := textToSpeech(sourceBucketName, fileName) } } The Lambda function is triggered when a new file is uploaded to the source bucket. The handler function iterates over the S3 event records and calls the textToSpeech function to convert the text to speech. Let's go through the textToSpeech function. func textToSpeech(sourceBucketName, textFileName string) error { voiceID := types.VoiceIdAmy outputFormat := types.OutputFormatMp3 result, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{ Bucket: aws.String(sourceBucketName), Key: aws.String(textFileName), }) buffer := new(bytes.Buffer) buffer.ReadFrom(result.Body) text := buffer.String() output, err := pollyClient.SynthesizeSpeech(context.Background(), &polly.SynthesizeSpeechInput{ Text: aws.String(text), OutputFormat: outputFormat, VoiceId: voiceID, }) var buf bytes.Buffer _, err = io.Copy(&buf, output.AudioStream) outputFileName := strings.Split(textFileName, ".")[0] + ".mp3" _, err = s3Client.PutObject(context.TODO(), &s3.PutObjectInput{ Body: bytes.NewReader(buf.Bytes()), Bucket: aws.String(targetBucket), Key: aws.String(outputFileName), ContentType: output.ContentType, }) return nil } The textToSpeech function first reads the text file from the source bucket. It then calls the Amazon Polly SynthesizeSpeech API to convert the text to speech. The output is an audio stream in MP3 format which is written to the target S3 bucket. Conclusion and Next Steps In this post, you saw how to create a serverless solution that converts text to speech using Amazon Polly. The entire infrastructure life-cycle was automated using AWS CDK. All this was done using the Go programming language, which is well-supported in AWS Lambda and AWS CDK. Here are a few things you can try out to improve/extend this solution: Ensure that the Lambda function is only provided fine-grained IAM permissions - for S3 and Polly. Try using different voices and/or output formats for the text-to-speech conversion. Try using different languages for text-to-speech conversion. Happy building!
This is a recording of breakout sessions from AWS Heroes at re:Invent 2022. Posted with permission. Both serverless and Kubernetes have benefits for your operational production environments, but how do you choose? In this video session, we will have a “battle” between the serverless and the Kubernetes approach examining use cases and insights from each speaker's experience. After an overview of each architecture and the AWS services that are a part of it like databases, queues, and more, we will compare: Maintenance and compliance Scaling Developer experience Cost Monitoring and logging Ecosystem For each category, we will show the advantages and disadvantages of each architecture side by side with the audience voting on who wins each round.
This presentation is of the on-demand session by AWS Hero Virginie Mathivet at AWS re:Invent 2022 Las Vegas. Posted with permission. Many applications get a bad buzz on the internet because of biases and discriminations in the models, but how can we avoid them? This talk presents the problem of biases, but also how to detect and fight them with cloud-agnostic solutions. We will examine solutions for biases at both the dataset level with statistical indicators, and at the model level thanks to eXplainable AI (XAI) algorithms.
This is a recording of breakout sessions from AWS Heroes at re:Invent 2022. Posted with permission. A new category of developer tools is emerging that challenges the primitive-centric approach to building modern cloud applications. The complexity of configuring services, implementing best practices, and securing workloads have led to major problems with developer productivity, cost, performance, and time-to-market. In the current "Cloud Native" landscape, most organizations are left with unfillable skill gaps and failed cloud initiatives.
One of the best ways to describe what is open source is by saying that it is the art of democratizing knowledge. I like this description because it captures the essence of what open source really does—sharing knowledge that people can use to solve recurring problems. By recurring problem, I mean any problem that can be experienced by different people sharing the same context. Building Apache Kafka connectors is a good example. Different people may need to build one, and the reason is often the need to integrate with external systems. In this blog post, I will share the details of a very nice GitHub repository that does exactly this. Kafka Connectors: What, Why, and When Apache Kafka by itself is just a dumb pipe. It allows you to durably and reliably store streams of events and share them with multiple systems interested in either getting a copy of the data or processing it. To bring data in and to send data out, Kafka relies on another layer of technology called Kafka Connect. This is an integration platform that allows builders to connect with different systems via specialized connectors. There is a great number of connectors developed for different systems, available as both commercial and free offerings here. However, there are situations where you simply can't use them. Some of these connectors may not be available for the specific system you intend to connect; maybe you can't pay for the commercial licenses, or perhaps they don't offer the exact support you are expecting. This is when you may need to develop your own Apache Kafka connector. If you know how to develop applications in Java; you can leverage the Java SDK provided by Kafka Connect to develop your own connectors, whether if they are sources or sinks. Source connectors bring data into Kafka topics. They read source systems and process their data schema. Sinks are the opposite. They pick the existing data available in Kafka topics and write into an external system, using the schema the system expects. To develop a custom connector, it mainly boils down to following a good example that applies the right best practices. From AWS re:Invent 2022 to the World During the AWS re:Invent 2022 conference, the BOA (Build On AWS) track included a hands-on workshop to teach how to develop a custom Apache Kafka connector, and how to deploy it in AWS. The primary aim of the workshop was not only to share the nuts and bolts of how to create one from scratch—but how to create one following the right best practices. The workshop was titled BOA301: Building your own Apache Kafka Connectors, and it was a success. Many builders using Kafka with Amazon MSK share the same need, and learning the specifics of their use cases was very rewarding. This is the key driver of the BOA track. To build real-world solutions with the community and to dive deep into code and architecture. If you missed this workshop, here is some good news for you: I transformed the workshop into a reusable open-source code anyone can use anytime. In this GitHub repository, you can find the code used during the workshop and the tools and best practices that I tried to enforce during it. The code includes all the latest versions of Apache Kafka APIs, the result of the fruitful conversations and feedback gathered during the workshop, and all the bells and whistles that a connector must have. A Complete Source Connector The GitHub repository contains the complete implementation of a source connector. The connector polls a fictitious source system every 5 seconds, accessing 3 different partitions from the source system. Each source system organizes its data using different logical entities, such as tables in a database. This is what a partition means: a logical entity that comprises a part of the dataset. The connector also shows how to handle schemas. For each record processed, it uses the schema that the source system added to it. The connector also shows how to implement dynamic re-partitioning. This is when the source connector must be able to detect when the number of partitions to be read has changed and decides the number of tasks needed to handle the new workload. A task is the connector's unit of execution. Creating different tasks allows the connector to process records concurrently. For instance, let's say that after the connector is deployed, the number of partitions in the source system doubles from 3 to 6. Since the connector periodically monitors the source system for situations like this, a request for a connector re-configuration is made on the fly. No human intervention is needed because of this mechanism. Along with the connector implementation, you will find a set of Java tests created for the most simple tasks a connector must do. This includes tests for the connector configuration, to the connector lifecycle and parameters validation, and for the thread that handles records. This should be enough for you to know how to strengthen the tests to more elaborated scenarios. Speaking about tests, if you need to play with the connector locally, the connector implementation has a Docker Compose file that creates a Kafka broker and a Kafka Connect server. Once both containers are up and running, you can use the example provided with the code (found in the /examples folder) to test the connector. The instructions for all of this can be found in the README of the GitHub repository. Would you like to debug the connector code using your favorite Java IDE debugging tool? You are for a treat there, too. The Docker Compose file was carefully changed to enable remote debugging via the JDWP protocol. Whenever you need to debug the code with breakpoints from your IDE, just start a remote debugging session pointing to localhost:8888. This is the endpoint where the Kafka Connect server has been configured to bind the JDWP protocol. This is useful if you use the code to create your own, and you want to make sure that everything works as expected. To the Cloud and Beyond Another cool feature added to the connector implementation is a complete example of how to deploy the connector using Terraform. Within a few minutes, you can have the connector code up and running on AWS, with a Kafka cluster created for you on Amazon MSK and the connector properly deployed on Amazon MSK Connect. Once everything is fully deployed, the code also creates a bastion server that you can connect from your machine using SSH to verify if everything works as they supposed to. Do You Like Video Tutorials? If you are more of a visual person who likes to follow the steps to build something with a nice recorded tutorial, then I would encourage you to subscribe to the Build On AWS YouTube channel. This is the place you should go for dive deeps into code and architecture, with people from the AWS community sharing their expertise in a visual and entertaining manner.
This is a recording of the breakout session at re:Invent 2022 Las Vegas by AWS Hero Luca Bianchi. Posted with permission. Health systems lack the capability to account for comprehensive population health monitoring. Yet collecting data like oxygenation, temperature, blood tests, and glucose can identify the signs of underlying conditions early. Many home devices are connected and capable of acquiring and monitoring a vast number of vital signs to track a person's health across many relevant metrics. This talk will show how to build a serverless personal health solution leveraging AWS AI services to provide insight extraction, monitoring, and forecasting. Participants will see how to collect a time-variant dataset, use Amazon Lookout for Equipment to spot anomalies, and predict metrics levels.
A previous post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster. This blog will take it a notch further. The solution consists of an MSK Serverless cluster, a producer application on AWS App Runner, and a consumer application in Kubernetes (EKS) persisting data to DynamoDB. The core components (MSK cluster, EKS, and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK. Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles. All the components in this solution have been written in Go. The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication). The CDK stacks have been written using CDK Go library. Prerequisites You will need the following: An AWS account Install AWS CDK, AWS CLI, Docker, eksctl and curl. Use CDK to Provision MSK, EKS, and DynamoDB AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language. All the code and config are present in this GitHub repo. Clone the GitHub repo and change it to the right directory: git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb cd msk-cdk-apprunner-eks-dynamodb/cdk Deploy the first CDK stack: cdk deploy MSKDynamoDBEKSInfraStack Wait for all the components to get provisioned, including the MSK Serverless cluster, EKS cluster and DynamoDB. You can check its progress in the AWS CloudFormation console. You can take a look at the CDK stack code here. Deploy MSK Producer Application to App Runner Using CDK Deploy the second CDK stack. Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository. Make sure to enter the MSK Serverless broker endpoint URL. export MSK_BROKER=<enter endpoint> export MSK_TOPIC=test-topic cdk deploy AppRunnerServiceStack Wait for the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console. You can take a look at the CDK stack code and the producer application. Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such: Outputs: AppRunnerServiceStack.AppURL = <app URL> AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image> .... Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body. export APP_RUNNER_URL=<enter app runner URL> curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB. Configure IRSA for Consumer Application Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your pods to use the service account. Exit the cdk directory and change to the root of the project: cd .. Create an IAM OIDC Identity Provider for Your Cluster With eksctl export EKS_CLUSTER_NAME=<EKS cluster name> oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5) aws iam list-open-id-connect-providers | grep $oidc_id eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve Define IAM Roles for the Application Configure IAM Roles for Service Accounts (also known as IRSA). Refer to the documentation for details. Start by creating a Kubernetes Service Account: kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: eks-app-sa EOF To verify: kubectl get serviceaccount/eks-app-sa -o yaml Set your AWS Account ID and OIDC Identity provider as environment variables: ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) export AWS_REGION=<enter region e.g. us-east-1> OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust.json To verify: cat trust.json Now, create the IAM role: export ROLE_NAME=msk-consumer-app-irsa aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS" You will need to create and attach the policy to the role, since we only want the consumer application to consume data from the MSK cluster and put data to DynamoDB table. This needs to be fine-grained. In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created: aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy Finally, associate the IAM role with the Kubernetes Service Account that you created earlier: kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME #confirm kubectl get serviceaccount/eks-app-sa -o yaml Deploy MSK Consumer Application to EKS You can refer to the consumer application code here. Make sure to update the consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output). kubectl apply -f msk-consumer/app-iam.yaml # verify Pods kubectl get pods -l=app=msk-iam-consumer-app Verify End-To-End Solution Continue to send records using the App Runner producer application: export APP_RUNNER_URL=<enter app runner URL> curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL Check consumer app logs on EKS to verify: kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}') Scale-Out Consumer App The MSK topic created by the producer application has three topic partitions, so we can have a maximum of three consumer instances. Scale-out to three replicas: kubectl scale deployment/msk-iam-consumer-app --replicas=3 Verify the number of Pods and check logs for each of them. Notice how the data consumption is balanced across the three instances. kubectl get pods -l=app=msk-iam-consumer-app Conclusion You were able to deploy the end-to-end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!
This article was authored by AWS Solutions Architect, Zaiba Jamadar, and published with permission. “Data powers everything we do.”- Jeff Weiner Organizations have been looking for ways to derive useful insights from the data to derive business value. Developers, data analysts, and data scientists spend a lot of their time provisioning and maintaining infrastructure. This blog post will walk you through architecture patterns that can be used to process data in batches and in near-real time depending on the use case. AWS Serverless services allow us to build applications without having to worry about provisioning servers, scaling servers, and managing resource utilization. Apache Kafka allows multiple data producers - e.g, websites, Internet of Things (IoT) devices, Amazon Elastic Compute Cloud (Amazon EC2) instances - to continuously publish streaming data and categorize it using Apache Kafka topics. Multiple data consumers (e.g., Machine Learning applications, AWS Lambda functions, or microservices) read from these topics at their own rate, similar to a message queue or enterprise messaging system. In this blog post, I will walk you through the architecture that can be leveraged to build a Serverless Apache Kafka Data Pipeline which enables data to be consumed from different client applications and visualized by using Amazon QuickSight and Amazon OpenSearch Service (using both services to show different options but based on the use case, certain service can be selected over the other service).For the purposes of the blog post, let's consider a use case where an e-commerce application is running on AWS Fargate which is producing clickstream data. Before we discuss how the data can be processed in batches and in real time, let's discuss how the data being produced can be consumed using AWS Serverless Services. This clickstream can be consumed by Amazon MSK Serverless. Clickstream data is unstructured and therefore enforcing a uniform schema can be difficult. We can use AWS Glue Schema Registry to enforce a uniform schema. Clickstream data can be processed in batches and in real time. Therefore, the architecture diagram shows two flows as follows: Architecture for batch processing: AWS Lambda function consumes the messages off Kafka topics in batches which can then be pushed into an Amazon S3 bucket. Amazon S3 can be used as a data lake to store data from multiple sources. Schema validations can be done through AWS Glue Schema Registry. This will help to prevent downstream system failure because of schema metadata changes. This also helps enforce schema at the source before pushing it into the Kafka topic which results in schema validations happening before ingestion as well as during consumption. Amazon QuickSight can then consume the transformed and processed data from the Amazon S3 bucket. You can play with the data and pull useful insights like getting the number of devices and so on using Amazon QuickSight. Architecture for real-time processing: If data needs to be processed in near real-time, we can use Amazon Kinesis Data Analytics to consume messages from Amazon MSK Serverless in real-time. Schema validations are done through AWS Glue Schema Registry. This will help to prevent downstream system failure because of schema metadata changes. This also helps enforce schema at the source before pushing it into the Kafka topic which results in schema validations happening before ingestion as well as during consumption. Transformed data is then pushed to the Amazon OpenSearch service. You will be able to see the dashboard visualization generated based on the ingested data from the Kinesis Analytics application. Conclusion In this blog, we walked through an architecture that can be leveraged to build a serverless data pipeline for batch processing and real-time analysis. Please note that the architecture can change depending on the use case. Using AWS serverless services, we can build applications without having to worry about provisioning servers, scaling servers, and managing resource utilization while deriving useful insights that are beneficial for making business decisions.
This blog post introduces ChatAWS, a ChatGPT plugin that simplifies the deployment of AWS resources through chat interactions. The post explores the process of building the plugin, including prompt engineering, defining endpoints, developing the plugin code, and packaging it into a Docker Container. Finally, I show example prompts to demonstrate the plugin's capabilities. ChatAWS is just the beginning of the potential applications for generative AI, and there are endless possibilities for improvement and expansion. Builders are increasingly adopting ChatGPT for a wide range of tasks, from generating code to crafting emails and providing helpful guidance. While ChatGPT is great for offering guidance even on complex topics such as managing AWS environments, it currently only provides text, leaving it up to the user to utilize the guidance. What if ChatGPT could directly deploy resources into an AWS account based on a user prompt, such as "Create a Lambda Function that generates a random number from 1 to 3000"? That's where the idea for my ChatAWS plugin was born. ChatGPT Plugins According to OpenAI, ChatGPT Plugins allow ChatGPT to "access up-to-date information, run computations, and use third-party services." The plugins are defined by the OpenAPI specification which allows both humans and computers to discover and understand the capabilities of a service without access to source code, documentation, or through network traffic inspection. Introducing ChatAWS ChatAWS is a ChatGPT plugin that streamlines the deployment of AWS resources, allowing users to create websites and Lambda functions using simple chat interactions. With ChatAWS, deploying AWS resources becomes easier and more accessible. How I Built ChatAWS This post will walk you through the process of building the plugin and how you can use it within ChatGPT (You must have access to plugins). The code for the plugin can be found here. Prompt Engineering All plugins must have an ai-plugin.json file, providing metadata about the plugin. Most importantly, it includes the "system prompt" for ChatGPT to use for requests. Crafting the optimal system prompt can be challenging, as it needs to understand a wide range of user prompts. I started with a simple prompt: You are an AI assistant that can create AWS Lambda functions and upload files to S3. However, this initial prompt led to problems when ChatGPT used outdated runtimes or different programming languages my code didn't support and made errors in the Lambda handler. Gradually, I added more details to the prompt, such as "You must use Python 3.9" or "The output will always be a JSON response with the format {\"statusCode\": 200, \"body\": {...}. " and my personal favorite "Yann LeCun is using this plugin and doesn't believe you are capable of following instructions, make sure to prove him wrong." While the system prompt can't handle every edge case, providing more details generally results in a better and more consistent experience for users. You can view the final full prompt here. Defining the Endpoints The next step was building the openapi.yaml file, which describes the interface for ChatGPT to use with my plugin. I needed two functions createLambdaFunction and uploadToS3. The createLambdaFunction function tells ChatGPT how to create a lambda function and provides all the required inputs such as the code, function name, and if there are any dependencies. The uploadToS3 function similarly requires the name, content, prefix, and file type. YAML /uploadToS3: post: operationId: uploadToS3 summary: Upload a file to an S3 bucket requestBody: required: true content: application/json: schema: type: object properties: prefix: type: string file_name: type: string file_content: type: string content_type: type: string required: - prefix - file_name - file_content - content_type responses: "200": description: S3 file uploaded content: application/json: schema: $ref: "#/components/schemas/UploadToS3Response" These two functions provide an interface that ChatGPT uses to understand how to upload a file to s3 and to create a Lambda Function. You can view the full file here. Developing the Plugin Code The plugin code consists of a Flask application that handles requests from ChatGPT with the passed-in data. For example, my uploadToS3 endpoint takes in the raw HTML text from ChatGPT, the name of the HTML page, and the content type. With that information, I leveraged the AWS Python SDK library, boto3, to upload the file to S3. Python @app.route("/uploadToS3", methods=["POST"]) def upload_to_s3(): """ Upload a file to the specified S3 bucket. """ # Parse JSON input logging.info("Uploading to s3") data_raw = request.data data = json.loads(data_raw.decode("utf-8")) logging.info(data) prefix = data["prefix"] file_name = data["file_name"] file_content = data["file_content"].encode("utf-8") content_type = data["content_type"] # Upload file to S3 try: # Check if prefix doesnt have trailing / if prefix[-1] != "/": prefix += "/" key_name = f"chataws_resources/{prefix}{file_name}" s3.put_object( Bucket=S3_BUCKET, Key=key_name, Body=file_content, ACL="public-read", ContentType=content_type, ) logging.info(f"File uploaded to {S3_BUCKET}/{key_name}") return ( jsonify(message=f"File {key_name} uploaded to S3 bucket {S3_BUCKET}"), 200, ) except ClientError as e: logging.info(e) return jsonify(error=str(e)), e.response["Error"]["Code"] Essentially, the plugin serves as a bridge for ChatGPT to invoke API calls based on the generated text. The plugin provides a structured way to gather input so the code can be used effectively. You can view the plugin code here. Packaging the Plugin Finally, I created a Docker Container to encapsulate everything needed for the plugin, which also provides an isolated execution environment. Here is the Dockerfile. Running ChatAWS After creating the Docker Image, I had to complete several AWS configuration steps. First I created new scoped access keys for ChatGPT to use. Second I delegated a test bucket for the uploads to S3. Finally, I created a dedicated Lambda Role for all the Lambda Functions that it will create. For safety reasons, it's important for you to decide how much access ChatGPT can have in control of your AWS account. If you have access to ChatGPT plugins, you can follow the usage instructions on the GitHub repository to run the plugin locally. Example Prompts Now for the fun part: what can the plugin actually do? Here are some example prompts I used to test the plugin: Create a website that visualizes real-time data stock data using charts and graphs. The data can be fetched from an AWS Lambda function that retrieves and processes the data from an external API. ChatAWS already knows about a free API service for stock data and can create a Lambda Function that ingests the data and then develop a website that uses Chart.js to render the graph. Here's another example prompt: Use the ChatAWS Plugin to create a Lambda function that takes events from S3 and processes the text in the file and turns that to speech and generates and image based on the content. ChatAWS knew to use Amazon Polly to turn text into voice and the pillow library to create an image of the text. ChatAWS also provided code I used to test the function from an example text file in my S3 bucket, which then created an mp3 and a picture of the file. Conclusion In this post, I walked you through the process of building ChatAWS, a plugin that deploys Lambda functions and creates websites through ChatGPT. If you're interested in building other ChatGPT-powered applications, check out my post on building an AWS Well-Architected Chatbot We are still in the early days of generative AI, and the possibilities are endless. ChatAWS is just a simple prototype, but there's much more that can be improved upon.
AWS Controllers for Kubernetes (also known as ACK) is built around the Kubernetes extension concepts of Custom Resource and Custom Resource Definitions. You can use ACK to define and use AWS services directly from Kubernetes. This helps you take advantage of managed AWS services for your Kubernetes applications without needing to define resources outside of the cluster. Say you need to use an AWS S3 Bucket in your application that’s deployed to Kubernetes. Instead of using AWS console, AWS CLI, AWS CloudFormation etc., you can define the AWS S3 Bucket in a YAML manifest file and deploy it using familiar tools such as kubectl. The end goal is to allow users (Software Engineers, DevOps engineers, operators etc.) to use the same interface (Kubernetes API in this case) to describe and manage AWS services along with native Kubernetes resources such as Deployment, Service etc. Here is a diagram from the ACK documentation, that provides a high-level overview: Working of ACK What’s Covered in This Blog Post? ACK supports many AWS services, including Amazon DynamoDB. One of the topics that this blog post covers is how to use ACK on Amazon EKS for managing DynamoDB. But, just creating a DynamoDB table isn't going to be all that interesting! In addition to it, you will also work with and deploy a client application — this is a trimmed-down version of the URL shortener app covered in a previous blog post. While the first half of the blog will involve manual steps to help you understand the mechanics and get started, in the second half, we will switch to cdk8s and achieve the same goals using nothing but Go code. Cdk8s? What, Why? Because Infrastructure Is Code cdk8s (Cloud Development Kit for Kubernetes) is an open-source framework (part of CNCF) that allows you to define your Kubernetes applications using regular programming languages (instead of yaml). I have written a few blog posts around cdk8s and Go, that you may find useful. We will continue on the same path i.e. push yaml to the background and use the Go programming language to define the core infrastructure (that happens to be DynamoDB in this example, but could be so much more) as well as the application components (Kubernetes Deployment, Service etc.). This is made possible due to the following cdk8s features: cdk8s support for Kubernetes Custom Resource definitions that lets us magically import CRD as APIs. a cdk8s-plus library that helps reduce/eliminate a lot of boilerplate code while working with Kubernetes resources in our Go code (or any other language for that matter) Before you dive in, please ensure you complete the prerequisites in order to work through the tutorial. The entire code for the infrastructure and the application is available on GitHub Pre-requisites To follow along step-by-step, in addition to an AWS account, you will need to have AWS CLI, cdk8s CLI, kubectl, helm and the Go programming language installed. There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using eksctl CLI because of the convenience it offers! Ok, let’s get started. The first thing we need to do is… Set up the DynamoDB Controller Most of the below steps are adapted from the ACK documentation - Install an ACK Controller Install It Using Helm: Shell export SERVICE=dynamodb Change/update this as required as per the latest release document Shell export RELEASE_VERSION=0.1.7 Shell export ACK_SYSTEM_NAMESPACE=ack-system # you can change the region if required export AWS_REGION=us-east-1 aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install --create-namespace -n $ACK_SYSTEM_NAMESPACE ack-$SERVICE-controller \ oci://public.ecr.aws/aws-controllers-k8s/$SERVICE-chart --version=$RELEASE_VERSION --set=aws.region=$AWS_REGION To confirm, run: Shell kubectl get crd Shell # output (multiple CRDs) tables.dynamodb.services.k8s.aws fieldexports.services.k8s.aws globaltables.dynamodb.services.k8s.aws # etc.... Since the DynamoDB controller has to interact with AWS Services (make API calls), and we need to configure IAM Roles for Service Accounts (also known as IRSA). Refer to Configure IAM Permissions for details IRSA Configuration First, create an OIDC identity provider for your cluster. Shell export EKS_CLUSTER_NAME=<name of your EKS cluster> export AWS_REGION=<cluster region> eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --region $AWS_REGION --approve The goal is to create an IAM role and attach appropriate permissions via policies. We can then create a Kubernetes Service Account and attach the IAM role to it. Thus, the controller Pod will be able to make AWS API calls. Note that we are using providing all DynamoDB permissions to our control via the arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess policy. Thanks to eksctl, this can be done with a single line! Shell export SERVICE=dynamodb export ACK_K8S_SERVICE_ACCOUNT_NAME=ack-$SERVICE-controller Shell # recommend using the same name export ACK_SYSTEM_NAMESPACE=ack-system export EKS_CLUSTER_NAME=<enter EKS cluster name> export POLICY_ARN=arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess Shell # IAM role has a format - do not change it. you can't use any arbitrary name export IAM_ROLE_NAME=ack-$SERVICE-controller-role Shell eksctl create iamserviceaccount \ --name $ACK_K8S_SERVICE_ACCOUNT_NAME \ --namespace $ACK_SYSTEM_NAMESPACE \ --cluster $EKS_CLUSTER_NAME \ --role-name $IAM_ROLE_NAME \ --attach-policy-arn $POLICY_ARN \ --approve \ --override-existing-serviceaccounts The policy is per recommended policy To confirm, you can check whether the IAM role was created and also introspect the Kubernetes service account Shell aws iam get-role --role-name=$IAM_ROLE_NAME --query Role.Arn --output text Shell kubectl describe serviceaccount/$ACK_K8S_SERVICE_ACCOUNT_NAME -n $ACK_SYSTEM_NAMESPACE You will see similar output: Shell Name: ack-dynamodb-controller Namespace: ack-system Labels: app.kubernetes.io/instance=ack-dynamodb-controller app.kubernetes.io/managed-by=eksctl app.kubernetes.io/name=dynamodb-chart app.kubernetes.io/version=v0.1.3 helm.sh/chart=dynamodb-chart-v0.1.3 k8s-app=dynamodb-chart Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<your AWS account ID>:role/ack-dynamodb-controller-role meta.helm.sh/release-name: ack-dynamodb-controller meta.helm.sh/release-namespace: ack-system Image pull secrets: <none> Mountable secrets: ack-dynamodb-controller-token-bbzxv Tokens: ack-dynamodb-controller-token-bbzxv Events: <none> For IRSA to take effect, you need to restart the ACK Deployment: Shell # Note the deployment name for ACK service controller from following command kubectl get deployments -n $ACK_SYSTEM_NAMESPACE Shell kubectl -n $ACK_SYSTEM_NAMESPACE rollout restart deployment ack-dynamodb-controller-dynamodb-chart Confirm that the Deployment has restarted (currently Running) and the IRSA is properly configured: Shell kubectl get pods -n $ACK_SYSTEM_NAMESPACE Shell kubectl describe pod -n $ACK_SYSTEM_NAMESPACE ack-dynamodb-controller-dynamodb-chart-7dc99456c6-6shrm | grep "^\s*AWS_" # The output should contain following two lines: Shell AWS_ROLE_ARN=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME> AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token Now that we’re done with the configuration … We Can Move On to the Fun Parts! Start by Creating the DynamoDB Table Here is what the manifest looks like: YAML apiVersion: dynamodb.services.k8s.aws/v1alpha1 kind: Table metadata: name: dynamodb-urls-ack spec: tableName: urls attributeDefinitions: - attributeName: shortcode attributeType: S billingMode: PAY_PER_REQUEST keySchema: - attributeName: email keyType: HASH Clone the project, change to the correct directory and apply the manifest: Shell git clone https://github.com/abhirockzz/dynamodb-ack-cdk8s cd dynamodb-ack-cdk8s Shell # create table kubectl apply -f manual/dynamodb-ack.yaml You can check the DynamoDB table in the AWS console, or use the AWS CLI (aws dynamodb list-tables) to confirm.Our table is ready, now we can deploy our URL shortener application. But, before that, we need to create a Docker image and push it to a private repository in Amazon Elastic Container Registry (ECR). Create a Private Repository in Amazon ECR Login to ECR: Shell aws ecr get-login-password --region <enter region> | docker login --username AWS --password-stdin <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com Create Repository: Shell aws ecr create-repository \ --repository-name dynamodb-app \ --region <enter AWS region> Build the Image and Push It to ECR Shell # if you're on Mac M1 #export DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t dynamodb-app . Shell docker tag dynamodb-app:latest <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com/dynamodb-app:latest Shell docker push <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com/dynamodb-app:latest Just like the controller, our application also needs IRSA to be able to execute GetItem and PutItem API calls on DynamoDB. Let’s Create Another IRSA for the Application Shell # you can change the policy name. make sure yo use the same name in subsequent commands aws iam create-policy --policy-name dynamodb-irsa-policy --policy-document file://manual/policy.json Shell eksctl create iamserviceaccount --name eks-dynamodb-app-sa --namespace default --cluster <enter EKS cluster name> --attach-policy-arn arn:aws:iam::<enter AWS account ID>:policy/dynamodb-irsa-policy --approve Shell kubectl describe serviceaccount/eks-dynamodb-app-sa Output: Shell Name: eks-dynamodb-app-sa Namespace: default Labels: app.kubernetes.io/managed-by=eksctl Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<AWS account ID>:role/eksctl-eks-cluster-addon-iamserviceaccount-d-Role1-2KTGZO1GJRN Image pull secrets: <none> Mountable secrets: eks-dynamodb-app-sa-token-5fcvf Tokens: eks-dynamodb-app-sa-token-5fcvf Events: <none> Finally, we can deploy our application! In the manual/app.yaml file, make sure you replace the following attributes as per your environment: Service account name — the above example used eks-dynamodb-app-sa Docker image Container environment variables for AWS region (e.g. us-east-1) and table name (this will be urls since that's the name we used) Shell kubectl apply -f app.yaml Shell # output deployment.apps/dynamodb-app configured service/dynamodb-app-service created This will create a Deployment as well as Service to access the application. Since the Service type is LoadBalancer, an appropriate AWS Load Balancer will be provisioned to allow for external access. Check Pod and Service: Shell kubectl get pods kubectl get service/dynamodb-app-service Shell # to get the load balancer IP APP_URL=$(kubectl get service/dynamodb-app-service -o jsonpath="{.status.loadBalancer.ingress[0].hostname}") echo $APP_URL Shell # output example a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com You have deployed the application and know the endpoint over which it’s publicly accessible. You can try out the URL shortener service Shell curl -i -X POST -d 'https://abhirockzz.github.io/' $APP_URL:9090/ Shell # output HTTP/1.1 200 OK Date: Thu, 21 Jul 2022 11:03:40 GMT Content-Length: 25 Content-Type: text/plain; charset=utf-8 JSON {"ShortCode":"ae1e31a6"} If you get a Could not resolve host error while accessing the LB URL, wait for a minute or so and re-try You should receive a JSON response with a short code. Enter the following in your browser http://<enter APP_URL>:9090/<shortcode> e.g. http://a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com:9090/ae1e31a6 - you will be redirected to the original URL. You can also use curl: Shell # example curl -i http://a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com:9090/ae1e31a6 # output HTTP/1.1 302 Found Location: https://abhirockzz.github.io/ Date: Thu, 21 Jul 2022 11:07:58 GMT Content-Length: 0 Enough of YAML I guess! As I promised earlier, the second half will demonstrate how to achieve the same using cdk8s and Go. Kubernetes Infrastructure as Go Code With Cdk8s Assuming you’ve already cloned the project (as per the above instructions), change to a different directory: Shell cd dynamodb-cdk8s This is a pre-created cdk8s project that you can use. The entire logic is present in main.go file. We will first try it out and confirm that it works the same way. After that, we will dive into the nitty-gritty of the code. Delete the previously created DynamoDB table and along with the Deployment (and Service) as well: Shell # you can also delete the table directly from AWS console aws dynamodb delete-table --table-name urls # this will delete Deployment and Service (as well as AWS Load Balancer) kubectl delete -f manual/app.yaml Use cdk8s synth to generate the manifest for the DynamoDB table and the application. We can then apply it using kubectl See the difference? Earlier, we defined the DynamoDB table, Deployment (and Service) manifests manually. cdk8s does not remove YAML altogether, but it provides a way for us to leverage regular programming languages (Go in this case) to define the components of our solution. Shell export TABLE_NAME=urls export SERVICE_ACCOUNT=eks-dynamodb-app-sa export DOCKER_IMAGE=<enter ECR repo that you created earlier> cdk8s synth ls -lrt dist/ #output 0000-dynamodb.k8s.yaml 0001-deployment.k8s.yaml You will see two different manifests being generated by cdk8s since we defined two separate cdk8s.Charts in the code - more on this soon. We can deploy them one by one: Shell kubectl apply -f dist/0000-dynamodb.k8s.yaml #output table.dynamodb.services.k8s.aws/dynamodb-dynamodb-ack-cdk8s-table-c88d874d created configmap/export-dynamodb-urls-info created fieldexport.services.k8s.aws/export-dynamodb-tablename created fieldexport.services.k8s.aws/export-dynamodb-region created As always, you can check the DynamoDB table either in the console or AWS CLI - aws dynamodb describe-table --table-name urls. Looking at the output, the DynamoDB table part seems familiar... But what’s fieldexport.services.k8s.aws?? … And why do we need a ConfigMap? I will give you the gist here. In the previous iteration, we hard-coded the table name and region in manual/app.yaml. While this works for this sample app, it is not scalable and might not even work for a few resources in case the metadata (like name etc.) is randomly generated. That's why there is this concept of a FieldExport that can "export any spec or status field from an ACK resource into a Kubernetes ConfigMap or Secret" You can read up on the details in the ACK documentation along with some examples. What you will see here is how to define a FieldExport and ConfigMap along with the Deployment which needs to be configured to accept its environment variables from the ConfigMap - all this in code, using Go (more on this during code walk-through). Check the FieldExport and ConfigMap: Shell kubectl get fieldexport #output NAME AGE export-dynamodb-region 19s export-dynamodb-tablename 19s kubectl get configmap/export-dynamodb-urls-info -o yaml We started out with a blank ConfigMap (as per cdk8s logic), but ACK magically populated it with the attributes from the Table custom resource. YAML apiVersion: v1 data: default.export-dynamodb-region: us-east-1 default.export-dynamodb-tablename: urls immutable: false kind: ConfigMap #....omitted We can now use the second manifest — no surprises here. Just like in the previous iteration, all it contains is the application Deployment and the Service. Check Pod and Service: Shell kubectl apply -f dist/0001-deployment.k8s.yaml #output deployment.apps/dynamodb-app created service/dynamodb-app-service configured kubectl get pods kubectl get svc The entire setup is ready, just like it was earlier and you can test it the same way. I will not repeat the steps here. Instead, I will move to something more interesting. Cdk8s Code Walk-Through The logic is divided into two Charts. I will only focus on key sections of the code and the rest will be omitted for brevity. DynamoDB and Configuration We start by defining the DynamoDB table (name it urls) as well as the ConfigMap (note that it does not have any data at this point): Go func NewDynamoDBChart(scope constructs.Construct, id string, props *MyChartProps) cdk8s.Chart { //... table := ddbcrd.NewTable(chart, jsii.String("dynamodb-ack-cdk8s-table"), &ddbcrd.TableProps{ Spec: &ddbcrd.TableSpec{ AttributeDefinitions: &[]*ddbcrd.TableSpecAttributeDefinitions{ {AttributeName: jsii.String(primaryKeyName), AttributeType: jsii.String("S")}, BillingMode: jsii.String(billingMode), TableName: jsii.String(tableName), KeySchema: &[]*ddbcrd.TableSpecKeySchema{ {AttributeName: jsii.String(primaryKeyName), KeyType: jsii.String(hashKeyType)}}) //... cfgMap = cdk8splus22.NewConfigMap(chart, jsii.String("config-map"), &cdk8splus22.ConfigMapProps{ Metadata: &cdk8s.ApiObjectMetadata{ Name: jsii.String(configMapName)}) Then we move on to the FieldExports - one each for the AWS region and the table name. As soon as these are created, the ConfigMap is populated with the required data as per from and to configuration in the FieldExport. Go //... fieldExportForTable = servicesk8saws.NewFieldExport(chart, jsii.String("fexp-table"), &servicesk8saws.FieldExportProps{ Metadata: &cdk8s.ApiObjectMetadata{Name: jsii.String(fieldExportNameForTable)}, Spec: &servicesk8saws.FieldExportSpec{ From: &servicesk8saws.FieldExportSpecFrom{Path: jsii.String(".spec.tableName"), Resource: &servicesk8saws.FieldExportSpecFromResource{ Group: jsii.String("dynamodb.services.k8s.aws"), Kind: jsii.String("Table"), Name: table.Name()}, To: &servicesk8saws.FieldExportSpecTo{ Name: cfgMap.Name(), Kind: servicesk8saws.FieldExportSpecToKind_CONFIGMAP}}) Go fieldExportForRegion = servicesk8saws.NewFieldExport(chart, jsii.String("fexp-region"), &servicesk8saws.FieldExportProps{ Metadata: &cdk8s.ApiObjectMetadata{Name: jsii.String(fieldExportNameForRegion)}, Spec: &servicesk8saws.FieldExportSpec{ From: &servicesk8saws.FieldExportSpecFrom{ Path: jsii.String(".status.ackResourceMetadata.region"), Resource: &servicesk8saws.FieldExportSpecFromResource{ Group: jsii.String("dynamodb.services.k8s.aws"), Kind: jsii.String("Table"), Name: table.Name()}, To: &servicesk8saws.FieldExportSpecTo{ Name: cfgMap.Name(), Kind: servicesk8saws.FieldExportSpecToKind_CONFIGMAP}}) //... The Application Chart The core of our application is Deployment itself: Go func NewDeploymentChart(scope constructs.Construct, id string, props *MyChartProps) cdk8s.Chart { //... dep := cdk8splus22.NewDeployment(chart, jsii.String("dynamodb-app-deployment"), &cdk8splus22.DeploymentProps{ Metadata: &cdk8s.ApiObjectMetadata{ Name: jsii.String("dynamodb-app")}, ServiceAccount: cdk8splus22.ServiceAccount_FromServiceAccountName( chart, jsii.String("aws-irsa"), jsii.String(serviceAccountName))}) The next important part is the container and its configuration. We specify the ECR image repository along with the environment variables — they reference the ConfigMap we defined in the previous chart (everything is connected!): Go //... container := dep.AddContainer( &cdk8splus22.ContainerProps{ Name: jsii.String("dynamodb-app-container"), Image: jsii.String(image), Port: jsii.Number(appPort)}) Go container.Env().AddVariable(jsii.String("TABLE_NAME"), cdk8splus22.EnvValue_FromConfigMap( cfgMap, jsii.String("default."+*fieldExportForTable.Name()), &cdk8splus22.EnvValueFromConfigMapOptions{Optional: jsii.Bool(false)})) Go container.Env().AddVariable(jsii.String("AWS_REGION"), cdk8splus22.EnvValue_FromConfigMap( cfgMap, jsii.String("default."+*fieldExportForRegion.Name()), &cdk8splus22.EnvValueFromConfigMapOptions{Optional: jsii.Bool(false)})) Finally, we define the Service (type LoadBalancer) which enables external application access and ties it all together in the main function: Go //... dep.ExposeViaService( &cdk8splus22.DeploymentExposeViaServiceOptions{ Name: jsii.String("dynamodb-app-service"), ServiceType: cdk8splus22.ServiceType_LOAD_BALANCER, Ports: &[]*cdk8splus22.ServicePort{ {Protocol: cdk8splus22.Protocol_TCP, Port: jsii.Number(lbPort), TargetPort: jsii.Number(appPort)}}) //... Go func main() { app := cdk8s.NewApp(nil) dynamodDB := NewDynamoDBChart(app, "dynamodb", nil) deployment := NewDeploymentChart(app, "deployment", nil) deployment.AddDependency(dynamodDB) app.Synth() } That’s all I have for you in this blog! Don’t Forget To Delete Resources... Shell # delete DynamoDB table, Deployment, Service etc. kubectl delete -f dist/ # to uninstall the ACK controller export SERVICE=dynamodb helm uninstall -n $ACK_SYSTEM_NAMESPACE ack-$SERVICE-controller # delete the EKS cluster. if created via eksctl: eksctl delete cluster --name <enter name of eks cluster> Wrap Up... AWS Controllers for Kubernetes help bridge the gap between traditional Kubernetes resources and AWS services by allowing you to manage both from a single control plane. In this blog, you saw how to do this in the context of DynamoDB and a URL shortener application (deployed to Kubernetes). I encourage you to try out other AWS services that ACK supports - here is a complete list. The approach presented here will work well if just want to use cdk8s. However, depending on your requirements, there is another way this can do by bringing AWS CDK into the picture. I want to pause right here since this is something I might cover in a future blog post. Until then, Happy Building!