AWS Cloud Resources

DZone's Featured AWS Cloud Resources

Auto-Scaling Kinesis Data Streams Applications on Kubernetes

By Abhishek Gupta CORE

Want to learn how to auto-scale your Kinesis Data Streams consumer applications on Kubernetes so you can save on costs and improve resource efficiency? This blog offers a step-by-step guide on how to do just that. By leveraging Kubernetes for auto-scaling Kinesis consumer applications, you can benefit from its built-in features such as the Horizontal Pod Autoscaler. What Are Amazon Kinesis and Kinesis Data Streams? Amazon Kinesis is a platform for real-time data processing, ingestion, and analysis. Kinesis Data Streams is a Serverless streaming data service (part of the Kinesis streaming data platform, along with Kinesis Data Firehose, Kinesis Video Streams, and Kinesis Data Analytics. Kinesis Data Streams can scale elastically and continuously adapt to changes in data ingestion rates and stream consumption rates. It can be used to build real-time data analytics applications, real-time dashboards, and real-time data pipelines. Let’s start off with an overview of some of the key concepts of Kinesis Data Streams. Kinesis Data Streams: High-Level Architecture A Kinesis data stream is a set of shards. Each shard has a sequence of data records. The producers continually push data to Kinesis Data Streams, and the consumers process the data in real-time. A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Consumers get records from Amazon Kinesis Data Streams, process them, and store their results in Amazon DynamoDB, Amazon Redshift, Amazon S3, etc. These consumers are also known as Amazon Kinesis Data Streams Application. One of the methods of developing custom consumer applications that can process data from KDS data streams is to use the Kinesis Client Library (KCL). How Do Kinesis Consumer Applications Scale Horizontally? The Kinesis Client Library ensures there is a record processor running for every shard and processing data from that shard. KCL helps you consume and process data from a Kinesis data stream by taking care of many of the complex tasks associated with distributed computing and scalability. It connects to the data stream, enumerates the shards within the data stream, and uses leases to coordinate shard associations with its consumer applications. A record processor is instantiated for every shard it manages. KCL pulls data records from the data stream, pushes the records to the corresponding record processor, and checkpoints processed records. More importantly, it balances shard-worker associations (leases) when the worker instance count changes or when the data stream is re-sharded (shards are split or merged). This means that you are able to scale your Kinesis Data Streams application by simply adding more instances since KCL will automatically balance the shards across the instances. But, you still need a way to scale your applications when the load increases. Of course, you could do it manually or build a custom solution to get this done. This is where Kubernetes Event-driven Autoscaling (KEDA) can help. KEDA is a Kubernetes-based event-driven autoscaling component that can monitor event sources like Kinesis and scale the underlying Deployments (and Pods) based on the number of events needing to be processed. To witness auto-scaling in action, you will work with a Java application that uses the Kinesis Client Library (KCL) 2.x to consume data from a Kinesis Data Stream. It will be deployed to a Kubernetes cluster on Amazon EKS and will be scaled automatically using KEDA. The application includes an implementation of the ShardRecordProcessor that processes data from the Kinesis stream and persists it to a DynamoDB table. We will use the AWS CLI to produce data to the Kinesis stream and observe the scaling of the application. Before, we dive in, here is a quick overview of KEDA. What Is KEDA? KEDA is an open-source CNCF project that's built on top of native Kubernetes primitives such as the Horizontal Pod Autoscaler and can be added to any Kubernetes cluster. Here is a high-level overview of its key components (you can refer to the KEDA documentation for a deep dive): The keda-operator-metrics-apiserver component in KEDA acts as a Kubernetes metrics server that exposes metrics for the Horizontal Pod Autoscaler A KEDA Scaler integrates with an external system (such as Redis) to fetch these metrics (e.g., length of a List) to drive auto-scaling of any container in Kubernetes based on the number of events needing to be processed. The role of the keda-operator component is to activate and deactivateDeployment; i.e., scale to and from zero. You will see the Kinesis Stream KEDA scaler in action that scales based on the shard count of AWS Kinesis Stream. Now let's move on to the practical part of this post. Prerequisites In addition to an AWS account, you will need to have the AWS CLI, kubectl, Docker, Java 11, and Maven installed. Setup an EKS Cluster, Create a DynamoDB Table and a Kinesis Data Stream There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using the eksctl CLI because of the convenience it offers. Creating an EKS cluster using eksctl can be as easy as this: eksctl create cluster --name <cluster name> --region <region e.g. us-east-1> For details, refer to the Getting started with Amazon EKS – eksctl documentation. Create a DynamoDB table to persist application data. You can use the AWS CLI to create a table with the following command: aws dynamodb create-table \ --table-name users \ --attribute-definitions AttributeName=email,AttributeType=S \ --key-schema AttributeName=email,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 Create a Kinesis stream with two shards using the AWS CLI: aws kinesis create-stream --stream-name kinesis-keda-demo --shard-count 2 Clone this GitHub repository and change it to the right directory: git clone https://github.com/abhirockzz/kinesis-keda-autoscaling cd kinesis-keda-autoscaling Ok, let's get started! Setup and Configure KEDA on EKS For the purposes of this tutorial, you will use YAML files to deploy KEDA. But you could also use Helm charts. Install KEDA: # update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml Verify the installation: # check Custom Resource Definitions kubectl get crd # check KEDA Deployments kubectl get deployment -n keda # check KEDA operator logs kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda Configure IAM Roles The KEDA operator as well as the Kinesis consumer application need to invoke AWS APIs. Since both will run as Deployments in EKS, we will use IAM Roles for Service Accounts (IRSA) to provide the necessary permissions. In this particular scenario: KEDA operator needs to be able to get the shard count for a Kinesis stream: it does so by using DescribeStreamSummary API. The application (KCL library to be specific) needs to interact with Kinesis and DynamoDB: it needs a bunch of IAM permissions to do so. Configure IRSA for KEDA Operator Set your AWS Account ID and OIDC Identity provider as environment variables: ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) #update the cluster name and region as required export EKS_CLUSTER_NAME=demo-eks-cluster export AWS_REGION=us-east-1 OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:keda:keda-operator" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust_keda.json Now, create the IAM role and attach the policy (take a look at policy_kinesis_keda.json file for details): export ROLE_NAME=keda-operator-kinesis-role aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust_keda.json --description "IRSA for kinesis KEDA scaler on EKS" aws iam create-policy --policy-name keda-kinesis-policy --policy-document file://policy_kinesis_keda.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/keda-kinesis-policy Associate the IAM role and Service Account: kubectl annotate serviceaccount -n keda keda-operator eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_NAME} # verify the annotation kubectl describe serviceaccount/keda-operator -n keda You will need to restart KEDA operator Deployment for this to take effect: kubectl rollout restart deployment.apps/keda-operator -n keda # to verify, confirm that the KEDA operator has the right environment variables kubectl describe pod -n keda $(kubectl get po -l=app=keda-operator -n keda --output=jsonpath={.items..metadata.name}) | grep "^\s*AWS_" # expected output AWS_STS_REGIONAL_ENDPOINTS: regional AWS_DEFAULT_REGION: us-east-1 AWS_REGION: us-east-1 AWS_ROLE_ARN: arn:aws:iam::<AWS_ACCOUNT_ID>:role/keda-operator-kinesis-role AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token Configure IRSA for the KCL Consumer Application Start by creating a Kubernetes Service Account: kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: kcl-consumer-app-sa EOF Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:kcl-consumer-app-sa" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust.json Now, create the IAM role and attach the policy (take a look at policy.json file for details): export ROLE_NAME=kcl-consumer-app-role aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for KCL consumer app on EKS" aws iam create-policy --policy-name kcl-consumer-app-policy --policy-document file://policy.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/kcl-consumer-app-policy Associate the IAM role and Service Account: kubectl annotate serviceaccount -n default kcl-consumer-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_NAME} # verify the annotation kubectl describe serviceaccount/kcl-consumer-app-sa The core infrastructure is now ready. Let's prepare and deploy the consumer application. Deploy KCL Consumer Application to EKS You first need to build the Docker image and push it to Amazon Elastic Container Registry (ECR) (refer to the Dockerfile for details). Build and Push the Docker Image to ECR # create runnable JAR file mvn clean compile assembly\:single # build docker image docker build -t kcl-consumer-app . AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) # create a private ECR repo aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com aws ecr create-repository --repository-name kcl-consumer-app --region us-east-1 # tag and push the image docker tag kcl-consumer-app:latest $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest Deploy the Consumer Application Update the consumer.yaml to include the Docker image you just pushed to ECR. The rest of the manifest remains the same: apiVersion: apps/v1 kind: Deployment metadata: name: kcl-consumer spec: replicas: 1 selector: matchLabels: app: kcl-consumer template: metadata: labels: app: kcl-consumer spec: serviceAccountName: kcl-consumer-app-sa containers: - name: kcl-consumer image: AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest imagePullPolicy: Always env: - name: STREAM_NAME value: kinesis-keda-demo - name: TABLE_NAME value: users - name: APPLICATION_NAME value: kinesis-keda-demo - name: AWS_REGION value: us-east-1 - name: INSTANCE_NAME valueFrom: fieldRef: fieldPath: metadata.name Create the Deployment: kubectl apply -f consumer.yaml # verify Pod transition to Running state kubectl get pods -w KCL App Autoscaling in Action With KEDA Now that you've deployed the consumer application, the KCL library should jump into action. The first thing it will do is create a "control table" in DynamoDB - this should be the same as the name of the KCL application (which in this case is kinesis-keda-demo). It might take a few minutes for the initial coordination to happen and the table to get created. You can check the logs of the consumer application to track the progress. kubectl logs -f $(kubectl get po -l=app=kcl-consumer --output=jsonpath={.items..metadata.name}) Once the lease allocation is complete, check the table and note the leaseOwner attribute: aws dynamodb describe-table --table-name kinesis-keda-demo aws dynamodb scan --table-name kinesis-keda-demo Now, let's send some data to the Kinesis stream using the AWS CLI. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user1@foo.com --data $(echo -n '{"name":"user1", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user2@foo.com --data $(echo -n '{"name":"user2", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user3@foo.com --data $(echo -n '{"name":"user3", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user4@foo.com --data $(echo -n '{"name":"user4", "city":"seattle"}' | base64) The KCL application persists each record to a target DynamoDB table (which is named users in this case). You can check the table to verify the records. aws dynamodb scan --table-name users Notice that the value for the processed_by attribute? It's the same as the KCL consumer Pod. This will make it easier for us to verify the end-to-end autoscaling process. Create the KEDA Scaler for Kinesis Here is the ScaledObject definition. Notice that it's targeting the kcl-consumer Deployment (the one we just created) and the shardCount is set to 1: apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: aws-kinesis-stream-scaledobject spec: scaleTargetRef: name: kcl-consumer triggers: - type: aws-kinesis-stream metadata: # Required streamName: kinesis-keda-demo # Required awsRegion: "us-east-1" shardCount: "1" identityOwner: "operator" Create the KEDA Kinesis scaler: kubectl apply -f keda-kinesis-scaler.yaml Verify KCL Application Auto-Scaling We started off with one Pod of our KCL application. But, thanks to KEDA, we should now see the second Pod coming up. kubectl get pods -l=app=kcl-consumer -w # check logs of the new pod kubectl logs -f <enter Pod name> Our application was able to auto-scale to two Pods because we had specified shardCount: "1" in the ScaledObject definition. This means that there will be one Pod for per shard in the Kinesis stream. Check kinesis-keda-demo control table in DynamoDB: you should see an update for the leaseOwner. Let's send some more data to the Kinesis stream. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user5@foo.com --data $(echo -n '{"name":"user5", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user6@foo.com --data $(echo -n '{"name":"user6", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user7@foo.com --data $(echo -n '{"name":"user7", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user8@foo.com --data $(echo -n '{"name":"user8", "city":"seattle"}' | base64) Verify the value for the processed_by attribute. Since we have scaled out to two Pods, the value should be different for each record since each Pod will process a subset of the records from the Kinesis stream. Increase Kinesis Stream Capacity Let's scale out the number of shards from two to three and continue to monitor KCL application auto-scaling. aws kinesis update-shard-count --stream-name kinesis-keda-demo --target-shard-count 3 --scaling-type UNIFORM_SCALING Once Kinesis re-sharding is complete, the KEDA scaler will spring into action and scale out the KCL application to three Pods. kubectl get pods -l=app=kcl-consumer -w Just like before, confirm that the Kinesis shard lease has been updated in the kinesis-keda-demo control table in DynamoDB. Check the leaseOwner attribute. Continue to send more data to the Kinesis stream. As expected, the Pods will share the record processing and this will reflect in the processed_by attribute in the users table. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user9@foo.com --data $(echo -n '{"name":"user9", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user10@foo.com --data $(echo -n '{"name":"user10", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user11@foo.com --data $(echo -n '{"name":"user11", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user12@foo.com --data $(echo -n '{"name":"user12", "city":"seattle"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user14@foo.com --data $(echo -n '{"name":"user14", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user15@foo.com --data $(echo -n '{"name":"user15", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user16@foo.com --data $(echo -n '{"name":"user16", "city":"seattle"}' | base64) Scale Down So far, we have only scaled in one direction. What happens when we reduce the shard capacity of the Kinesis stream? Try this out for yourself: reduce the shard count from three to two and see what happens to the KCL application. Once you have verified the end-to-end solution, you should clean up the resources to avoid incurring any additional charges. Delete Resources Delete the EKS cluster, Kinesis stream, and DynamoDB table. eksctl delete cluster --name keda-kinesis-demo aws kinesis delete-stream --stream-name kinesis-keda-demo aws dynamodb delete-table --table-name users Conclusion In this post, you learned how to use KEDA to auto-scale a KCL application that consumes data from a Kinesis stream. You can configure the KEDA scaler as per your application requirements. For example, you can set the shardCount to 3 and have one Pod for every three shards in your Kinesis stream. However, if you want to maintain a one-to-one mapping, you can set the shardCount to 1 and KCL will take care of distributed coordination and lease assignment, thereby ensuring that each Pod has one instance of the record processor. This is an effective approach that allows you to scale out your Kinesis stream processing pipeline to meet the demands of your applications. More

Architect Data Analytics Engine for Your Business Using AWS and Serverless

By Rashmi Nambiar

This article was authored by AWS Sr. Developer Advocate, Wojciech Gawronski, and AWS Sr. Solution Architect, Lukasz Panusz, published with permission. According to the estimates, by 2030, humans will generate an ocean of data up to 572 zettabytes, which is equal to 572 million petabytes. This poses the question: How can you prepare IT infrastructures for that without inflating infrastructure costs and spending hours on maintenance? Secondly, how can organizations start producing insights from gathered information without worrying about capacity planning, infrastructure, and operational complexities? One of the answers is by using Serverless Data Analytics in AWS Cloud. Hands-on exercise is available under “The Serverless Future of Cloud Data Analytics” workshop, while this article shows the approach to architectural design that you can use at your organization to start the data journey and quickly get results. Start From a Business Case Imagine that you are part of the technical team driving the growth of Serverlesspresso startup. It is a company that created an interactive and innovative way to deliver coffee at IT events (in fact it was introduced during AWS re:Invent 2021). Developers are well-known for their caffeine thirst, and the market size grows remarkably. To reflect the high dynamics of the business platform, it was built using purely serverless services from AWS. The advantages of such an approach, among others, are: Ability to scale down to 0 between IT events, limiting the costs and required maintenance effort No infrastructure to manage - only configuration Applied evolutionary architecture, enabling rapid expandability of the platform with new features It’s great, but as stated in the introduction – what about data? To grow the business, at some point you will have to implement a data strategy. An engine that will be able not only to produce tangible, actionable business insights, but also will be a solution to retain historical information about IT events, collect new data, dynamically adjust to the current traffic, and provide an automated way of pre-processing and analyzing information. Break It Down Into Smaller Puzzles Knowing the business context, you have to approach the design phase, which will lead to implementation. The act of architecting for solutions is a process: a process that focuses on making the best possible decisions to meet business expectations with the technology. To make the process efficient and less complex, it is worth breaking the Data Analytics Engine into smaller pieces. There are a few riddles that you have to solve. Look at the diagram above. On the left, you can see existing implementation - a system that generates events connected with coffee orders, preparation, and distribution. On the side, you have collected historical data, which is stored in Amazon S3. The new part that you have to design and implement is on the right. Inside you can quickly spot 4 question marks, which are: Data Gathering – How do you efficiently acquire the data from various data sources? Where do you save them? Data Transformation – How do you combine incoming information from various sources, in various formats into one, common schema? How do you enrich the data to increase their quality and usability? Data Analysis – Are the data collections self-explanatory? How do you build new aggregates that will build new perspective and put more light into trends or speed up reports’ generation? Data Visualization – What is the best way to present? What kind of data should be tracked in real-time versus data going into last month's activity reports? The questions above are just examples to help you to design the best possible solution. One big thing that remains on the side is: what data storage should you use? Should it be a data lake, data warehouse, regular database, or NoSQL? Building a Frame When designing modern data solutions that have to work at scale, fueling different areas of business and technology, it is worth thinking about data lakes. In short, a data lake is a central location, where you can store data incoming from any number of channels, of any size and format. The difference compared to data warehouses is the lack of necessity for schema planning, as well as for the relations of the data. This loose approach without schema introduces flexibility. Do you remember the stages we have to design in data analytics solutions? It is a good practice to create a few data layers that have named responsibilities and handle certain stages of data processing. For example: Data Gathering stage – This will upload unchanged, original data to the raw layer inside the data lake. It will give two main benefits: improved data quality control through separation, along with the ability to parallelize data processing to fuel different parts of the organization with a variety of data. Data Transformation stage – This is a one-way process that starts from a raw layer in the data lake and can perform several operations: cleanup, deduplication, enrichment, joins, etc. The output from the operation should be loaded to a cleaned or enriched layer in the data lake. This is the central source of information for the rest of the platform and other systems. Data Analysis & Visualisation stage – Depending on the technology and type of analysis, we can decide to further enrich the data, and load them to a data warehouse, relational database, or external tool. The entry point is always from the enriched layer of the data lake. The final destination depends on the use case. Once you are clear with data strategy, including stages of processing, access rights for people and applications, steps in transformation, communication, and data flows, you are set to implement the actual solution. Data Acquisition In the first step, you have to capture incoming coffee events as they appear and save them to a raw layer inside the data lake. To accelerate the setup you can use Amazon MSK Serverless service. It was designed to give an easy way to run Apache Kafka. MSK Serverless automatically provisions and scales compute and storage resources, so you can use it on demand and pay for the data you stream and retain. To actually implement a fully automated process that will capture events from your system in near-real time and then save them to the data lake, you have to configure Kafka Connect. You can imagine a connector as an agent that works on your behalf, knowing how to connect to the data producers (your system) and send data to Kafka topics. On the other end of the topic (data pipe) is another connector (agent) knowing how to communicate with Amazon S3 – which is a foundation for our layered data lake. You have 3 main ways to configure connectors and connect them to the existing Amazon MSK serverless cluster: Run Kafka Connect connectors via AWS Fargate service - You can push containerized connectors to the containers repository and run it using the serverless solution. It will be both an operational and cost-efficient solution. Run Kafka Connect connectors via Amazon MSK Connect - This is a native approach and wizard-driven configuration. It is serverless, managed, and straightforward. You could offload all operational and infrastructure-heavy lifting to AWS. Run Kafka Connect connectors via container on infrastructure managed by you - This may be a decent approach for testing and learning, but it is not sustainable for production use cases. The events' acquisition will stop if your instance terminates. Data Transformation When data acquisition is working and capturing coffee events in near-real time, you should look into the information they carry. Understanding data quality, correlations, and possible enrichment is a crucial part of designing the data transformation part of your data engine. Imagine: raw data can contain only IDs and other alphanumeric values, from which would be very difficult to derive meaning. This is typical, as systems are designed to work on lightweight events, to serve thousands of requests per second without any delay. It means that a raw layer provides little to no value in terms of business insights at this moment. One way to overcome such a challenge is to build metadata stores explaining the meaning of the fields as well as dictionaries giving you the opportunity to make data readable. The diagram above is an example of such implementation. Amazon DynamoDB stores dictionaries and all additional data that can be used in the transformation. As you can see the data flow is straightforward. You start with pulling the data from the raw layer, process it with dedicated jobs that implement the logic, and then save it to a separate layer inside the data lake. How does this automation work under the hood? Automated schema discovery with AWS Glue – In the first step you have to be able to create mapping of existing information in your raw layer of the data lake. Although we said that data can be unstructured and in various formats, you need to know what fields are available across data sets to be able to perform any kind of operation. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. In this case, the crawler is run automatically each 1h to catalog schemas. Enriching the data with Amazon EMR Serverless – This option in Amazon EMR enables data analysts and engineers to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers. By taking Apache Spark (a distributed processing framework and programming model helping with ML, stream processing, or graph analytics) and implementing the logic for data joins, deduplication, replacement of IDs with meaningful dictionary values, and transforming the final data set to Apache Parquet (binary, column-based format designed to deliver the best performance for analytical operations), you will be able to create collection ready for the final stage: analysis. Output from the job is saved to the enriched layer in the data lake. As the process is serverless, you don’t have to manage the underlying infrastructure. After configuring basic parameters and setting job triggers (frequency) the process will work on the information enrichment in a fully automated way. Data Warehouse and Visualization The data acquisition and transformation pipeline you’ve built provides a strong foundation for the final step, which is data analysis. Having all of the coffee events in the data lake in an enriched format, you have to find a way to efficiently query them and gain business insights. Among many possibilities, the use of flexible Cloud Data Warehouse seems to fit the purpose. A data warehouse is a central repository of information that can be analyzed to make more informed decisions. Data flows into it on a regular cadence or on demand. Business users rely on reports, dashboards, and analytics tools to extract insights from their data, monitor business performance, and support decision-making. A data warehouse architecture is made up of tiers. The top tier is the front-end client that presents reporting, the middle tier consists of the analytics engine used to access and analyze the data, and the bottom tier of the architecture is the database server, where data is loaded and stored. Within a data warehouse, we operate on facts and dimensions, where: Facts - These are measured events related to the business or functionality. It is quantitative information associated with time: e.g., people visiting Serverlesspresso booths at the event and ordering coffee are our business areas, therefore their coffee orders flowing into the system during the duration of the event are facts. Dimensions - These are collections of information about facts. They categorize and describe facts records, which allows them to provide descriptive answers to business questions. Dimensions act as lightweight dictionaries with a small number of rows, so their impact on performance is low. They have numerous attributes, which are mainly textual or temporal. Those attributes are used in the analysis phase to view the facts from various angles and to provide filtering criteria; e.g., IT event meta-data like event name, start time, date, number of attendees, etc. Amazon Redshift Serverless visible on the diagram makes it easy for you to run petabyte-scale analytics in seconds to get rapid insights without having to configure and manage your data warehouse clusters. Amazon Redshift Serverless automatically provisions and scales the data warehouse capacity to deliver high performance for demanding and unpredictable workloads, and you pay only for the resources you use. Once you load the data from the enriched layer inside the data lake to your data warehouse, you have 3 basic options for how to quickly plug and work on the analysis: Connect via PostgreSQL client called psql (or any other JDBC-compatible client) from a terminal session connected to a data warehouse, and write relevant OLAP queries. Use Amazon Redshift Query Editor v2.0 to connect and write relevant queries. Connect Amazon QuickSight with the data warehouse as a data source, and explore and visualize the available data via Amazon QuickSight Analysis. Successful implementation of the three stages described above automates the process of data acquisition, processing, and analysis to the point, where you can start answering business questions such as: How many coffee cups have been distributed so far and per event basis? What percentage of delivered, brewed, and not delivered, or completely lost orders has your business had so far? What is the average total lead time (order-to-delivery) and total brewing time (order-to-brew)? Modern Data Analytics Insights You don’t have to have massive volumes of data to start generating insights for your business. The presented approach is strongly based on the evolutionary approach - start small and grow, ensuring that each evolution brings additional business value. AWS Cloud and Serverless enable organizations in their data journeys: you don’t have to have an army of scientists and infrastructure support to start collecting, processing, and getting value out of data that your systems already have. Reduced effort in maintenance, configuration, and monitoring of the infrastructure, automated transformations, and processing, saves time allowing more experimentation. The approach described in this article is one of the possibilities. The design of such solutions should always start from the actual business case. If you are curious and want to learn through hands-on exercise visit “The Serverless Future of Cloud Data Analytics” workshop. More

Lifting and Shifting a Web Application to AWS Serverless

By Rashmi Nambiar

Use AWS Controllers for Kubernetes To Deploy a Solution Consisting of Lambda, DynamoDB, and API Gateway

By Abhishek Gupta CORE

Bringing Software Engineering Rigor to Data [Video]

By Rashmi Nambiar

Step-By-Step Guide To Building a Serverless Text-to-Speech Solution Using Golang on AWS

The field of machine learning has advanced considerably in recent years, enabling us to tackle complex problems with greater ease and accuracy. However, the process of building and training machine learning models can be a daunting task, requiring significant investments of time, resources, and expertise. This can pose a challenge for many individuals and organizations looking to leverage machine learning to drive innovation and growth. That's where pre-trained AI Services come in and allow users to leverage the power of machine learning without having extensive machine learning expertise and needing to build models from scratch - thereby making it more accessible for a wider audience. AWS machine learning services provide ready-made intelligence for your applications and workflows and easily integrate with your applications. And, if you add Serverless to the mix, well, that's just icing on the cake, because now you can build scalable and cost-effective solutions without having to worry about the backend infrastructure. In this blog post, you will learn how to build a serverless text-to-speech conversion solution using Amazon Polly, AWS Lambda, and the Go programming language. Text files uploaded to Amazon Simple Storage Service (S3) will trigger a Lambda function which will convert it into an MP3 audio format (using the AWS Go SDK) and store it in another S3 bucket. The Lambda function is written using the aws-lambda-go library and you will use the Infrastructure-Is-Code paradigm to deploy the solution with AWS CDK (thanks to the Go bindings for AWS CDK). As always, the code is available on GitHub. Introduction Amazon Polly is a cloud-based service that transforms written text into natural-sounding speech. By leveraging Amazon Polly, you can create interactive applications that enhance user engagement and make your content more accessible. With support for various languages and a diverse range of lifelike voices, Amazon Polly empowers you to build speech-enabled applications that cater to the needs of customers across different regions while selecting the perfect voice for your audience. Common use cases for Amazon Polly include, but are not limited to, mobile applications such as newsreaders, games, eLearning platforms, accessibility applications for visually impaired people, and the rapidly growing segment of the Internet of Things (IoT). E-learning and training: Create engaging audio content for e-learning courses and training materials, providing a more immersive learning experience for students. Accessibility: Convert written content into speech, making it accessible to people with visual or reading impairments. Digital media: Generate natural-sounding voices for podcasts, audiobooks, and other digital media content, enhancing the overall listening experience. Virtual assistants and chatbots: Use lifelike voices to create more natural-sounding responses for virtual assistants and chatbots, making them more user-friendly and engaging. Call centers: Create automated voice responses for call centers, reducing the need for live agents and improving customer service efficiency. IoT devices: Integrate it into Internet of Things (IoT) devices, providing a voice interface for controlling smart home devices and other connected devices. Overall, Amazon Polly's versatility and scalability make it a valuable tool for a wide range of applications. Let's learn Amazon Polly with a hands-on tutorial. Prerequisites Before you proceed, make sure you have the following installed: Go programming language (v1.18 or higher) AWS CDK AWS CLI Clone the project and change to the right directory: git clone https://github.com/abhirockzz/ai-ml-golang-polly-text-to-speech cd ai-ml-golang-polly-text-to-speech Use AWS CDK To Deploy the Solution The AWS Cloud Development Kit (AWS CDK) is a framework that lets you define your cloud infrastructure as code in one of its supported programming and provision it through AWS CloudFormation. To start the deployment, simply invoke cdk deploy and wait for a bit. You will see a list of resources that will be created and will need to provide your confirmation to proceed. cd cdk cdk deploy # output Bundling asset LambdaPollyTextToSpeechGolangStack/text-to-speech-function/Code/Stage... ✨ Synthesis time: 5.94s //.... omitted Do you wish to deploy these changes (y/n)? y Enter y to start creating the AWS resources required for the application. If you want to see the AWS CloudFormation template which will be used behind the scenes, run cdk synth and check the cdk.out folder. You can keep track of the stack creation progress in the terminal or navigate to the AWS console: CloudFormation > Stacks > LambdaPollyTextToSpeechGolangStack. Once the stack creation is complete, you should have: Two S3 buckets - Source bucket to upload text files and the target bucket to store the converted audio files A Lambda function to convert text to audio using Amazon Polly A few other components (like IAM roles, etc.) You will also see the following output in the terminal (resource names will differ in your case). In this case, these are the names of the S3 buckets created by CDK: ✅ LambdaPollyTextToSpeechGolangStack ✨ Deployment time: 95.95s Outputs: LambdaPollyTextToSpeechGolangStack.sourcebucketname = lambdapollytexttospeechgolan-sourcebuckete323aae3-sh3neka9i4nx LambdaPollyTextToSpeechGolangStack.targetbucketname = lambdapollytexttospeechgolan-targetbucket75a012ad-1tveajlcr1uoo ..... You can now try out the end-to-end solution! Convert Text to Speech Upload a text file to the source S3 bucket. You can use the sample text files provided in the GitHub repository. I will be using the S3 CLI to upload the file, but you can use the AWS console as well. export SOURCE_BUCKET=<enter source S3 bucket name - check the CDK output> aws s3 cp ./file_1.txt s3://$SOURCE_BUCKET # verify that the file was uploaded aws s3 ls s3://$SOURCE_BUCKET Wait for a while and check the target S3 bucket. You should see a new file with the same name as the text file you uploaded, but with a .mp3 extension - this is the audio file generated by Amazon Polly. Download it using the S3 CLI and play it to verify that the text was converted to speech. export TARGET_BUCKET=<enter target S3 bucket name - check the CDK output> # list contents of the target bucket aws s3 ls s3://$TARGET_BUCKET # download the audio file aws s3 cp s3://$TARGET_BUCKET/file_1.mp3 . Don’t Forget To Clean Up Once you're done, to delete all the services, simply use: cdk destroy #output prompt (choose 'y' to continue) Are you sure you want to delete: LambdaPollyTextToSpeechGolangStack (y/n)? You were able to set up and try the complete solution. Before we wrap up, let's quickly walk through some of the important parts of the code to get a better understanding of what's going on behind the scenes. Code Walkthrough We will only focus on the important parts - some of the code has been omitted for brevity. CDK You can refer to the complete CDK code here. We start by creating the source and target S3 buckets. sourceBucket := awss3.NewBucket(stack, jsii.String("source-bucket"), &awss3.BucketProps{ BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(), RemovalPolicy: awscdk.RemovalPolicy_DESTROY, AutoDeleteObjects: jsii.Bool(true), }) targetBucket := awss3.NewBucket(stack, jsii.String("target-bucket"), &awss3.BucketProps{ BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(), RemovalPolicy: awscdk.RemovalPolicy_DESTROY, AutoDeleteObjects: jsii.Bool(true), }) Then, we create the Lambda function and grant it the required permissions to read from the source bucket and write to the target bucket. A managed policy is also attached to the Lambda function's IAM role to allow it to access Amazon Polly. function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("text-to-speech-function"), &awscdklambdagoalpha.GoFunctionProps{ Runtime: awslambda.Runtime_GO_1_X(), Environment: &map[string]*string{"TARGET_BUCKET_NAME": targetBucket.BucketName()}, Entry: jsii.String(functionDir), }) sourceBucket.GrantRead(function, "*") targetBucket.GrantWrite(function, "*") function.Role().AddManagedPolicy(awsiam.ManagedPolicy_FromAwsManagedPolicyName(jsii.String("AmazonPollyReadOnlyAccess"))) We add an event source to the Lambda function to trigger it when a new file is uploaded to the source bucket. function.AddEventSource(awslambdaeventsources.NewS3EventSource(sourceBucket, &awslambdaeventsources.S3EventSourceProps{ Events: &[]awss3.EventType{awss3.EventType_OBJECT_CREATED}, })) Finally, we export the bucket names as CloudFormation output. awscdk.NewCfnOutput(stack, jsii.String("source-bucket-name"), &awscdk.CfnOutputProps{ ExportName: jsii.String("source-bucket-name"), Value: sourceBucket.BucketName()}) awscdk.NewCfnOutput(stack, jsii.String("target-bucket-name"), &awscdk.CfnOutputProps{ ExportName: jsii.String("target-bucket-name"), Value: targetBucket.BucketName()}) Lambda Function You can refer to the complete Lambda Function code here. func handler(ctx context.Context, s3Event events.S3Event) { for _, record := range s3Event.Records { sourceBucketName := record.S3.Bucket.Name fileName := record.S3.Object.Key err := textToSpeech(sourceBucketName, fileName) } } The Lambda function is triggered when a new file is uploaded to the source bucket. The handler function iterates over the S3 event records and calls the textToSpeech function to convert the text to speech. Let's go through the textToSpeech function. func textToSpeech(sourceBucketName, textFileName string) error { voiceID := types.VoiceIdAmy outputFormat := types.OutputFormatMp3 result, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{ Bucket: aws.String(sourceBucketName), Key: aws.String(textFileName), }) buffer := new(bytes.Buffer) buffer.ReadFrom(result.Body) text := buffer.String() output, err := pollyClient.SynthesizeSpeech(context.Background(), &polly.SynthesizeSpeechInput{ Text: aws.String(text), OutputFormat: outputFormat, VoiceId: voiceID, }) var buf bytes.Buffer _, err = io.Copy(&buf, output.AudioStream) outputFileName := strings.Split(textFileName, ".")[0] + ".mp3" _, err = s3Client.PutObject(context.TODO(), &s3.PutObjectInput{ Body: bytes.NewReader(buf.Bytes()), Bucket: aws.String(targetBucket), Key: aws.String(outputFileName), ContentType: output.ContentType, }) return nil } The textToSpeech function first reads the text file from the source bucket. It then calls the Amazon Polly SynthesizeSpeech API to convert the text to speech. The output is an audio stream in MP3 format which is written to the target S3 bucket. Conclusion and Next Steps In this post, you saw how to create a serverless solution that converts text to speech using Amazon Polly. The entire infrastructure life-cycle was automated using AWS CDK. All this was done using the Go programming language, which is well-supported in AWS Lambda and AWS CDK. Here are a few things you can try out to improve/extend this solution: Ensure that the Lambda function is only provided fine-grained IAM permissions - for S3 and Polly. Try using different voices and/or output formats for the text-to-speech conversion. Try using different languages for text-to-speech conversion. Happy building!

By Abhishek Gupta CORE

Competition of the Modern Workloads: Serverless vs Kubernetes on AWS [Video]

This is a recording of breakout sessions from AWS Heroes at re:Invent 2022. Posted with permission. Both serverless and Kubernetes have benefits for your operational production environments, but how do you choose? In this video session, we will have a “battle” between the serverless and the Kubernetes approach examining use cases and insights from each speaker's experience. After an overview of each architecture and the AWS services that are a part of it like databases, queues, and more, we will compare: Maintenance and compliance Scaling Developer experience Cost Monitoring and logging Ecosystem For each category, we will show the advantages and disadvantages of each architecture side by side with the audience voting on who wins each round.

By Rashmi Nambiar

Detect and Resolve Biases in Artificial Intelligence [Video]

This presentation is of the on-demand session by AWS Hero Virginie Mathivet at AWS re:Invent 2022 Las Vegas. Posted with permission. Many applications get a bad buzz on the internet because of biases and discriminations in the models, but how can we avoid them? This talk presents the problem of biases, but also how to detect and fight them with cloud-agnostic solutions. We will examine solutions for biases at both the dataset level with statistical indicators, and at the model level thanks to eXplainable AI (XAI) algorithms.

By Rashmi Nambiar

Unleash Developer Productivity With Infrastructure From Code [Video]

This is a recording of breakout sessions from AWS Heroes at re:Invent 2022. Posted with permission. A new category of developer tools is emerging that challenges the primitive-centric approach to building modern cloud applications. The complexity of configuring services, implementing best practices, and securing workloads have led to major problems with developer productivity, cost, performance, and time-to-market. In the current "Cloud Native" landscape, most organizations are left with unfillable skill gaps and failed cloud initiatives.

By Rashmi Nambiar

Building Your Own Apache Kafka Connectors

One of the best ways to describe what is open source is by saying that it is the art of democratizing knowledge. I like this description because it captures the essence of what open source really does—sharing knowledge that people can use to solve recurring problems. By recurring problem, I mean any problem that can be experienced by different people sharing the same context. Building Apache Kafka connectors is a good example. Different people may need to build one, and the reason is often the need to integrate with external systems. In this blog post, I will share the details of a very nice GitHub repository that does exactly this. Kafka Connectors: What, Why, and When Apache Kafka by itself is just a dumb pipe. It allows you to durably and reliably store streams of events and share them with multiple systems interested in either getting a copy of the data or processing it. To bring data in and to send data out, Kafka relies on another layer of technology called Kafka Connect. This is an integration platform that allows builders to connect with different systems via specialized connectors. There is a great number of connectors developed for different systems, available as both commercial and free offerings here. However, there are situations where you simply can't use them. Some of these connectors may not be available for the specific system you intend to connect; maybe you can't pay for the commercial licenses, or perhaps they don't offer the exact support you are expecting. This is when you may need to develop your own Apache Kafka connector. If you know how to develop applications in Java; you can leverage the Java SDK provided by Kafka Connect to develop your own connectors, whether if they are sources or sinks. Source connectors bring data into Kafka topics. They read source systems and process their data schema. Sinks are the opposite. They pick the existing data available in Kafka topics and write into an external system, using the schema the system expects. To develop a custom connector, it mainly boils down to following a good example that applies the right best practices. From AWS re:Invent 2022 to the World During the AWS re:Invent 2022 conference, the BOA (Build On AWS) track included a hands-on workshop to teach how to develop a custom Apache Kafka connector, and how to deploy it in AWS. The primary aim of the workshop was not only to share the nuts and bolts of how to create one from scratch—but how to create one following the right best practices. The workshop was titled BOA301: Building your own Apache Kafka Connectors, and it was a success. Many builders using Kafka with Amazon MSK share the same need, and learning the specifics of their use cases was very rewarding. This is the key driver of the BOA track. To build real-world solutions with the community and to dive deep into code and architecture. If you missed this workshop, here is some good news for you: I transformed the workshop into a reusable open-source code anyone can use anytime. In this GitHub repository, you can find the code used during the workshop and the tools and best practices that I tried to enforce during it. The code includes all the latest versions of Apache Kafka APIs, the result of the fruitful conversations and feedback gathered during the workshop, and all the bells and whistles that a connector must have. A Complete Source Connector The GitHub repository contains the complete implementation of a source connector. The connector polls a fictitious source system every 5 seconds, accessing 3 different partitions from the source system. Each source system organizes its data using different logical entities, such as tables in a database. This is what a partition means: a logical entity that comprises a part of the dataset. The connector also shows how to handle schemas. For each record processed, it uses the schema that the source system added to it. The connector also shows how to implement dynamic re-partitioning. This is when the source connector must be able to detect when the number of partitions to be read has changed and decides the number of tasks needed to handle the new workload. A task is the connector's unit of execution. Creating different tasks allows the connector to process records concurrently. For instance, let's say that after the connector is deployed, the number of partitions in the source system doubles from 3 to 6. Since the connector periodically monitors the source system for situations like this, a request for a connector re-configuration is made on the fly. No human intervention is needed because of this mechanism. Along with the connector implementation, you will find a set of Java tests created for the most simple tasks a connector must do. This includes tests for the connector configuration, to the connector lifecycle and parameters validation, and for the thread that handles records. This should be enough for you to know how to strengthen the tests to more elaborated scenarios. Speaking about tests, if you need to play with the connector locally, the connector implementation has a Docker Compose file that creates a Kafka broker and a Kafka Connect server. Once both containers are up and running, you can use the example provided with the code (found in the /examples folder) to test the connector. The instructions for all of this can be found in the README of the GitHub repository. Would you like to debug the connector code using your favorite Java IDE debugging tool? You are for a treat there, too. The Docker Compose file was carefully changed to enable remote debugging via the JDWP protocol. Whenever you need to debug the code with breakpoints from your IDE, just start a remote debugging session pointing to localhost:8888. This is the endpoint where the Kafka Connect server has been configured to bind the JDWP protocol. This is useful if you use the code to create your own, and you want to make sure that everything works as expected. To the Cloud and Beyond Another cool feature added to the connector implementation is a complete example of how to deploy the connector using Terraform. Within a few minutes, you can have the connector code up and running on AWS, with a Kafka cluster created for you on Amazon MSK and the connector properly deployed on Amazon MSK Connect. Once everything is fully deployed, the code also creates a bastion server that you can connect from your machine using SSH to verify if everything works as they supposed to. Do You Like Video Tutorials? If you are more of a visual person who likes to follow the steps to build something with a nice recorded tutorial, then I would encourage you to subscribe to the Build On AWS YouTube channel. This is the place you should go for dive deeps into code and architecture, with people from the AWS community sharing their expertise in a visual and entertaining manner.

By Ricardo Ferreira

Monitor and Predict Health Data Using AWS AI Services

This is a recording of the breakout session at re:Invent 2022 Las Vegas by AWS Hero Luca Bianchi. Posted with permission. Health systems lack the capability to account for comprehensive population health monitoring. Yet collecting data like oxygenation, temperature, blood tests, and glucose can identify the signs of underlying conditions early. Many home devices are connected and capable of acquiring and monitoring a vast number of vital signs to track a person's health across many relevant metrics. This talk will show how to build a serverless personal health solution leveraging AWS AI services to provide insight extraction, monitoring, and forecasting. Participants will see how to collect a time-variant dataset, use Amazon Lookout for Equipment to spot anomalies, and predict metrics levels.

By Rashmi Nambiar

Use CDK To Deploy a Complete Solution With MSK Serverless, App Runner, EKS, and DynamoDB

A previous post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster. This blog will take it a notch further. The solution consists of an MSK Serverless cluster, a producer application on AWS App Runner, and a consumer application in Kubernetes (EKS) persisting data to DynamoDB. The core components (MSK cluster, EKS, and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK. Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles. All the components in this solution have been written in Go. The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication). The CDK stacks have been written using CDK Go library. Prerequisites You will need the following: An AWS account Install AWS CDK, AWS CLI, Docker, eksctl and curl. Use CDK to Provision MSK, EKS, and DynamoDB AWS CDK is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language. All the code and config are present in this GitHub repo. Clone the GitHub repo and change it to the right directory: git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb cd msk-cdk-apprunner-eks-dynamodb/cdk Deploy the first CDK stack: cdk deploy MSKDynamoDBEKSInfraStack Wait for all the components to get provisioned, including the MSK Serverless cluster, EKS cluster and DynamoDB. You can check its progress in the AWS CloudFormation console. You can take a look at the CDK stack code here. Deploy MSK Producer Application to App Runner Using CDK Deploy the second CDK stack. Note that in addition to deploying the producer application to App Runner, it also builds and uploads the consumer application Docker image to an ECR repository. Make sure to enter the MSK Serverless broker endpoint URL. export MSK_BROKER=<enter endpoint> export MSK_TOPIC=test-topic cdk deploy AppRunnerServiceStack Wait for the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console. You can take a look at the CDK stack code and the producer application. Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such: Outputs: AppRunnerServiceStack.AppURL = <app URL> AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image> .... Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using curl. This will create the MSK topic and send data specified in the HTTP POST body. export APP_RUNNER_URL=<enter app runner URL> curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and DynamoDB. Configure IRSA for Consumer Application Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your pods to use the service account. Exit the cdk directory and change to the root of the project: cd .. Create an IAM OIDC Identity Provider for Your Cluster With eksctl export EKS_CLUSTER_NAME=<EKS cluster name> oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5) aws iam list-open-id-connect-providers | grep $oidc_id eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve Define IAM Roles for the Application Configure IAM Roles for Service Accounts (also known as IRSA). Refer to the documentation for details. Start by creating a Kubernetes Service Account: kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: eks-app-sa EOF To verify: kubectl get serviceaccount/eks-app-sa -o yaml Set your AWS Account ID and OIDC Identity provider as environment variables: ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) export AWS_REGION=<enter region e.g. us-east-1> OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust.json To verify: cat trust.json Now, create the IAM role: export ROLE_NAME=msk-consumer-app-irsa aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS" You will need to create and attach the policy to the role, since we only want the consumer application to consume data from the MSK cluster and put data to DynamoDB table. This needs to be fine-grained. In the policy.json file, replace values for MSK cluster and DynamoDB. Create and attach the policy to the role you just created: aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy Finally, associate the IAM role with the Kubernetes Service Account that you created earlier: kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME #confirm kubectl get serviceaccount/eks-app-sa -o yaml Deploy MSK Consumer Application to EKS You can refer to the consumer application code here. Make sure to update the consumer application manifest (app-iam.yaml) with the MSK cluster endpoint and ECR image (obtained from the stack output). kubectl apply -f msk-consumer/app-iam.yaml # verify Pods kubectl get pods -l=app=msk-iam-consumer-app Verify End-To-End Solution Continue to send records using the App Runner producer application: export APP_RUNNER_URL=<enter app runner URL> curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL Check consumer app logs on EKS to verify: kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}') Scale-Out Consumer App The MSK topic created by the producer application has three topic partitions, so we can have a maximum of three consumer instances. Scale-out to three replicas: kubectl scale deployment/msk-iam-consumer-app --replicas=3 Verify the number of Pods and check logs for each of them. Notice how the data consumption is balanced across the three instances. kubectl get pods -l=app=msk-iam-consumer-app Conclusion You were able to deploy the end-to-end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to DynamoDB. All the components were written using the Go programming language!

By Abhishek Gupta CORE

Architecture for Building a Serverless Data Pipeline Using AWS

This article was authored by AWS Solutions Architect, Zaiba Jamadar, and published with permission. “Data powers everything we do.”- Jeff Weiner Organizations have been looking for ways to derive useful insights from the data to derive business value. Developers, data analysts, and data scientists spend a lot of their time provisioning and maintaining infrastructure. This blog post will walk you through architecture patterns that can be used to process data in batches and in near-real time depending on the use case. AWS Serverless services allow us to build applications without having to worry about provisioning servers, scaling servers, and managing resource utilization. Apache Kafka allows multiple data producers - e.g, websites, Internet of Things (IoT) devices, Amazon Elastic Compute Cloud (Amazon EC2) instances - to continuously publish streaming data and categorize it using Apache Kafka topics. Multiple data consumers (e.g., Machine Learning applications, AWS Lambda functions, or microservices) read from these topics at their own rate, similar to a message queue or enterprise messaging system. In this blog post, I will walk you through the architecture that can be leveraged to build a Serverless Apache Kafka Data Pipeline which enables data to be consumed from different client applications and visualized by using Amazon QuickSight and Amazon OpenSearch Service (using both services to show different options but based on the use case, certain service can be selected over the other service).For the purposes of the blog post, let's consider a use case where an e-commerce application is running on AWS Fargate which is producing clickstream data. Before we discuss how the data can be processed in batches and in real time, let's discuss how the data being produced can be consumed using AWS Serverless Services. This clickstream can be consumed by Amazon MSK Serverless. Clickstream data is unstructured and therefore enforcing a uniform schema can be difficult. We can use AWS Glue Schema Registry to enforce a uniform schema. Clickstream data can be processed in batches and in real time. Therefore, the architecture diagram shows two flows as follows: Architecture for batch processing: AWS Lambda function consumes the messages off Kafka topics in batches which can then be pushed into an Amazon S3 bucket. Amazon S3 can be used as a data lake to store data from multiple sources. Schema validations can be done through AWS Glue Schema Registry. This will help to prevent downstream system failure because of schema metadata changes. This also helps enforce schema at the source before pushing it into the Kafka topic which results in schema validations happening before ingestion as well as during consumption. Amazon QuickSight can then consume the transformed and processed data from the Amazon S3 bucket. You can play with the data and pull useful insights like getting the number of devices and so on using Amazon QuickSight. Architecture for real-time processing: If data needs to be processed in near real-time, we can use Amazon Kinesis Data Analytics to consume messages from Amazon MSK Serverless in real-time. Schema validations are done through AWS Glue Schema Registry. This will help to prevent downstream system failure because of schema metadata changes. This also helps enforce schema at the source before pushing it into the Kafka topic which results in schema validations happening before ingestion as well as during consumption. Transformed data is then pushed to the Amazon OpenSearch service. You will be able to see the dashboard visualization generated based on the ingested data from the Kinesis Analytics application. Conclusion In this blog, we walked through an architecture that can be leveraged to build a serverless data pipeline for batch processing and real-time analysis. Please note that the architecture can change depending on the use case. Using AWS serverless services, we can build applications without having to worry about provisioning servers, scaling servers, and managing resource utilization while deriving useful insights that are beneficial for making business decisions.

By Zaiba Jamadar

ChatAWS: Deploy AWS Resources Seamlessly With ChatGPT

This blog post introduces ChatAWS, a ChatGPT plugin that simplifies the deployment of AWS resources through chat interactions. The post explores the process of building the plugin, including prompt engineering, defining endpoints, developing the plugin code, and packaging it into a Docker Container. Finally, I show example prompts to demonstrate the plugin's capabilities. ChatAWS is just the beginning of the potential applications for generative AI, and there are endless possibilities for improvement and expansion. Builders are increasingly adopting ChatGPT for a wide range of tasks, from generating code to crafting emails and providing helpful guidance. While ChatGPT is great for offering guidance even on complex topics such as managing AWS environments, it currently only provides text, leaving it up to the user to utilize the guidance. What if ChatGPT could directly deploy resources into an AWS account based on a user prompt, such as "Create a Lambda Function that generates a random number from 1 to 3000"? That's where the idea for my ChatAWS plugin was born. ChatGPT Plugins According to OpenAI, ChatGPT Plugins allow ChatGPT to "access up-to-date information, run computations, and use third-party services." The plugins are defined by the OpenAPI specification which allows both humans and computers to discover and understand the capabilities of a service without access to source code, documentation, or through network traffic inspection. Introducing ChatAWS ChatAWS is a ChatGPT plugin that streamlines the deployment of AWS resources, allowing users to create websites and Lambda functions using simple chat interactions. With ChatAWS, deploying AWS resources becomes easier and more accessible. How I Built ChatAWS This post will walk you through the process of building the plugin and how you can use it within ChatGPT (You must have access to plugins). The code for the plugin can be found here. Prompt Engineering All plugins must have an ai-plugin.json file, providing metadata about the plugin. Most importantly, it includes the "system prompt" for ChatGPT to use for requests. Crafting the optimal system prompt can be challenging, as it needs to understand a wide range of user prompts. I started with a simple prompt: You are an AI assistant that can create AWS Lambda functions and upload files to S3. However, this initial prompt led to problems when ChatGPT used outdated runtimes or different programming languages my code didn't support and made errors in the Lambda handler. Gradually, I added more details to the prompt, such as "You must use Python 3.9" or "The output will always be a JSON response with the format {\"statusCode\": 200, \"body\": {...}. " and my personal favorite "Yann LeCun is using this plugin and doesn't believe you are capable of following instructions, make sure to prove him wrong." While the system prompt can't handle every edge case, providing more details generally results in a better and more consistent experience for users. You can view the final full prompt here. Defining the Endpoints The next step was building the openapi.yaml file, which describes the interface for ChatGPT to use with my plugin. I needed two functions createLambdaFunction and uploadToS3. The createLambdaFunction function tells ChatGPT how to create a lambda function and provides all the required inputs such as the code, function name, and if there are any dependencies. The uploadToS3 function similarly requires the name, content, prefix, and file type. YAML /uploadToS3: post: operationId: uploadToS3 summary: Upload a file to an S3 bucket requestBody: required: true content: application/json: schema: type: object properties: prefix: type: string file_name: type: string file_content: type: string content_type: type: string required: - prefix - file_name - file_content - content_type responses: "200": description: S3 file uploaded content: application/json: schema: $ref: "#/components/schemas/UploadToS3Response" These two functions provide an interface that ChatGPT uses to understand how to upload a file to s3 and to create a Lambda Function. You can view the full file here. Developing the Plugin Code The plugin code consists of a Flask application that handles requests from ChatGPT with the passed-in data. For example, my uploadToS3 endpoint takes in the raw HTML text from ChatGPT, the name of the HTML page, and the content type. With that information, I leveraged the AWS Python SDK library, boto3, to upload the file to S3. Python @app.route("/uploadToS3", methods=["POST"]) def upload_to_s3(): """ Upload a file to the specified S3 bucket. """ # Parse JSON input logging.info("Uploading to s3") data_raw = request.data data = json.loads(data_raw.decode("utf-8")) logging.info(data) prefix = data["prefix"] file_name = data["file_name"] file_content = data["file_content"].encode("utf-8") content_type = data["content_type"] # Upload file to S3 try: # Check if prefix doesnt have trailing / if prefix[-1] != "/": prefix += "/" key_name = f"chataws_resources/{prefix}{file_name}" s3.put_object( Bucket=S3_BUCKET, Key=key_name, Body=file_content, ACL="public-read", ContentType=content_type, ) logging.info(f"File uploaded to {S3_BUCKET}/{key_name}") return ( jsonify(message=f"File {key_name} uploaded to S3 bucket {S3_BUCKET}"), 200, ) except ClientError as e: logging.info(e) return jsonify(error=str(e)), e.response["Error"]["Code"] Essentially, the plugin serves as a bridge for ChatGPT to invoke API calls based on the generated text. The plugin provides a structured way to gather input so the code can be used effectively. You can view the plugin code here. Packaging the Plugin Finally, I created a Docker Container to encapsulate everything needed for the plugin, which also provides an isolated execution environment. Here is the Dockerfile. Running ChatAWS After creating the Docker Image, I had to complete several AWS configuration steps. First I created new scoped access keys for ChatGPT to use. Second I delegated a test bucket for the uploads to S3. Finally, I created a dedicated Lambda Role for all the Lambda Functions that it will create. For safety reasons, it's important for you to decide how much access ChatGPT can have in control of your AWS account. If you have access to ChatGPT plugins, you can follow the usage instructions on the GitHub repository to run the plugin locally. Example Prompts Now for the fun part: what can the plugin actually do? Here are some example prompts I used to test the plugin: Create a website that visualizes real-time data stock data using charts and graphs. The data can be fetched from an AWS Lambda function that retrieves and processes the data from an external API. ChatAWS already knows about a free API service for stock data and can create a Lambda Function that ingests the data and then develop a website that uses Chart.js to render the graph. Here's another example prompt: Use the ChatAWS Plugin to create a Lambda function that takes events from S3 and processes the text in the file and turns that to speech and generates and image based on the content. ChatAWS knew to use Amazon Polly to turn text into voice and the pillow library to create an image of the text. ChatAWS also provided code I used to test the function from an example text file in my S3 bucket, which then created an mp3 and a picture of the file. Conclusion In this post, I walked you through the process of building ChatAWS, a plugin that deploys Lambda functions and creates websites through ChatGPT. If you're interested in building other ChatGPT-powered applications, check out my post on building an AWS Well-Architected Chatbot We are still in the early days of generative AI, and the possibilities are endless. ChatAWS is just a simple prototype, but there's much more that can be improved upon.

By Banjo Obayomi

Write Your Kubernetes Infrastructure as Go Code - Manage AWS Services

AWS Controllers for Kubernetes (also known as ACK) is built around the Kubernetes extension concepts of Custom Resource and Custom Resource Definitions. You can use ACK to define and use AWS services directly from Kubernetes. This helps you take advantage of managed AWS services for your Kubernetes applications without needing to define resources outside of the cluster. Say you need to use an AWS S3 Bucket in your application that’s deployed to Kubernetes. Instead of using AWS console, AWS CLI, AWS CloudFormation etc., you can define the AWS S3 Bucket in a YAML manifest file and deploy it using familiar tools such as kubectl. The end goal is to allow users (Software Engineers, DevOps engineers, operators etc.) to use the same interface (Kubernetes API in this case) to describe and manage AWS services along with native Kubernetes resources such as Deployment, Service etc. Here is a diagram from the ACK documentation, that provides a high-level overview: Working of ACK What’s Covered in This Blog Post? ACK supports many AWS services, including Amazon DynamoDB. One of the topics that this blog post covers is how to use ACK on Amazon EKS for managing DynamoDB. But, just creating a DynamoDB table isn't going to be all that interesting! In addition to it, you will also work with and deploy a client application — this is a trimmed-down version of the URL shortener app covered in a previous blog post. While the first half of the blog will involve manual steps to help you understand the mechanics and get started, in the second half, we will switch to cdk8s and achieve the same goals using nothing but Go code. Cdk8s? What, Why? Because Infrastructure Is Code cdk8s (Cloud Development Kit for Kubernetes) is an open-source framework (part of CNCF) that allows you to define your Kubernetes applications using regular programming languages (instead of yaml). I have written a few blog posts around cdk8s and Go, that you may find useful. We will continue on the same path i.e. push yaml to the background and use the Go programming language to define the core infrastructure (that happens to be DynamoDB in this example, but could be so much more) as well as the application components (Kubernetes Deployment, Service etc.). This is made possible due to the following cdk8s features: cdk8s support for Kubernetes Custom Resource definitions that lets us magically import CRD as APIs. a cdk8s-plus library that helps reduce/eliminate a lot of boilerplate code while working with Kubernetes resources in our Go code (or any other language for that matter) Before you dive in, please ensure you complete the prerequisites in order to work through the tutorial. The entire code for the infrastructure and the application is available on GitHub Pre-requisites To follow along step-by-step, in addition to an AWS account, you will need to have AWS CLI, cdk8s CLI, kubectl, helm and the Go programming language installed. There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using eksctl CLI because of the convenience it offers! Ok, let’s get started. The first thing we need to do is… Set up the DynamoDB Controller Most of the below steps are adapted from the ACK documentation - Install an ACK Controller Install It Using Helm: Shell export SERVICE=dynamodb Change/update this as required as per the latest release document Shell export RELEASE_VERSION=0.1.7 Shell export ACK_SYSTEM_NAMESPACE=ack-system # you can change the region if required export AWS_REGION=us-east-1 aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install --create-namespace -n $ACK_SYSTEM_NAMESPACE ack-$SERVICE-controller \ oci://public.ecr.aws/aws-controllers-k8s/$SERVICE-chart --version=$RELEASE_VERSION --set=aws.region=$AWS_REGION To confirm, run: Shell kubectl get crd Shell # output (multiple CRDs) tables.dynamodb.services.k8s.aws fieldexports.services.k8s.aws globaltables.dynamodb.services.k8s.aws # etc.... Since the DynamoDB controller has to interact with AWS Services (make API calls), and we need to configure IAM Roles for Service Accounts (also known as IRSA). Refer to Configure IAM Permissions for details IRSA Configuration First, create an OIDC identity provider for your cluster. Shell export EKS_CLUSTER_NAME=<name of your EKS cluster> export AWS_REGION=<cluster region> eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --region $AWS_REGION --approve The goal is to create an IAM role and attach appropriate permissions via policies. We can then create a Kubernetes Service Account and attach the IAM role to it. Thus, the controller Pod will be able to make AWS API calls. Note that we are using providing all DynamoDB permissions to our control via the arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess policy. Thanks to eksctl, this can be done with a single line! Shell export SERVICE=dynamodb export ACK_K8S_SERVICE_ACCOUNT_NAME=ack-$SERVICE-controller Shell # recommend using the same name export ACK_SYSTEM_NAMESPACE=ack-system export EKS_CLUSTER_NAME=<enter EKS cluster name> export POLICY_ARN=arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess Shell # IAM role has a format - do not change it. you can't use any arbitrary name export IAM_ROLE_NAME=ack-$SERVICE-controller-role Shell eksctl create iamserviceaccount \ --name $ACK_K8S_SERVICE_ACCOUNT_NAME \ --namespace $ACK_SYSTEM_NAMESPACE \ --cluster $EKS_CLUSTER_NAME \ --role-name $IAM_ROLE_NAME \ --attach-policy-arn $POLICY_ARN \ --approve \ --override-existing-serviceaccounts The policy is per recommended policy To confirm, you can check whether the IAM role was created and also introspect the Kubernetes service account Shell aws iam get-role --role-name=$IAM_ROLE_NAME --query Role.Arn --output text Shell kubectl describe serviceaccount/$ACK_K8S_SERVICE_ACCOUNT_NAME -n $ACK_SYSTEM_NAMESPACE You will see similar output: Shell Name: ack-dynamodb-controller Namespace: ack-system Labels: app.kubernetes.io/instance=ack-dynamodb-controller app.kubernetes.io/managed-by=eksctl app.kubernetes.io/name=dynamodb-chart app.kubernetes.io/version=v0.1.3 helm.sh/chart=dynamodb-chart-v0.1.3 k8s-app=dynamodb-chart Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<your AWS account ID>:role/ack-dynamodb-controller-role meta.helm.sh/release-name: ack-dynamodb-controller meta.helm.sh/release-namespace: ack-system Image pull secrets: <none> Mountable secrets: ack-dynamodb-controller-token-bbzxv Tokens: ack-dynamodb-controller-token-bbzxv Events: <none> For IRSA to take effect, you need to restart the ACK Deployment: Shell # Note the deployment name for ACK service controller from following command kubectl get deployments -n $ACK_SYSTEM_NAMESPACE Shell kubectl -n $ACK_SYSTEM_NAMESPACE rollout restart deployment ack-dynamodb-controller-dynamodb-chart Confirm that the Deployment has restarted (currently Running) and the IRSA is properly configured: Shell kubectl get pods -n $ACK_SYSTEM_NAMESPACE Shell kubectl describe pod -n $ACK_SYSTEM_NAMESPACE ack-dynamodb-controller-dynamodb-chart-7dc99456c6-6shrm | grep "^\s*AWS_" # The output should contain following two lines: Shell AWS_ROLE_ARN=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME> AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token Now that we’re done with the configuration … We Can Move On to the Fun Parts! Start by Creating the DynamoDB Table Here is what the manifest looks like: YAML apiVersion: dynamodb.services.k8s.aws/v1alpha1 kind: Table metadata: name: dynamodb-urls-ack spec: tableName: urls attributeDefinitions: - attributeName: shortcode attributeType: S billingMode: PAY_PER_REQUEST keySchema: - attributeName: email keyType: HASH Clone the project, change to the correct directory and apply the manifest: Shell git clone https://github.com/abhirockzz/dynamodb-ack-cdk8s cd dynamodb-ack-cdk8s Shell # create table kubectl apply -f manual/dynamodb-ack.yaml You can check the DynamoDB table in the AWS console, or use the AWS CLI (aws dynamodb list-tables) to confirm.Our table is ready, now we can deploy our URL shortener application. But, before that, we need to create a Docker image and push it to a private repository in Amazon Elastic Container Registry (ECR). Create a Private Repository in Amazon ECR Login to ECR: Shell aws ecr get-login-password --region <enter region> | docker login --username AWS --password-stdin <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com Create Repository: Shell aws ecr create-repository \ --repository-name dynamodb-app \ --region <enter AWS region> Build the Image and Push It to ECR Shell # if you're on Mac M1 #export DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t dynamodb-app . Shell docker tag dynamodb-app:latest <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com/dynamodb-app:latest Shell docker push <enter aws_account_id>.dkr.ecr.<enter region>.amazonaws.com/dynamodb-app:latest Just like the controller, our application also needs IRSA to be able to execute GetItem and PutItem API calls on DynamoDB. Let’s Create Another IRSA for the Application Shell # you can change the policy name. make sure yo use the same name in subsequent commands aws iam create-policy --policy-name dynamodb-irsa-policy --policy-document file://manual/policy.json Shell eksctl create iamserviceaccount --name eks-dynamodb-app-sa --namespace default --cluster <enter EKS cluster name> --attach-policy-arn arn:aws:iam::<enter AWS account ID>:policy/dynamodb-irsa-policy --approve Shell kubectl describe serviceaccount/eks-dynamodb-app-sa Output: Shell Name: eks-dynamodb-app-sa Namespace: default Labels: app.kubernetes.io/managed-by=eksctl Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<AWS account ID>:role/eksctl-eks-cluster-addon-iamserviceaccount-d-Role1-2KTGZO1GJRN Image pull secrets: <none> Mountable secrets: eks-dynamodb-app-sa-token-5fcvf Tokens: eks-dynamodb-app-sa-token-5fcvf Events: <none> Finally, we can deploy our application! In the manual/app.yaml file, make sure you replace the following attributes as per your environment: Service account name — the above example used eks-dynamodb-app-sa Docker image Container environment variables for AWS region (e.g. us-east-1) and table name (this will be urls since that's the name we used) Shell kubectl apply -f app.yaml Shell # output deployment.apps/dynamodb-app configured service/dynamodb-app-service created This will create a Deployment as well as Service to access the application. Since the Service type is LoadBalancer, an appropriate AWS Load Balancer will be provisioned to allow for external access. Check Pod and Service: Shell kubectl get pods kubectl get service/dynamodb-app-service Shell # to get the load balancer IP APP_URL=$(kubectl get service/dynamodb-app-service -o jsonpath="{.status.loadBalancer.ingress[0].hostname}") echo $APP_URL Shell # output example a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com You have deployed the application and know the endpoint over which it’s publicly accessible. You can try out the URL shortener service Shell curl -i -X POST -d 'https://abhirockzz.github.io/' $APP_URL:9090/ Shell # output HTTP/1.1 200 OK Date: Thu, 21 Jul 2022 11:03:40 GMT Content-Length: 25 Content-Type: text/plain; charset=utf-8 JSON {"ShortCode":"ae1e31a6"} If you get a Could not resolve host error while accessing the LB URL, wait for a minute or so and re-try You should receive a JSON response with a short code. Enter the following in your browser http://<enter APP_URL>:9090/<shortcode> e.g. http://a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com:9090/ae1e31a6 - you will be redirected to the original URL. You can also use curl: Shell # example curl -i http://a0042d5b5b0ad40abba9c6c42e6342a2-879424587.us-east-1.elb.amazonaws.com:9090/ae1e31a6 # output HTTP/1.1 302 Found Location: https://abhirockzz.github.io/ Date: Thu, 21 Jul 2022 11:07:58 GMT Content-Length: 0 Enough of YAML I guess! As I promised earlier, the second half will demonstrate how to achieve the same using cdk8s and Go. Kubernetes Infrastructure as Go Code With Cdk8s Assuming you’ve already cloned the project (as per the above instructions), change to a different directory: Shell cd dynamodb-cdk8s This is a pre-created cdk8s project that you can use. The entire logic is present in main.go file. We will first try it out and confirm that it works the same way. After that, we will dive into the nitty-gritty of the code. Delete the previously created DynamoDB table and along with the Deployment (and Service) as well: Shell # you can also delete the table directly from AWS console aws dynamodb delete-table --table-name urls # this will delete Deployment and Service (as well as AWS Load Balancer) kubectl delete -f manual/app.yaml Use cdk8s synth to generate the manifest for the DynamoDB table and the application. We can then apply it using kubectl See the difference? Earlier, we defined the DynamoDB table, Deployment (and Service) manifests manually. cdk8s does not remove YAML altogether, but it provides a way for us to leverage regular programming languages (Go in this case) to define the components of our solution. Shell export TABLE_NAME=urls export SERVICE_ACCOUNT=eks-dynamodb-app-sa export DOCKER_IMAGE=<enter ECR repo that you created earlier> cdk8s synth ls -lrt dist/ #output 0000-dynamodb.k8s.yaml 0001-deployment.k8s.yaml You will see two different manifests being generated by cdk8s since we defined two separate cdk8s.Charts in the code - more on this soon. We can deploy them one by one: Shell kubectl apply -f dist/0000-dynamodb.k8s.yaml #output table.dynamodb.services.k8s.aws/dynamodb-dynamodb-ack-cdk8s-table-c88d874d created configmap/export-dynamodb-urls-info created fieldexport.services.k8s.aws/export-dynamodb-tablename created fieldexport.services.k8s.aws/export-dynamodb-region created As always, you can check the DynamoDB table either in the console or AWS CLI - aws dynamodb describe-table --table-name urls. Looking at the output, the DynamoDB table part seems familiar... But what’s fieldexport.services.k8s.aws?? … And why do we need a ConfigMap? I will give you the gist here. In the previous iteration, we hard-coded the table name and region in manual/app.yaml. While this works for this sample app, it is not scalable and might not even work for a few resources in case the metadata (like name etc.) is randomly generated. That's why there is this concept of a FieldExport that can "export any spec or status field from an ACK resource into a Kubernetes ConfigMap or Secret" You can read up on the details in the ACK documentation along with some examples. What you will see here is how to define a FieldExport and ConfigMap along with the Deployment which needs to be configured to accept its environment variables from the ConfigMap - all this in code, using Go (more on this during code walk-through). Check the FieldExport and ConfigMap: Shell kubectl get fieldexport #output NAME AGE export-dynamodb-region 19s export-dynamodb-tablename 19s kubectl get configmap/export-dynamodb-urls-info -o yaml We started out with a blank ConfigMap (as per cdk8s logic), but ACK magically populated it with the attributes from the Table custom resource. YAML apiVersion: v1 data: default.export-dynamodb-region: us-east-1 default.export-dynamodb-tablename: urls immutable: false kind: ConfigMap #....omitted We can now use the second manifest — no surprises here. Just like in the previous iteration, all it contains is the application Deployment and the Service. Check Pod and Service: Shell kubectl apply -f dist/0001-deployment.k8s.yaml #output deployment.apps/dynamodb-app created service/dynamodb-app-service configured kubectl get pods kubectl get svc The entire setup is ready, just like it was earlier and you can test it the same way. I will not repeat the steps here. Instead, I will move to something more interesting. Cdk8s Code Walk-Through The logic is divided into two Charts. I will only focus on key sections of the code and the rest will be omitted for brevity. DynamoDB and Configuration We start by defining the DynamoDB table (name it urls) as well as the ConfigMap (note that it does not have any data at this point): Go func NewDynamoDBChart(scope constructs.Construct, id string, props *MyChartProps) cdk8s.Chart { //... table := ddbcrd.NewTable(chart, jsii.String("dynamodb-ack-cdk8s-table"), &ddbcrd.TableProps{ Spec: &ddbcrd.TableSpec{ AttributeDefinitions: &[]*ddbcrd.TableSpecAttributeDefinitions{ {AttributeName: jsii.String(primaryKeyName), AttributeType: jsii.String("S")}, BillingMode: jsii.String(billingMode), TableName: jsii.String(tableName), KeySchema: &[]*ddbcrd.TableSpecKeySchema{ {AttributeName: jsii.String(primaryKeyName), KeyType: jsii.String(hashKeyType)}}) //... cfgMap = cdk8splus22.NewConfigMap(chart, jsii.String("config-map"), &cdk8splus22.ConfigMapProps{ Metadata: &cdk8s.ApiObjectMetadata{ Name: jsii.String(configMapName)}) Then we move on to the FieldExports - one each for the AWS region and the table name. As soon as these are created, the ConfigMap is populated with the required data as per from and to configuration in the FieldExport. Go //... fieldExportForTable = servicesk8saws.NewFieldExport(chart, jsii.String("fexp-table"), &servicesk8saws.FieldExportProps{ Metadata: &cdk8s.ApiObjectMetadata{Name: jsii.String(fieldExportNameForTable)}, Spec: &servicesk8saws.FieldExportSpec{ From: &servicesk8saws.FieldExportSpecFrom{Path: jsii.String(".spec.tableName"), Resource: &servicesk8saws.FieldExportSpecFromResource{ Group: jsii.String("dynamodb.services.k8s.aws"), Kind: jsii.String("Table"), Name: table.Name()}, To: &servicesk8saws.FieldExportSpecTo{ Name: cfgMap.Name(), Kind: servicesk8saws.FieldExportSpecToKind_CONFIGMAP}}) Go fieldExportForRegion = servicesk8saws.NewFieldExport(chart, jsii.String("fexp-region"), &servicesk8saws.FieldExportProps{ Metadata: &cdk8s.ApiObjectMetadata{Name: jsii.String(fieldExportNameForRegion)}, Spec: &servicesk8saws.FieldExportSpec{ From: &servicesk8saws.FieldExportSpecFrom{ Path: jsii.String(".status.ackResourceMetadata.region"), Resource: &servicesk8saws.FieldExportSpecFromResource{ Group: jsii.String("dynamodb.services.k8s.aws"), Kind: jsii.String("Table"), Name: table.Name()}, To: &servicesk8saws.FieldExportSpecTo{ Name: cfgMap.Name(), Kind: servicesk8saws.FieldExportSpecToKind_CONFIGMAP}}) //... The Application Chart The core of our application is Deployment itself: Go func NewDeploymentChart(scope constructs.Construct, id string, props *MyChartProps) cdk8s.Chart { //... dep := cdk8splus22.NewDeployment(chart, jsii.String("dynamodb-app-deployment"), &cdk8splus22.DeploymentProps{ Metadata: &cdk8s.ApiObjectMetadata{ Name: jsii.String("dynamodb-app")}, ServiceAccount: cdk8splus22.ServiceAccount_FromServiceAccountName( chart, jsii.String("aws-irsa"), jsii.String(serviceAccountName))}) The next important part is the container and its configuration. We specify the ECR image repository along with the environment variables — they reference the ConfigMap we defined in the previous chart (everything is connected!): Go //... container := dep.AddContainer( &cdk8splus22.ContainerProps{ Name: jsii.String("dynamodb-app-container"), Image: jsii.String(image), Port: jsii.Number(appPort)}) Go container.Env().AddVariable(jsii.String("TABLE_NAME"), cdk8splus22.EnvValue_FromConfigMap( cfgMap, jsii.String("default."+*fieldExportForTable.Name()), &cdk8splus22.EnvValueFromConfigMapOptions{Optional: jsii.Bool(false)})) Go container.Env().AddVariable(jsii.String("AWS_REGION"), cdk8splus22.EnvValue_FromConfigMap( cfgMap, jsii.String("default."+*fieldExportForRegion.Name()), &cdk8splus22.EnvValueFromConfigMapOptions{Optional: jsii.Bool(false)})) Finally, we define the Service (type LoadBalancer) which enables external application access and ties it all together in the main function: Go //... dep.ExposeViaService( &cdk8splus22.DeploymentExposeViaServiceOptions{ Name: jsii.String("dynamodb-app-service"), ServiceType: cdk8splus22.ServiceType_LOAD_BALANCER, Ports: &[]*cdk8splus22.ServicePort{ {Protocol: cdk8splus22.Protocol_TCP, Port: jsii.Number(lbPort), TargetPort: jsii.Number(appPort)}}) //... Go func main() { app := cdk8s.NewApp(nil) dynamodDB := NewDynamoDBChart(app, "dynamodb", nil) deployment := NewDeploymentChart(app, "deployment", nil) deployment.AddDependency(dynamodDB) app.Synth() } That’s all I have for you in this blog! Don’t Forget To Delete Resources... Shell # delete DynamoDB table, Deployment, Service etc. kubectl delete -f dist/ # to uninstall the ACK controller export SERVICE=dynamodb helm uninstall -n $ACK_SYSTEM_NAMESPACE ack-$SERVICE-controller # delete the EKS cluster. if created via eksctl: eksctl delete cluster --name <enter name of eks cluster> Wrap Up... AWS Controllers for Kubernetes help bridge the gap between traditional Kubernetes resources and AWS services by allowing you to manage both from a single control plane. In this blog, you saw how to do this in the context of DynamoDB and a URL shortener application (deployed to Kubernetes). I encourage you to try out other AWS services that ACK supports - here is a complete list. The approach presented here will work well if just want to use cdk8s. However, depending on your requirements, there is another way this can do by bringing AWS CDK into the picture. I want to pause right here since this is something I might cover in a future blog post. Until then, Happy Building!

By Abhishek Gupta CORE

AWS Cloud

DZone's Featured AWS Cloud Resources

The Latest AWS Cloud Topics