Software Design and Architecture

In this blog post, you will be using AWS Controllers for Kubernetes on an Amazon EKS cluster to put together a solution wherein data from an Amazon SQS queue is processed by an AWS Lambda function and persisted to a DynamoDB table. AWS Controllers for Kubernetes (also known as ACK) leverage Kubernetes Custom Resource and Custom Resource Definitions and give you the ability to manage and use AWS services directly from Kubernetes without needing to define resources outside of the cluster. The idea behind ACK is to enable Kubernetes users to describe the desired state of AWS resources using the Kubernetes API and configuration language. ACK will then take care of provisioning and managing the AWS resources to match the desired state. This is achieved by using Service controllers that are responsible for managing the lifecycle of a particular AWS service. Each ACK service controller is packaged into a separate container image that is published in a public repository corresponding to an individual ACK service controller. There is no single ACK container image. Instead, there are container images for each individual ACK service controller that manages resources for a particular AWS API. This blog post will walk you through how to use the SQS, DynamoDB, and Lambda service controllers for ACK. Prerequisites To follow along step-by-step, in addition to an AWS account, you will need to have AWS CLI, kubectl, and Helm installed. There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using eksctl CLI because of the convenience it offers. Creating an EKS cluster using eksctl can be as easy as this: eksctl create cluster --name my-cluster --region region-code For details, refer to Getting started with Amazon EKS – eksctl. Clone this GitHub repository and change it to the right directory: git clone https://github.com/abhirockzz/k8s-ack-sqs-lambda cd k8s-ack-sqs-lambda Ok, let's get started! Setup the ACK Service Controllers for AWS Lambda, SQS, and DynamoDB Install ACK Controllers Log into the Helm registry that stores the ACK charts: aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws Deploy the ACK service controller for Amazon Lambda using the lambda-chart Helm chart: RELEASE_VERSION_LAMBDA_ACK=$(curl -sL "https://api.github.com/repos/aws-controllers-k8s/lambda-controller/releases/latest" | grep '"tag_name":' | cut -d'"' -f4) helm install --create-namespace -n ack-system oci://public.ecr.aws/aws-controllers-k8s/lambda-chart "--version=${RELEASE_VERSION_LAMBDA_ACK}" --generate-name --set=aws.region=us-east-1 Deploy the ACK service controller for SQS using the sqs-chart Helm chart: RELEASE_VERSION_SQS_ACK=$(curl -sL "https://api.github.com/repos/aws-controllers-k8s/sqs-controller/releases/latest" | grep '"tag_name":' | cut -d'"' -f4) helm install --create-namespace -n ack-system oci://public.ecr.aws/aws-controllers-k8s/sqs-chart "--version=${RELEASE_VERSION_SQS_ACK}" --generate-name --set=aws.region=us-east-1 Deploy the ACK service controller for DynamoDB using the dynamodb-chart Helm chart: RELEASE_VERSION_DYNAMODB_ACK=$(curl -sL "https://api.github.com/repos/aws-controllers-k8s/dynamodb-controller/releases/latest" | grep '"tag_name":' | cut -d'"' -f4) helm install --create-namespace -n ack-system oci://public.ecr.aws/aws-controllers-k8s/dynamodb-chart "--version=${RELEASE_VERSION_DYNAMODB_ACK}" --generate-name --set=aws.region=us-east-1 Now, it's time to configure the IAM permissions for the controller to invoke Lambda, DynamoDB, and SQS. Configure IAM Permissions Create an OIDC Identity Provider for Your Cluster For the steps below, replace the EKS_CLUSTER_NAME and AWS_REGION variables with your cluster name and region. export EKS_CLUSTER_NAME=demo-eks-cluster export AWS_REGION=us-east-1 eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --region $AWS_REGION --approve OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f2- | cut -d '/' -f2-) Create IAM Roles for Lambda, SQS, and DynamoDB ACK Service Controllers ACK Lambda Controller Set the following environment variables: ACK_K8S_SERVICE_ACCOUNT_NAME=ack-lambda-controller ACK_K8S_NAMESPACE=ack-system AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) Create the trust policy for the IAM role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:sub": "system:serviceaccount:${ACK_K8S_NAMESPACE}:${ACK_K8S_SERVICE_ACCOUNT_NAME}" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust_lambda.json Create the IAM role: ACK_CONTROLLER_IAM_ROLE="ack-lambda-controller" ACK_CONTROLLER_IAM_ROLE_DESCRIPTION="IRSA role for ACK lambda controller deployment on EKS cluster using Helm charts" aws iam create-role --role-name "${ACK_CONTROLLER_IAM_ROLE}" --assume-role-policy-document file://trust_lambda.json --description "${ACK_CONTROLLER_IAM_ROLE_DESCRIPTION}" Attach IAM policy to the IAM role: # we are getting the policy directly from the ACK repo INLINE_POLICY="$(curl https://raw.githubusercontent.com/aws-controllers-k8s/lambda-controller/main/config/iam/recommended-inline-policy)" aws iam put-role-policy \ --role-name "${ACK_CONTROLLER_IAM_ROLE}" \ --policy-name "ack-recommended-policy" \ --policy-document "${INLINE_POLICY}" Attach ECR permissions to the controller IAM role. These are required since Lambda functions will be pulling images from ECR. aws iam put-role-policy \ --role-name "${ACK_CONTROLLER_IAM_ROLE}" \ --policy-name "ecr-permissions" \ --policy-document file://ecr-permissions.json Associate the IAM role to a Kubernetes service account: ACK_CONTROLLER_IAM_ROLE_ARN=$(aws iam get-role --role-name=$ACK_CONTROLLER_IAM_ROLE --query Role.Arn --output text) export IRSA_ROLE_ARN=eks.amazonaws.com/role-arn=$ACK_CONTROLLER_IAM_ROLE_ARN kubectl annotate serviceaccount -n $ACK_K8S_NAMESPACE $ACK_K8S_SERVICE_ACCOUNT_NAME $IRSA_ROLE_ARN Repeat the steps for the SQS controller. ACK SQS Controller Set the following environment variables: ACK_K8S_SERVICE_ACCOUNT_NAME=ack-sqs-controller ACK_K8S_NAMESPACE=ack-system AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) Create the trust policy for the IAM role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:sub": "system:serviceaccount:${ACK_K8S_NAMESPACE}:${ACK_K8S_SERVICE_ACCOUNT_NAME}" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust_sqs.json Create the IAM role: ACK_CONTROLLER_IAM_ROLE="ack-sqs-controller" ACK_CONTROLLER_IAM_ROLE_DESCRIPTION="IRSA role for ACK sqs controller deployment on EKS cluster using Helm charts" aws iam create-role --role-name "${ACK_CONTROLLER_IAM_ROLE}" --assume-role-policy-document file://trust_sqs.json --description "${ACK_CONTROLLER_IAM_ROLE_DESCRIPTION}" Attach IAM policy to the IAM role: # for sqs controller, we use the managed policy ARN instead of the inline policy (unlike the Lambda controller) POLICY_ARN="$(curl https://raw.githubusercontent.com/aws-controllers-k8s/sqs-controller/main/config/iam/recommended-policy-arn)" aws iam attach-role-policy --role-name "${ACK_CONTROLLER_IAM_ROLE}" --policy-arn "${POLICY_ARN}" Associate the IAM role to a Kubernetes service account: ACK_CONTROLLER_IAM_ROLE_ARN=$(aws iam get-role --role-name=$ACK_CONTROLLER_IAM_ROLE --query Role.Arn --output text) export IRSA_ROLE_ARN=eks.amazonaws.com/role-arn=$ACK_CONTROLLER_IAM_ROLE_ARN kubectl annotate serviceaccount -n $ACK_K8S_NAMESPACE $ACK_K8S_SERVICE_ACCOUNT_NAME $IRSA_ROLE_ARN Repeat the steps for the DynamoDB controller. ACK DynamoDB Controller Set the following environment variables: ACK_K8S_SERVICE_ACCOUNT_NAME=ack-dynamodb-controller ACK_K8S_NAMESPACE=ack-system AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) Create the trust policy for the IAM role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:sub": "system:serviceaccount:${ACK_K8S_NAMESPACE}:${ACK_K8S_SERVICE_ACCOUNT_NAME}" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust_dynamodb.json Create the IAM role: ACK_CONTROLLER_IAM_ROLE="ack-dynamodb-controller" ACK_CONTROLLER_IAM_ROLE_DESCRIPTION="IRSA role for ACK dynamodb controller deployment on EKS cluster using Helm charts" aws iam create-role --role-name "${ACK_CONTROLLER_IAM_ROLE}" --assume-role-policy-document file://trust_dynamodb.json --description "${ACK_CONTROLLER_IAM_ROLE_DESCRIPTION}" Attach IAM policy to the IAM role: # for dynamodb controller, we use the managed policy ARN instead of the inline policy (like we did for Lambda controller) POLICY_ARN="$(curl https://raw.githubusercontent.com/aws-controllers-k8s/dynamodb-controller/main/config/iam/recommended-policy-arn)" aws iam attach-role-policy --role-name "${ACK_CONTROLLER_IAM_ROLE}" --policy-arn "${POLICY_ARN}" Associate the IAM role to a Kubernetes service account: ACK_CONTROLLER_IAM_ROLE_ARN=$(aws iam get-role --role-name=$ACK_CONTROLLER_IAM_ROLE --query Role.Arn --output text) export IRSA_ROLE_ARN=eks.amazonaws.com/role-arn=$ACK_CONTROLLER_IAM_ROLE_ARN kubectl annotate serviceaccount -n $ACK_K8S_NAMESPACE $ACK_K8S_SERVICE_ACCOUNT_NAME $IRSA_ROLE_ARN Restart ACK Controller Deployments and Verify the Setup Restart the ACK service controller Deployment using the following commands. It will update service controller Pods with IRSA environment variables. Get list of ACK service controller deployments: export ACK_K8S_NAMESPACE=ack-system kubectl get deployments -n $ACK_K8S_NAMESPACE Restart Lambda, SQS, and DynamoDB controller Deployments: DEPLOYMENT_NAME_LAMBDA=<enter deployment name for lambda controller> kubectl -n $ACK_K8S_NAMESPACE rollout restart deployment $DEPLOYMENT_NAME_LAMBDA DEPLOYMENT_NAME_SQS=<enter deployment name for sqs controller> kubectl -n $ACK_K8S_NAMESPACE rollout restart deployment $DEPLOYMENT_NAME_SQS DEPLOYMENT_NAME_DYNAMODB=<enter deployment name for dynamodb controller> kubectl -n $ACK_K8S_NAMESPACE rollout restart deployment $DEPLOYMENT_NAME_DYNAMODB List Pods for these Deployments. Verify that the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables exist for your Kubernetes Pod using the following commands: kubectl get pods -n $ACK_K8S_NAMESPACE LAMBDA_POD_NAME=<enter Pod name for lambda controller> kubectl describe pod -n $ACK_K8S_NAMESPACE $LAMBDA_POD_NAME | grep "^\s*AWS_" SQS_POD_NAME=<enter Pod name for sqs controller> kubectl describe pod -n $ACK_K8S_NAMESPACE $SQS_POD_NAME | grep "^\s*AWS_" DYNAMODB_POD_NAME=<enter Pod name for dynamodb controller> kubectl describe pod -n $ACK_K8S_NAMESPACE $DYNAMODB_POD_NAME | grep "^\s*AWS_" Now that the ACK service controller has been set up and configured, you can create AWS resources! Create SQS Queue, DynamoDB Table, and Deploy the Lambda Function Create SQS Queue In the file sqs-queue.yaml, replace the us-east-1 region with your preferred region as well as the AWS account ID. This is what the ACK manifest for the SQS queue looks like: apiVersion: sqs.services.k8s.aws/v1alpha1 kind: Queue metadata: name: sqs-queue-demo-ack annotations: services.k8s.aws/region: us-east-1 spec: queueName: sqs-queue-demo-ack policy: | { "Statement": [{ "Sid": "__owner_statement", "Effect": "Allow", "Principal": { "AWS": "AWS_ACCOUNT_ID" }, "Action": "sqs:SendMessage", "Resource": "arn:aws:sqs:us-east-1:AWS_ACCOUNT_ID:sqs-queue-demo-ack" }] } Create the queue using the following command: kubectl apply -f sqs-queue.yaml # list the queue kubectl get queue Create DynamoDB Table This is what the ACK manifest for the DynamoDB table looks like: apiVersion: dynamodb.services.k8s.aws/v1alpha1 kind: Table metadata: name: customer annotations: services.k8s.aws/region: us-east-1 spec: attributeDefinitions: - attributeName: email attributeType: S billingMode: PAY_PER_REQUEST keySchema: - attributeName: email keyType: HASH tableName: customer You can replace the us-east-1 region with your preferred region. Create a table (named customer) using the following command: kubectl apply -f dynamodb-table.yaml # list the tables kubectl get tables Build Function Binary and Create Docker Image GOARCH=amd64 GOOS=linux go build -o main main.go aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws docker build -t demo-sqs-dynamodb-func-ack . Create a private ECR repository, tag and push the Docker image to ECR: AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com aws ecr create-repository --repository-name demo-sqs-dynamodb-func-ack --region us-east-1 docker tag demo-sqs-dynamodb-func-ack:latest $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/demo-sqs-dynamodb-func-ack:latest docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/demo-sqs-dynamodb-func-ack:latest Create an IAM execution Role for the Lambda function and attach the required policies: export ROLE_NAME=demo-sqs-dynamodb-func-ack-role ROLE_ARN=$(aws iam create-role \ --role-name $ROLE_NAME \ --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}' \ --query 'Role.[Arn]' --output text) aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Since the Lambda function needs to write data to DynamoDB and invoke SQS, let's add the following policies to the IAM role: aws iam put-role-policy \ --role-name "${ROLE_NAME}" \ --policy-name "dynamodb-put" \ --policy-document file://dynamodb-put.json aws iam put-role-policy \ --role-name "${ROLE_NAME}" \ --policy-name "sqs-permissions" \ --policy-document file://sqs-permissions.json Create the Lambda Function Update function.yaml file with the following info: imageURI - The URI of the Docker image that you pushed to ECR, e.g., <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/demo-sqs-dynamodb-func-ack:latest role - The ARN of the IAM role that you created for the Lambda function, e.g., arn:aws:iam::<AWS_ACCOUNT_ID>:role/demo-sqs-dynamodb-func-ack-role This is what the ACK manifest for the Lambda function looks like: apiVersion: lambda.services.k8s.aws/v1alpha1 kind: Function metadata: name: demo-sqs-dynamodb-func-ack annotations: services.k8s.aws/region: us-east-1 spec: architectures: - x86_64 name: demo-sqs-dynamodb-func-ack packageType: Image code: imageURI: AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/demo-sqs-dynamodb-func-ack:latest environment: variables: TABLE_NAME: customer role: arn:aws:iam::AWS_ACCOUNT_ID:role/demo-sqs-dynamodb-func-ack-role description: A function created by ACK lambda-controller To create the Lambda function, run the following command: kubectl create -f function.yaml # list the function kubectl get functions Add SQS Trigger Configuration Add SQS trigger which will invoke the Lambda function when an event is sent to the SQS queue. Here is an example using AWS Console: Open the Lambda function in the AWS Console and click on the Add trigger button. Select SQS as the trigger source, select the SQS queue, and click on the Add button. Now you are ready to try out the end-to-end solution! Test the Application Send a few messages to the SQS queue. For the purposes of this demo, you can use the AWS CLI: export SQS_QUEUE_URL=$(kubectl get queues/sqs-queue-demo-ack -o jsonpath='{.status.queueURL}') aws sqs send-message --queue-url $SQS_QUEUE_URL --message-body user1@foo.com --message-attributes 'name={DataType=String, StringValue="user1"}, city={DataType=String,StringValue="seattle"}' aws sqs send-message --queue-url $SQS_QUEUE_URL --message-body user2@foo.com --message-attributes 'name={DataType=String, StringValue="user2"}, city={DataType=String,StringValue="tel aviv"}' aws sqs send-message --queue-url $SQS_QUEUE_URL --message-body user3@foo.com --message-attributes 'name={DataType=String, StringValue="user3"}, city={DataType=String,StringValue="new delhi"}' aws sqs send-message --queue-url $SQS_QUEUE_URL --message-body user4@foo.com --message-attributes 'name={DataType=String, StringValue="user4"}, city={DataType=String,StringValue="new york"}' The Lambda function should be invoked and the data should be written to the DynamoDB table. Check the DynamoDB table using the CLI (or AWS console): aws dynamodb scan --table-name customer Clean Up After you have explored the solution, you can clean up the resources by running the following commands: Delete SQS queue, DynamoDB table and the Lambda function: kubectl delete -f sqs-queue.yaml kubectl delete -f function.yaml kubectl delete -f dynamodb-table.yaml To uninstall the ACK service controllers, run the following commands: export ACK_SYSTEM_NAMESPACE=ack-system helm ls -n $ACK_SYSTEM_NAMESPACE helm uninstall -n $ACK_SYSTEM_NAMESPACE <enter name of the sqs chart> helm uninstall -n $ACK_SYSTEM_NAMESPACE <enter name of the lambda chart> helm uninstall -n $ACK_SYSTEM_NAMESPACE <enter name of the dynamodb chart> Conclusion and Next Steps In this post, we have seen how to use AWS Controllers for Kubernetes to create a Lambda function, SQS, and DynamoDB table and wire them together to deploy a solution. All of this (almost) was done using Kubernetes! I encourage you to try out other AWS services supported by ACK. Here is a complete list. Happy building!

Security in one's information system has always been among the most critical non-functional requirements. Transport Layer Security, aka TLS and formerly SSL, is among its many pillars. In this post, I'll show how to configure TLS for the Apache APISIX API Gateway. TLS in a Few Words TLS offers several capabilities: Server authentication: The client is confident that the server it exchanges data with is the right one. It avoids sending data, which might be confidential, to the wrong actor. Optional client authentication: The other way around: the server only allows clients whose identities can be verified. Confidentiality: No third party can read the data exchanged between the client and the server. Integrity: No third party can tamper with the data. TLS works through certificates. A certificate is similar to an ID, proving the certificate's holder identity. Just like an ID, you need to trust who delivered it. Trust is established through a chain: if I trust Alice, who trusts Bob, who in turn trusts Charlie, who delivered the certificate, then I trust the latter. In this scenario, Alice is known as the root certificate authority. TLS authentication is based on public key cryptography. Alice generates a public key/private key pair and publishes the public key. If one encrypts data with the public key, only the private key that generated the public key can decrypt them. The other usage is for one to encrypt data with the private key and everybody with the public key to decrypt it, thus proving their identity. Finally, mutual TLS, aka mTLS, is the configuration of two-way TLS: server authentication to the client, as usual, but also the other way around, client authentication to the server. We now have enough understanding of the concepts to get our hands dirty. Generating Certificates With cert-manager A couple of root CA are installed in browsers by default. That's how we can browse HTTPS websites safely, trusting that is the site they pretend to be. The infrastructure has no pre-installed certificates, so we must start from scratch. We need at least one root certificate. In turn, it will generate all other certificates. While it's possible to do every manually, I'll rely on cert-manager in Kubernetes. As its name implies, cert-manager is a solution to manage certificates. Installing it with Helm is straightforward: Shell helm repo add jetstack https://charts.jetstack.io #1 helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ #2 --create-namespace \ #2 --version v1.11.0 \ --set installCRDs=true \ --set prometheus.enabled=false #3 Add the charts' repository. Install the objects in a dedicated namespace. Don't monitor; in the scope of this post. We can make sure that everything works as expected by looking at the pods: Shell kubectl get pods -n cert-manager Plain Text cert-manager-cainjector-7f694c4c58-fc9bk 1/1 Running 2 (2d1h ago) 7d cert-manager-cc4b776cf-8p2t8 1/1 Running 1 (2d1h ago) 7d cert-manager-webhook-7cd8c769bb-494tl 1/1 Running 1 (2d1h ago) 7d cert-manager can sign certificates from multiple sources: HashiCorp Vault, Let's Encrypt, etc. To keep things simple: We will generate our dedicated root certificate, i.e., Self-Signed. We won't handle certificates rotation. Let's start with the following: YAML apiVersion: cert-manager.io/v1 kind: ClusterIssuer #1 metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: v1 kind: Namespace metadata: name: tls #2 --- apiVersion: cert-manager.io/v1 kind: Certificate #3 metadata: name: selfsigned-ca namespace: tls spec: isCA: true commonName: selfsigned-ca secretName: root-secret issuerRef: name: selfsigned-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer #4 metadata: name: ca-issuer namespace: tls spec: ca: secretName: root-secret Certificate authority that generates certificates cluster-wide Create a namespace for our demo. Namespaced root certificate using the cluster-wide issuer. Only used to create a namespaced issuer. Namespaced issuer: Used to create all other certificates in the post. After applying the previous manifest, we should be able to see the single certificate that we created: Shell kubectl get certificate -n tls Plain Text NAME READY SECRET AGE selfsigned-ca True root-secret 7s The certificate infrastructure is ready; let's look at Apache APISIX. Quick Overview of a Sample Apache APISIX Architecture Apache APISIX is an API Gateway. By default, it stores its configuration in etcd, a distributed key-value store - the same one used by Kubernetes. Note that in real-world scenarios, we should set up etcd clustering to improve the resiliency of the solution. For this post, we will limit ourselves to a single etcd instance. Apache APISIX offers an admin API via HTTP endpoints. Finally, the gateway forwards calls from the client to an upstream. Here's an overview of the architecture and the required certificates: Let's start with the foundational bricks: etcd and Apache APISIX. We need two certificates: one for etcd, in the server role, and one for Apache APISIX, as the etcd client. Let's set up certificates from our namespaced issuer: YAML apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: etcd-server #1 namespace: tls spec: secretName: etcd-secret #2 isCA: false usages: - client auth #3 - server auth #3 dnsNames: - etcd #4 issuerRef: name: ca-issuer #5 kind: Issuer --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: apisix-client #6 namespace: tls spec: secretName: apisix-client-secret isCA: false usages: - client auth emailAddresses: - apisix@apache.org #7 issuerRef: name: ca-issuer #5 kind: Issuer Certificate for etcd Kubernetes Secret name, see below Usages for this certificate Kubernetes Service name, see below Reference the previously namespaced issuer created earlier Certificate for Apache APISIX as a client of etcd Mandatory attribute for clients After applying the above manifest, we can list the certificates in the tls namespace: Shell kubectl get certificates -n tls Plain Text NAME READY SECRET AGE selfsigned-ca True root-secret 8m59s //1 apisix-client True apisix-client-secret 8m22s //2 etcd-server True etcd-secret 8m54s //2 Previously created certificate Newly-created certificates signed by selfsigned-ca cert-manager's Certificates So far, we have created Certificate objects, but we didn't explain what they are. Indeed, they are simple Kubernetes CRDs provided by cert-manager. Under the cover, cert-manager creates a Kubernetes Secret from a Certificate. It manages the whole lifecycle, so deleting a Certificate deletes the bounded Secret. The secretName attribute in the above manifest sets the Secret name. Shell kubectl get secrets -n tls Plain Text NAME TYPE DATA AGE apisix-client-secret kubernetes.io/tls 3 35m etcd-secret kubernetes.io/tls 3 35m root-secret kubernetes.io/tls 3 35m Let's look at a Secret, e.g., apisix-client-secret: Shell kubectl describe apisix-client-secret -n tls Plain Text Name: apisix-client-secret Namespace: tls Labels: controller.cert-manager.io/fao=true Annotations: cert-manager.io/alt-names: cert-manager.io/certificate-name: apisix-client cert-manager.io/common-name: cert-manager.io/ip-sans: cert-manager.io/issuer-group: cert-manager.io/issuer-kind: Issuer cert-manager.io/issuer-name: ca-issuer cert-manager.io/uri-sans: Type: kubernetes.io/tls Data ==== ca.crt: 1099 bytes tls.crt: 1115 bytes tls.key: 1679 bytes A Secret created by a Certificate provides three attributes: tls.crt: The certificate itself tls.key: The private key ca.crt: The signing certificate in the certificate chain, i.e., root-secret/tls.crt Kubernetes encodes Secret content in base 64. To get any of the above in plain text, one should decode it, e.g.: Shell kubectl get secret etcd-secret -n tls -o jsonpath='{ .data.tls\.crt }' | base64 Plain Text -----BEGIN CERTIFICATE----- MIIDBjCCAe6gAwIBAgIQM3JUR8+R0vuUndjGK/aOgzANBgkqhkiG9w0BAQsFADAY MRYwFAYDVQQDEw1zZWxmc2lnbmVkLWNhMB4XDTIzMDMxNjEwMTYyN1oXDTIzMDYx NDEwMTYyN1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQpMj/0 giDVOjOosSRRKUwTzl1Wo2R9YYAeteOW3fuMiAd+XaBGmRO/+GWZQN1tyRQ3pITM ezBgogYAUUNcuqN/UAsgH/JM58niMjZdjRKn4+it94Nj1e24jFL4ts2snCn7FfKJ 3zRtY9tyS7Agw3tCwtXV68Xpmf3CsfhPmn3rGdWHXyYctzAZhqYfEswN3hxpJZxR YVeb55WgDoPo5npZo3+yYiMtoOimIprcmZ2Ye8Wai9S4QKDafUWlvU5GQ65VVLzH PEdOMwbWcwiLqwUv889TiKiC5cyAD6wJOuPRF0KKxxFnG+lHlg9J2S1i5sC3pqoc i0pEQ+atOOyLMMECAwEAAaNkMGIwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAU2ZaAdEficKUWPFRjdsKSEX/l gbMwEgYDVR0RAQH/BAgwBoIEZXRjZDANBgkqhkiG9w0BAQsFAAOCAQEABcNvYTm8 ZJe3jUq6f872dpNVulb2UvloTpWxQ8jwXgcrhekSKU6pZ4p9IPwfauHLjceMFJLp t2eDi5fSQ1upeqXOofeyKSYjjyA/aVf1zMI8ReCCQtQuAVYyJWBlNLc3XMMecbcp JLGtd/OAZnKDeYYkUX7cJ2wN6Wl/wGLM2lxsqDhEHEZwvGL0DmsdHw7hzSjdVmxs 0Qgkh4jVbNUKdBok5U9Ivr3P1xDPaD/FqGFyM0ssVOCHxtPxhOUA/m3DSr6klfEF McOfudZE958bChOrJgVrUnY3inR0J335bGQ1luEp5tYwPgyD9dG4MQEDD3oLwp+l +NtTUqz8WVlMxQ== -----END CERTIFICATE----- Configuring mTLS Between etcd and APISIX With the certificates available, we can now configure mutual TLS between etcd and APISIX. Let's start with etcd: YAML apiVersion: v1 kind: Pod metadata: name: etcd namespace: tls labels: role: config spec: containers: - name: etcd image: bitnami/etcd:3.5.7 ports: - containerPort: 2379 env: - name: ETCD_TRUSTED_CA_FILE #1 value: /etc/ssl/private/ca.crt - name: ETCD_CERT_FILE #2 value: /etc/ssl/private/tls.crt - name: ETCD_KEY_FILE #3 value: /etc/ssl/private/tls.key - name: ETCD_ROOT_PASSWORD value: whatever - name: ETCD_CLIENT_CERT_AUTH #4 value: "true" - name: ETCD_LISTEN_CLIENT_URLS value: https://0.0.0.0:2379 volumeMounts: - name: ssl mountPath: /etc/ssl/private #5 volumes: - name: ssl secret: secretName: etcd-secret #5 Set the trusted CA Set the certificate Set the private key Require clients to pass their certificate, hence ensuring mutual authentication Mount the previously generated secret in the container for access Now, it's Apache APISIX's turn: YAML apiVersion: v1 kind: ConfigMap #1 metadata: name: apisix-config namespace: tls data: config.yaml: >- apisix: ssl: ssl_trusted_certificate: /etc/ssl/certs/ca.crt #2 deployment: etcd: host: - https://etcd:2379 tls: cert: /etc/ssl/certs/tls.crt #2 key: /etc/ssl/certs/tls.key #2 admin: allow_admin: - 0.0.0.0/0 https_admin: true #3 admin_api_mtls: admin_ssl_cert: /etc/ssl/private/tls.crt #3 admin_ssl_cert_key: /etc/ssl/private/tls.key #3 admin_ssl_ca_cert: /etc/ssl/private/ca.crt #3 --- apiVersion: v1 kind: Pod metadata: name: apisix namespace: tls labels: role: gateway spec: containers: - name: apisix image: apache/apisix:3.2.0-debian ports: - containerPort: 9443 #4 - containerPort: 9180 #5 volumeMounts: - name: config #1 mountPath: /usr/local/apisix/conf/config.yaml subPath: config.yaml - name: ssl #6 mountPath: /etc/ssl/private - name: etcd-client #7 mountPath: /etc/ssl/certs volumes: - name: config configMap: name: apisix-config - name: ssl #6,8 secret: secretName: apisix-server-secret - name: etcd-client #7,8 secret: secretName: apisix-client-secret Apache APISIX doesn't offer configuration via environment variables. We need to use a ConfigMap that mirrors the regular config.yaml file. Configure client authentication for etcd Configure server authentication for the Admin API Regular HTTPS port Admin HTTPS port Certificates for server authentication Certificates for client authentication Two sets of certificates are used: one for server authentication for the Admin API and regular HTTPS, and one for client authentication for etcd. At this point, we can apply the above manifests and see the two pods communicating. When connecting, Apache APISIX sends its apisix-client certificate via HTTPS. Because an authority signs the certificate that etcd trusts, it allows the connection. I've omitted the Service definition for brevity's sake, but you can check them in the associated GitHub repo. Plain Text NAME READY STATUS RESTARTS AGE apisix 1/1 Running 0 179m etcd 1/1 Running 0 179m Client Access Now that we've set up the basic infrastructure, we should test accessing it with a client. We will use our faithful curl, but any client that allows configuring certificates should work, e.g, httpie. The first step is to create a dedicated certificate-key pair for the client: YAML apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: curl-client namespace: tls spec: secretName: curl-secret isCA: false usages: - client auth emailAddresses: - curl@localhost.dev issuerRef: name: ca-issuer kind: Issuer curl requires a path to the certificate file instead of the content. We can go around this limitation through the magic of zsh: the =( ... ) syntax allows the creation of a temporary file. If you're using another shell, you'll need to find the equivalent syntax or download the files manually. Let's query the Admin API for all existing routes. This simple command allows checking that Apache APISIX is connected to etcd, and it can read its configuration from there. Shell curl --resolve 'admin:32180:127.0.0.1' https://admin:32180/apisix/admin/routes \ #1 --cert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.crt }' | base64 -d) \ #2 --key =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.key }' | base64 -d) \ #2 --cacert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.ca\.crt }' | base64 -d) \ #2 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' --resolve avoids polluting one's /etc/hosts file. curl will translate admin to localhost, but the query is sent to admin inside the Kubernetes cluster, thus using the correct Service. Get the required data inside the Secret, decode it, and use it as a temporary file. If everything works, and it should, the result should be the following: JSON {"total":0,"list":[]} No routes are available so far because we have yet to create any. TLS With Upstreams Last but not least, we should configure TLS for upstreams. In the following, I'll use a simple nginx instance that responds with static content. Use it as an illustration for more complex upstreams. The first step, as always, is to generate a dedicated Certificate for the upstream. I'll skip how to do it as we already created a few. I call it upstream-server and its Secret, unimaginatively, upstream-secret. We can now use the latter to secure nginx: YAML apiVersion: v1 kind: ConfigMap #1 metadata: name: nginx-config namespace: tls data: nginx.conf: >- events { worker_connections 1024; } http { server { listen 443 ssl; server_name upstream; ssl_certificate /etc/ssl/private/tls.crt; #2 ssl_certificate_key /etc/ssl/private/tls.key; #2 root /www/data; location / { index index.json; } } } --- apiVersion: v1 kind: Pod metadata: name: upstream namespace: tls labels: role: upstream spec: containers: - name: upstream image: nginx:1.23-alpine ports: - containerPort: 443 volumeMounts: - name: config mountPath: /etc/nginx/nginx.conf #1 subPath: nginx.conf - name: content mountPath: /www/data/index.json #3 subPath: index.json - name: ssl #2 mountPath: /etc/ssl/private volumes: - name: config configMap: name: nginx-config - name: ssl #2 secret: secretName: upstream-secret - name: content #3 configMap: name: nginx-content NGINX doesn't allow configuration via environment variables; we need to use the ConfigMap approach. Use the key-certificate pair created via the Certificate. Some static content unimportant in the scope of this post The next step is to create the route with the help of the Admin API. We prepared everything in the previous step; now we can use the API: Shell curl --resolve 'admin:32180:127.0.0.1' https://admin:32180/apisix/admin/routes/1 \ --cert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.crt }' | base64 -d) \ #1 --key =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.key }' | base64 -d) \ #1 --cacert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.ca\.crt }' | base64 -d) \ #1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d "{ \"uri\": \"/\", \"upstream\": { \"scheme\": \"https\", #2 \"nodes\": { \"upstream:443\": 1 }, \"tls\": { \"client_cert\": \"$(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.crt }' | base64 -d)\", #3 \"client_key\": \"$(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.key }' | base64 -d)\" #3 } } }" Client auth for Admin API, as above Use HTTPS for the upstream Configure key-certificate pair for the route. Apache APISIX stores the data in etcd and will use them when you call the route. Alternatively, you can keep the pair as a dedicated object and use the newly-created reference (just like for upstreams). It depends on how many routes the certificate needs. For more information, check the SSL endpoint. Finally, we can check it works as expected: Shell curl --resolve 'upstream:32443:127.0.0.1' https://upstream:32443/ \ --cert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.crt }' | base64 -d) \ --key =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.tls\.key }' | base64 -d) \ --cacert =(kubectl get secret curl-secret -n tls -o jsonpath='{ .data.ca\.crt }' | base64 -d) And it does: JSON { "hello": "world" } Conclusion In this post, I've described a working Apache APISIX architecture and implemented mutual TLS between all the components: etcd and APISIX, client and APISIX, and finally, client and upstream. I hope it will help you to achieve the same. The complete source code for this post can be found on GitHub (linked above in the "Configuring mTLS Between etcd and APISIX" section). To Go Further: How to Easily Deploy Apache APISIX in Kubernetes cert-manager A Simple CA Setup with Kubernetes Cert Manager Mutual TLS Authentication

The benefits of adopting cloud-native practices have been talked about by industry professionals ad nauseam, with everyone extolling its ability to lower costs, easily scale, and fuel innovation like never before. Easier said than done. Companies want to go cloud-native, but they’re still trying to solve the metaphorical “security puzzle” that’s at the heart of Kubernetes management. They start with people and processes, shift corporate culture to match cloud-native security best-practices, and then sort of hit a wall. They understand they need a way to embed security into the standard developer workflows and cluster deployments, but creating continuous and secure GitOps is—in a word—hard. What Makes Kubernetes Security So Difficult? For starters, Kubernetes is usually managed by developers. They can spin up clusters quickly, but they may not be responsible for managing or securing them. Enter the Site-Reliability Engineers (SREs), who feel the need to slow down developer velocity to better manage clusters and make sure they know what’s going into production. Though not ideal, slowing things down is often seen as a reasonable tradeoff for eliminating unmanaged clusters and the kinds of risks that make IT security professionals shudder. Another challenge of Kubernetes security is that its default configurations aren’t very secure. Kubernetes is built for speed, performance, and scalability. Though each piece of the Kubernetes puzzle has security capabilities built-in, they’re often not turned on by default. That usually means developers forgo security features to move faster. As if that weren’t challenging enough, Kubernetes has complex privilege-management features that can easily become everyone’s greatest pain point. Kubernetes comes with its own Role-Based Access Controls (RBAC), but they add yet another layer of complexity. Any changes made to the Kubernetes puzzle need to be reflected in the others. Four Tenets of Kubernetes Security Although all these challenges roll up to something pretty daunting, all hope is not lost. Enter the four tenets of Kubernetes security: Easy-to-digest best-practices you can implement—today—that will make your Kubernetes and cloud-native infrastructure more secure, and help you manage your cyber exposure. Tenet #1: Manage Kubernetes Misconfigurations With Solid Policies Applying policies and associated industry benchmarks consistently throughout the development lifecycle is an essential first step in Kubernetes security. For example, you might have a policy that says you’re not allowed to run any containers with root privileges in your clusters, or that your Kubernetes API server cannot be accessible to the public. If your policies are violated, that means you have misconfigurations that will lead to unacceptable security risks. Leveraging policy frameworks, like OPA, and industry benchmarks, like CIS, can help harden Kubernetes environments and prevent misconfigurations from going into production. Set your policies first, and don’t just park them on a shelf. Revisit them on a regular schedule to ensure they continue to suit your needs. Tenet #2: Implement Security Guardrails Kubernetes security starts in the development process. Most teams developing for cloud-native platforms are using Infrastructure-as-Code (IaC) to provision and configure their systems. They should expand that to policy-as-code. That means you and your team can apply the same security policies across the software-development lifecycle. For example, developers can scan their code for security problems on their workstations, in CI/CD pipelines, container image registries, and the Kubernetes environment itself. Leveraging developer-friendly tools can help seamlessly integrate security into the development process. Open-source IaC static code-scanners, like Terrascan, can help ensure only secure code enters your environment. Tenet #3: Understand and Remediate Container Image Vulnerabilities Remediating container image vulnerabilities can be a challenge because it can be hard to see what’s actually running in your environment. Kubernetes dashboards don’t tell you much; you need to know what your images contain and how they’re built. Many developers use common base images and call it good. Unfortunately, unscanned base images can leave you vulnerable, and the risks are compounded at the registry and host levels. Developers usually skip this sort of image-scanning because it slows them down. The fact is, if they’re not in the habit of being on the lookout for outdated OS images, misconfigured settings, permission issues, embedded credentials and secrets, deprecated or unverified packages, unnecessary services, and exposed ports, they’re just handing off the problem to someone else. Yes, their work may go more quickly, but their practices are slowing the entire software-delivery process. Some common security risks: In images: Outdated packages, embedded secrets, and the use of untrusted images. In registries: Stale images, lax authentication, and lack of testing. In hosts: Lax network policies, weak access controls, unencrypted data volumes and storage, and insecure host infrastructure. Tenet #4: Exposure Management Once you’ve implemented the first three tenets, it’s time to start looking at your infrastructure holistically. Not all policies apply in all cases, so how are you applying your exclusions? Not every vulnerability is critical, so how do you prioritize fixes and automate remediation? These questions can guide you as you work toward becoming less reactive and more proactive. At the end of the day, visibility is central to managing security in Kubernetes and cloud-native environments. You need to be able to recognize when your configurations have drifted from your secure baselines, and readily identify failing policies and misconfigurations. Only then can you get the full picture of your attack surface. Where Do You Go From Here? The Journey to Comprehensive Cloud Security You might be feeling a little overwhelmed by the sheer number of attack vectors in cloud environments. But, with a solid game-plan and taking to heart the best-practices described above, cloud-native security isn’t just about the destination—your runtimes—but the journey (the entire development lifecycle). Shifting left to catch vulnerabilities using policy-as-code is a great first step toward ensuring the reliability and security of your cloud assets—and wrangling all the pieces of the Kubernetes puzzle.

Secrets management in Docker is a critical security concern for any business. When using Docker containers, it is essential to keep sensitive data, such as passwords, API keys, and other credentials, secure. This article will discuss some best practices for managing secrets in Docker, including how to store them securely and minimize their exposure. We will explore multiple solutions: using Docker Secrets with Docker Swarm, Docker Compose, or Mozilla SOPS. Feel free to choose what’s more appropriate to your use case. But most importantly is to remember to never hard-code your Docker secrets in plain text in your Dockerfile! Following these guidelines ensures your organization’s sensitive information remains safe even when running containerized services. 4 Ways To Store and Manage Secrets in Docker 1. Using Docker Secrets and Docker Swarm Docker Secrets and Docker Swarm are two official and complimentary tools allowed to securely manage secrets when running containerized services. Docker Secrets provides a secure mechanism for storing and retrieving secrets from the system without exposing them in plain text. It enables users to keep their credentials safe by encrypting the data with a unique key before passing it to the system. Docker Swarm is a powerful tool for managing clusters of nodes for distributed applications. It provides an effective means of deploying containerized applications at scale. With this tool, you can easily manage multiple nodes within a cluster and automatically distribute workloads among them. This helps ensure your application has enough resources available at all times, even during peak usage periods or unexpected traffic spikes. Together, these two tools provide an effective way to ensure your organization’s sensitive information remains safe despite ever-evolving security needs. Let’s see how to create and manage an example secret. Creating a Secret To create a secret, we need to first initialize Docker Swarm. You can do so using the following command: docker swarm init Once the service is initialized, we can use the docker secret create command to create the secret: ssh-keygen -t rsa -b 4096 -N "" -f mykey docker secret create my_key mykey rm mykey In these commands, we first create an SSH key using the ssh-keygen command and write it to mykey. Then, we use the Docker secret command to generate the secret. Ensure you delete the mykey file to avoid any security risks. You can use the following command to confirm the secret is created successfully: docker secret ls We can now use this secret in our Docker containers. One way is to pass this secret with –secret flag when creating a service: docker service create --name mongodb --secret my_mongodb_secret redis:latest We can also pass this secret to the docker-compose.yml file. Let’s take a look at an example file: version: '3.7' services: myapp: image: mydummyapp:latest secrets: - my_secret volumes: - type: bind source: my_secret_key target: /run/secrets/my_secret read_only: true secrets: my_secret: external: true In the example compose file, the secrets section defines a secret named my_secret_key (discussed earlier). The myapp service definition specifies that it requires my_secret_key , and mounts it as a file at /run/secrets/my_secret in the container. 2. Using Docker Compose Docker Compose is a powerful tool for defining and running multi-container applications with Docker. A stack is defined by a docker-compose file allowing you to define and configure the services that make up your application, including their environment variables, networks, ports, and volumes. With Docker Compose, it is easy to set up an application in a single configuration file and deploy it quickly and consistently across multiple environments. Docker Compose provides an effective solution for managing secrets for organizations handling sensitive data such as passwords or API keys. You can read your secrets from an external file (like a TXT file). But be careful not to commit this file with your code: version: '3.7' services: myapp: image: myapp:latest secrets: - my_secret secrets: my_secret: file: ./my_secret.txt 3. Using a Sidecar Container A typical strategy for maintaining and storing secrets in a Docker environment is to use sidecar containers. Secrets can be sent to the main application container via the sidecar container, which can also operate a secrets manager or another secure service. Let’s understand this using a Hashicorp Vault sidecar for a MongoDB container: First, create a Docker Compose (docker-compose.yml) file with two services: mongo and secrets. In the secrets service, use an image containing your chosen secret management tool, such as a vault. Mount a volume from the secrets container to the mongo container so the mongo container can access the secrets stored in the secrets container. In the mongo service, use environment variables to set the credentials for the MongoDB database, and reference the secrets stored in the mounted volume. Here is the example compose file: version: '3.7' services: mongo: image: mongo volumes: - secrets:/run/secrets environment: MONGO_INITDB_ROOT_USERNAME_FILE: /run/secrets/mongo-root-username MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo-root-password secrets: image: vault volumes: - ./secrets:/secrets command: ["vault", "server", "-dev", "-dev-root-token-id=myroot"] ports: - "8200:8200" volumes: secrets: 4. Using Mozilla SOPS Mozilla SOPS (Secrets Ops) is an open-source platform that provides organizations with a secure and automated way to manage encrypted secrets in files. It offers a range of features designed to help teams share secrets in code in a safe and practical way. The following assumes you are already familiar with SOPS, if that’s not the case, start here. Here is an example of how to use SOPS with docker-compose.yml: version: '3.7' services: myapp: image: myapp:latest environment: API_KEY: ${API_KEY} secrets: - mysecrets sops: image: mozilla/sops:latest command: ["sops", "--config", "/secrets/sops.yaml", "--decrypt", "/secrets/mysecrets.enc.yaml"] volumes: - ./secrets:/secrets environment: # Optional: specify the path to your PGP private key if you encrypted the file with PGP SOPS_PGP_PRIVATE_KEY: /secrets/myprivatekey.asc secrets: mysecrets: external: true In the above, the myapp service requires a secret called API_KEY. The secrets section uses a secret called mysecrets, which is expected to be stored in an external key/value store, such as Docker Swarm secrets or HashiCorp Vault. The sops service uses the official SOPS Docker image to decrypt the mysecrets.enc.yaml file, which is stored in the local ./secrets directory. The decrypted secrets are mounted to the myapp service as environment variables. Note: Make sure to create the secrets directory and add the encrypted mysecrets.enc.yaml file and the sops.yaml configuration file (with SOPS configuration) in that directory. Scan for Secrets in Your Docker Images Hard coding secrets in Docker is a significant security risk, making them vulnerable to attackers. We have seen different best practices to avoid hard-coding secrets in plain text in your Docker images, but security doesn’t stop there. You Should Also Scan Your Images for Secrets All Dockerfiles start with a FROM directive that defines the base image. It’s important to understand when you use a base image, especially from a public registry like Docker Hub, you are pulling external code that may contain hardcoded secrets. More information is exposed than visible in your single Dockerfile. Indeed, it’s possible to retrieve a plain text secret hard-coded in a previous layer starting from your image. In fact, many public Docker images are concerned: in 2021, we estimated that **7% of the Docker Hub images contained at least one secret.** Fortunately, you can easily detect them with ggshield (GitGuardian CLI). For example: ggshield secret scan docker ubuntu:22.04 Conclusion Managing secrets in Docker is a crucial part of preserving the security of your containerized apps. Docker includes several built-in tools for maintaining secrets, such as Docker Secrets and Docker Compose files. Additionally, organizations can use third-party solutions, like HashiCorp Vault and Mozilla SOPS, to manage secrets in Docker. These technologies offer extra capabilities, like access control, encryption, and audit logging, to strengthen the security of your secret management. Finally, finding and limiting accidental or unintended exposure of sensitive information is crucial to handling secrets in Docker. Companies are invited to use secret scanning tools, such as GitGuardian, to scan the Docker images built in their CI/CD pipelines as mitigation to prevent supply-chain attacks. If you want to know more about Docker security, we also summarized some of the best practices in a cheat sheet.

Microservice architecture is a popular software design pattern that involves dividing a large application into smaller, independent services that can be developed, deployed, and scaled autonomously. Each microservice is responsible for a specific function within the larger application and communicates with other services via well-defined interfaces, often leveraging lightweight communication protocols such as REST or message queues. Advantages of Microservice Architecture Microservice architecture provides several advantages. Here are some benefits of using microservices: Maintainability: Smaller services are more manageable and easier to understand, leading to more stable and reliable applications. Technology Diversity: Microservices can be built using different technologies, providing greater flexibility and innovation in the development process. Scalability: Microservices can be scaled independently of each other, making it easier to allocate resources based on demand Flexibility: Microservices can be developed and deployed independently, allowing for more agile development processes and faster time-to-market Resilience: Microservices are independent of each other; failures in one service do not necessarily affect other services. Complexity: Microservices can introduce additional complexity in managing and coordinating interactions between services, and it can be more challenging to test and debug a distributed system. Disadvantages of Microservice Architecture Microservice architecture has its own set of challenges and drawbacks, which include: Complexity: The complexity increases as the number of microservices increases, making it harder to manage and coordinate interactions between services. Skilled developers: Working with microservices architecture requires skilled developers who can identify the microservices and manage their inter-communications effectively. Deployment: The independent deployment of microservices can be complicated, requiring additional coordination and management. Network usage: Microservices can be costly in terms of network usage as they need to interact with each other, resulting in network latency and increased traffic. Security: Microservices may be less secure than monolithic applications due to the inter-services communication over the network, which can be vulnerable to security breaches. Debugging: Debugging can be difficult as control flows over multiple microservices, and identifying where and why an error occurred can be a challenging task. Difference Between Monolithic and Microservice Architecture Monolithic architecture refers to a traditional approach where an entire application is built as a single, unified system, with all the functions tightly coupled together. In this approach, changes to one part of the application can have a ripple effect on other parts, making it difficult to scale and maintain over time. In addition, monolithic architecture is typically deployed as a single, large application, and it can be difficult to add new features or make changes to the system without affecting the entire application. In contrast, microservice architecture is a more modern approach where an application is broken down into smaller, independent services that can be developed, deployed, and scaled independently of each other. Each microservice focuses on a specific function within the larger application, and communication between services is typically accomplished through lightweight communication protocols such as REST or message queues. This approach offers more flexibility and scalability, as changes to one microservice do not necessarily affect the other microservices. Why Golang for Microservices? Go, also known as Golang, is an ideal language for developing microservices due to its unique characteristics. Here are some reasons why you consider Go for building microservices: Performance: Go is a fast and efficient language, which makes it perfect for building microservices that need to handle high volumes of traffic and requests. It also provides excellent runtime performance. Benchmarks game test: fannkuch-redux source secs mem gz cpu secs Go 8.25 10,936 969 32.92 Java 40.61 39,396 1257 40.69 Python 913.87 11,080 385 913.83 Benchmarks game test: n-body source secs mem gz cpu secs Go 6.36 11,244 1200 6.37 Java 7.46 39,992 1430 7.5 Python 383.12 11,096 1196 383.11 Source: Benchmarks Game Concurrency: Go provides built-in support for concurrency, which allows it to handle multiple requests simultaneously, making it a great choice for building scalable microservices. Small footprint: Go binaries have a small footprint and are lightweight, which makes it easier to deploy and manage microservices in a containerized environment. Simple and readable syntax: Go has a simple and intuitive syntax that is easy to learn and read. This makes it a great language for building maintainable microservices that are easy to understand and modify. Strong standard library: Go has a strong standard library that provides many useful built-in functions and packages for building microservices, such as networking, serialization, and logging. Large community: Go has a large and active community that provides support, resources, and third-party libraries for building microservices. Cross-platform support: Go is designed to be cross-platform, which means that you can build and deploy microservices on a variety of operating systems and environments. Overall, Go is an excellent choice for building microservices that require high performance, scalability, and efficiency. Its simple syntax, strong standard library, and large community make it a popular choice among developers for building microservices, and its small footprint and cross-platform support make it easy to deploy and manage microservices in a containerized environment. Golang Frameworks Here are a few frameworks build in Golang: • Go Micro – Go Micro is an open-source platform designed for API-driven development. It is also the most popular remote procedure call (RPC) framework today. It provides load balancing, synchronous and asynchronous communication, message encoding, service discovery, and RPC client/server packages. • Go Kit – Go Kit is a library that fills the gaps left by the standard library and provides solutions for most operational and infrastructure concerns. It makes Go a first-class language for writing microservices in any organization. These frameworks provide a variety of features and functionalities that can help developers build and manage microservices more efficiently. Choosing the right framework will depend on the specific requirements of the project and the expertise of the development team. Why Do Companies Use Golang? Two main reasons why Google's programming language is used by companies of various tiers: Designed for multi-core processing: Golang was specifically designed for cloud computing and takes advantage of modern hardware's parallelism and concurrency capabilities. Golang's goroutines and channel-based approach make it easy to utilize all available CPU cores and handle parallel I/O without adding complexity to development. Built for big projects: Golang's code is simple and easy to read, making it ideal for large-scale projects. Real-Life Examples of Using Go to Build Microservices Here are some real-life examples of companies using Go to build microservices: Netflix: Netflix uses Go for several of its microservices, including its platform engineering and cloud orchestration services. Go's performance and concurrency features help Netflix handle the scale and complexity of its infrastructure. Docker: Docker is a popular containerization platform that uses Go to power many of its core components, including the Docker daemon and Docker CLI. Go's small footprint, and cross-platform support make it well-suited for building containerized applications. Conclusion Microservices have become a popular choice for startups and large companies due to their scalability, sustainability, and ability to deploy them independently. Among programming languages, Go stands out as an excellent choice for microservices. Its easy-to-write code, high level of security, fast execution speed, and low entry threshold make it unmatched for this architecture. Go's performance, concurrency, and simplicity make it an increasingly popular option for building scalable and reliable microservices architectures.

Setting up a VPN server to allow remote connections can be challenging if you set this up for the first time. In this post, I will guide you through the steps to set up your own VPN Server and connect to it using a VPN Client. Additionally, I will also show how to set up a free Radius server and a plugin to implement multi-factor authentication for additional security. 1. Installation OpenVPN Server on Linux (Using a Centos Stream 9 Linux) # yum update # curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh # chmod +x openvpn-install.sh # ./openvpn-install.sh Accept defaults for installation of OpenVPN and, in the end, provide a Client name e.g. demouser. I have chosen a passwordless client, but if you want, you can also add an additional password to protect your private key. PowerShell Client name: demouser Do you want to protect the configuration file with a password? (e.g. encrypt the private key with a password) 1) Add a passwordless client 2) Use a password for the client Select an option [1-2]: 1 ... ... The configuration file has been written to /root/demouser.ovpn. Download the .ovpn file and import it in your OpenVPN client. Finally, a client configuration file is ready to be imported into the VPN Client. 2. Installation of OpenVPN Client for Windows Download the OpenVPN Client software. Install the OpenVPN Client: Once the installation is finished, we can import the configuration file demouser.ovpn which was generated on the OpenVPN server, but before importing, we need to modify the IP address of our OpenVPN server within this file: client proto udp explicit-exit-notify remote 192.168.0.150 1194 dev tun resolv-retry infinite nobind persist-key persist-tun ... Normally the remote IP by default will be the address of your public IP which is normal if you have your VPN server on your local network and need remote access from outside this network. You can leave the public IP address in the config, but then you will have to open up the correct port and set the routing on your internet access point. Finally, we can test the VPN connection. The first time the connection will probably fail as the firewall on the OpenVPN Linux server is blocking the access. To quickly test this, we can just disable the firewall using the command: # systemctl stop firewalld Alternatively, configure Linux firewall for OpenVPN connectivity: # sudo firewall-cmd --add-service=openvpn # sudo firewall-cmd --permanent --add-service=openvpn # sudo firewall-cmd --add-masquerade # sudo firewall-cmd --permanent --add-masquerade # sudo firewall-cmd --permanent --add-port=1194/udp # sudo firewall-cmd --reload Now the connection should work: On the windows client, you should now also get an additional VPN adapter configured with a default IP address of 10.8.0.2 (this subnet is defined within the file /etc/openvpn/server.conf). 3. How To Use Radius With OpenVPN First, we will install the IBM Security Verify Gateway for Radius on a Windows machine. This package can be downloaded from the IBM Security AppExchange (you will need to use your IBMid to log in). Extract and run the installation using setup_radius.exe. Edit the Radius configuration file c:\Program Files\IBM\IbmRadius\IbmRadiusConfig.json: Find the clients section in the configuration file. The default file has three example client definitions. Delete these definitions and replace them with the single definition shown above. This definition will match any Radius client connecting from the network used by the test machines. The secret authenticates the client. Save the file and close the editor. JSON { "address":"::", "port":1812, /* "trace-file":"c:/tmp/ibm-auth-api.log", "trace-rollover":12697600, */ "ibm-auth-api":{ "client-id":"???????", "obf-client-secret":"???????", /* See IbmRadius -obf "the-secret" */ "protocol":"https", "host":"???????.verify.ibm.com", "port":443, "max-handles":16 }, "clients":[ { "name": "OpenVPN", "address": "192.168.0.0", "mask": "255.255.0.0", "secret": "Passw0rd", "auth-method": "password-and-device", "use-external-ldap": false, "reject-on-missing-auth-method": true, "device-prompt": "A push notification has been sent to your device:[%D].", "poll-device": true, "poll-timeout": 60 } ] } Complete the fields client-id, obf-client-secret and host with the correct information to point to your IBM Verify Saas API. Before we can do this, we will need to set up API access in IBM Verify Saas. Login to your environment or go for a trial account if you don’t have one. From the main menu, select Security > API Access > Add API client Create a new API Client : Specify the entitlements by selecting the check bow from the list: Authenticate any user Read authenticator registrations for all users Read users and groups Read second-factor authentication enrollment for all users Click next on the following screens and finally give the API client a name: e.g. MFA-Client A Client ID and Secret will automatically be created for you. Use this information to complete the Radius config. Use the c:\Program Files\IBM\IbmRadius\IbmRadius.exe -obf command to generate the obfuscated secret value. Finally, configure the IBM Radius service to startup automatically and start the service: Test Radius Authentication using the Radius tool : NTRadPing You should get a push notification on the IBM Verify app on the mobile device. (Make sure you test with a userid that is known in IBM Verify Saas and is enrolled for OTP) 4. Install OpenVPN Radius Plugin Log in to the Linux OpenVPN server and launch the following commands: # wget https://www.nongnu.org/radiusplugin/radiusplugin_v2.1a_beta1.tar.gz # tar -xvf radiusplugin_v2.1a_beta1.tar.gz # cd radiusplugin_v2.1a_beta1 # yum install libgcrypt libgcrypt-devel gcc-c++ # make Copy the Radius plugin files to /etc/openvpn # cp /root/radiusplugin_v2.1a_beta1/radiusplugin.cnf /etc/openvpn # cp /root/radiusplugin_v2.1a_beta1/radiusplugin.so /etc/openvpn Edit the file /etc/openvpn/server.conf and add the following line to activate the Radius plugin: plugin /etc/openvpn/radiusplugin.so /etc/openvpn/radiusplugin.cnf Edit the file /etc/openvpn/radiusplugin.cnf and modify the ip address of the Radius server and set the sharedsecret to Passw0rd (this is the secret that was also configured on the Radius server side). Make sure to set nonfatalaccounting=true because the Radius server does not support Radius accounting. C ... NAS-IP-Address=<IP Address of the OpenVPN Server> ... nonfatalaccounting=true ... Server { # The UDP port for RADIUS accounting. acctport=1813 # The UDP port for RADIUS authentication. authport=1812 # The name or ip address of the RADIUS server. name=<IP Address of the RADIUS Server> # How many times should the plugin send the if there is no response? retry=1 # How long should the plugin wait for a response? wait=60 # The shared secret. sharedsecret=Passw0rd } Save the file and restart the OpenVPN server using the command : # systemctl restart openserver-server@server.service Finally, edit the OpenVPN client file demouser.ovpn and add a line auth-user-pass : client proto udp auth-user-pass explicit-exit-notify remote 192.168.0.150 1194 dev tun resolv-retry infinite nobind persist-key persist-tun ... This will allow the user to enter a username and password when initiating the VPN connection. These credentials will be authenticated against the IBM Verify Saas directory, and this should result in a challenge request on the IBM Verify Mobile app. The wait=60 will allow the plugin to wait for a response from the user who has to respond to the challenge using the IBM Verify App on his phone. If you prefer to use a TOTP challenge instead, you can modify the Radius configuration file on Windows (IBMRadiusConfig.json) and set the auth-method to password-and-totp. Then you can open the client VPN connection and use 123456:password instead of the normal password.

Container orchestration is a critical aspect of modern software development, enabling organizations to deploy and manage large-scale containerized applications. In this article, we will discuss what container orchestration is, why it is important, and some of the popular container orchestration tools available today. What Is Container Orchestration? Container orchestration is the process of automating the deployment, scaling, and management of containerized applications. Containers are lightweight, portable software units that can run anywhere, making them ideal for modern, distributed applications. However, managing containerized applications can be complex, as they typically consist of multiple containers that must be deployed, configured, and managed as a single entity. Container orchestration tools provide a platform for automating tasks, enabling organizations to manage large-scale containerized applications easily. They typically provide features such as automated deployment, load balancing, service discovery, scaling, and monitoring, making it easier to manage complex containerized applications. One of the most popular container orchestration tools is Kubernetes, which was developed by Google. Kubernetes provides a platform for automating the deployment, scaling, and management of containerized applications and has a large and active community. Other popular container orchestration tools include Docker Swarm, Apache Mesos, and Nomad. Container orchestration is important for organizations that develop and deploy modern, distributed applications. Containerization provides several benefits, including improved portability, scalability, and agility. However, managing containerized applications can be challenging, particularly as the number of containers and applications increases. Container orchestration tools provide a way to automate the management of containerized applications, enabling organizations to deploy and manage complex applications with ease. They also help ensure applications are highly available, scalable, and reliable, making it easier to deliver high-quality services to customers. Why Is Container Orchestration Important? Container orchestration is important for several reasons, particularly for organizations that develop and deploy modern, distributed applications. Here are some of the key reasons why container orchestration is important: Automation Container orchestration tools enable organizations to automate the deployment, scaling, and management of containerized applications. This reduces the need for manual intervention, making it easier to manage large-scale applications. Scalability Container orchestration tools provide features, such as automatic scaling and load balancing, which make it easier to scale applications up or down as demand changes. Container orchestration platforms make it easy to scale applications horizontally by adding or removing containers based on demand. Availability Container orchestration tools help ensure applications are highly available and reliable by providing features such as service discovery and self-healing. Portability Containers are portable, meaning they can be run anywhere, from local development environments to public cloud platforms. Container orchestration tools enable organizations to manage containerized applications across different environments and platforms, making it easier to move applications between different infrastructure providers. Container orchestration platforms provide a high degree of portability, enabling developers to run their applications in any environment, from on-premises data centers to public cloud environments. Flexibility Container orchestration tools provide a flexible and modular platform for managing containerized applications, making it easier to customize and extend the platform to meet specific requirements. Efficiency Container orchestration platforms automate many of the tasks involved in managing containerized applications, which can save developers time and reduce the risk of errors. Resiliency Container orchestration platforms offer self-healing capabilities that ensure applications remain available and responsive even with failures. Overall, container orchestration is essential for organizations that are developing and deploying modern, distributed applications. By automating the deployment, scaling, and management of containerized applications, container orchestration tools enable organizations to deliver high-quality services to customers, while also reducing the complexity and cost of managing containerized applications. Popular Container Orchestration Tools There are several container orchestration tools available, each with its own strengths and weaknesses. The most popular container orchestration tool is Kubernetes, which is an open-source platform for managing containerized applications. Kubernetes provides a robust set of features for managing containers, including container deployment, scaling, and health monitoring. Other popular container orchestration tools include Docker Swarm, which is a simple and lightweight orchestration tool, and Apache Mesos, which is a highly scalable and flexible orchestration tool. Kubernetes Kubernetes is one of the most popular container orchestration tools and is widely used in production environments. It provides a rich set of features, including automatic scaling, load balancing, service discovery, and self-healing. Docker Swarm Docker Swarm is a container orchestration tool that is tightly integrated with the Docker ecosystem. It provides a simple and easy-to-use platform for managing containerized applications but has fewer features than Kubernetes. Apache Mesos Apache Mesos is a distributed systems kernel that provides a platform for managing resources across clusters of machines. It can be used to manage a wide range of workloads, including containerized applications. Nomad Nomad is a container orchestration tool developed by HashiCorp. It provides a simple and flexible platform for managing containerized applications and can be used to manage containers and non-container workloads. OpenShift OpenShift is a container application platform developed by Red Hat. It is based on Kubernetes but provides additional features and capabilities, such as integrated developer tools and enterprise-grade security. Amazon ECS Amazon Elastic Container Service (ECS) is a fully managed container orchestration service provided by Amazon Web Services. It provides a simple and easy-to-use platform for managing containerized applications on the AWS cloud platform. Google Cloud Run Google Cloud Run is a fully managed serverless container platform provided by Google Cloud. It allows developers to run containerized applications without the need to manage the underlying infrastructure. Overall, the choice of container orchestration tool will depend on a range of factors, including the specific requirements of the organization, the size and complexity of the application, and the preferred infrastructure platform. Container Orchestration Best Practices To ensure successful container orchestration, there are several best practices that organizations should follow. These include: Standardize Container Images Use standardized container images to ensure consistency and repeatability in deployments. Monitor Container Health Use container monitoring tools to ensure containers are healthy and performing as expected. Automate Deployments Use automated deployment tools to reduce the risk of human error and ensure consistent deployments. Implement Resource Quotas Implement resource quotas to ensure containerized applications are not overprovisioned and to optimize resource utilization. Plan for Disaster Recovery Plan for disaster recovery by implementing backup and restore processes, and testing disaster recovery plans regularly. Conclusion Container orchestration is an essential aspect of modern software development, enabling organizations to manage large-scale containerized applications with ease. By automating the deployment, scaling, and management of containerized applications, container orchestration tools enable organizations to deliver high-quality services to customers, while also reducing the complexity and cost of managing containerized applications. With several popular container orchestration tools available, organizations have a wide range of options for managing containerized applications and can choose the platform that best meets their needs. Container orchestration is a critical element of modern software development and deployment. It enables organizations to manage containerized applications at scale, ensuring they are highly available and resilient. By following best practices and leveraging container orchestration tools like Kubernetes, organizations can optimize resource utilization, accelerate the software development lifecycle, and reduce the risk of human error.

Before we jump into the details, let’s answer the question: “What is cloud integration architecture?” Cloud integration architecture is the combination of the cloud components and several cloud providers that allows exchanging of data across several data centers. For example, imagine that we have a large banking solution. Some banks don’t want to migrate their core and secured data. They want to keep it in their on-premises data centers. However, they can’t avoid using clouds. They migrate existing projects, new projects that are starting, and even the whole call centers and back office to the cloud. Building this complex architecture requires knowledge of the integration service and tools that the clouds provide. Let’s see what components AWS, Google Cloud, and Azure provide. Common Integration Services Knowing general cloud integration best practices and patterns is crucial. However, knowing what exactly each cloud provider offers is also important. Here we will get into the top integration services that AWS, Google Cloud, and Azure provide. Also, we look at what benefits and drawbacks each service has. AWS AWS has several integration services that provide powerful features alongside simplicity. SQS (Simple Queue Service) SQS is a simple query service or distributed queuing system. It’s a queue that stores the messages that should be processed later. SQS is a pull-based service that allows the decoupling of cloud components and microservices. Pros: Includes dead-letter queue: Automatically separates unprocessed messages and pushes them into the dead-letter queue The choice between a standard queue and FIFO Automatic duplication removal: SQS will automatically detect and remove duplicated messages. Cons: Doesn’t support one-to-many message broadcasting SWF (Simple Workflow Service) SWF is a task coordination and workflow service. It allows the building of parallels and sequential workflows. It allows for building a reliable distributed system where the tasks are logical units or functions of your application. SWF is suitable for processing human-based operations like order workflow or procedural requests. Pros: Reliability: Workflows and state tracking are reliable as they run at high-availability data centers Flexibility and logical separation: SWF allows architects and users to modify workflow elements easily. Also, SWF separates the control flow of your background job Cons: A lot of bugs in the AWS Management Console and a lack of general support The setup process is difficult and takes a lot of time Lack of limit control tracking: It can lead to throttling issues AWS Step Functions AWS step functions allow us to create a serverless workflow or state machines; for example, to automate call center tasks. Step Functions is an orchestration function that orchestrates other Lambda functions. Pros: Decreasing application complexity by decoupling workflow logic from business logic Cons: Step Functions uses a specific state machine to define workflows, helpful only for the Step Functions service. This may lead to vendor lock-in. Google Cloud (GC) The GC has an integration ecosystem, an Integration-Platform-as-a-Service (iPaaS). It provides a set of tools and services to manage and connect applications. The Google Cloud iPaaS contains the following services: Integration DesignerIntegration designer is a no-code UI tool that provides the ability to build workflows that consist of triggers and tasks (see below). Tasks and triggers are connected with an edge and fork and join. This service looks pretty similar to SWF. Triggers Triggers are an essential workflow element. Triggers are usually an entry point for the workflow and represent an event that initiates the task. For example, API trigger or pub/sub trigger. Tasks Tasks are units of work (or functions) that comes after triggers. One example is Data Mapping, where you can integrate one data model into another. Some predefined tasks include the Call Integration task, For Each Parallel task, For Each Loop task, Timer task, and many others. Alongside many advantages, Google Cloud has some drawbacks: Lack of data centers compared to Azure and AWS It is a relatively new framework. Some features may be in preview (or in beta) and contain some bugs. Azure Azure offers an Azure integration service set. It contains a variety of services to set up strong integration between applications. Below are set of the most powerful integrations services Azure offers. API Management API Management is an API gateway service that integrates several APIs and connects Azure Functions, Logic Apps, and Service Bus. Pros: API Management provides many features like authentication, authorization, cache responses, and fine-grain quotas and limits. Flexible customization through the API policies Cons: The developer portal contains many usability issues and bugs. Scaling costs can be too high. API management scales automatically; however, there is no good transparency on scaling costs. Logic Apps Logic Apps is a serverless, low-code, workflow integration tool (like GC Integration designer and AWS SWF). Pros: The service can connect SQL Server, Event Grid, Oracle, ServiceNow, and Office 365. We can select more than 200 connectors. Ability to host maps and schemas locally Cons: Difficult to manage and maintain complex integration workflows Issues with user interface, especially with complex workflows Service Bus Service Bus is a service that implements the event bus pattern. It provides an event bus based on the queue and publish/subscribe integration principle when one component produces a message, and others consume it. Pros: The service also supports translations, dead letters, multi-consumer topic-based logic, and other features. Powerful message routing and filtering capabilities Cons: Slow performance of the HTTPs connection Option to monitor messages, queues, and topics is quite limited No integrated UI tool to debug messages, queries, and filters Event Grid With the Event Grid, you can join cloud resources that produce events (publishers) and resources that handle the events (subscribers). The event grid looks similar to AWS SNS. It is also based on the push approach. Event Grid listens to the events from one component and can trigger other when it reserves messages. For example, one application can send a message to the Azure queue and an email notification. Pros: Enterprise integration: Supports almost all Azure event sources; e.g., Functions, Event/IoT Hubs, Logic Apps, Service Bus, SignalR, and many more Contains options to monitor and debug messages Includes automatic dead-letter queue Lower cost: A pay-per-event price model results in very low cost. Cons: Transparency on how to store and enable dead-letter can be improved. Azure Arc Azure Arc is a service that allows us to connect services from different cloud providers and implement multi-cloud integrations. This service will as a main integration service banking multi-cloud application. Pros: Supports services like SQL Server and Kubernetes Supports Windows and Linux operation systems Cons: Sometimes, the agent connection can be unstable and hard to debug. Conclusion In the article, we’ve gone through the top components to build cloud integration. However, integration solutions don’t have to be enterprise-level: they can meddle or even be small. For example, we have application A which requires the exchange of data between application B. We can create integration middleware with Azure Durable Functions or AWS Step Functions. With these services, we can create workflows for data mapping, notification, and even some plain analytics.

Along with the extensively discussed technological trending topics like AI, hyper-automation, blockchain, edge computing, and hyper-automation, Cloud computing is the central component in the upcoming years of various firms' IT strategies. These days the benefits of flexibility, alertness, fast speed, and cost efficiency have become essential for various CIOs. A few businesses are currently working on refining their overall IT cloud strategy. They take fundamental considerations such as which plan of action to opt Whether they should go for public, private, or a mixture of both. Others have progressed even further. They are working with full efforts to modify their applications. Moreover, they are taking advantage of different PaaS capabilities provided by the cloud to maximize benefits. Challenges Faced by Cloud Computing Such firms can also overcome the essential issues of Cloud computing, such as safety, data coherence, flexibility, and functional coherence, by focusing on the main elements of the cloud: simplifying the cloud performance. The frequent question in the area of cloud performance engineering is which execution can be achieved by the relocated and modified system in comparison to a pure on-site landscape. Is it going to be less, similar, or even higher and better performance? Cloud Scalability Options Many experts claim that in dynamic scalability possibilities in the cloud, it is simple to grow the system linearly just by amplifying the number of machines. It is unquestionably the initial Step that should be observed. Same as on-site systems, the vertical scalability capabilities are first employed traditional hardware capacities like CPUs and Read-Only-Memory raised. However, larger firms' IT systems with high output, access rates, and peak loads are reaching the breaking point. When ambitious expansion strategies combined with disorganized application might result in IT needs that exceed Moore's Law. Thus, requisite hardware is not yet accessible. Next-Generation and Upgradation of Cloud Hardware On one side, CIOs can aspire that the upcoming generation of hardware is ready to enter the market and can be provided to its users soon. On the other side, the subject of horizontal scaling has also achieved a lot of traction. Different from increasing servers for similar sections of the application. In many situations, this needs substantial changes in the application itself, like on-site systems. In particular, databases need an elaborated concept permitting the data to persist autonomously across many servers. In this situation, there might be an alternative for applications. That is an increasing number of read-only transactions. To gain execution goals in the absence of "real" horizontal scaling, implementing such PaaS offerings can be a solution. Such as, Microsoft provides the so-called Hyper-scale service for SQL databases, which dynamically scales the computing power through caching techniques and divides it horizontally to read replicas used as images of the database. AWS Cloud also provides read copies for RDS MySQL, PostgreSQL, MariaDB, Amazon Aurora, and Oracle Cloud. They depend on their popular Oracle RAC. Classical Approach There are other possibilities beyond vertical and horizontal scalability offered by Cloud performance engineering. There are still many well-known options available in the cloud compared to the availability on-premise. The most common classic approach is to boost your indexes, which helps to determine I/O performance for over 80% of your performance activity. However, if any one indicator is missing, the performance of the entire IT system may suffer. As a result, cloud performance engineers should always prioritize database indexing. In addition, topics related to batch processing and session handling, the definition of maximum batch sizes, connection durations, read frequencies, idle times, and possible connection pooling of, for example, SSL connection, can be decisive for the performance of the system. Due to this, your interface partner's CPUs from being overloaded by opening a new connection for each HTTPS request. It says that it is desirable to reduce the number of requests to the database and actively apply caching mechanisms. Similarly, the number of instances, the number of threads, and the hardware itself can be varied until a self-defined level of perfection is reached. Elasticity In cloud computing, scalability is just one aspect of performance engineering. One of the features that the cloud promises are fully automated elasticity, allowing resources to be dynamically adjusted to meet every demand. The hurdle is that on-premises applications are usually designed with static environments in mind, so they need to respond to dynamic scaling first. As a result, it requires defining and testing different test scenarios for the cloud. Attention should be on the interaction between the cloud and the applications. An essential metric is how well the application responds to the dynamic scaling of the cloud, whether it doesn't lose connections or experience other unusual behavior, and whether it doesn't suffer from the usual performance degradation that occurs on a system. Additional Features Cloud service providers present numerous new possibilities to quickly create test environments and analyze and evaluate performance KPIs at runtime. The best way to cover planned testing concepts in the cloud is to combine existing testing tools with new testing options in the cloud. It is always preferable to consider a complete rebuild of old applications instead of heavily customizing an existing application. This approach works when various functional, non-functional, and technical requirements are not implemented in the current application. Support of IT Organizations IT organizations are playing a vital role here by supporting them in the best possible ways. They support all the activities and improve the performance of the cloud, which has been recognized and benefited by agile processes and architectures of modular containers and not only to them but also to the new concept that is time-to-market, for instance, CI/CD pipelines. Most of the time, it is beneficial to implement such ideas beforehand before opting for the cloud. Conclusion Lastly, even though shifting to the cloud offers multiple opportunities and benefits, cloud performance engineering is a challenge that needs to be defeated by approved and new methods. The feature of automatically usable scalability in the cloud has to be opposed by many large-scale companies. The budget and time frame plan customization needed during the implementation. It is because well-planned, high-level supervision is highly recommended to obtain the best possible reaction among the users and facilitate them in the best possible way. Apart from this, there are other activities of testing. Those are data integrity, security, and resilience. These are significant to provide exclusive performance to the world. A good connection between all the teams involved in this regard, like the CEO, CIO, architects, cloud experts, and performance engineering specialist, is essential to achieve the shift to the cloud and successfully convey this new topic, cloud performance engineering, to the world.

A few weeks ago, Kelsey Hightower wrote a tweet and held a live discussion on Twitter about whether it's a good idea or not to run a database on Kubernetes. This happened to be incredibly timely for me since we at QuestDB are about to launch our own cloud database service (built on top of k8s)! "Rubbing Kubernetes on Postgres Won't Turn it Into Cloud SQL" You can run databases on Kubernetes because it's fundamentally the same as running a database on a VM. The biggest challenge is understanding that rubbing Kubernetes on Postgres won't turn it into Cloud SQL. https://t.co/zdFobm4ijy One of the biggest takeaways from this discussion is that there seems to be a misconception about the features that k8s actually provides. While newcomers to k8s may expect that it can handle complex application lifecycle features out-of-the-box, it, in fact, only provides a set of cloud-native primitives (or building blocks) for you to configure and use to deploy your workflows. Any functionality outside of these core building blocks needs to be implemented somehow in additional orchestration code (usually in the form of an operator) or config. K8s Primitives When working with databases, the obvious concern is data persistence. Earlier in its history, k8s really shined in the area of orchestrating stateless workloads, but support for stateful workflows was limited. Eventually, primitives like StatefulSets, PersistentVolumes (PVs), and PersistentVolumeClaims (PVCs) were developed to help orchestrate stateful workloads on top of the existing platform. PersistentVolumes are abstractions that allow for the management of raw storage, ranging from local disk to NFS, cloud-specific block storage, and more. These work in concert with PersistentVolumeClaims that represent requests for a pod to access the storage managed by a PV. A user can bind a PVC to a PV to make an ownership claim on a set of raw disk resources encompassed by the PV. Then, you can add that PVC to any pod spec as a volume, effectively allowing you to mount any kind of persistent storage medium to a particular workload. The separation of PV and PVC also allows you to fully control the lifecycle of your underlying block storage, including mounting it to different workloads or freeing it all together once the claim expires. StatefulSets manage the lifecycles of pods that require more stability than what exists in other primitives like Deployments and ReplicaSets. By creating a StatefulSet, you can guarantee that when you remove a pod, the storage managed by its mounted PVCs does not get deleted along with it. You can imagine how useful this property is if you're hosting a database! StatefulSets also allow for ordered deployment, scaling, and rolling updates, all of which create more predictability (and thus stability) in our workloads. This is also something that seems to go hand-in-hand with what you want out of your database's infrastructure. What Else? While StatefulSets, PVs, and PVCs do quite a bit of work for us, there are still many administration and configuration actions that you need to perform on a production-level database. For example, how do you orchestrate backups and restores? These can get quite complex when dealing with high-traffic databases that include functionality such as WALs. What about clustering and high availability? Or version upgrades? Are these operations zero-downtime? Every database deals with these features in different ways, many of which require precise coordination between components to succeed. Kubernetes alone can't handle this. For example, you can't have a StatefulSet automatically set up your average RDBMS in a read-replica mode very easily without some additional orchestration. Not only do you have to implement many of these features yourself, but you also need to deal with the ephemeral nature of Kubernetes workloads. To ensure peak performance, you have to guarantee that the k8s scheduler places your pods on nodes that are already pre-tuned to run your database, with enough free resources to run it properly. If you're dealing with clustering, how are you handling networking to ensure that database nodes are able to connect to each other (ideally in the same cloud region)? This brings me to my next point... Pets, Not Cattle Once you start accounting for things like node performance-tuning and networking, along with the requirement to store data persistently in-cluster, all of a sudden, your infrastructure starts to grow into a set of carefully groomed pet servers instead of nameless herds of cattle. But one of the main benefits of running your application in k8s is the exact ability to treat your infrastructure like cattle instead of pets! All of the most common abstractions like Deployments, Ingresses, and Services, along with features like vertical and horizontal autoscaling, are made possible because you can run your workloads on a high-level set of infrastructure components, so you don't have to worry about your physical infrastructure layer. These abstractions allow you to focus more on what you're trying to achieve with your infrastructure instead of how you're going to achieve it. Then Why Even Bother With K8s? Despite these rough edges, there are plenty of reasons to want to run your database on k8s. There's no denying that k8s' popularity has increased tremendously over the past few years across both startups and enterprises. The k8s ecosystem is under constant development so that its feature set continues to expand and improve regularly. And its operator model allows end users to programmatically manage their workloads by writing code against the core k8s APIs to automatically perform tasks that would previously have to be done manually. K8s allows for easy GitOps-style management so you can leverage battle-tested software development practices when managing infrastructure in a reproducible and safe way. While vendor lock-in still exists in the world of k8s, its effect can be minimized to make it easier for you to go multi-cloud (or even swap one for another). So what can we do if we want to take advantage of all the benefits that k8s has to offer while using it to host our database? What Do You Need to Build an RDS on K8s? Toward the end of the live chat, someone asked Kelsey, "what do you actually need to build an RDS on k8s?" He jokingly answered with expertise, funding, and customers. While we're certainly on the right track with these at QuestDB, I think that this can be better phrased in that you need to implement Day 2 Operations to get to what a typical managed database service would provide. Day 2 Operations Day 2 Operations encompass many of the items that I've been discussing; backups, restore, stop/start, replication, high availability, and clustering. These are the features that differentiate a managed database service from a simple database hosted on k8s primitives, which is what I would call a Day 1 Operation. While k8s and its ecosystem can make it very easy to install a database in your cluster, you're going to eventually need to start thinking about Day 2 Operations once you get past the prototype phase. Here, I'll jump into more detail about what makes these operations so difficult to implement and why special care must be taken when implementing them, either by a database admin or a managed database service provider. Stop/Start Stopping and starting databases is a common operation in today's DevOps practices and is a must-have for any fully-featured managed database service. It is pretty easy to find at least one reason for wanting to stop and start a database. For example, you may want to have a database used for running integration tests that run on a pre-defined schedule. Or you maybe have a shared instance that's used by a development team for live QA before merging a commit. You could always create and delete database instances on-demand, but it is sometimes easier to have a reference to a static database connection string and URL in your test harness or orchestration code. While stop/start can be automated in k8s (perhaps by simply setting a StatefulSet's replica count to 0), there are still other aspects that need to be considered. If you're shutting down a database to save some money, will you also be spinning down any infrastructure? If so, how can you ensure that this infrastructure will be available when you start the database backup? K8s provides primitives like node affinity and taints to help solve this problem, but everyone's infrastructure provisioning situation and budget are different, and there's no one-size-fits-all approach to this problem. Backup and Restore One interesting point that Kelsey made in his chat was that having the ability to start an instance from scratch (moving from a stopped -> running state), is not trivial. Many challenges need to be solved, including finding the appropriate infrastructure to run the database, setting up network connectivity, mounting the correct volume, and ensuring data integrity once the volume has been mounted. In fact, this is such an in-depth topic that Kelsey compares going from 0 -> 1 running instance to an actual backup-and-restore test. If you can indeed spin up an instance from scratch while loading up pre-existing data, you have successfully completed a live restore test! Even if you have restores figured out, backups have their own complexities. K8s provides some useful building blocks like Jobs and CronJobs, which you can use if you want to take a one-off backup or create a backup schedule, respectively. But you need to ensure that these jobs are configured correctly in order to access raw database storage. Or if your database allows you to perform a backup using a CLI, then these jobs also need secure access to credentials to even connect to the database in the first place. From an end-user standpoint, you need an easy way to manage existing backups, which includes creating an index, applying data retention policies, and RBAC policies. Again, while k8s can help us build out these backup-and-restore components, a lot of these features are built on top of the infrastructure primitives that k8s provides. Replication, HA, and Clustering These days, you can get very far by simply vertically scaling your database. The performance of modern databases can be sufficient for almost anyone's use case if you throw enough resources at the problem. But once you've reached a certain scale or require features like high availability, there is a reason to enable some of the more advanced database management features like clustering and replication. Once you start down this path, the amount of infrastructure orchestration complexity can increase exponentially. You need to start thinking more about networking and physical node placement to achieve your desired goal. If you don't have a centralized monitoring, logging, and telemetry solution, you're now going to need one if you want to easily diagnose issues and get the best performance out of your infrastructure. Based on its architecture and feature set, every database can have different options for enabling clustering, many of which require intimate knowledge of the inner workings of the database to choose the correct settings. Vanilla k8s know nothing of these complexities. Instead, these all need to be orchestrated by an administrator or operator (human or automated). If you're working with production data, changes may need to happen with close-to-zero downtime. This is where managed database services shine. They can make some of these features as easy to configure as a single web form with a checkbox or two and some input fields. Unless you're willing to invest the time into developing these solutions yourself, or leverage existing open-source solutions if they exist, sometimes it's worth giving up some level of control for automated expert assistance when configuring a database cluster. Orchestration For your Day 2 Operations to work as they would in a managed database service such as RDS, they need to not just work but also be automated. Luckily for us, there are several ways to build automation around your database on k8s. Helm and YAML Tools Won't Get Us There Since k8s configuration is declarative, it can be very easy to get from 0 -> 1 with traditional YAML-based tooling like Helm or cdk8s. Many industry-leading k8s tools install into a cluster with a simple helm install or kubectl apply command. These are sufficient for Day 1 Operations and non-scalable deployments. But as soon as you start to move into more vendor-specific Day 2 Operations that require more coordination across system components, the usefulness of traditional YAML-based tools starts to degrade quickly since some imperative programming logic is required. Provisioners One pattern that you can use to automate database management is a provisioner process. We've even used this approach to build v1 of our managed cloud solution. When a user wants to make a change to an existing database's state, our backend sends a message to a queue that is eventually picked up by a provisioner. The provisioner reads the message, uses its contents to determine which actions to perform on the cluster, and performs them sequentially. Where appropriate, each action contains a rollback step in case of a kubectl apply error to leave the infrastructure in a predictable state. Progress is reported back to the application on a separate gossip queue, providing almost-immediate feedback to the user on the progress of each state change. While this has grown to be a powerful tool for us, there is another way to interact with the k8s API that we are now starting to leverage... Operators K8s has an extensible Operator pattern that you can use to manage your own Custom Resources (CRs) by writing and deploying a controller that reconciles your current cluster state into its desired state, as specified by CR YAML spec files that are applied to the cluster. This is also how the functionality of the basic k8s building blocks are implemented, which just further emphasizes how powerful this model can be. Operators have the ability to hook into the k8s API server and listen for changes to resources inside a cluster. These changes get processed by a controller, which then kicks off a reconciliation loop where you can add your custom logic to perform any number of actions, ranging from simple resource existence to complex Day 2 Operations. This is an ideal solution to our management problem; we can offload much of our imperative code into a native k8s object, and database-specific operations appear to be as seamless as the standard set of k8s building blocks. Many existing database products use operators to accomplish this, and more are currently in development (see the Data on Kubernetes community for more information on these efforts). As you can imagine, coordinating activities like backups, restores, and clustering inside a mostly stateless and idempotent reconciliation loop isn't the easiest. Even if you follow best practices by writing a variety of simple controllers, with each managing its own clearly-defined CR, the reconciliation logic can still be very error-prone and time-consuming to write. While frameworks like Operator SDK exist to help you with scaffolding your operator and libraries like Kubebuilder provide a set of incredibly useful controller libraries, it's still a lot of work to undertake. K8s Is Just a Tool At the end of the day, k8s is a single tool in the DevOps engineer's toolkit. These days, it's possible to host workloads in a variety of ways, using managed services (PaaS), k8s, VMs, or even running on a bare metal server. The tool that you choose depends on a variety of factors, including time, experience, performance requirements, ease of use, and cost. While hosting a database on k8s might be a fit for your organization, it just as easily could create even more overhead and instability if not done carefully. Implementing the Day 2 features that I described above is time-consuming and costly to get right. Testing is incredibly important since you want to be absolutely sure that your (and your customers') precious data is kept safe and accessible when it's needed. If you just need a reliable database to run your application on top of, then maybe all of the work required to run a database on k8s might be too much for you to undertake. But if your database has strong k8s support (most likely via an operator), or you are doing something unique (and at scale) with your storage layer, it might be worth it to look more into managing your stateful databases on k8s. Just be prepared for a large time investment and ensure that you have the requisite in-house knowledge (or support) so that you can be confident that you're performing your database automation activities correctly and safely.