Cloud Architecture Resources

DZone's Featured Cloud Architecture Resources

Member Spotlight: Srinivas Chippagiri

By Dominique Pugh

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here. More

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

By Soham Sengupta

Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article. More

A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

By Xavier Portilla Edo

CORE

Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade

By Garima Agarwal

A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces

By Ahmed Tariq

Deploying Small Language Models on Sovereign Cloud

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

By Anuj Ashok Potdar

Closing the Drift Loop: An AI Agent That Drafts Terraform Fixes

Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket. Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal. The Gap The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer. The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review. I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live. Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched. Two Kinds of Drift When plan says a resource differs from state, there are two directions to go: DirectionMeaningActionRevertThe out-of-band change was wrong; code is the truthNo HCL change; apply the existing configCodifyThe out-of-band change was right; code is staleUpdate HCL to match reality, so apply produces no diff A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not. The Workflow Four stages. Only the second one is new. Stage 1: Detection A scheduled workflow runs a plan and captures a machine-readable output. YAML # .github/workflows/drift-detect.yml (abridged) on: schedule: [{ cron: "0 */6 * * *" }] jobs: plan: permissions: { id-token: write, contents: read, actions: write, issues: write } strategy: fail-fast: false matrix: include: # CloudTrail LookupEvents is regional; the agent runs where the resource lives - { stack: network/prod-use1, region: us-east-1 } - { stack: network/prod-use2, region: us-east-2 } steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.DRIFT_READONLY_ROLE } # reads + lock-table write; plan holds the lock aws-region: ${{ matrix.region } - name: Plan (refresh-only) id: plan run: | set +e terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \ -out=drift.tfplan -input=false code=$?; set -e echo "code=$code" >> "$GITHUB_OUTPUT" [ "$code" -eq 1 ] && exit 1 # plan error: fail loudly terraform show -json drift.tfplan > drift.json - name: Count drifted resources # refresh-only plans report under resource_drift id: count run: | n=$(python3 -c "import json; d=json.load(open('drift.json')); print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))") echo "n=$n" >> "$GITHUB_OUTPUT" - name: Drift-storm guard # >20 drifted resources is a provider/state problem, not drift if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20 run: gh issue create --title "[drift] storm in ${{ matrix.stack }" --body "Paging a human." - name: Dispatch remediation if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20 run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack } -f region=${{ matrix.region } -f run_id=${{ github.run_id } Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent. Stage 2: The Remediation Agent The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply. YAML # .github/workflows/drift-remediate.yml (abridged) on: workflow_dispatch: inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } } jobs: remediate: permissions: { contents: write, pull-requests: write, id-token: write, actions: read } steps: - uses: actions/download-artifact@v4 # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step with: { name: drift-${{ inputs.stack }, run-id: ${{ inputs.run_id }, github-token: ${{ secrets.GITHUB_TOKEN } } - uses: aws-actions/configure-aws-credentials@v4 with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }, aws-region: ${{ inputs.region } } - name: Analyze drift # model + AWS read access; no GitHub write token here run: python agent/remediate.py analyze --stack "${{ inputs.stack }" --region "${{ inputs.region }" --plan drift.json --out proposal/ env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL } } - name: Open PR # bot token only; see below for why not GITHUB_TOKEN run: python agent/remediate.py propose --stack "${{ inputs.stack }" --proposal proposal/ env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN } } The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” The tool surface is small, and every tool is read-only except the last: Python TOOLS = [ read_plan_json, # resource_drift entries from drift.json read_hcl, # repo files, capped per file read_tags, # tags on the drifted ARN cloudtrail_lookup, # LookupEvents in the stack's region, last 14 days describe_resource, # aws ec2/rds/iam describe-* search_issues, # GitHub issues/PRs mentioning the resource ID propose_fix, # the ONLY write: stages a branch + PR body under proposal/ ] Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs. Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code. The system prompt sets the contract: You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like. That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible. Stage 3: The PR as the Gate Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform: [drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band Recommendation: Codify (evidence: strong) Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state. Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG. Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr. Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that. If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring. The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges. The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim: YAML - name: Plan on PR run: | set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false echo "code=$?" >> "$GITHUB_OUTPUT"; set -e terraform show -json pr.tfplan > pr.json - name: Judge only the claimed resource # codify: the claimed resource is no-op/absent in pr.json # revert: the claimed fingerprint still appears under resource_drift run: python agent/gate.py --plan pr.json - name: Policy check run: conftest test pr.json -p policy/ # Branch protection: 1 approval + Code Owners review. The bot can't approve # its own PR, and "Allow GitHub Actions reviews to count" stays off. Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to. Stage 4: Apply (Unchanged) On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point. Guardrails One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one. That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent. Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one. Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops. Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up. Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code. Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent. What This Doesn’t Solve It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps. Conclusion The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply.

By Sidharth Shah

MQTT, Kafka, NATS, or Zenoh? Choosing Edge-to-Cloud Messaging by Traffic Pattern

Edge to cloud is not one integration problem. It is four: telemetry going up, control going down, sites syncing sideways, and data reaching people. Each has different physics. Most failed projects picked one technology for all four. Every enterprise with factories, stores, vehicles, hospitals, or devices faces the same architectural question. How does data move between the edge and the data center or cloud, in both directions, reliably, securely, and at acceptable cost? The vendor landscape answers with dozens of overlapping options: MQTT platforms, hyperscaler services, event streaming, event mesh, messaging fabrics, delivery platforms, and protocol proxies. This article structures the option space by traffic pattern instead of by product category, maps the vendor categories onto those patterns, and closes with a decision framework you can apply to your own architecture. The categories here sit alongside the wider integration landscape rather than inside it. For that broader map across event streaming, API, and batch, see the Data Integration Landscape 2026. What Does Edge-to-Cloud Communication Mean? Edge-to-cloud communication is the movement of data between distributed physical locations and centralized computing environments, in both directions, across networks the architect does not control. The word edge hides four very different tiers: The device edge means microcontrollers, sensors, and actuators with kilobytes of memory.The gateway edge means industrial PCs, vehicle computers, and protocol converters.The site edge means a factory, store, hospital, or ship with local compute and its own network boundary.The near edge means regional data centers close to many sites. A technology that fits one tier often fails at another. Edge to cloud splits into four traffic patterns: northbound telemetry, southbound control, site-to-site sync, and outbound delivery to people. The traffic crossing these tiers falls into four patterns: Northbound telemetry moves sensor readings, machine states, transactions, and logs from the edge toward central platforms for analytics, AI, and business systems.Southbound control moves commands, configuration, software updates, and increasingly AI models from central platforms down to machines and gateways.Site-to-site traffic synchronizes data between sites, between a site and a core data center, or between clouds.Outbound delivery pushes live data from central platforms to the applications people use: dashboards, mobile apps, and trading screens. Direction plus consumer type plus constraint determines the right technology. Vendor preference does not. Why Do Network Constraints Decide the Edge-to-Cloud Architecture? The network between edge and cloud is the one component you cannot refactor. Its properties dictate the architecture before any product evaluation starts. Connectivity is unreliable. Ships, vehicles, and remote sites go offline for hours or days. Any viable architecture is offline first: local operation continues, data buffers locally, and synchronization resumes when the link returns. This is why store and forward is the single most important capability on any edge evaluation checklist. Inbound connections are forbidden. Industrial sites, retail networks, and vehicle fleets sit behind NAT, firewalls, and cellular carriers. Security teams do not open inbound ports for messaging. Every serious edge technology therefore uses the same trick: the edge dials out. MQTT clients dial out to brokers. NATS leaf nodes dial out to hub clusters. Azure IoT Operations dials out from the factory to the cloud broker. Different stacks, identical answer, because the constraint is identical. Hardware is constrained and heterogeneous. An 8-bit microcontroller, a fanless gateway PC, and a site server need different footprints. Some technologies scale down to bare microcontrollers. Others assume a Kubernetes cluster. Matching footprint to tier is half the selection process. Footprint is not the only axis. In fixed installations like plants and warehouses, the device edge can stay thin, because the uplink is reliable and most intelligence lives upstream. In the field, agriculture, cranes, rural grid, remote or contested sites, the uplink is wireless and often degraded, saturated, expensive, or absent for days. There, the edge cannot just buffer and forward. It has to run enough local intelligence, pattern matching, image recognition, autonomous decisions, to operate correctly on its own and sync opportunistically when the link returns. That pushes compute onto the device tier for reasons that have nothing to do with protocol size, and it changes the pattern, not just the product. Data residency is now an architecture requirement, not a legal afterthought. European manufacturers increasingly must prove what data leaves the site, country, or jurisdiction. Topic design, routing rules, and retention policies become compliance controls. And one constraint is usually invented rather than measured: latency. Teams assert real-time requirements that the business case does not contain. A large share of edge traffic is perfectly fine as periodic batch over a metered link. Keep that thought. It returns in the site-to-site section. Northbound: How Does Telemetry Get from OT and Devices into the Core? Northbound telemetry is the pattern the industry knows best, and the one with the most settled consensus. At the machine level, OPC UA provides the semantic information model. It defines what a temperature, a batch, or a machine state means. MQTT provides the efficient transport, and Sparkplug adds topic structure, birth and death certificates, and state awareness on top of MQTT. The two standards stopped being rivals. OPC UA Pub/Sub over MQTT is the convergence point: OPC UA semantics carried over MQTT infrastructure. Edge gateways and industrial DataOps products, for example HighByte or Ignition with its MQTT modules, convert brownfield protocols like Modbus and S7 into that stack. The Unified Namespace (UNS) pattern then organizes all site data as a live, hierarchical topic structure on a central broker. I covered UNS design and its trade-offs in depth in Unified Namespace vs. Data Product in IT/OT for Industrial IoT and the broader MQTT market in MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration, so this section stays short. The real decision is the broker and its operating model. Self-managed platforms carry the northbound load for most industrial estates: HiveMQ and EMQX at the enterprise end, Mosquitto and NanoMQ where a lightweight or embedded footprint matters. They give you deployment control, protocol depth, and edge variants that run on gateways. Managed cloud services remove the operations burden and integrate natively with cloud analytics. The dividing line is usually sovereignty and site autonomy: if the broker must run inside the factory, on the ship, or in a specific jurisdiction, self-managed wins. If all sites have reliable connectivity and the data may leave, managed wins on cost of operations. The hyperscalers themselves tell an asymmetric story. Google retired Cloud IoT Core in 2023 and exited the category. AWS IoT Core remains the steady incumbent for managed device connectivity. Azure went furthest architecturally: Azure IoT Operations runs a distributed MQTT broker on Arc-enabled Kubernetes at the site edge and bridges it to the Event Grid MQTT broker in the cloud, which now supports Sparkplug natively. That is a symmetric edge plus cloud MQTT pair from a single vendor, and it is a new pattern. One hyperscaler retreated, one held, one doubled down. The lesson for architects: managed convenience is real, but so is the risk that a cloud provider deprecates your ingestion layer. The open standards underneath, MQTT and OPC UA, are the insurance policy. Choose products that speak them, and migration remains possible. Southbound and Outbound: How Do Data and Commands Flow Back to Machines and People? Almost all edge content covers ingestion. The return direction is harder, and it is where architectures break, because the return direction has two completely different consumers. Machines need commands and state. Request-response over pub/sub topics handles commands: reboot this gateway, close this valve, change this setpoint. Retained messages and device twins handle state: a device that reconnects must immediately learn its desired configuration without waiting for the next command. Fleet-wide rollout needs staged delivery, and the payloads keep growing. Configuration files became firmware images, and firmware images are becoming AI models. Shipping a quantized model to ten thousand gateways is a southbound data movement problem, not an MLOps footnote. The same fabric increasingly carries traffic for agents at the edge: AI agents that perceive local data, act locally, and coordinate through the messaging layer. NATS and the MQTT platforms are both positioning for exactly this workload. People need the current picture. A plant manager opening a dashboard, an operator reconnecting after a network drop, and a trader loading a screen mid-session all have the same requirement: current state on connect, then live updates. A log is not a user interface. Machines just consume the stream. People do not. This is the last-mile delivery problem, and it has its own vendor category: Lightstreamer, Ably, and PubNub sit between the central platform and many concurrent browsers and mobile apps, handling fan-out, per-client throttling, and entitlement checks. The category is maturing fast. For instance, Lightstreamer's Kafka Connector 2.0, released in July 2026, moved snapshot management into the connector itself, so every reconnecting client receives materialized state followed by the live stream. Delivery to humans deserves a dedicated article. Here it earns its place as the fourth traffic pattern that most edge architectures forget until the first user complains. Site-to-Site: How Do Sites, Data Centers, and Clouds Stay in Sync? The third direction is sideways: keeping data consistent between many sites and one core, or between multiple clouds. For durable event streams, Kafka-style replication is the default. Open source MirrorMaker 2 or vendor-specific cluster linking copies topics between an edge cluster and the central cluster, preserving ordering and history. Apache Kafka is the open-source option; Confluent, Amazon MSK, Aiven, and other solutions are the managed ones. This suits sites large enough to operate a streaming cluster, such as a factory with local analytics that also feeds the global platform. For thousands of small sites, the streaming cluster per site model collapses under its own operational weight. This is where NATS is architecturally distinct rather than just another broker. The entire server is a single small binary with no external dependencies. Each store, vehicle, or clinic runs a leaf node that dials out to the central cluster, keeps serving local applications when the WAN drops, and uses JetStream mirroring to synchronize persisted streams when connectivity returns. Retail chains run this pattern across thousands of locations. The same design serves both the intermittent case, hours or days offline with a bulk sync on reconnect, and the scheduled-batch case where sites replicate on a timer rather than continuously. Its own maintainers warn against one trap: do not stretch a single stream cluster across unstable edge links, because the cluster's consensus breaks every time the network does. Instead, let each site persist its data locally and keep serving through an outage, then mirror to the center asynchronously when the link returns. For enterprises whose sites run applications rather than device fleets, an event mesh covers the same ground at the application layer. Solace links its brokers across data centers and clouds so an event published anywhere reaches any interested subscriber, with topic routing and governance built in. It is the natural fit when the endpoints are SAP, mainframes, and microservices rather than sensors. And a large share of site-to-site traffic does not need streaming, even if streaming is often the stronger long-term architecture. Nightly file synchronization, batch uploads over metered links, and change data capture from edge databases remain perfectly valid when there is no real-time requirement. This is especially true when the existing OT and IT systems on both ends are batch-oriented anyway, and no business case justifies the cost and effort of making them real time. Real time is a requirement, not a default. Stream when you need fresh data, or when you build greenfield and carry no legacy batch systems to preserve. An architecture that streams everything pays streaming costs for batch problems. Which Category Serves Which Edge to Cloud Pattern? Seven vendor categories map onto the four patterns. No single category covers all of them, and messaging fabrics, event mesh, and protocol proxies deliberately span several. No single vendor covers every direction. Seven categories divide the field, and the real picture is that some sit in one pattern while others deliberately span several. MQTT platforms (EMQX, HiveMQ, Mosquitto, NanoMQ) serve both northbound telemetry and southbound control, because MQTT is bidirectional by design. Hyperscaler services (AWS IoT Core, Azure IoT Operations) do the same for the managed-cloud case, ingesting telemetry and pushing desired state through device twins. Event streaming (Apache Kafka, Confluent, Aiven, and more in the data streaming landscape) owns site-to-site replication. Last-mile delivery(Ably, Lightstreamer, PubNub) owns outbound delivery to people, and it is the only pattern with a category built solely for it. Three categories span multiple patterns. An event mesh (Solace) routes site-to-site and out to applications. Messaging fabrics (NATS, Zenoh) carry northbound, southbound, and site-to-site in a single fabric. Protocol proxies (Zilla) translate across all four, replacing several specialized components with one layer. The underlying distinction is worth stating plainly: a protocol standard is not a deployment architecture. MQTT standardizes the wire, which is why so many products speak it and your devices stay portable. NATS and Zenoh standardize the topology, which reduces moving parts and increases commitment. Zenoh, born from DDS research and adopted by automotive initiatives including General Motors' uProtocol and by the ROS 2 robotics ecosystem, goes furthest: it routes named data along the shortest path between producer and consumer, aiming to make the cloud versus edge placement debate irrelevant. These spanning options are not equivalent. A fabric like NATS or Zenoh reduces moving parts and unifies security by being one technology end to end, but that reach is also the lock-in: the more it carries, the more expensive it is to leave. A protocol proxy like Zilla does the opposite. It spans the patterns by translating between existing technologies, which keeps your exit options open at the cost of another component in the path. The right choice depends on how many patterns you actually need and how much lock-in you can accept, not on which vendor markets hardest. How to Choose: Five Questions That Pick Your Edge-to-Cloud Architecture Product evaluations start too late. Answer five questions first, and the shortlist mostly writes itself. Five-question decision framework for choosing an edge-to-cloud architecture, from consumer type to data residency Who consumes the data, machines or people? People need current state on connect, then live updates, which means a last-mile delivery layer over the backbone. Machines just consume the stream.Does this flow actually need real time? Measure before you architect. If a fifteen-minute batch satisfies the business case, use file sync or change data capture. Stream when you need fresh data, because streaming infrastructure you did not need is the most expensive kind.What is the most constrained node in the path? A microcontroller means MQTT or Zenoh, the two with clients small enough for the device itself, not a JVM. A gateway opens the field to lightweight NATS or full brokers, and a site server adds event meshes or a small streaming cluster.How many sites, and can each one operate a cluster? A few large sites with local analytics justify edge Kafka clusters with MirrorMaker or cluster linking. Hundreds or thousands of small sites demand hub-and-spoke: NATS leaf nodes or hierarchical MQTT bridges, with store and forward at every spoke. Enterprises whose sites run applications rather than devices fit an event mesh.Where must the data live? If residency or sovereignty rules apply, the broker runs on site or in jurisdiction, self-managed, and routing policy becomes a compliance control. If data may leave and connectivity is solid, managed cloud services win on operations cost. Two checks apply on almost every branch, at every step. The edge must dial out and never accept inbound connections to the site. And local operation must survive offline, with a buffer sized deliberately, not discovered in the first outage. Five questions, four patterns, one map. The technologies then sort themselves into it. Moving Data Is Not Automating the Business Every pattern and category in this article solves the same problem: moving data between edge, data center, and cloud. That movement is necessary but not sufficient. When telemetry arrives, something must decide what happens next: trigger the maintenance workflow, roll back the failed firmware deployment, replicate the dataset, open the ticket, retrain the model. Deciding what happens next is workflow orchestration. It spans the same edge-to-cloud boundary as the data movement beneath it. Data integration moves the events. Orchestration, with the process intelligence to see how those flows actually run, decides what the business does about them. And increasingly, agentic AI acts on those decisions at the edge. These are not three separate topics. They are one stack, and edge to cloud is where they meet the physical world. Architectures that move data and improvise everything above it end up with the automation logic scattered across broker plugins, cron jobs, and cloud functions. That is the integration spaghetti this whole discipline set out to remove. Edge to cloud is not one problem. It is four. Architect for the pattern, not the product.

By Kai Wähner

CORE

Vision-Language Models in 2026: From Research Breakthroughs to Cloud-Native Production Systems

Over the past few years, vision-language models (VLMs) have moved from experimental research projects to early production use. These systems take in both images and text, so they can read documents, interpret photographs, answer questions about visual content, and reason across modalities in ways that were not practical a few years ago. Better architectures and training methods explain part of this progress. Better ways of deploying models explain at least as much. In 2026, the defining question for enterprises is no longer whether VLMs work, but whether they can run reliably, safely and cost-effectively in production. The answer depends as much on cloud infrastructure, observability and governance as on the models themselves. This article traces how VLMs evolved, why cloud computing became the turning point for adoption, where reliability still falls short, and which cloud-native practices organizations need in order to deploy them responsibly at scale. From Research Breakthroughs to Foundation Models The modern VLM era began with CLIP (Contrastive Language–Image Pretraining). It showed that training on hundreds of millions of image-text pairs could align visual and textual representations in a shared embedding space. That alignment enabled zero-shot image classification and made large-scale multimodal search possible. DeepMind's Flamingo built on this foundation. It introduced cross-attention layers that connect a frozen vision encoder to a large language model, so a single model could learn new tasks from only a few examples. BLIP-2 then showed that a lightweight querying module could bridge frozen image encoders and frozen language models, which sharply reduced training cost. Instruction tuning was the next step. LLaVA (Large Language and Vision Assistant) showed that fine-tuning on visual instruction data produced conversational assistants that could follow complex multimodal prompts. OpenAI's GPT-4V brought these capabilities to a mass audience through a commercial product. Open-weight models such as Qwen2-VL later extended high-resolution perception to the open-source community. Taken together, these advances in cross-attention mechanisms, instruction fine-tuning, and large-scale multimodal pretraining allowed VLMs to combine perception and reasoning in a single system. The Cloud as the Deployment Turning Point The research milestones were essential, but the real turning point for adoption has been cloud deployment. VLMs are computationally demanding: they need high-memory GPUs, efficient batching, and low-latency serving. Few organizations can justify building that infrastructure in-house for workloads that are still evolving. Cloud platforms solve this problem. They offer GPU-backed compute on demand, managed model endpoints, and API-based access that fits directly into existing applications. As a result, VLMs now support practical enterprise workflows: Document intelligence: extracting data from invoices, contracts, and formsVisual inspection: spotting defects in manufacturingMultimodal search: retrieving content by combining images with natural-language queries Elastic scaling also lets organizations match capacity to demand, so they don't pay for idle hardware during quiet periods. Benchmarks Reveal Uneven Reliability Despite these advances, VLM reliability remains inconsistent. MMBench evaluates models across a wide range of perception and reasoning skills, and its results show large differences in performance from task to task. The VERIFY benchmark focuses on visual reasoning fidelity, testing whether models reason correctly about what they see rather than simply producing plausible answers. It exposes significant gaps even in leading systems. Industry reviews of recent computer vision research, including Voxel51's analysis of ICCV 2025 papers, point to the same ongoing weaknesses in fine-grained perception and consistency. A model that does well on one benchmark can still struggle on a closely related task. The practical lesson is clear: choose models based on the specific use case and workload, not on general leaderboard rankings. Choosing Between Open-Source and Managed Models Organizations face a strategic choice between open-source and managed VLMs. Open-source models such as LLaVA and Qwen2-VL can run on Kubernetes clusters or in a private cloud. They give full control over data residency, customization, and cost, which suits regulated industries and sensitive data. The trade-off is that they require in-house expertise in GPU orchestration, model serving, and ongoing maintenance. Managed services offer elastic scaling, built-in security, and much less operational complexity, at the cost of less control and possible vendor lock-in. Many enterprises use a hybrid approach. They route sensitive workloads to self-hosted models and send general-purpose tasks to managed endpoints. Hidden Failure Modes at Scale A major concern in production is hidden failure modes. The most widely discussed is hallucination, where a model describes objects or details that are not in the image, and research on object hallucination shows it remains a persistent problem. Other risks include: OCR errors on low-quality scansSmall objects being missed in dense scenesSpatial reasoning failures, such as confusing left and right or misjudging relative positions Inside automated pipelines, these errors are especially dangerous. One hallucinated invoice total or missed defect can repeat across thousands of transactions before anyone notices. At scale, small error rates add up to significant business and compliance risk. Cloud-Native Guardrails for Production To manage these risks, organizations need guardrails built into their cloud-native architecture: Confidence thresholds send uncertain outputs for further review instead of accepting them automatically.Human-in-the-loop review gives a person oversight of high-stakes decisions.Retrieval-based grounding ties model outputs to verified source data, which reduces hallucination.Audit logging records inputs, outputs, and decisions for traceability and compliance.API gateway policies enforce rate limits, access controls, and content rules consistently across applications. Frameworks such as the NIST AI Risk Management Framework give a useful structure for applying these controls. They help organizations map, measure, and manage AI risk across the lifecycle. Observability tools that track drift, latency, and error patterns complete the picture, so teams can detect problems before they spread. These controls are what move VLMs from impressive demonstrations to dependable production systems. Conclusion The VLM story is ultimately about systems as much as models. Breakthroughs from CLIP to Qwen2-VL made multimodal understanding possible, but cloud computing made it practical. Combining multimodal AI with cloud-native infrastructure has let organizations build AI workflows that are scalable, observable, and governed. VLMs are not yet perfect. Benchmarks show uneven reliability, and hallucination remains a real risk. Even so, they have matured enough for teams to deploy them at scale, provided those teams take careful model selection, rigorous evaluation, and strong guardrails seriously. The organizations that succeed in 2026 will treat VLMs as one part of a well-architected system with governance built in from the start.

By Srinivas Chippagiri

CORE

Google Cloud KMS vs. AWS KMS: Java Encryption Guide

Every system that stores sensitive data needs keys that never leave a hardened boundary, an audit trail of every use, and rotation that doesn't mean re-encrypting petabytes. On Google Cloud, that's Cloud KMS; on AWS, it's AWS KMS. Both store symmetric and asymmetric keys, both use HSMs, and both are built for envelope encryption. But AWS KMS vs. Google Cloud KMS comes down to the choices that show up in your code: how keys are named, what rotation means, where authorization lives, and whether the service hands you data keys. This article walks through Cloud KMS in Java and then maps it onto AWS KMS. The Cloud KMS Resource Model Cloud KMS organizes keys in a four-level hierarchy under a project: Location. Where the key material lives: a region such as us-east1, a multi-region such as us, or global. Key material stays in the location you pick for its whole lifecycle.Key ring. A named group of keys in one location. It exists for organization and IAM inheritance.CryptoKey. The key you reference in code. It has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, and a few more) and a rotation schedule.CryptoKeyVersion. The actual key material. A key has one or more versions, and for symmetric keys one of them is the primary, the version that new encryptions use. A symmetric key with three versions: new data uses version 3, data written under version 2 still decrypts, and version 1 is on its way out. A key is identified only by its full resource name, projects/my-project/locations/us-east1/keyRings/payments/cryptoKeys/card-data, optionally followed by /cryptoKeyVersions/2. There are no aliases. Nothing can move after it's created: the location is fixed when you create the key ring, and key rings themselves can't be deleted. Each key version also has a protection level: SOFTWARE: keys are protected by Google's software crypto (BoringCrypto, FIPS 140-3 Level 1).HSM: keys live in Cloud HSM, which is FIPS 140-2 Level 3 validated. HSM_SINGLE_TENANT gives you a dedicated HSM.EXTERNAL and EXTERNAL_VPC: Cloud EKM keeps the key material in a supported external key manager that you operate, reached over the internet or a VPC. Google never holds the key. Creating a Key That Rotates Itself The Java client is KeyManagementServiceClient. It uses Application Default Credentials, so on GKE, Cloud Run, or GCE, the workload's service account authenticates with no keys in the code. This creates a key ring and an HSM-backed symmetric key that rotates every 90 days: Java try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { KeyRing ring = kms.createKeyRing( LocationName.of("my-project", "us-east1"), "payments", KeyRing.getDefaultInstance()); CryptoKey key = CryptoKey.newBuilder() .setPurpose(CryptoKeyPurpose.ENCRYPT_DECRYPT) .setVersionTemplate(CryptoKeyVersionTemplate.newBuilder() .setAlgorithm(CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION) .setProtectionLevel(ProtectionLevel.HSM)) .setRotationPeriod(Duration.newBuilder().setSeconds(90L * 24 * 3600)) // every 90 days .setNextRotationTime(Timestamp.newBuilder() .setSeconds(Instant.now().plusSeconds(90L * 24 * 3600).getEpochSecond())) .build(); kms.createCryptoKey(KeyRingName.parse(ring.getName()), "card-data", key); } The rotation period can be anywhere from one day to 100 years. When it fires, Cloud KMS creates a new version and makes it primary. Earlier versions stay enabled, so old ciphertext still decrypts, and nothing is re-encrypted. To rotate by hand, create a version with CreateCryptoKeyVersion and promote it with UpdateCryptoKeyPrimaryVersion. Automatic rotation applies only to symmetric keys; asymmetric keys are rotated manually. The AWS KMS key rotation process differs in how key material is managed, as described below. Envelope Encryption KMS keys aren't meant to encrypt your data directly. A single Encrypt call accepts at most 64 KiB of plaintext (8 KiB of plaintext plus AAD for HSM keys), and every call is a network round trip. The standard pattern is envelope encryption, also used with AWS KMS envelope encryption: encrypt the data locally with a data encryption key (DEK), then have KMS encrypt (wrap) the DEK with a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. Envelope encryption: The payload never goes to KMS; only the 32-byte DEK does. Cloud KMS has no data-key API, so you generate the DEK yourself. Google's guidance is to generate a new DEK every time you write data, never share one across users, and prefer Tink, which packages this whole flow. Doing it by hand shows what's happening: Java CryptoKeyName kek = CryptoKeyName.of("my-project", "us-east1", "payments", "card-data"); ByteString aad = ByteString.copyFromUtf8("tenant=acme"); // must be supplied again to decrypt try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { // 1. Generate a fresh DEK locally. Cloud KMS has no GenerateDataKey call. KeyGenerator gen = KeyGenerator.getInstance("AES"); gen.init(256); SecretKey dek = gen.generateKey(); // ... encrypt the payload locally with dek (AES-GCM), then discard dek ... // 2. Wrap the DEK with the KEK, with CRC32C checks against corruption in transit. ByteString dekBytes = ByteString.copyFrom(dek.getEncoded()); EncryptResponse wrapped = kms.encrypt(EncryptRequest.newBuilder() .setName(kek.toString()) .setPlaintext(dekBytes) .setPlaintextCrc32C(crc32c(dekBytes)) .setAdditionalAuthenticatedData(aad) .setAdditionalAuthenticatedDataCrc32C(crc32c(aad)) .build()); if (!wrapped.getVerifiedPlaintextCrc32C() || wrapped.getCiphertextCrc32C().getValue() != crc32c(wrapped.getCiphertext()).getValue()) { throw new IllegalStateException("integrity check failed"); } ByteString wrappedDek = wrapped.getCiphertext(); // store next to the encrypted payload String usedVersion = wrapped.getName(); // the primary version that wrapped it // 3. Later: unwrap. Pass the key, not the version; Cloud KMS picks the version. ByteString unwrapped = kms.decrypt(DecryptRequest.newBuilder() .setName(kek.toString()) .setCiphertext(wrappedDek) .setAdditionalAuthenticatedData(aad) .build()) .getPlaintext(); SecretKey restored = new SecretKeySpec(unwrapped.toByteArray(), "AES"); } static Int64Value crc32c(ByteString data) { return Int64Value.of(Hashing.crc32c().hashBytes(data.toByteArray()).padToLong()); } Encrypt uses the key's current primary version. Decrypt also takes the key, not a version: the ciphertext records which version produced it, so rotation is invisible to readers. The optional CRC32C fields let the server confirm it received your plaintext intact, and let you confirm the ciphertext you got back. The AAD (tenant=acme) is bound to the ciphertext, so one tenant's wrapped DEK can't be swapped into another tenant's record. Disabling, destroying, and deleting Lifecycle operations act on versions, not keys. You disable a version (ENABLED to DISABLED) to stop it from being used, and re-enable it later. There's no switch that disables a whole key; you disable each version. Destroying a version is deliberately slow. DestroyCryptoKeyVersion moves it to DESTROY_SCHEDULED for 30 days by default. The duration is set when the key is created, and an organization policy can enforce a minimum. During that window, RestoreCryptoKeyVersion brings it back in the DISABLED state. After the window, the material is gone and data encrypted by that version can't be recovered. Destroyed versions are no longer billed. Cloud KMS has also added permanent deletion: you can delete key versions and then the key itself, though a deleted key's name can't be reused. Access control is just IAM Cloud KMS has no policy language of its own. Access is ordinary IAM, granted at the organization, folder, project, key ring, or key level and inherited downward. The predefined roles split duties cleanly: roles/cloudkms.admin manages keys but can't encrypt or decrypt.roles/cloudkms.cryptoKeyEncrypterDecrypter, cryptoKeyEncrypter, and cryptoKeyDecrypter can use keys but not manage them.roles/cloudkms.signerVerifier and publicKeyViewer cover asymmetric keys. The operator who rotates keys never sees plaintext, and the service that decrypts can't destroy its key. Grant usage roles on the specific key, not the project. Two more controls round this out. Cloud Audit Logs record admin operations (create, rotate, destroy) automatically, but Encrypt and Decrypt are Data Access logs, which are off by default. If you want a record of every key use, enable them for Cloud KMS. Key Access Justifications, used with Cloud EKM keys, attaches a reason code to every Google request for your key, and your external key manager can refuse it. Bringing your own key To import key material, you create an import job with a wrapping method, such as RSA_OAEP_3072_SHA256_AES_256 or one of the newer post-quantum HPKE methods. The job gives you a public key, and it expires after three days. You wrap your key material with that public key and import it as a new version of a key. For a key that must only ever hold imported material, set importOnly, which means rotation is manual: import new material and make it primary. If you destroy an imported version, you can re-import the same material later. AWS KMS for comparison AWS KMS covers the same ground with a flatter, version-less model: Keys, not versions. A KMS key has a key ID (a UUID), an ARN, and optional aliases such as alias/card-data. There's no hierarchy above it beyond the account and Region.Rotation is hidden. Automatic rotation is off by default for customer managed keys. When enabled, it runs every 365 days by default, configurable from 90 to 2,560 days, and RotateKeyOnDemand allows up to 25 extra rotations per key. The key ID never changes; AWS keeps old material internally and picks it on decrypt.All keys in HSMs. Keys in the default key store live in HSMs validated at FIPS 140-3 Level 3. There's no software tier. CloudHSM key stores and external key stores (XKS) cover the dedicated and hold-your-own-key cases.A data-key API. GenerateDataKey returns a fresh DEK in plaintext along with a copy wrapped under your key, so step 1 and step 2 above are one call.Encryption context. The AWS equivalent of AAD is a map of key-value pairs. Unlike Cloud KMS AAD, it's recorded in CloudTrail and can be used in policy conditions such as kms:EncryptionContext:tenant.Three authorization layers. Every key has exactly one key policy, a resource policy that supports Deny and conditions. IAM policies work only if the key policy enables them, and grants add temporary, programmatic permissions on top.Whole-key lifecycle. DisableKey and ScheduleKeyDeletion act on the whole key. The deletion window is 7 to 30 days, and CancelKeyDeletion backs it out.Replication. Multi-Region keys share the same key ID and material across Regions, and ReplicateKey copies one to a new Region. You choose this at creation; a single-Region key can't be converted. Here's the same envelope flow on AWS: Java Map<String, String> context = Map.of("tenant", "acme"); // logged in CloudTrail, usable in policy try (KmsClient kms = KmsClient.create()) { GenerateDataKeyResponse dataKey = kms.generateDataKey(GenerateDataKeyRequest.builder() .keyId("alias/card-data") .keySpec(DataKeySpec.AES_256) .encryptionContext(context) .build()); byte[] dek = dataKey.plaintext().asByteArray(); // encrypt locally, then discard SdkBytes wrappedDek = dataKey.ciphertextBlob(); // store next to the payload byte[] unwrapped = kms.decrypt(DecryptRequest.builder() .keyId("alias/card-data") // optional for symmetric keys .ciphertextBlob(wrappedDek) .encryptionContext(context) .build()) .plaintext().asByteArray(); } KeyId is optional on Decrypt for symmetric keys because the ciphertext blob identifies the key, but AWS recommends passing it so you don't decrypt with a key you didn't expect. Cloud KMS requires the key name on every Decrypt. Side by side ConceptGoogle Cloud KMSAWS KMSKey identifierResource name projects/…/cryptoKeys/k, no aliasesKey ID, ARN, aliasesVersionsFirst-class CryptoKeyVersion with its own stateHidden behind the key IDProtectionSOFTWARE, HSM (FIPS 140-2 L3), single-tenant HSM, EXTERNAL, EXTERNAL_VPCAll HSM-backed (FIPS 140-3 L3), plus CloudHSM and external key storesMax plaintext per call64 KiB (8 KiB with AAD for HSM keys)4,096 bytesData keysGenerate locally (or with Tink)GenerateDataKeyAuthenticated dataAAD bytes, not loggedEncryption context map, logged and usable in policiesAuto-rotation1 day to 100 years90 to 2,560 days, off by default, plus 25 on-demandDisablePer versionWhole keyDestroyPer version, 30 days scheduled by default, restorableWhole key, 7–30 days, cancellableAccess controlIAM roles on the hierarchy, admin and usage splitKey policy (with Deny) + IAM + grantsUsage loggingData Access audit logs, off by defaultCloudTrail management eventsMulti-regionChoose a multi-region location for the key ringMulti-Region keys, ReplicateKeyBYOKImport job, expires in 3 daysGetParametersForImport, token valid 24 hoursThroughput quota (default)Token-based per region; about 60,000 software operations a minute10,000 to 100,000 symmetric operations a second, by Region What the AWS KMS vs. Google Cloud KMS encryption comparison means in practice: Cloud KMS is simpler to reason about for access. It's one IAM model with roles that separate managing keys from using them. AWS is more expressive: a key policy can Deny, and conditions on encryption context can tie a key's use to a tenant. That power comes with the risk of a key policy that locks out its own administrators.Versions are explicit on Google Cloud. You can disable or destroy just the key material that was compromised and leave the rest working. On AWS you disable or delete the whole key, or rotate and keep the old material.AWS logs every use by default. On Google Cloud, enable Data Access logs for Cloud KMS on day one, or you won't have a record of who decrypted what.Plan for quotas. AWS gives each account and Region a large shared pool of cryptographic operations. Cloud KMS's default is far smaller per project and region, so cache unwrapped DEKs for hot reads where your threat model allows, and request a quota increase before launch. Both quotas are adjustable.Both want envelope encryption. Neither service is built to encrypt bulk data directly. On Google Cloud, Tink makes the local half trivial. On AWS, the AWS Encryption SDK plays the same role. Conclusion Google Cloud KMS treats keys as resources in your project hierarchy: versions are visible, protection level is chosen per key, and IAM is the whole authorization story. That's easy to audit and gives fine-grained control over key material, as long as you turn on Data Access logs and size your quotas. AWS KMS hides versions and puts authorization in a per-key policy language that's more expressive, especially with encryption context, and its data-key API saves a step. These differences are worth considering when reviewing AWS KMS best practices alongside Google Cloud KMS key management. On both clouds, the envelope pattern is the common ground: generate a DEK, wrap it, store it next to the data, and let KMS guard only the key that guards everything else.

By Sandeep Pal

How to Keep AI Agents From Going Rogue Using Docker Sandbox

An AI agent is an "autonomous software program that uses artificial intelligence to make a plan, execute tools, and complete an action." Autonomy (through AI) plus tools is what makes an agent powerful, and these tools can range from a simple calculator to querying a sophisticated database. While autonomy makes things easier for humans, it only opens a big security hole. Because an agent can do almost anything, without proper guardrails, it can cause production outages. For example, in June 2025, a Replit agent deleted an entire database belonging to a SaaStr founder even though it was explicitly told to request permissions. This is why it is important to put proper guardrails around your agent. Guardrails can take the form of rules that prevent destructive tool calls or allow calls only to a specific list of domains. Another name for such a thing is "sandbox." A sandbox is a "security mechanism for separating running programs, usually in an effort to mitigate system failures and/or software vulnerabilities from spreading." There are multiple agent sandbox offerings. For example: Docker SandboxLangsmith SandboxOpen Sandbox Docker Sandbox Today, we will discuss Docker Sandbox. It has gained a lot of traction recently and plays an important role in Docker's AI strategy. Simply put, Docker Sandbox helps you run agents in an isolated environment. These sandboxes can run on your laptop or be hosted in the cloud. To understand how these sandboxes restrict agents, let's look at the architecture. First, there is a host machine. As we discussed earlier, this can be either your host or cloud infrastructure. When you launch a Docker Sandbox, it is primarily made up of two components. An AI agentDocker engine private to its own Sandbox. Each sandbox has its own Docker engine. There are multiple ways to implement a Sandbox architecture, but the industry is pivoting towards microVMs. This is primarily because of: Low resource overheadFaster bootup timesStrong isolationsReduced attack surfaceHyper-scaling A popular microVM tech stack involves Firecracker, Cloud hypervisors, and Kata containers. Now, coming to the communications, let's take a simple example. Imagine there is a hotel booking agent that searches and reserves hotels for customers. When a request comes in: Step 1: The agent makes a tool call to search for hotels. Step 2: The tool call leaves the micro VM to be verified against the Network policy. PowerShell sbx policy allow network api.search.foo.com sbx policy allow network api.search.bar.com Step 3: Above is an example of how you can define network policies for a Sandbox. In this case, we are allowing the agent to search under the foo domain and restrict its access to bar. Step 4: If the request comes for the foo domain, the network policy allows it to be forwarded to the particular website that serves the request. Let's look at how to create these Sandboxes on your laptop. Step 1: Installation PowerShell brew trust docker/tap brew install docker/tap/sbx Step 2: Sign in to your Docker account. PowerShell sbx login If you don't have one, you can create a free Docker account. Step 3: Run the sbx command. PowerShell sbx The first time you run it, you will see the following text: PowerShell Waiting for authentication... Signed in as sirivarma. Daemon started (PID: 58801, socket: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/sandboxd.sock) Logs: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/daemon.log Select a default network policy for your sandboxes: 1. Open — All network traffic allowed, no restrictions. ❯ 2. Balanced — Default deny, with common dev sites allowed. 3. Locked Down — All network traffic blocked unless you allow it. Use ↑/↓ or 1–3 to navigate, Enter to confirm, Esc to cancel. Network policy set to "Balanced". Default deny, with common dev sites allowed. I picked balanced so things like GitHub are on the allow list and everything else is denied by default. When you run sbx from a project folder, the sandbox has read/write permissions to everything in that folder. Step 4: Running sbx ls in a different shell will give the following output. PowerShell sbx ls PowerShell SANDBOX AGENT STATUS PORTS WORKSPACE docker-agent-svegiraju docker-agent running /Users/svegiraju Step 5: Running sbx policy ls will show the current list of allow and deny rules with the "Balanced" path we have chosen. PowerShell sbx policy ls PowerShell NAME TYPE DECISION RESOURCES default-ai-services network allow **.chatgpt.com:443 **.oaistatic.com:443 **.oaiusercontent.com:443 **.openai.com:443 api.anthropic.com:443 api.perplexity.ai:443 cdn.openaimerge.com:443 chatgpt.com:443 gemini.google.com:443 generativelanguage.googleapis.com:443 models.dev:443 nanoclaw.dev:443 platform.claude.com:443 play.googleapis.com:443 statsig.anthropic.com:443 default-package-managers network allow **.bun.sh:443 **.gradle.org:443 **.packagist.org:443 **.yarnpkg.com:443 apache.org:443 Step 6: In order to stop and remove the sandbox, issue the following commands PowerShell sbx stop my-sandbox sbx rm my-sandbox Step 7: Of course, you may want to diagnose issues within the sandbox or while running it. This is when we will use the "diagnose" command. PowerShell sbx diagnose Conclusion What we discussed so far shows why sandboxes matter and how to create one with Docker Sandbox. It is a free, open-source offering from Docker for individual use. In the upcoming blogs, we will discuss Cloud Sandboxes (running sandboxes in the cloud infrastructure instead of locally) and more complex configurations like Credentials, GPU passthrough, and environment files. All in all, Sandbox is an interesting project from Docker. Go try it out.

By Siri Varma Vegiraju

CORE

Hot-Reloading Kubernetes Secrets and ConfigMaps in Go With mamori (No Pod Restarts)

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these: kubectl rollout restart deployment/my-app and eat the disruption.Install Reloader so it restarts your pods for you when a Secret changes.Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot). All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string. I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything. New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut. The Two Schemes You'll Use mamori's Kubernetes provider gives you two ref schemes: RefReads fromNotesk8s-secret://<namespace>/<name>#<key>a Secretvalue is base64-decoded for youk8s-cm://<namespace>/<name>#<key>a ConfigMapplain string values Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples: Plain Text k8s-secret://prod/db-creds#password # one key from a Secret, base64-decoded k8s-secret://prod/tls#ca.crt # a certificate as []byte k8s-cm://prod/app-config#log_level # one key from a ConfigMap k8s-cm://prod/app-config # the whole ConfigMap as a JSON map Setup The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes): Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/k8s Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori: Go import _ "github.com/xavidop/mamori/providers/k8s" In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: mamori-reader namespace: prod rules: - apiGroups: [""] resources: ["secrets", "configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: mamori-reader namespace: prod subjects: - kind: ServiceAccount name: my-app namespace: prod roleRef: kind: Role name: mamori-reader apiGroup: rbac.authorization.k8s.io The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls. Load Once, Then Watch Forever Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/k8s" ) type Config struct { DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"` LogLevel string `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"` MaxConns int `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"` CACert []byte `source:"k8s-secret://prod/tls#ca.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } _ = cfg // fully resolved, defaulted, validated } Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal(). Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts: Go w, err := mamori.Watch[Config](ctx, mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // Rebuild the pool with the rotated password — live, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // lock-free snapshot; always the last VALID config w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state. Informers, Not Polling This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds. Practically, that means: Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely. Compare that to the alternatives: ApproachHow a change is picked upRestart?kubectl rollout restartyou, manuallyyesReloader / Stakateroperator restarts the podyesSecret mounted as volumekubelet re-syncs file (~60s), if your app re-reads itno, but you must re-readmamoriinformer pushes change → validated atomic swap → callbackno Whole ConfigMaps and JSON Blobs Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout. A Note on TLS Certificates Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref: Go type TLS struct { Cert []byte `source:"k8s-secret://prod/tls#tls.crt?debounce=0"` Key secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"` } Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret. When You Don't Need This Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all. mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod. Wrapping Up Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout. Shell go get github.com/xavidop/mamori/providers/k8s Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

By Xavier Portilla Edo

CORE

Node and Capacity Spillover: What Happens When Kubernetes Runs Out of Room

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

By Kishor Patil

Azure Key Vault Secrets That Reconcile Themselves in Go, With mamori

Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned; you get soft-delete and purge protection; access is governed by RBAC or access policies; and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity/Workload Identity on AKS) with zero code changes. And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination. mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret, and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running. This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault. The azure-kv Scheme The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]: Plain Text azure-kv://my-vault/db-password # latest enabled version azure-kv://my-vault/api-key?version=abc123 # pin to a specific version id azure-kv://my-vault/creds#password # one key out of a JSON secret One Key Vault quirk to keep in mind: versions are identified by opaque hex IDs, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/azure Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order: Go import _ "github.com/xavidop/mamori/providers/azure" The identity you bind needs read access to the vault's secrets. With Key Vault's RBAC model, that's the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets): Shell az role assignment create \ --role "Key Vault Secrets User" \ --assignee "<managed-identity-object-id>" \ --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault" If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets. Load, Then Watch the Rotation Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/azure" ) type Config struct { // Tracks the latest enabled version: rotations flow in automatically. DBPassword secret.String `source:"azure-kv://my-vault/db-password"` // Pinned to an immutable version id: deliberate, reproducible. SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"` // One field out of a JSON secret. APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart log.Println("Key Vault: db-password rotated and applied") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // last VALID config, lock-free _ = cfg } Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String. What Reconciliation Guarantees When a new version of db-password becomes the latest enabled one, mamori's poll cycle detects the version change, fetches it, and then: Re-validates the entire struct – a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can't break your service.Swaps atomically – w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.Fires a diff-aware callback – ev.Fields tells you exactly which fields moved, so you only rebuild what changed. Key Vault has no push channel for "a new version was created", so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn't hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap. If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip. Structured Secrets With flatten Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct: Go type Config struct { DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"` } type DBConfig struct { Host string `mapstructure:"host"` Port int `mapstructure:"port"` Password secret.String `mapstructure:"password"` } One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself. When You Don't Need This If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you'd rather rebuild a client than recycle a pod. Wrapping Up Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen. Shell go get github.com/xavidop/mamori/providers/azure Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

By Xavier Portilla Edo

CORE

Rotate AWS Secrets Manager Secrets Without Restarting Your Go Service, With mamori

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo

CORE

Cloud Architecture

DZone's Featured Cloud Architecture Resources

Monthly Top Cloud Architecture Experts

The Latest Cloud Architecture Topics