Cloud Architecture

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these: kubectl rollout restart deployment/my-app and eat the disruption.Install Reloader so it restarts your pods for you when a Secret changes.Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot). All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string. I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything. New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut. The Two Schemes You'll Use mamori's Kubernetes provider gives you two ref schemes: RefReads fromNotesk8s-secret://<namespace>/<name>#<key>a Secretvalue is base64-decoded for youk8s-cm://<namespace>/<name>#<key>a ConfigMapplain string values Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples: Plain Text k8s-secret://prod/db-creds#password # one key from a Secret, base64-decoded k8s-secret://prod/tls#ca.crt # a certificate as []byte k8s-cm://prod/app-config#log_level # one key from a ConfigMap k8s-cm://prod/app-config # the whole ConfigMap as a JSON map Setup The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes): Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/k8s Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori: Go import _ "github.com/xavidop/mamori/providers/k8s" In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: mamori-reader namespace: prod rules: - apiGroups: [""] resources: ["secrets", "configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: mamori-reader namespace: prod subjects: - kind: ServiceAccount name: my-app namespace: prod roleRef: kind: Role name: mamori-reader apiGroup: rbac.authorization.k8s.io The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls. Load Once, Then Watch Forever Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/k8s" ) type Config struct { DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"` LogLevel string `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"` MaxConns int `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"` CACert []byte `source:"k8s-secret://prod/tls#ca.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } _ = cfg // fully resolved, defaulted, validated } Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal(). Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts: Go w, err := mamori.Watch[Config](ctx, mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // Rebuild the pool with the rotated password — live, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // lock-free snapshot; always the last VALID config w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state. Informers, Not Polling This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds. Practically, that means: Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely. Compare that to the alternatives: ApproachHow a change is picked upRestart?kubectl rollout restartyou, manuallyyesReloader / Stakateroperator restarts the podyesSecret mounted as volumekubelet re-syncs file (~60s), if your app re-reads itno, but you must re-readmamoriinformer pushes change → validated atomic swap → callbackno Whole ConfigMaps and JSON Blobs Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout. A Note on TLS Certificates Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref: Go type TLS struct { Cert []byte `source:"k8s-secret://prod/tls#tls.crt?debounce=0"` Key secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"` } Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret. When You Don't Need This Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all. mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod. Wrapping Up Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout. Shell go get github.com/xavidop/mamori/providers/k8s Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

By Xavier Portilla Edo CORE

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned; you get soft-delete and purge protection; access is governed by RBAC or access policies; and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity/Workload Identity on AKS) with zero code changes. And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination. mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret, and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running. This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault. The azure-kv Scheme The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]: Plain Text azure-kv://my-vault/db-password # latest enabled version azure-kv://my-vault/api-key?version=abc123 # pin to a specific version id azure-kv://my-vault/creds#password # one key out of a JSON secret One Key Vault quirk to keep in mind: versions are identified by opaque hex IDs, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/azure Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order: Go import _ "github.com/xavidop/mamori/providers/azure" The identity you bind needs read access to the vault's secrets. With Key Vault's RBAC model, that's the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets): Shell az role assignment create \ --role "Key Vault Secrets User" \ --assignee "<managed-identity-object-id>" \ --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault" If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets. Load, Then Watch the Rotation Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/azure" ) type Config struct { // Tracks the latest enabled version: rotations flow in automatically. DBPassword secret.String `source:"azure-kv://my-vault/db-password"` // Pinned to an immutable version id: deliberate, reproducible. SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"` // One field out of a JSON secret. APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart log.Println("Key Vault: db-password rotated and applied") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // last VALID config, lock-free _ = cfg } Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String. What Reconciliation Guarantees When a new version of db-password becomes the latest enabled one, mamori's poll cycle detects the version change, fetches it, and then: Re-validates the entire struct – a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can't break your service.Swaps atomically – w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.Fires a diff-aware callback – ev.Fields tells you exactly which fields moved, so you only rebuild what changed. Key Vault has no push channel for "a new version was created", so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn't hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap. If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip. Structured Secrets With flatten Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct: Go type Config struct { DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"` } type DBConfig struct { Host string `mapstructure:"host"` Port int `mapstructure:"port"` Password secret.String `mapstructure:"password"` } One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself. When You Don't Need This If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you'd rather rebuild a client than recycle a pod. Wrapping Up Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen. Shell go get github.com/xavidop/mamori/providers/azure Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

By Xavier Portilla Edo CORE

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo CORE

A pod goes into CrashLoopBackOff. You pull the logs expecting a stack trace, a panic, an error string - anything that points you somewhere. Instead, you get one line: Plain Text Loading config... And then nothing. No error. No exit message. The container is just gone, and a few seconds later it’s back, prints the exact same line, and disappears again. Magic. This is the story of chasing that silence to its root cause. TCP connection that was never going to succeed, and never going to fail either. At least not on any timescale a Kubernetes health check was willing to wait for. The Setup We were migrating backend services from a legacy message queue to Kafka. The new consumers ran side by side with the old ones in a “shadow mode.” That let us compare behavior before the real cutover. Part of that work meant pointing a dev environment at a managed Kafka cluster. (Think AWS MSK — the specifics don’t matter here.) We also updated the broker endpoint in config. In shadow mode, we send to both old and new queues, but only one of them processes the message. The other queue infrastructure just logs what it receives. The change looked trivial: swap one connection string for another, restart the pods, watch them come up. Instead, every pod that touched Kafka went straight into a crash loop. The only clue was that single “Loading config” line. Repeated forever. Why “No Error” Is the Error The instinct when a service crashes is to look for what it logged right before dying. Here that instinct is a trap. The absence of any further log output isn’t a hint — it’s the symptom itself. Two things had to be true simultaneously for this to happen: Something blocked the process long enough that Kubernetes’ health checks gave up on it and sent SIGKILL.Whatever the process wanted to log about being blocked never made it out of its internal buffers before the kill. That second point matters more than it looks. Go’s standard logger writes to os.Stdout. How a container runtime attaches to that stream determines whether output appears immediately or sits in a buffer. Buffering is common under load, or when the write target isn’t a real TTY. Consider a process blocked inside a library call, say dialing a broker. It never gets back to the point in its code where it would flush or print the next line. SIGKILL doesn’t give a process the chance to clean up. Whatever was sitting in a buffer is gone. From the outside, a service that’s actually deep in a hung network call looks identical to one that exited silently. Both just print “Loading config” and stop. The lesson here generalizes past Kafka. If a container’s logs stop dead with no error and no clean shutdown message, assume a hang-then-kill. Not a fast crash. Until proven otherwise. Reaching for the Network Layer Once “look at the application logs” stopped being useful, the next step was to get underneath the application entirely. Shelling into a node and watching the raw traffic (tcpdump) tells you what actually happened at the OS level. So does tracing the process’s syscalls with strace. Neither depends on whether the application ever got to log anything about it. What that showed: a TCP handshake that started and never finished. A SYN packet went out toward the broker; no SYN-ACK ever came back, and critically, no RST came back either. That distinction is the whole story. Connection refused is fast and loud. The remote host, or a firewall in front of it, actively sends back an RST packet. Your client’s connect() call fails almost immediately.Connection blackholed is slow and silent. Packets go out, and nothing comes back. The OS has no way to know if the remote end is down, unreachable, or just very far away. So it retransmits the SYN a few times with exponential backoff, then gives up. The kernel’s default TCP connect timeout can be well over a minute. In this case, the broker endpoint we’d configured was a private, VPC-internal address. It was reachable from some parts of the network, but not from the specific node group these pods landed on. No security group or routing rule was actively rejecting the connection – the packets were simply going nowhere. That’s the worst kind of network failure to debug from inside an application. Everything about it looks like the process is just slow, right up until it isn’t. Where the Health Check Made Things Worse None of this would have been quite so opaque if the failure had surfaced immediately. But the service’s startup path connected to Kafka before reporting itself healthy. On top of that, the Kubernetes startup probe carried a generous timeout, meant to avoid flapping on slow boots. That combination left the platform with no opinion about what was wrong. It just saw a container that hadn’t become healthy in time. So it did the only thing it can do here: kill it and try again. The pod restart count climbed. The backoff delay between restarts grew too - Kubernetes doubles it after repeated failures, up to roughly five minutes. Every fix we tried afterward seemed to take forever to take effect. That’s because we were still watching a container that hadn’t actually restarted yet. It was just waiting out its backoff window. Deleting the pod outright forced an immediate restart. That turned out to be the fastest way to test each hypothesis, rather than waiting for the backoff timer. The Fix, and the More Useful Part The actual fix was almost anticlimactic: switch to the broker’s public endpoint. In a real production setup, you’d instead fix the VPC routing or peering. That makes the private endpoint reachable from every node group that needs it. Once the TCP path was real, the connection succeeded instantly, and the crash loop stopped. The useful part isn’t the fix. It’s the checklist that could have saved us time: Handy Checklist Treat “one log line then silence” as a hang, not a crash. A clean crash logs an error. A silent one usually means something upstream killed the process mid-blocking-call.Go to the network layer early, not last. tcpdump or strace on the affected node will show you a stuck SYN in seconds. That’s far faster than adding print statements and waiting through several crash-loop cycles.Know the difference between “refused” and “blackholed” in your bones. An RST means someone answered and said no. Check credentials, ports, and application-level config. Silence means the packet never arrived. Check routing, VPC peering, security groups, and whether you’re using the right endpoint for the network you’re actually in.Set explicit, short connect timeouts in your client libraries. Don’t let a startup path inherit the OS’s default TCP connect timeout. The OS optimizes that default for general robustness, not for failing fast during a health check window.Make sure your logger flushes before anything that can block indefinitely. If a call to an external system can hang, log “attempting to connect to X” first. Then make sure that line is actually out the door, synchronously if necessary, before making the call. It costs you nothing when the call succeeds and saves you hours when it doesn’t.When you’re mid-debug, delete the pod instead of waiting out the backoff. Kubernetes’ exponential backoff on repeated CrashLoopBackOff restarts is helpful in production and actively annoying when you’re iterating on a fix. None of this is exotic — it’s TCP fundamentals and container basics that everyone technically knows. What makes it worth writing down is how convincingly a blackholed connection disguises itself as an application bug. Right up until you stop looking at the application and start looking at the wire.

Most agent projects start the same way: one model, one prompt loop, three or four tools. That's the right place to start, and for a lot of workloads it's also the right place to stop. The trouble shows up later, when the same agent is asked to inspect infrastructure, parse OpenAPI specs, write docs, answer developer questions, and call operational tools. The system prompt turns into a wall of text. Tool selection gets flaky in ways you can't reproduce. Someone tunes the prompt to fix documentation quality and quietly breaks spec analysis, and nobody notices for a week. Splitting the agent is the obvious fix. But the agents were never the hard part. The hard part is the contract between them: how one agent finds another, how it hands over work, where authentication lives, and how you trace a request that crossed three runtimes before it failed. This article builds a small multi-agent system on Amazon Bedrock AgentCore Runtime using the Agent-to-Agent (A2A) protocol: an orchestrator, an OpenAPI spec analyzer, and a documentation generator. Three agents are a toy. The decisions are not. A note on versions. AgentCore and A2A are both moving fast. Everything here was checked against the AWS documentation and AgentCore CLI 0.25.x in August 2026. Pin your CLI and Python dependencies, and re-check the docs before you deploy anything you care about. Is a Second Agent Actually Worth It? Multi-agent isn't a maturity level. Every remote handoff you add buys you latency, an extra auth boundary, and a new class of failure that only shows up under load. Keep one agent when one team owns the workflow, the tools sit inside the same security boundary, and a single prompt can plausibly cover the job. Don't split because the architecture diagram looks better with more boxes in it. Split when responsibilities genuinely diverge. The signals are usually organizational before they're technical: two teams own two capabilities, the specialists want different models, the release cadences don't match, or one capability touches data the other has no business seeing. At that point the agent stops being a prompt and starts being a service, and it needs a service's contract. The arithmetic argument for a shared protocol is the familiar one. Pairwise integrations grow with the square of the number of participants, and each one tends to invent its own payload shape, auth handling, timeout, and error format. A standard protocol doesn't make any of those problems disappear. It just means every participant expresses them in the same place, which is most of the benefit. MCP and A2A Aren't Competing These two get discussed as alternatives. They aren't: they sit on different edges of the architecture. MCP connects an agent to tools and data.A2A connects an agent (or an application) to another agent. A spec analyzer might pull an OpenAPI document out of a repo over MCP, then return its analysis to an orchestrator over A2A. Both, same request. In the code below, the OpenAPI document is passed inline so the A2A flow stays readable. One caveat about "framework-agnostic," since it gets oversold: A2A is framework-agnostic at the protocol level. That doesn't mean two arbitrary SDKs will talk to each other on the first try. Both sides still need a compatible A2A implementation, transport, protocol version, and auth configuration. Once those line up, the internals genuinely don't matter; Strands on one side, LangGraph on the other, fine. What Runtime Gives You Configure an AgentCore Runtime for A2A, and it becomes a transparent proxy in front of your container. Your side of the contract is narrow: a stateless, streamable HTTP server on 0.0.0.0:9000, mounted at the root path, speaking JSON-RPC 2.0. AgentCore passes the JSON-RPC payload straight through without touching it, and adds session isolation, authentication, scaling, and observability around it. Note the port. A2A is 9000 at /, MCP is 8000 at /mcp, plain HTTP is 8080 at /invocations. Getting this wrong is the most common reason a container deploys fine and then never answers. The A2A vocabulary you need for the rest of this: A2A client: whoever is sending work. Here is the orchestrator.A2A server: the agent endpoint receiving it.Agent Card: metadata at /.well-known/agent-card.json describing identity, endpoint, skills, capabilities, and auth requirements. This is how discovery works.Message: one turn, carrying one or more parts (text, structured data).Task and artifact: a task tracks work at one remote agent; artifacts carry its output. An orchestrator fanning out to three specialists creates three separate tasks. For inbound calls, Runtime supports IAM SigV4 or OAuth 2.0 bearer tokens (JWT). A runtime version uses one of them at a time, so pick based on who's calling and where the security boundary sits. If a client authenticates with the wrong one, you get a 403 that looks nothing like an agent error. The Shape of the System The system has a hub and spoke architecture. The orchestrator is an A2A server to the user and an A2A client to the specialists, which is worth stating explicitly because it's easy to forget when you're debugging. Hub and Spoke Achitecture The orchestrator should know what each specialist advertises. It should not know how any of them work. The test: adding a fourth agent a changelog writer, say should mean one new config entry, not edits inside the agents you already shipped. Building a Specialist Prerequisites Python 3.10+Node.js 20+ (the AgentCore CLI ships as an npm package; the A2A tutorial page still says 18, but the CLI's own docs say 20)AWS CDK installed, the CLI deploys through itAWS credentials, plus model access enabled in Bedrocknpm install -g @aws/agentcore If you still have the old Python starter toolkit installed, uninstall it first. Both provide an agentcore command, and you will lose an afternoon to that. Scaffold an A2A project and pick Strands when prompted: Shell bash agentcore create --protocol A2A The generated project gives you the config, the model loader, and the A2A server wiring. The specialist below does one thing: read an OpenAPI document and pull out a compact, honest view of its operations. Python # app/SpecAnalyzer/main.py from typing import Any import yaml from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model HTTP_METHODS = { "get", "put", "post", "delete", "options", "head", "patch", "trace", } def _auth_mode(security: Any) -> str: if security is None: return "unspecified" if security == []: return "none" if isinstance(security, list) and any(item == {} for item in security): return "optional" return "required" @tool def summarize_endpoints(openapi_yaml: str) -> list[dict[str, Any]]: """Return a compact list of operations from an OpenAPI document.""" spec = yaml.safe_load(openapi_yaml) if not isinstance(spec, dict): raise ValueError("The OpenAPI document must contain a YAML object.") paths = spec.get("paths", {}) if not isinstance(paths, dict): raise ValueError("The OpenAPI 'paths' field must be an object.") global_security = spec.get("security") operations: list[dict[str, Any]] = [] for path, path_item in paths.items(): if not isinstance(path_item, dict): continue for method, operation in path_item.items(): if method.lower() not in HTTP_METHODS: continue if not isinstance(operation, dict): continue security = operation.get("security", global_security) operations.append( { "path": path, "method": method.upper(), "operation_id": operation.get("operationId"), "summary": operation.get("summary", ""), "auth": _auth_mode(security), } ) return operations agent = Agent( model=load_model(), system_prompt=( "You are an API specification analyst. " "Call summarize_endpoints before reasoning. " "Report endpoints, authentication behavior, and concrete " "specification risks. Never invent fields that aren't present." ), tools=[summarize_endpoints], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) serve_a2a is the AgentCore SDK helper that does the boring, mandatory parts: /ping, Agent Card serving, AGENTCORE_RUNTIME_URL handling, Bedrock header propagation, and binding to port 9000. The validation inside that tool is the part I'd argue for hardest. OpenAPI path items legally contain keys that aren't HTTP methods parameters, servers, $ref, orsummary. Loop over every key as though it were an operation and you'll either crash on a string or, worse, emit a confident summary containing an endpoint called PARAMETERS. Then the doc generator writes it up, and it ships. Demos never surface this because demo specs are clean. Direct dependencies for both specialists: Shell bedrock-agentcore[a2a] strands-agents[a2a] a2a-sdk PyYAML httpx The a2a extras matter. Without them, serve_a2a and the client classes aren't there. a2a-sdk is what the orchestrator's client code imports from, so it's a direct dependency even though the server side pulls it in transitively. The documentation generator uses the same wrapper with a narrower job: Python # app/DocGenerator/main.py from strands import Agent from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model agent = Agent( model=load_model(), system_prompt=( "You write developer documentation from an API request and a " "specification analysis. Produce Markdown with an overview, " "authentication notes, request examples, error handling, and a " "short Python quickstart. Never invent endpoints, fields, or " "credentials." ), ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) That one's prompt-only on purpose. If your docs have to match a house format, this is where you add output validation or template tools instead of asking a model to remember a style guide. Test It Locally Shell agentcore dev # or, without the inspector UI: python main.py Then send it a JSON-RPC message at the root path: JSON curl -X POST http://localhost:9000/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": "req-001", "method": "message/send", "params": { "message": { "role": "user", "parts": [ { "kind": "text", "text": "Analyze the authentication on this API." } ], "messageId": "12345678-1234-1234-1234-123456789012" } } }' | jq . Pull the Agent Card too, before you deploy anything: Shell curl http://localhost:9000/.well-known/agent-card.json | jq . Thirty seconds of curl catches a missing discovery endpoint or a card whose skills describe an agent you deleted two commits ago. Card problems are miserable to debug from the client side, because what you see is a resolver failure, not a bad card. Deploy the Specialists The CLI defaults to IAM SigV4. Configure bearer-token auth when the caller can't sign requests, the AWS tutorial walks through Cognito for this, though Cognito isn't a requirement; any OIDC provider works. Deploy each specialist as its own runtime: Shell agentcore deploy The CLI packages the code, uploads the artifact to S3, creates the runtime, and hands back an ARN: Shell arn:aws:bedrock-agentcore:us-west-2:<account-id>:runtime/SpecAnalyzer-xyz123 Separate runtimes are half the point of doing this at all. Each specialist scales, releases, and rolls back on its own, without a coordinated deploy across the whole system. The URL Nobody Warns You About Here's the step that eats the most time. The invocation URL isn't the ARN, and it isn't a friendly hostname. It's the ARN, URL-encoded, embedded in a path: https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{url-encoded-arn}/invocations/ Every colon and slash in the ARN gets escaped with quote(arn, safe='') in Python, or let the SDK do it: Python from bedrock_agentcore.runtime import build_runtime_url url = build_runtime_url( "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/SpecAnalyzer-xyz123" ) The Agent Card sits under that path, at .../invocations/.well-known/agent-card.json. Keep the trailing slash on the base URL. A missing slash or an unescaped ARN both produce a 404, which reads like the runtime doesn't exist. Calling a Remote Agent This client assumes bearer auth. For an IAM runtime, sign with SigV4 or use a client that signs for you. Python # app/Orchestrator/client.py import os from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, TextPart DEFAULT_TIMEOUT = 300 def _message(text: str) -> Message: return Message( kind="message", role=Role.user, parts=[Part(TextPart(kind="text", text=text))], message_id=uuid4().hex, ) async def call_remote_agent( runtime_url: str, text: str, session_id: str | None = None, ) -> str: headers = { "Authorization": f"Bearer {os.environ['BEARER_TOKEN']}", "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": session_id or str(uuid4()), } async with httpx.AsyncClient( timeout=DEFAULT_TIMEOUT, headers=headers, ) as http: resolver = A2ACardResolver(httpx_client=http, base_url=runtime_url) card = await resolver.get_agent_card() config = ClientConfig(httpx_client=http, streaming=False) client = ClientFactory(config).create(card) async for event in client.send_message(_message(text)): if isinstance(event, Message): return event.model_dump_json(exclude_none=True) if isinstance(event, tuple) and len(event) == 2: task, _ = event return task.model_dump_json(exclude_none=True) raise RuntimeError("The remote agent returned no A2A result.") With streaming=False, you get exactly one result, which is either a Message or a (Task, UpdateEvent) tuple. Handle both. The tuple form is what trips people up on the first run. Returning raw JSON keeps the protocol visible for a walkthrough, but don't ship it. Normalize messages and artifacts into your own response type at this boundary. Otherwise, every downstream prompt has to understand the full A2A envelope, and you end up with models reasoning about kind fields instead of about your problem. Session IDs deserve a real decision, not a uuid4() you stopped thinking about. Fresh ID for an isolated one-shot task. Same ID when you're continuing a multi-turn conversation with the same remote agent. And keep a separate correlation ID of your own, because one user request may span several runtime sessions and you'll want to stitch them back together in the traces. The Orchestrator Each remote agent becomes a tool. Endpoints come from config; the client resolves the Agent Card at the endpoint before sending work. Python # app/Orchestrator/main.py import os from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model from client import call_remote_agent REMOTE_AGENTS = { "spec_analyzer": os.environ["SPEC_ANALYZER_URL"], "doc_generator": os.environ["DOC_GENERATOR_URL"], } @tool async def delegate(agent_name: str, instruction: str) -> str: """Send work to a named A2A specialist.""" runtime_url = REMOTE_AGENTS.get(agent_name) if runtime_url is None: allowed = ", ".join(sorted(REMOTE_AGENTS)) return f"Unknown agent '{agent_name}'. Available agents: {allowed}." return await call_remote_agent(runtime_url, instruction) orchestrator = Agent( model=load_model(), system_prompt=( "You coordinate two API specialists. Use spec_analyzer for OpenAPI " "parsing and risk analysis. Use doc_generator for documentation and " "SDK examples. When both are needed, analyze the specification " "first, then include that analysis in the documentation request. " "Never assign work outside an agent's advertised capability." ), tools=[delegate], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(orchestrator)) Two small things in there matter more than they look. The tool is async, which avoids wrapping the remote call in asyncio.run() that blows up when the agent is already running inside an event loop, and the traceback doesn't point anywhere near the cause. And returning a friendly string for an unknown agent, rather than raising, gives the model something it can recover from instead of a dead turn. The URLs belong in deployment config, a service catalog, or an agent registry. Not in source control. Let the Model Route, or Hard-Code the Path? The code above lets the orchestrator model decide who to call and in what order. That's genuinely useful when requests vary. It's also the default choice for the wrong reasons. If every documentation request must be analyzed before generation, always, no exceptions encode that sequence in Python and use the model only inside each step. Deterministic orchestration is easier to test, retry, and audit, and it doesn't drift when someone edits a prompt. Save model-driven routing for routes that are actually ambiguous. It shouldn't be there to make a fixed workflow feel more autonomous. Running It With all three runtimes deployed, send the user request to the orchestrator. Something like "Review this customer API spec and create a Python quickstart" should cause it to call the analyzer, feed that analysis to the doc generator, and return one answer. Should. That's expected behavior, not a guarantee the protocol hands you. Whether the right handoffs happen comes down to your orchestrator prompt, your tool descriptions, your eval cases, and your workflow code. A2A standardizes the conversation. It has no opinion about whether your orchestration logic is correct. Before You Call It Production Least privilege per runtime. Each runtime gets permissions for its own tools and data, nothing more. The failure mode to watch for is the orchestrator's role slowly accumulating every permission any specialist ever needed. Distinguish auth failures from agent failures. A2A application errors come back as JSON-RPC error objects, frequently with HTTP 200. Auth and authorization failures come back as native HTTP errors AccessDeniedException is a 403. AgentCore also maps runtime exceptions onto JSON-RPC codes: -32051 for not found (404), -32053 for throttling (429), -32054 for conflict (409), -32055 for a runtime client error (424). Inspect the JSON-RPC shape. A 200 is not a success. Retry deliberately. 429 and retryable-conflict 409 are worth retrying with backoff. A 424 usually means your container failed and the answer is in CloudWatch, not in a second attempt. Never blind-retry non-idempotent work, and pass a task or request identifier so a specialist can recognize a duplicate. Trace every handoff. AgentCore Observability gives you sessions, traces, and spans. Propagate your own business correlation data on top, or you'll have three sets of traces and no way to prove they belong to the same user request. Set lifecycle values on purpose. Defaults are a 15-minute idle timeout and an 8-hour max lifetime, both configurable (60 to 28,800 seconds). Long-running tasks may justify raising them; a bigger number should be a decision, not a leftover. Also watch that your A2A sessions actually go idle — a server that holds a connection open after the client is gone will happily keep a microVM warm on your bill. Cache Agent Cards carefully. Resolving on every call is simple and costs you a round trip per hop. Cache with a short TTL and invalidate when an agent's version or capabilities change. A stale card advertising a skill that no longer exists is a genuinely confusing bug. Keep protocol and SDK versions aligned. Pin dependencies, validate cards, and test a mixed-framework setup rather than assuming it works. Use private connectivity if you need it. Runtime can attach to a VPC for internal resources, and interface VPC endpoints via PrivateLink keep AgentCore API calls off the public internet. A Failure Test Matrix Before you call it done, break the handoffs rather than admiring the happy path: The specialist's Agent Card is unreachable or malformed.The bearer token expires between card resolution and invocation.The analyzer returns a valid task with no usable artifact.The doc generator times out after the analyzer succeeded.The orchestrator picks an agent that doesn't exist, or one it isn't allowed to call.A retry re-runs a task that already had an external side effect. Each of these teaches you something a successful demo can't. They also make the division of labor concrete: Runtime hosts and protects the endpoint, A2A defines the conversation, and workflow correctness is still entirely yours. Closing The value here isn't the agent count. It's that each capability gets a clear boundary, a narrow permission set, and its own release cycle. Runtime provides the managed execution boundary; A2A provides a common way to discover agents and pass work between them. So, start with one agent. Split it when responsibilities, ownership, or security boundaries justify a network hop, and not before. Where the sequence is fixed, keep the orchestration deterministic. Where routing is genuinely variable, let the orchestrator reason over well-described capabilities, and instrument every handoff you add. References AWS: Deploy A2A servers in AgentCore RuntimeAWS: A2A protocol contractAWS: AgentCore Runtime authentication and authorizationAWS: LifecycleConfiguration API referenceAWS: Configure AgentCore Runtime for VPC accessAgentCore CLIAgent2Agent protocol specification

As Large Language Models (LLMs) continue their rapid trajectory of development, software engineers and cloud architects regularly run into two systemic challenges: The Fixed Knowledge Cutoff: Model intelligence is inherently restricted to its training data window, making it blind to real-time changes. The "Air-Gap" Limitation: Out of the box, LLMs cannot securely interact with external systems or private APIs on their own. Historically, developers bypassed these hurdles by writing fragile, ad-hoc API wrappers or custom orchestrators. Enter the Model Context Protocol (MCP): an open standard designed to standardize how AI applications safely connect to external data sources and execution environments.In this article, we will explore the core architecture of MCP, look at why the Azure MCP Server is a game-changer for cloud engineers, and walk through a step-by-step guide to configuring it inside Visual Studio Code. The Core Architecture of MCP At its heart, MCP establishes a uniform "language" that allows AI applications (Hosts) to talk to external resources (Servers). Rather than building custom integrations for every new LLM or tool, developers can rely on a single, clean architecture: MCP Architecture The standard is built around four fundamental building blocks: MCP Host: The runtime environment or user interface where the AI agent operates (e.g., VS Code, Claude Desktop, Cursor).MCP Client: The architectural component within the host that initiates and maintains the active connection.MCP Server: A lightweight, modular helper service that exposes specific resources, prompts, and tools.Transport Layer: The underlying protocol facilitating communication, typically utilizing JSON-RPC 2.0 over standard input/output (stdio) or HTTPS. MCP Components How the MCP Handshake Works Instead of executing raw, unpredictable bash scripts, the interaction is highly structured: Initialization: The client connects to the server and queries its capabilities. Declaration: The server returns a structured schema listing the specific tools it supports. Execution Request: When an LLM determines it needs external data, the host requests a specific tool execution from the server. Context Injection: The server executes the local process, gathers the result, and returns a semantic JSON response to the host, which is then cleanly formatted for the user. MCP in Action Why Use the Azure MCP Server? If you are managing infrastructure on Microsoft Azure, the Azure MCP Server bridges the gap between your local AI assistant and your active cloud resources. Operating as a secure local process, it natively integrates with the Azure command-line context. The server supports over 40 Azure services and more than 170 tools out of the box, spanning critical cloud primitives: Compute & Containers: Azure Container Apps and Azure Kubernetes Service (AKS). Storage & Resource Management: Azure Storage (blobs and containers) and Azure Resource Groups. Infrastructure as Code: Integrated Azure Terraform Best Practices. Key Advantages Over Raw CLI Executions While you could technically let an AI agent execute arbitrary commands in a terminal, using a dedicated MCP server provides several major structural benefits: Rich Semantics: Instead of parsing messy, unstructured terminal stdout text, the MCP server passes rich semantic data blocks directly back to the LLM. Strict Governance & Scope: You can explicitly configure the server to run in read-only mode or expose only a select subset of tools, preventing the AI from accidentally deleting production infrastructure. Interactive Guardrails: The protocol requires explicit user confirmation before executing tools that touch sensitive data or perform mutative actions. Azure MCP Server Setup and Configuration Modes You can run the Azure MCP server locally across multiple development setups using stdio transport. Below are the three most common configuration schemas. NuGet Configuration For .NET teams, the server can be dynamically fetched and started using the dnx toolchain: JSON { "mcpServers": { "Azure MCP Server": { "command": "dnx", "args": [ "Azure.Mcp", "--source", "https://api.nuget.org/v3/index.json", "--yes", "--", "azmcp", "server", "start" ], "type": "stdio" } } } Node.js Configuration If you are developing in a standard JavaScript/TypeScript ecosystem, you can spin up the server dynamically using the latest npm package via npx: JSON { "mcpServers": { "azure-mcp-server": { "command": "npx", "args": [ "-y", "@azure/mcp@latest", "server", "start" ] } } } Docker Configuration For isolated development environments, you can run the server in a container. Note that you must provide a local environment file (.env) containing your Azure Service Principal credentials: JSON { "mcpServers": { "Azure MCP Server": { "comman { d": "docker", "args": [ "run", "-i", "--rm", "--env-file", "/full/path/to/.env", "mcr.microsoft.com/azure-sdk/azure-mcp:latest" ] } } } Step-by-Step Visual Studio Code Integration A great feature for those who want to work within Visual Studio Code is that they can also manage the Azure MCP Server through an exclusive extension available directly in their editor. The following steps walk you through the setup. Step 1: Install the Extension Search for and install the Azure MCP Server extension directly from the Visual Studio Code marketplace. Install Azure MCP Server Extension Step 2: Initialize and Verify Open the Command Palette (Cmd + Shift + P on macOS or Ctrl + Shift + P on Windows) and search for the extension commands to verify the server is active and running. Select an MCP Server Step 3: Configure Your Tool Accessibility Open your integrated AI chat window and select the Tools icon. Here, you will see a list of all active tools provided by the Azure MCP Server. You can toggle specific permissions on or off—for instance, disabling write operations while keeping read operations active. Select Your Tools Step 4: Interact Natively Now, your AI chat assistant can securely call Azure tools in the background. You can ask complex queries like: "Are there any inactive containers running in my resource group?""Upload our local configuration file directly to our Azure storage container blob storage." Any further interactions that reference an Azure account would use the MCP tools. Confirm Tools and Resources Real-World Engineering Use Cases To see the power of this setup, let’s look at how this changes day-to-day operations: Use Case A: Automated AKS Incident Troubleshooting When an incident occurs in an Azure Kubernetes Service (AKS) cluster, engineers typically run dozens of diagnostic commands. With the Azure MCP server connected, you can simply ask the LLM: "Investigate why the pods in our production namespace are crash-looping." The agent will call the relevant AKS tools, inspect the logs, identify the misconfiguration, and suggest the fix - all in seconds. Use Case B: Continuous Terraform and Compliance Audits Before deploying infrastructure, you can point your local AI agent to your code directory. Because the server incorporates Azure Terraform Best Practices, the agent can audit your configuration files, cross-reference them against your live Azure Resource Groups, and warn you if you are violating security compliance rules or generating drift. Conclusion The Model Context Protocol represents a major step forward in AI-assisted development. By standardizing the communication layer, the Azure MCP Server enables software engineers to transform static, isolated LLMs into active, context-aware cloud operators. Whether you are monitoring active Kubernetes clusters, auditing Terraform configurations, or automating file uploads to blob storage, MCP gives your AI assistant safe and highly scoped access to the "live" Azure ecosystem.

The Foundry model catalog covers a lot of ground, but it doesn't cover everything. If you need a model trained on your own proprietary, tabular data, a churn predictor built on your actual customer history, a fraud score trained on your actual transaction patterns, that's not a model catalog problem. That's a real machine learning problem, and on Microsoft's stack it belongs to Azure Machine Learning, a genuinely separate platform from Foundry with its own SDK, its own workspace concept, and its own deployment model. This is a hands-on build of the whole path: train a model with Azure ML's SDK, register it, deploy it behind a managed endpoint, and then wire that endpoint into a Foundry agent as a function tool, so a conversational agent can call your custom model mid-conversation the same way it would call any other tool. The Mental Model First Azure ML and Foundry don't share a runtime. They share an ecosystem, and a genuine amount of engineering effort has gone into making the seam between them small, but it's worth being precise about where that seam actually is: A command job is how training happens. You point Azure ML's SDK at a script, a compute target, and a set of inputs, and it runs your training code as a tracked, reproducible job.The model registry is where a trained model becomes a named, versioned artifact, independent of the job that produced it. This is what lets you promote a specific version to production without recreating the training run.A Managed Online Endpoint is how a registered model actually serves predictions. It's a real, always-on (or autoscaling) piece of infrastructure with its own URL, its own auth, and its own cost, running inside the Azure ML workspace, not inside Foundry.A Foundry agent's FunctionTool is the bridge. Foundry doesn't know or care that the function it's calling happens to invoke an Azure ML endpoint. From the agent's perspective, it's just a tool with a name, a schema, and a result. Prerequisites An Azure ML workspace and a Foundry project, in the same or different resource groups; it doesn't matter which, since nothing about this integration requires them to share infrastructure.Python 3.9+ with both SDKs installed. Python pip install azure-ai-ml azure-ai-projects azure-identity Python from azure.ai.ml import MLClient from azure.identity import DefaultAzureCredential ml_client = MLClient( DefaultAzureCredential(), subscription_id="<subscription-id>", resource_group_name="<resource-group>", workspace_name="<aml-workspace-name>", ) Step 1: Train the Model A command job wraps a training script, here a plain scikit-learn classifier, and runs it on managed compute. Python from azure.ai.ml import command, Input, Output train_job = command( code="./src", command="python train.py --data ${{inputs.training_data} --model_output ${{outputs.model_output}", inputs={"training_data": Input(type="uri_folder", path="azureml://datastores/workspaceblobstore/paths/churn-training/")}, outputs={"model_output": Output(type="uri_folder")}, environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest", compute="cpu-cluster", display_name="churn-model-training", ) returned_job = ml_client.jobs.create_or_update(train_job) ml_client.jobs.stream(returned_job.name) ml_client.jobs.stream blocks and prints logs until the job finishes, which is worth doing in any script you're actually going to run rather than fire-and-forget, since a training job failing silently in the background is a bad way to find out your pipeline is broken. Step 2: Register the Trained Model Registration turns the job's output into a named, versioned artifact you can reference independently of the job. Python from azure.ai.ml.entities import Model from azure.ai.ml.constants import AssetTypes model = ml_client.models.create_or_update( Model( path=f"azureml://jobs/{returned_job.name}/outputs/model_output", name="churn-classifier", type=AssetTypes.MLFLOW_MODEL, description="Customer churn classifier, trained on 18 months of account history.", ) ) Using MLFLOW_MODEL as the type here isn't incidental. If your training script logs the model with MLflow's autologging, Azure ML's managed endpoints can deploy it with a built-in scoring container, no custom score.py inference script required. That's a real time saver worth designing your training script around from the start rather than discovering after the fact. Step 3: Deploy It Behind a Managed Endpoint Python from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment endpoint = ManagedOnlineEndpoint(name="churn-endpoint", auth_mode="key") ml_client.online_endpoints.begin_create_or_update(endpoint).result() deployment = ManagedOnlineDeployment( name="blue", endpoint_name="churn-endpoint", model=model, instance_type="Standard_DS3_v2", instance_count=1, ) ml_client.online_deployments.begin_create_or_update(deployment).result() endpoint.traffic = {"blue": 100} ml_client.online_endpoints.begin_create_or_update(endpoint).result() The blue deployment name isn't a convention you have to follow, but it's worth keeping, since it sets up the pattern you'll want the first time you deploy a new model version: create a green deployment alongside blue, split traffic between them, and shift fully once you trust the new version, rather than replacing blue outright and hoping. Step 4: Confirm It Works Before Anything Else Touches It Python import json test_input = {"input_data": {"columns": ["tenure_months", "monthly_spend", "support_tickets"], "data": [[14, 89.50, 3]]} response = ml_client.online_endpoints.invoke( endpoint_name="churn-endpoint", request_file=None, deployment_name="blue", input_data=json.dumps(test_input), ) print(response) Get a real prediction back here before wiring anything else into it. Debugging a broken endpoint through the extra layer of an agent's function-calling loop is meaningfully harder than debugging it directly. Step 5: Wrap the Endpoint as a Foundry Agent Function Tool This is the actual bridge. The Foundry agent doesn't call Azure ML directly; your application code does, in response to the agent asking for it. Python import os import requests from azure.ai.projects import AIProjectClient from azure.ai.projects.models import PromptAgentDefinition, Tool, FunctionTool from azure.identity import DefaultAzureCredential def get_churn_score(tenure_months: int, monthly_spend: float, support_tickets: int) -> dict: payload = {"input_data": {"columns": ["tenure_months", "monthly_spend", "support_tickets"], "data": [[tenure_months, monthly_spend, support_tickets]]} resp = requests.post( "https://churn-endpoint.<region>.inference.ml.azure.com/score", headers={"Authorization": f"Bearer {os.environ['AML_ENDPOINT_KEY']}", "Content-Type": "application/json"}, json=payload, timeout=10, ) resp.raise_for_status() return {"churn_probability": resp.json()[0]} func_tool = FunctionTool( name="get_churn_score", description="Predict churn probability for a customer given tenure, spend, and support ticket history.", parameters={ "type": "object", "properties": { "tenure_months": {"type": "integer", "description": "How many months the customer has been active."}, "monthly_spend": {"type": "number", "description": "Average monthly spend in dollars."}, "support_tickets": {"type": "integer", "description": "Number of support tickets in the last 90 days."}, }, "required": ["tenure_months", "monthly_spend", "support_tickets"], "additionalProperties": False, }, strict=True, ) project = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential()) tools: list[Tool] = [func_tool] agent = project.agents.create_version( agent_name="retention-agent", definition=PromptAgentDefinition( model="gpt-4.1-mini", instructions="Help the team assess churn risk. Call get_churn_score whenever specific customer numbers are provided.", tools=tools, ), ) Step 6: Run It End to End Python import json from openai.types.responses.response_input_param import FunctionCallOutput openai_client = project.get_openai_client() conversation = openai_client.conversations.create() response = openai_client.responses.create( input="A customer's been with us 14 months, spends about $90/month, and filed 3 tickets recently. Churn risk?", conversation=conversation.id, extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}, ) for item in response.output: if item.type == "function_call" and item.name == "get_churn_score": result = get_churn_score(**json.loads(item.arguments)) follow_up = openai_client.responses.create( input=[FunctionCallOutput(type="function_call_output", call_id=item.call_id, output=json.dumps(result))], conversation=conversation.id, extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}, ) print(follow_up.output_text) The model decides whether the question warrants calling get_churn_score at all; your code executes the actual HTTP call to Azure ML when it does, and the result flows back into the same conversation for the model to finish answering. Nothing about the agent definition or the Responses API calls changes based on what the function actually does behind the scenes, whether it hits a database, a Lambda function, or, as here, a completely separate Azure ML workspace. Where This Fits in the Bigger Picture Worth being explicit about something the SDKs don't make obvious on their own: Azure ML's managed endpoint and the Foundry project endpoint are two different resources with two different auth boundaries, and nothing about this integration merges them. The Azure ML endpoint has its own key or managed identity, scoped to that workspace. The Foundry project has its own credential, scoped separately. FunctionTool doesn't create a trust relationship between the two; it's just a name and a schema the model uses to ask your code to do something. The actual authentication to Azure ML happens entirely in your own function implementation, same as it would if that function called any other external API. That's a feature, not a limitation. It means the Azure ML side of this can be owned, secured, and rotated independently of the Foundry side, by a different team if that's how your organization is structured, without either side needing write access to the other's resources. Production Considerations Before You Commit Process expires 10 minutes after a function call is issued. Submit the tool's output back to the conversation before that window closes, or the run fails. If get_churn_score's HTTP call to Azure ML is slow, that's a hard deadline, not a soft one; budget for it.Prefer a managed identity over a static endpoint key once this is more than a prototype. A key baked into an environment variable is fine for local testing and a real liability in anything that runs unattended. Grant the identity running your function tool code the AzureML Data Scientist or a more narrowly scoped custom role against just this endpoint, not the whole workspace.Blue/green the endpoint; don't overwrite it. Standing up a new deployment alongside the old one and shifting traffic gradually is the only way to catch a regression in a newly trained model before it's serving 100% of real requests.Treat strict=True on the FunctionTool schema as a correctness feature, not boilerplate. It's what keeps the model from calling get_churn_score with a malformed or partial argument set that would otherwise fail inside your function rather than being caught before the call.Online endpoint compute is billed whether or not it's actively scoring. Unlike a serverless model call, a Managed Online Endpoint with instance_count=1 running around the clock costs money at idle. If call volume is low and bursty, look at scale-to-zero options or batch endpoints instead of defaulting to always-on instance count 1.Model drift doesn't announce itself. Nothing in this pipeline retrains automatically when the real-world distribution shifts away from what the model was trained on. Log the inputs and outputs of every get_churn_score call somewhere you can actually review, and revisit training data on a real cadence, not only when someone notices predictions have gotten worse. Where This Leaves You Foundry's model catalog is the right tool for the overwhelming majority of generative AI work, and for teams that never need a model trained on their own proprietary data, it's genuinely the whole story. But the moment the problem is "predict something specific about our own customers, from our own historical data," that's Azure ML's job, and pretending otherwise means either forcing a language model to approximate a task it was never built for, or building your own training and serving infrastructure by hand. The actual integration effort here is small: a training job, a registered model, a managed endpoint, and one FunctionTool definition. The two platforms don't need to merge for that to work. They just both need to keep doing the one thing each is actually good at. References Microsoft Learn. "Train models with the Python SDK v2." Azure Machine Learning. learn.microsoft.com/en-us/azure/machine-learning/how-to-train-model?view=azureml-api-2Microsoft Learn. "Deploy machine learning models to online endpoints." Azure Machine Learning. learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?view=azureml-api-2Microsoft Learn. "Use function calling with Microsoft Foundry agents." learn.microsoft.com/en-us/azure/foundry/agents/how-to/tools/function-callingMicrosoft Learn. "Explore Microsoft Foundry Models in Azure Machine Learning." learn.microsoft.com/en-us/azure/machine-learning/foundry-models-overview?view=azureml-api-2Microsoft Learn. "Get started with Microsoft Foundry SDKs and endpoints." learn.microsoft.com/en-us/azure/foundry/how-to/develop/sdk-overview

By Jubin Soni, FBCS CORE

In my previous article, I walked through running coding agents inside Docker Sandboxes on a local machine. We installed the sbx CLI, started with a small project, and covered the commands needed to run, stop, and remove a sandbox. This time, I want to take that same workflow off the laptop. Docker added cloud sandboxes in version 0.42.0. You can now use sbx --cloud to run an agent on Docker-managed infrastructure instead of using your machine for the sandbox’s compute. The command is simple to use. The part that is worth understanding is how you get your code into that environment, work with the agent, and bring the changes back locally. That is what we will do here. Nothing complicated; we will start with a small Python project, one coding task, and a cloud sandbox. We will remove the sandbox when we are done with the work. What Changes With a Cloud Sandbox? The sbx CLI still runs in your terminal. With --cloud, supported commands target Docker’s cloud service rather than your local sandbox environment. For example: PowerShell sbx ls Lists your local sandboxes. PowerShell sbx --cloud ls Lists your cloud sandboxes. That distinction matters throughout this walkthrough. If you forget --cloud, you are not asking about the same environment. Cloud sandboxes also have separate credentials and network policies. Do not assume that an agent login or network policy you configured locally is already available in the cloud. For this example, we will copy individual files explicitly. That keeps it easy to see what we send to the sandbox and what we bring back. Before You Start You will need: An updated sbx CLI with cloud support, introduced in version 0.42.0.A Docker account with an active Docker Agentic Platform plan for cloud compute.Authentication for the coding agent you want to use. This walkthrough uses Claude.Python 3 available in the sandbox image for the example. Note: The free sbx CLI does not mean cloud compute is free. Docker bills cloud compute based on usage, and your model provider bills inference separately. Check your account’s pricing before starting. Also, use a small sample project first. Running remotely means sending code off your machine. For company repositories, make sure that is allowed before uploading anything. The host-side commands below use PowerShell. Paths inside the cloud sandbox use Linux-style paths. Step 1: Sign In and Configure the Agent First, check your installed version: PowerShell sbx version If you are still using an older version from the previous walkthrough, update it before continuing. Sign in to Docker: PowerShell sbx login For Claude, Docker documents a cloud OAuth flow: PowerShell sbx --cloud secret set anthropic --oauth Complete the provider sign-in with an account that has the required access. Notice the --cloud flag here, too. These credentials are stored for cloud use, separately from your local sandbox credentials. There is no reason to put a token in our Python files or paste it into an agent prompt. Step 2: Create a Small Project Let us give the agent something specific to fix. Create a project folder: PowerShell New-Item -ItemType Directory -Path .\cloud-sandbox-demo Set-Location .\cloud-sandbox-demo Inside it, create a file named slug.py: Python def make_slug(text): return text.lower().replace(" ", "-") This converts "Docker Sandboxes" into "docker-sandboxes". It works for that input, but it does not handle whitespace very well. Leading spaces become leading hyphens. Repeated spaces become repeated hyphens. Tabs are not handled at all. Now create test_slug.py: Python import unittest from slug import make_slug class SlugTests(unittest.TestCase): def test_two_words(self): self.assertEqual(make_slug("Docker Sandboxes"),"docker-sandboxes") if __name__ == "__main__": unittest.main() We have one passing case and a clear improvement to make. The point is not that this function needs cloud compute. It is small enough that we can focus on the sandbox workflow without spending half the article explaining an application. Step 3: Start a Cloud Sandbox Run the following command: PowerShell sbx --cloud run --detached --name cloud-demo --ttl 1h claude This creates a cloud sandbox and starts the agent without attaching your terminal to it. The flags in the above command are for doing useful things: --cloud selects the cloud environment.--detached returns control to your terminal.--name cloud-demo gives the sandbox a recognizable name.--ttl 1h requests a one-hour lifetime. Important: The documented default action when the TTL expires is deletion. Treat this as a disposable environment, and copy your work out before the deadline. The command prints a sandbox ID. You can also find it with: PowerShell sbx --cloud ls Copy that ID into a PowerShell variable: PowerShell $sandbox = "PASTE_YOUR_SANDBOX_ID_HERE" Use the real ID returned by Docker, not the placeholder above. Keep using this terminal for the remaining commands. One detail to remember is a detached cloud run creates a new sandbox. It is not the command to run repeatedly when you want to reconnect to the same one. Step 4: Copy the Project Into the Sandbox Create a directory for our example: PowerShell sbx --cloud exec $sandbox mkdir -p /workspace/demo The mkdir command runs inside the Linux sandbox, not on Windows. Now copy the two files: PowerShell sbx --cloud cp .\slug.py "${sandbox}:/workspace/demo/slug.py" sbx --cloud cp .\test_slug.py "${sandbox}:/workspace/demo/test_slug.py" The ${sandbox} syntax is intentional. In PowerShell, it separates the variable name from the colon used in Docker’s SANDBOX:PATH format. This is also why I am copying individual files rather than uploading the entire folder. We do not need a virtual environment, local configuration, or an accidentally included .env file for this task. Run the existing test inside the sandbox: PowerShell sbx --cloud exec --workdir /workspace/demo $sandbox python3 -m unittest discover -v If your selected image does not include Python 3, add it inside the sandbox before continuing. The existing test only covers two words separated by one space. Passing it does not mean the whitespace handling is correct yet. Step 5: Give the Agent a Narrow Task Attach to the running cloud sandbox: PowerShell sbx --cloud attach $sandbox Now give Claude a concrete task: Plain Text Work on the Python project in /workspace/demo. Update make_slug so that: - The output remains lowercase. - Leading and trailing whitespace is removed. - Consecutive whitespace becomes a single hyphen. - Spaces, tabs, and newlines are handled consistently. - Empty input returns an empty string. Add unit tests for these cases using unittest. Keep the existing test. Do not add third-party dependencies or modify files outside this project. Run the tests and summarize which files you changed. This is much more useful than asking the agent to “improve the project.” We have told it what the function should do, which edge cases matter, and how much freedom it has. There is no reason for it to introduce a framework or reorganize the project. The prompt is task guidance, though — not a security policy. File access, network access, and credentials still need the appropriate sandbox controls. Once the agent finishes, use Ctrl + backslash to detach and return to your local terminal. Detaching does not stop the cloud sandbox. Step 6: Run the Tests and Bring the Changes Back Run the test command again from your terminal: PowerShell sbx --cloud exec --workdir /workspace/demo $sandbox python3 -m unittest discover -v This executes inside the cloud sandbox. It is not running against your original local files. For this task, a straightforward implementation could look like: Python def make_slug(text): return "-".join(text.lower().split()) Calling split() without a separator handles consecutive whitespace and removes leading and trailing whitespace. Joining those words with a hyphen gives us the requested behavior. The agent may arrive at a different implementation. Read it rather than assuming that passing tests makes every change worth keeping. Create a separate folder for the returned files: PowerShell New-Item -ItemType Directory -Path .\review Copy the modified files into it: PowerShell sbx --cloud cp "${sandbox}:/workspace/demo/slug.py" .\review\slug.py sbx --cloud cp "${sandbox}:/workspace/demo/test_slug.py" .\review\test_slug.py Your original files are still untouched. If you have Git installed, compare the versions: PowerShell git diff --no-index -- .\slug.py .\review\slug.py git diff --no-index -- .\test_slug.py .\review\test_slug.py You can also compare them in your editor. Look at the tests as closely as the implementation. Did the agent actually add cases for tabs and newlines? Did it keep the original test? Did it add anything unrelated? For a real repository, I would bring the changes into a working branch and use the normal review process. The sandbox changes where the agent works. It does not replace code review. What About Web Applications? Our Python example does not start a server. If you use a web project instead, cloud sandboxes can expose an application through a public HTTPS URL. For an application already listening on sandbox port 3000: PowerShell sbx --cloud ports $sandbox --publish 3000 sbx --cloud ports $sandbox Use the URL returned by Docker. This is different from publishing a local port such as localhost:3000. In cloud mode, the command accepts the sandbox port, and Docker assigns the public URL. Note: Publicly reachable is not the same as private. Do not expose an unauthenticated admin page, secrets, or sensitive test data. Remove the exposure when you no longer need it: PowerShell sbx --cloud ports $sandbox --unpublish 3000 Step 7: Clean Up the Cloud Sandbox Before cleanup, make sure the files you want to keep are on your machine. If you want to pause rather than delete, Docker documents cloud stop as preserving the sandbox’s memory and disk state: PowerShell sbx --cloud stop $sandbox Do not assume that preserved resources have no cost. Check your plan’s billing terms. For this small exercise, we have already copied the results out, so we can remove the sandbox: PowerShell sbx --cloud rm $sandbox Confirm the removal when prompted, then list your cloud sandboxes: PowerShell sbx --cloud ls There is an important difference from my earlier article: sbx --cloud rm --all is intentionally disabled. Cloud cleanup requires explicit sandbox identifiers. That is a useful safeguard. A cloud credential may have access to more than the one environment you were experimenting with. A Few Things That Can Slow You Down If the agent cannot authenticate, check its cloud credentials. A successful local session does not prove that cloud authentication is configured. If it cannot reach a service, check the cloud network policy. Do not immediately open access to everything just to make an error disappear. If your local files have not changed, remember the workflow we used: we copied files into the cloud and copied the results back. Those copies are not a live synchronization mechanism. And if you are coming back to a running sandbox, use attach. Repeating the detached creation command gives you another sandbox, not another connection to the original one. Conclusion What I like about this addition is that it keeps the workflow familiar. We are still using sbx, still giving the agent a specific project, and still deciding what work to keep. The difference is where that work happens. Start small. Send only the files the agent needs, give it one clear task, and bring the results back into your normal development process. Once that feels comfortable, move on to a larger repository or a task that actually benefits from remote compute. And copy the changes back before the sandbox expires. A useful fix is not very useful if the only copy disappears with the environment.

By Naga Santhosh Reddy Vootukuri CORE

The Diagram-to-Deployment Gap Cloud security architecture often begins with a strong design, account boundaries are defined, identity federation and vending patterns are selected, centralized security services are planned, and diagrams show how telemetry, governance, and incident response should work together. The design may be reviewed by experienced architects and approved by risk stakeholders. Yet the most difficult part starts after deployment. Cloud environments are not static. New accounts are created, workloads are modernized, emergency changes are made, teams adopt new services, and temporary exceptions accumulate. Over time, the deployed environment can diverge from the approved design even when no single team intends to weaken security. A diagram captures architectural intent while the running cloud environment represents operational reality. A mature security program must continuously reconcile the two. The central challenge is therefore not only whether an organization can design a secure cloud architecture. It is whether the architecture can continuously determine that its assumptions still hold. In practice, applying the operating model in this article reduced the time to detect architecture drift from weeks to hours - because divergence from intent is checked continuously against explicit invariants rather than discovered during periodic reviews or audits. I have watched this play out on several architectures I have helped design over the years. The architecture was approved, the launch was clean, and a couple of quarters later the environment no longer matched the diagram we had signed off on. No single team broke it. It drifted organically through dozens of individually reasonable decisions, each of which looked fine on its own review. This article presents a five-stage Continuous Security Architecture Loop — Define, Prevent, Observe, Validate, and Improve — for turning architecture from a one-time deliverable into an operating system for cloud assurance. Why Traditional Security Architecture Becomes Static Traditional architecture processes are strongest at design time. Teams conduct threat modeling, select controls, review trust boundaries, approve exceptions, and publish reference patterns. Once a workload is approved, responsibility often shifts to platform engineering, application teams, security operations, compliance, and audit. The handoff creates a structural gap where architecture defines the intended state, while operations manages the actual state. This gap becomes particularly visible in multi-stakeholder environments. A central team may define a standard account structure, but application teams control many day-to-day decisions inside each account. A security service may be enabled at launch but disconnected later. A centralized log destination may exist while selected workloads stop delivering critical events. A temporary administrative role may become a permanent access path. Each change may look like a configuration issue, yet the combined effect can break the original trust model. Configuration Drift vs. Architecture Drift Configuration drift and architecture drift are different problems. Configuration drift means a resource no longer matches an expected setting. Architecture drift means a security property of the system is no longer true. Logging may still be enabled while workload administrators can now alter the evidence. Encryption may still be present while key access violates the intended separation of duties. The resources look compliant, but the architecture has quietly lost an assumption it depended on. None of this means engineering teams are careless. Drift is what you get when architectural decisions are never wired to enforceable boundaries, current evidence, and operational feedback. Defining Continuous Security Architecture Continuous security architecture is an operating model that translates security principles into testable requirements, applies those requirements through preventive and detective mechanisms, validates deployed environments against architectural intent, and feeds operational findings back into future designs. It is related to continuous compliance, policy as code, infrastructure as code, cloud security posture management, and security monitoring, but it is not equivalent to any one of them. Continuous compliance asks whether a defined requirement is satisfied; continuous security architecture asks a broader question: does the implemented environment continue to preserve the trust assumptions, control objectives, and security outcomes of the approved architecture? This distinction matters because an architecture can satisfy many individual configuration checks and still fail as a system. A useful model must therefore connect four elements: the principle being protected, the architecture requirement derived from that principle, the controls that implement the requirement, and the evidence used to determine whether the outcome remains true. How continuous security architecture relates to adjacent practices: practicecore questionrelationship to this loop Policy as code Is this rule encoded and enforced? A mechanism used inside Prevent and Validate - not the model itself. Continuous compliance Is a defined requirement satisfied? A subset: answers per-control conformance, not whether the system’s trust assumptions still hold. CSPM Are resources misconfigured against a benchmark? Feeds the Observe stage; scores configurations, not architectural invariants. Continuous security architecture Do the architecture’s trust assumptions still hold in production? The superset - connects principle, requirement, control, and evidence across all five stages. The Continuous Security Architecture Loop The Continuous Security Architecture Loop contains five stages. The stages are not a maturity sequence that an organization completes once. They form a recurring operating cycle. Each stage produces information needed by the next, and the final stage feeds learning back into the beginning. Figure 1. The Continuous Security Architecture Loop connects architectural intent with prevention, evidence, validation, and operational learning. Running example throughout this section: We follow one invariant, “Security logs cannot be modified by workload administrators” (INV-LOG-001), through all five stages, so “invariant” stops being an abstract term and becomes a single property each stage acts on. 1. Define: Convert Principles Into Testable Requirements Security principles are often written as apply least privilege, centralize visibility, minimize blast radius, protect administrative access, or encrypt sensitive data. These are directionally correct, but they do not specify what evidence proves implementation, so different teams interpret the same principle differently. The Define stage converts broad principles into testable architecture requirements. Consider centralizing security visibility. A testable requirement might state that security-relevant activity from every account must be delivered to a centrally governed logging environment. That statement decomposes into measurable conditions: required audit sources are enabled, destinations are centrally controlled, workload teams cannot delete retained evidence, delivery failures generate alerts, newly created accounts are automatically enrolled, and retention supports investigation needs. You are not trying to turn every architecture document into a long checklist. Rather, you are identifying the properties that materially support the security model. Each requirement should describe the intended outcome, identify the evidence needed to validate it, specify an owner, and state whether the implementation is preventive, detective, responsive, or compensating. The strongest requirements are technology-aware without being tool-bound: state the security property first, then map it to the chosen cloud services, so the design survives a change of implementation technology. Express the requirement as a structured, adoptable artifact rather than prose: YAML # invariant: logs-immutable-by-workload id: INV-LOG-001 principle: Centralize security visibility requirement: > Security-relevant events from every account are delivered to a centrally governed logging destination that workload administrators cannot alter or delete. outcome: Log evidence remains complete and tamper-resistant for investigation. control_type: preventive + detective owners: control: platform-engineering evidence: platform-engineering risk: security-architecture evidence_sources: - cloudtrail: organization trail delivery status - config: S3 bucket policy + KMS key policy on the log destination - scp: effective policy on workload OUs validation_frequency: near-real-time # log-delivery failure is high-consequence exception_policy: allowed: false # no standing exceptions to this invariant residual_risk_target: none Define — running example: The principle centralize security visibility becomes the spec above (INV-LOG-001). The security property is stated first, and then the AWS services are mapped underneath it. Figure 2. A security principle becomes continuously testable only when it is connected to an architecture requirement, control implementation, and current validation evidence. 2. Prevent: Enforce High-Confidence Architectural Boundaries Some architectural decisions should not depend on detection after a violation occurs. Disabling central logging, moving data into an unapproved region, disconnecting an account from governance, or creating an unmanaged administrative path can undermine the security model immediately. Preventive controls make selected decisions non-optional. In a multi-account environment, platform teams can use organizational policies, account foundations, identity boundaries, deployment controls, and protected service configurations to prevent workload accounts from changing central audit destinations, leaving the organization, disabling designated security services, or creating resources in restricted locations. The goal is to protect the boundaries that preserve the architecture. A single service control policy (SCP) makes the boundary concrete-the action is simply unavailable to workload accounts: JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "ProtectCentralLogDestination", "Effect": "Deny", "Action": [ "s3:DeleteBucket", "s3:PutBucketPolicy", "s3:PutEncryptionConfiguration", "s3:PutLifecycleConfiguration" ], "Resource": "arn:aws:s3:::org-central-security-logs*", "Condition": { "StringNotEquals": { "aws:PrincipalArn": "arn:aws:iam::*:role/PlatformLoggingAdmin" } } }, { "Sid": "PreventLeavingOrgAndDisablingAudit", "Effect": "Deny", "Action": [ "organizations:LeaveOrganization", "cloudtrail:StopLogging", "cloudtrail:DeleteTrail" ], "Resource": "*" } ] } The design challenge is selectivity. Excessive preventive control makes the platform brittle, blocks legitimate engineering work, and creates pressure that leads to bypasses. Not every deviation has the same consequence. A useful rule: prevent actions that would materially break the security architecture, and detect-and-remediate lower-risk deviations where flexibility is necessary. Only a small number of actions, like those above, warrant a hard Deny. Before enforcing a preventive control, evaluate service behavior, failure modes, exception needs, and recovery paths. A guardrail that cannot be safely changed during an incident introduces a different form of risk. Prevent — running example: The SCP above, applied to all workload OUs, denies StopLogging, DeleteTrail, and mutating actions on the central bucket for every principal except the platform logging role. INV-LOG-001’s boundary is now unavailable, not merely monitored. 3. Observe: Collect Evidence About the Deployed Architecture Architecture cannot be validated using resource configuration alone. The evidence layer may need configuration state, identity activity, network flows, deployment events, security findings, data-access records, control exceptions, and account-lifecycle events. Observation turns a running environment into evidence that can be compared with architectural intent. Consider an approved privileged-access model requiring federation, strong authentication, time-limited role assumption, and centralized activity logging. A configuration scan will happily confirm that the administrative role exists. What it will not show you is the engineer who skips that path entirely - using a long-lived key, an alternate role, or direct access from an unmanaged identity. That only shows up in behavioral evidence. So the questions worth asking are practical ones: does the property still hold, what proves it, how fresh is that proof, and who gets paged when it goes missing? Treat absence of telemetry as a finding in its own right — a gap in the logs is a gap in your ability to say anything true about that account. Centralization should not eliminate local ownership. Workload teams still need visibility into their findings and operational context, while the organization needs a protected evidence plane that cannot be selectively altered by the systems being observed. Observe — running example: For INV-LOG-001, the platform collects trail delivery status, bucket and key policy state, and every S3 mutation event against the log destination. Absent delivery telemetry is itself recorded as a finding. 4. Validate: Compare Deployed Reality With Architectural Intent Security programs often report findings as isolated events: one missing log source, one excessive role, one unmanaged connection, or one failed service enrollment. Continuous architecture validation asks whether those findings indicate that a larger architectural property is no longer true. Validation can be organized around architectural invariants which represent properties that must remain true regardless of application changes. Examples include: security logs cannot be modified by workload administrators; production identities originate only from approved sources; all production accounts inherit baseline governance; privileged access is time-bound and attributable; and external connectivity passes through approved control points. An invariant is machine-checkable. A query for INV-LOG-001, where a zero-row result is the passing state: MS SQL -- Did any non-platform principal touch the central log destination? SELECT eventtime, useridentity.arn AS principal, eventname, requestparameters FROM security_events WHERE eventsource = 's3.amazonaws.com' AND eventname IN ('PutBucketPolicy','DeleteBucket', 'PutEncryptionConfiguration','PutLifecycleConfiguration') AND element_at(requestparameters, 'bucketName') LIKE 'org-central-security-logs%' AND useridentity.arn NOT LIKE '%role/PlatformLoggingAdmin' AND eventtime > date_add('day', -1, now()); A returned row is an architecture-conformance failure rather than a lone configuration finding. Even when the SCP already blocked the action, the attempt itself is a signal the Improve stage should see. Each invariant should be connected to a validation record containing the expected state, evidence sources, current state, owner, validation frequency, exception status, residual risk, and remediation target. The record is not simply an audit artifact; it provides a shared language for architecture, platform, operations, and workload teams. A single failing record communicates the thesis better than a page of prose: fieldexample value Invariant Security logs cannot be modified by workload administrators (INV-LOG-001) Expected state Central log bucket + KMS key policies deny workload principals; org trail delivering Evidence sources Config rule s3-log-bucket-policy; org CloudTrail delivery status; effective SCP Current state FAIL — Account 1234 delivering, but bucket policy drifted to allow WorkloadAdmin Owner Platform engineering (control) / Security architecture (risk) Validation frequency Near-real-time Residual risk High until remediated — evidence integrity not guaranteed Remediation target 4 hours Validation frequency should reflect risk and rate of change. Public exposure, privileged access, and log-delivery failures may require near-real-time evaluation. Account ownership may be checked daily. Exception reviews may occur monthly or quarterly. Reference architectures should also be reviewed when new services, threats, incidents, or business models invalidate prior assumptions. Validate — running example: A daily and event-driven check finds Account 1234’s bucket policy has drifted to grant a workload role write access. It is recorded as an architecture-conformance failure against INV-LOG-001 - with owner, residual risk, and a 4-hour remediation target - not as a standalone S3 finding. 5. Improve: Feed Operational Learning Back Into Architecture Security incidents and recurring findings are frequently remediated at the workload level. The immediate problem is fixed, but the reusable architecture remains unchanged, allowing the same design weakness to appear in other environments. The Improve stage treats operational events as feedback about the architecture itself. Incidents, near misses, recurring findings, control bypasses, exception patterns, deployment failures, new threat intelligence, and cloud-service changes should all influence future design decisions. Suppose an incident reveals that a workload role could redirect security logs. Correcting the role is necessary but not sufficient. The organization should ask whether the reference architecture clearly separates log ownership, whether organizational controls should protect the destination, whether other accounts share the condition, whether new validation logic is required, and whether the account-vending process should be updated. A practical rule: every significant incident should produce both a workload-level corrective action and an architecture-level learning decision—a revised reference pattern, a new preventive control, an additional validation test, clearer ownership, an improved deployment template, or an explicit acceptance of residual risk. I remember a stretch where three newly vended accounts failed the same detection-enrollment step inside a single month. Fixing them one at a time felt like progress until the pattern became impossible to ignore. It revealed that the account vending pipeline was the defect, not the accounts. Once we corrected the pipeline, the failure class stopped appearing. Improve — running example: Root cause of the Account 1234 drift: the account-vending template applied the bucket policy once but did not protect it against later edits. The fix is not just repairing Account 1234 - it is (a) adding the mutating actions to the SCP, (b) adding a Config rule to catch policy drift, and (c) updating the vending template so future accounts start protected. The invariant, not the incident, drives the change. From Control Deployment to Control Effectiveness A control being deployed does not prove that it is effective. Logging may be enabled while important events are excluded. Encryption may be enabled while key access is broader than intended. Threat detection may be active while findings have no response owner. Backup policies may exist while restoration has never been tested. Organizational guardrails may be present while alternative paths bypass the intended restriction. Continuous assurance therefore needs more than a binary deployed/not-deployed status. A practical evaluation can examine five dimensions: dimensionquestion it answers Coverage How much of the intended environment is protected. Correctness Whether the implementation matches the requirement. Resilience Whether the control can be bypassed, altered, or disabled. Responsiveness Whether failure produces timely action. Outcome Whether the control measurably reduces the intended risk. These dimensions should not be collapsed into a universal score without context. A high coverage percentage can conceal a critical gap, while a small number of exceptions may carry disproportionate risk. The architecture team should define what effective means for each important control objective and how that effectiveness will be demonstrated. The specific dimension I have seen fail most quietly is Resilience. A control can be present, correct, and even alerting, and a workload role can still disable it or route around it. Coverage and Correctness are the easy numbers to put on a slide, while Resilience is the one that decides whether those numbers mean anything. Central Governance With Distributed Ownership No central team can manage every workload configuration, and no workload team can set enterprise-wide requirements on its own. The shared responsibility that works in practice is that architecture owns the principles, reference patterns, invariants, and the hard exception calls. Platform turns those into the account foundations, guardrails, enrollment workflows, and evidence collection everyone else inherits. Workload teams own what is specific to their application — the controls, the context behind a finding, and their own exceptions. Security operations watches for threats and control failures and feeds what it learns back into the architecture. Risk and compliance tie the evidence to obligations and to whatever risk has been formally accepted. The model must distinguish among control ownership, evidence ownership, and risk ownership. The platform team may operate centralized logging, while the workload owner remains accountable for producing the application events needed for investigation. Security operations may own an alerting process, while architecture owns the invariant that the process is meant to protect. Ambiguity at these boundaries is a common cause of unaddressed findings. Figure 3. Continuous assurance requires explicit coordination among architecture, platform, security operations, and workload teams. End-to-End Scenario: Creating a New Production Account Consider the creation of a new production account. The organization defines several requirements: the account must join the production governance hierarchy, use approved identity federation, deliver security logs to a protected destination, enroll in centralized detection, and restrict deployment to approved regions. During Prevent, organizational policies enforce these boundaries with concrete mechanisms. SCPs on the production OU deny organizations:LeaveOrganization, deny mutating actions on the central log bucket, and deny non-approved regions via an aws:RequestedRegion condition. Account Factory (or an equivalent vending pipeline) provisions the account directly into the production OU so the guardrails apply from creation, not after. During Observe, the security platform collects the literal events: CloudTrail CreateAccount and the account’s move into the OU, the AWS Config recorder status, GuardDuty/Security Hub enrollment state, identity-federation configuration, and organization-trail delivery status. Validation compares the account with the production architecture invariants. Suppose the account is delivering logs but has not enrolled in centralized detection. The check returns a failing record: YAML invariant: all-prod-accounts-inherit-baseline-governance (INV-GOV-002) account: 1234 expected_state: GuardDuty + Security Hub enrolled via delegated admin current_state: FAIL - Security Hub not enrolled (Config recorder ON, trail OK) owner: platform-engineering (control) / security-architecture (risk) residual_risk: medium - threat findings not aggregated for this account remediation_by: 24h This is recorded as an architecture-conformance issue rather than an isolated configuration finding. The record names the account owner, the missing evidence, the remediation target, and any approved exception. The Improve stage examines patterns across accounts. If several new accounts fail at the same enrollment step, the organization does not continue fixing them individually. It treats the repeated failure as evidence that the account-vending or enrollment architecture is incomplete, and the reusable process is corrected so future accounts begin in the expected state. This example illustrates the essential shift where continuous architecture is not a larger collection of controls; it is a system that connects design intent, platform implementation, operational evidence, validation, and learning. Common Anti-Patterns Architecture by diagram. There is a beautiful target-state picture on the wiki, and no way to answer the only question that matters: does the running environment still look like it?Guardrail accumulation. Controls pile up over years. Nobody removes them, nobody re-checks whether they still fire, and eventually the platform is so encrusted that engineers route around it - which is its own risk.Dashboard assurance. The board is green, so leadership feels safe - even though the dashboard only measures the handful of configurations someone remembered to wire up.Permanent “temporary” exceptions. The exception was granted for two weeks in 2023. It has no expiry, no owner, no compensating control, and no evidence the original risk still exists. It is now load-bearing.Finding-by-finding remediation. Teams close tickets faster than the architecture produces them, treating each symptom as new while the weakness that generates them stays untouched.Tool-defined architecture. The security model quietly shrinks to whatever the chosen product happens to detect. The tool should serve an architecture you defined independently - not the other way around. Adopting the Loop Incrementally Continuous security architecture does not require building all five stages at once. A workable sequence can be: Start with 3-5 invariants, not a catalog. Pick the properties whose failure would most damage the trust model, such as log integrity, production identity origin, privileged-access time-bounding, network egress control, baseline governance inheritance. Write each as a spec like INV-LOG-001.Validate before you observe everything. You do not need a complete evidence lake to begin. For each invariant, identify the single signal that proves it and check that. Breadth of telemetry comes later.Prevent only the highest-consequence actions first. A small set of well-chosen Deny guardrails, like leaving the org, disabling logging, or altering the log destination, protects more than a large, brittle policy set. Add preventive controls where a violation is irreversible; detect-and-remediate everywhere else.Wire the feedback loop early, even if manual. A monthly review that turns recurring findings into architecture changes delivers most of the Improve-stage value before any automation exists.Automate by consequence and rate of change. Move the highest-risk, fastest-changing invariants to near-real-time checks first. Slower-moving properties can stay on a daily or weekly cadence. A team can reach a useful state with five invariants, a handful of SCPs, one validation query per invariant, and a recurring review, and then expand coverage as the model proves its value. This is also how the weeks-to-hours improvement in drift detection is realized in practice: near-real-time validation of a few high-consequence invariants, not full automation on day one. When I have taken teams through this, the first few invariants we picked mattered far more than the tooling around them. Choose the properties whose failure would genuinely hurt, prove those, and resist the pull to boil the ocean on day one. Conclusion: Architecture as an Operating System A reference architecture captures intended security design. Continuous security architecture is how you find out whether that intent still holds as systems, teams, threats, and cloud services change. The loop gives you a practical model: Define testable requirements, Prevent the actions that would break critical boundaries, Observe the evidence you need to understand the environment, Validate reality against intent, and Improve the design from what operations teaches you. The value of a security architecture is not determined by the quality of its diagram. It is determined by how reliably the organization preserves its security assumptions in production and how quickly it learns when those assumptions no longer hold. Applied to a multi-account environment, this model cut architecture-drift detection from weeks to hours, turning drift from a condition found in periodic reviews into one that is continuously observed.