Deployment

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these: kubectl rollout restart deployment/my-app and eat the disruption.Install Reloader so it restarts your pods for you when a Secret changes.Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot). All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string. I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything. New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut. The Two Schemes You'll Use mamori's Kubernetes provider gives you two ref schemes: RefReads fromNotesk8s-secret://<namespace>/<name>#<key>a Secretvalue is base64-decoded for youk8s-cm://<namespace>/<name>#<key>a ConfigMapplain string values Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples: Plain Text k8s-secret://prod/db-creds#password # one key from a Secret, base64-decoded k8s-secret://prod/tls#ca.crt # a certificate as []byte k8s-cm://prod/app-config#log_level # one key from a ConfigMap k8s-cm://prod/app-config # the whole ConfigMap as a JSON map Setup The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes): Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/k8s Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori: Go import _ "github.com/xavidop/mamori/providers/k8s" In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: mamori-reader namespace: prod rules: - apiGroups: [""] resources: ["secrets", "configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: mamori-reader namespace: prod subjects: - kind: ServiceAccount name: my-app namespace: prod roleRef: kind: Role name: mamori-reader apiGroup: rbac.authorization.k8s.io The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls. Load Once, Then Watch Forever Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/k8s" ) type Config struct { DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"` LogLevel string `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"` MaxConns int `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"` CACert []byte `source:"k8s-secret://prod/tls#ca.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } _ = cfg // fully resolved, defaulted, validated } Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal(). Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts: Go w, err := mamori.Watch[Config](ctx, mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // Rebuild the pool with the rotated password — live, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // lock-free snapshot; always the last VALID config w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state. Informers, Not Polling This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds. Practically, that means: Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely. Compare that to the alternatives: ApproachHow a change is picked upRestart?kubectl rollout restartyou, manuallyyesReloader / Stakateroperator restarts the podyesSecret mounted as volumekubelet re-syncs file (~60s), if your app re-reads itno, but you must re-readmamoriinformer pushes change → validated atomic swap → callbackno Whole ConfigMaps and JSON Blobs Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout. A Note on TLS Certificates Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref: Go type TLS struct { Cert []byte `source:"k8s-secret://prod/tls#tls.crt?debounce=0"` Key secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"` } Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret. When You Don't Need This Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all. mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod. Wrapping Up Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout. Shell go get github.com/xavidop/mamori/providers/k8s Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

By Xavier Portilla Edo CORE

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

If you need to generate prerequisite test data, such as adding dummy records through an API, before executing your test suite, using Global Setup in Playwright is a clean, maintainable, and scalable approach. It ensures all required data is prepared upfront, keeping your test execution structured and reliable. What Is Global Setup? The Global Setup in Playwright runs once before the entire test suite begins execution. It is typically used for activities such as creating test data, seeding the database, generating authentication tokens, preparing the test environment, or cleaning up old data. Since it executes only once at the start of the run, it helps ensure efficient and consistent test initialization across the suite. How to Create a Global Setup in a Playwright TypeScript Project The following steps need to be followed to create a globalSetup in the Playwright project: Step 1: Create a global-setup.ts file in the Root Folder of the Project TypeScript import orders from './test_data/orders.json' with {type: 'json'}; async function globalSetup() { console.log('Running Global Setup...'); const apiContext = await request.newContext(); const response = await apiContext.post("http://localhost:3004/addOrder/", { data: orders, }); if(response.status() !=201) { throw new Error ("Failed to create setup test data!"); } console.log('Test data created successfully'); } export default globalSetup; The globalSetup() function will insert test data into the RESTful e-commerce demo application. The test data for creating new orders is supplied using the following orders.json file. JSON [ { "user_id": "1", "product_id": "79", "product_name": "5 star 10gm Chocobar", "product_amount": 5, "qty": 1, "tax_amt": 0.5, "total_amt": 5.5 } ] The data is required to validate the GET API tests, which will execute immediately after the Playwright test command is triggered. Step 2: Register globalSetup in playwright.config.ts TypeScript import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: './global-setup', }); The globalSetup property tells Playwright to execute the file once before the entire test suite runs. This setup file is typically used to perform pre-test activities such as creating test data or preparing the test environment, ensuring that all tests start with the required initial state. tsconfig.json: JSON { "compilerOptions": { "module": "NodeNext", "moduleResolution": "NodeNext", "resolveJsonModule": true, "types": ["node", "@playwright/test"] } } In this case, “require.resolve()” is not used because the project is configured to use ES Modules ("module": "NodeNext" and "moduleResolution": "NodeNext" in tsconfig.json). Since the project runs in ESM mode, Playwright can directly resolve the setup file using a string path ('./global-setup') instead of relying on CommonJS-based require.resolve(). Step 3: Run the Test The tests are already in place; only global-setup.ts needs to run to feed data into the system before running them. The script for running the tests is updated in the package.json file: JSON "scripts": { "test": "npx playwright test" }, Let’s run the tests by executing the following command in the terminal: Plain Text npm run test The following is a screenshot of the output after running the tests: It can be seen that the test data was created using the global setup before running the tests. How This Helps in Your API Testing Instead of creating the order in the first test and relying on test execution order, we can use a more robust approach by setting up the required data beforehand. By creating test data once during global setup, the tests remain independent, reducing flaky dependencies and unintended failures. This approach aligns with Playwright best practices and helps build a stable, maintainable automation test suite. Important Points to Note About the Global Setup It runs once per test run.It does NOT share variables directly with tests.Use files or environment variables to pass data.Assertions should not be put in the global setup, except for setup validation. Check out how to verify response data with Playwright TypeScript for a detailed understaning on assertions. Summary Global setup in Playwright runs once before the entire test suite and is primarily used for preparing the test environment before running the tests. It helps keep tests independent and avoids relying on execution order. However, since it runs only once and does not directly share variables with tests, any required data must be passed through files, environment variables, or external storage mechanisms. Happy testing!

By Faisal Khatri CORE

Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned; you get soft-delete and purge protection; access is governed by RBAC or access policies; and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity/Workload Identity on AKS) with zero code changes. And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination. mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret, and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running. This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault. The azure-kv Scheme The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]: Plain Text azure-kv://my-vault/db-password # latest enabled version azure-kv://my-vault/api-key?version=abc123 # pin to a specific version id azure-kv://my-vault/creds#password # one key out of a JSON secret One Key Vault quirk to keep in mind: versions are identified by opaque hex IDs, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/azure Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order: Go import _ "github.com/xavidop/mamori/providers/azure" The identity you bind needs read access to the vault's secrets. With Key Vault's RBAC model, that's the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets): Shell az role assignment create \ --role "Key Vault Secrets User" \ --assignee "<managed-identity-object-id>" \ --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault" If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets. Load, Then Watch the Rotation Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/azure" ) type Config struct { // Tracks the latest enabled version: rotations flow in automatically. DBPassword secret.String `source:"azure-kv://my-vault/db-password"` // Pinned to an immutable version id: deliberate, reproducible. SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"` // One field out of a JSON secret. APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart log.Println("Key Vault: db-password rotated and applied") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // last VALID config, lock-free _ = cfg } Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String. What Reconciliation Guarantees When a new version of db-password becomes the latest enabled one, mamori's poll cycle detects the version change, fetches it, and then: Re-validates the entire struct – a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can't break your service.Swaps atomically – w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.Fires a diff-aware callback – ev.Fields tells you exactly which fields moved, so you only rebuild what changed. Key Vault has no push channel for "a new version was created", so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn't hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap. If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip. Structured Secrets With flatten Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct: Go type Config struct { DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"` } type DBConfig struct { Host string `mapstructure:"host"` Port int `mapstructure:"port"` Password secret.String `mapstructure:"password"` } One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself. When You Don't Need This If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you'd rather rebuild a client than recycle a pod. Wrapping Up Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen. Shell go get github.com/xavidop/mamori/providers/azure Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

By Xavier Portilla Edo CORE

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo CORE

Context The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP. One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed. The service used placement constraints, max replicas per node = 1, and a Traefik health check configured for /actuator/health on port 8080. This meant we relied on external health checking and strictly controlled replica placement across nodes. Swarm-level load balancing was intentionally disabled. We needed routing control at the Traefik layer, along with the ability to quickly remove unhealthy backends based on their health check status. For this production setup, that provided more flexibility than the built-in Swarm load-balancing logic. What Happened The issue was neither a widespread outage nor a failure of the entire service. One replica became unreachable to Traefik over the overlay network: a Traefik instance on one of the manager nodes could not establish a connection to that replica’s backend endpoint. The second replica continued to work normally, so the service as a whole remained available. For regular traffic, the incident did not affect service availability: Traefik continued routing requests to the healthy replica and removed the unavailable backend from routing. During targeted manual testing of the unhealthy replica’s backend endpoint through Traefik, requests ended in a timeout or HTTP 503. The container appeared healthy, but only locally. The service responded inside the container, and /actuator/health returned UP. However, other containers and Traefik instances on remote nodes could not establish a connection to that replica through its IP address in mgp_default. Initial Checks The container had the same DNS name in both networks. This ruled out name mismatch as an independent cause of the unavailability: the issue occurred specifically at the network-connectivity level to the replica IP in mgp_default. First, we identified the problematic container and inspected its state: Shell docker ps --filter "name=mgp_router-app" --q docker inspect 6bd15b62f891 Next, we checked the networks connected to the container: Shell docker inspect 6bd15b62f891 --format '{{json .NetworkSettings.Networks}' | jq The container was attached to two networks: ingress with IP 10.22.3.137/22;mgp_default with IP 10.29.72.232/13. We then entered the container and checked whether the application was listening on the expected port: Shell docker exec -it 6bd15b62f891 bash ss -tuln The health endpoint also responded correctly: Shell curl "http://10.29.72.232:8080/actuator/health" The expected response was: Shell {"status":"UP"} At this point, it was clear that the issue was not the application itself, its startup, or its bind address inside the container. The service was alive, but that alone said nothing about its reachability from outside the container. Checking Connectivity Between Nodes The next step was critical: We checked whether another node in the cluster could reach the container using its overlay IP. Shell docker exec -it 373354916bfc bash ping 10.22.3.137 This was the IP from the ingress network, and the test confirmed ICMP reachability of the address in that network. That indicated the issue did not look like a complete loss of inter-node connectivity. The situation was different for the mgp_default IP: pinging it from other nodes failed, even though this overlay network was supposed to provide inter-container communication in Docker Swarm. This distinction was important. ICMP reachability of the address in ingress remained available, while connectivity to the replica through mgp_default was broken. Therefore, the problem was not the container itself or a complete network failure; it was a specific break in the network path used to reach that replica. What Docker Logs Showed The dockerd logs on the affected node provided another useful signal. They contained repeated messages related to Swarm network synchronization: Shell Jun 02 11:08:42 node-router2 dockerd: time="2025-06-02T11:08:42.462965233Z" level=warning msg="bulk sync to node 091d01df89ae failed: failed to send a TCP message during bulk sync: dial tcp 10.170.0.145:7946: i/o timeout" Jun 02 11:17:02 node-router2 dockerd: time="2025-06-02T11:17:02.581936915Z" level=error msg="Bulk sync to node 091d01df89ae timed out" The logs also contained NetworkDB stats entries: Shell Jun 02 11:07:08 node-router2 dockerd: time="2025-06-02T11:07:08.012509772Z" level=info msg="NetworkDB stats node-router2(2642aeb5172b) - netID:q9athb3moj967oxmb3y8xw4lj leaving:false netPeers:58 entries:13563 Queue qLen:2 netMsg/s:7" These messages do not prove a specific bug or root cause, but they indicate problems with exchanging network state between nodes. Combined with the endpoint being unreachable through mgp_default, they strengthened the working hypothesis of an overlay-networking issue. This was an important diagnostic signal: the Docker daemon was running, and the container was started, but the cluster’s network state was not ideal and was likely becoming inconsistent during node-to-node state exchange. Investigating the Hypotheses Hypothesis 1: The container itself was the problem. This hypothesis was not confirmed. A curl request inside the container to http://10.29.72.232:8080/actuator/health returned {"status":"UP"}, and ss -tuln showed that the service was listening on the required port. Hypothesis 2: The problem was port publishing or binding. This hypothesis was not confirmed. Even though port 10005 was reachable externally, that did not explain why Traefik could not reach port 8080 over the overlay network. The container worked locally, while the issue occurred on the network path between nodes. Hypothesis 3: Traefik itself was the problem. Probably not. Traefik was not the source of the failure; it was the first component that detected that the backend was unavailable from its node. Hypothesis 4: The Docker Swarm overlay network was the problem. This became the primary working hypothesis. The replica was reachable from inside its container but was not properly reachable from other nodes through its overlay IP. Shell curl http://10.29.72.232:8080 nc -vz 10.29.72.232 8080 The curl request worked locally, while nc -vz 10.29.72.232 8080 failed from other nodes. This indicated that traffic was not being routed correctly through the overlay network. An MTU/VXLAN issue was also considered. This is a plausible class of problems for Docker overlay networks, especially when there is an MTU mismatch or a Path MTU Discovery issue. In this case, however, it remained only a hypothesis rather than a confirmed root cause. Recovery We restored the problematic replica by forcing Docker Swarm to recreate the task through a rolling update: Shell docker service update --force \ --update-parallelism 1 \ --update-order start-first \ --update-delay 30s \ --detach=true \ mgp_router-app --force triggered an update even though neither the image nor the service configuration had changed. Swarm recreated the task for the affected replica and reattached it to mgp_default; after that, Traefik could reach it again over the overlay network. The rollout parameters were selected to avoid affecting service availability: --update-parallelism 1 – update one replica at a time;--update-order start-first – start the new task before stopping the old one;--update-delay 30s – give the new task time to start and pass its checks. Restarting Docker on the node and placing the node into drain mode were considered fallback measures, but they were not required to resolve this incident. Why It Happened The exact upstream root cause cannot be stated without additional evidence. However, the most well-supported explanation is that a local network-state inconsistency occurred in the Docker Swarm overlay network. As a result, one replica remained operational inside its container, but became unreachable through its overlay IP. The most plausible explanations include: the overlay network on the node was not fully ready when the container started;an issue in Docker libnetwork;an outdated VXLAN routing table;an iptables issue or a race condition while updating network rules;a node restart without restarting the Docker daemon;an issue with a Swarm endpoint that was not properly advertised or connected. These situations are particularly unpleasant because Docker continues to consider the container running and its internal health check green, while the network path to the replica is already partially broken. Key Takeaways The main conclusion is simple: a healthy container does not necessarily mean a reachable replica. This is critical in production because: a local curl inside the container does not prove external reachability;an internal health check does not guarantee that Traefik can reach the backend;Swarm may remain formally healthy even when one replica has dropped out of normal network connectivity. Therefore, in such clusters, it is necessary to check not only container state but also inter-node reachability over the overlay network. What to Monitor To prevent similar incidents, it is useful to monitor not only service health but also the cluster’s network plane. The minimum set of checks includes: reachability of container overlay IPs from other nodes;health checks through Traefik;bulk sync and NetworkDB errors in Docker logs;recurring issues on the same node;consistency between a container’s local health and its external reachability. It is also useful to have a script that specifically checks connectivity between nodes through the overlay network, not just through ingress or localhost. Conclusion This case demonstrated several important points. First, a container can be healthy locally while remaining unreachable externally. This is especially important for services that handle user traffic. Second, Traefik is a good early indicator of network problems. It can reveal that a backend on one node has stopped responding. Third, Docker Swarm overlay networking can degrade locally. In that case, the cluster may appear operational, while one replica has already dropped out of normal routing. Fourth, such incidents must be diagnosed along the network path, not only through application state. First, check the container from the inside, then verify reachability from other nodes, then inspect Docker logs, and only after that draw conclusions about the root cause.

CI failures often expose a narrow defect through a broad stream of evidence such as compiler output, test reports, stack traces, changed files, dependency metadata, and build logs. Large language models can turn that evidence into candidate repairs, and recent industrial research has shown that LLM-assisted CI repair can resolve a meaningful share of compilation failures when the model receives carefully selected context. The difficult part is not generating a patch. It is ensuring that automated repair changes only code justified by the failure, leaves protected files untouched, and proves that the original failure is actually fixed. Automated program repair research has repeatedly identified patch overfitting as a core risk: a patch can satisfy available tests without being semantically correct. Capture Failure Evidence Before Asking for a Fix A repair agent should start from structured CI evidence rather than a raw prompt containing an entire build log. GitHub Actions artifacts can persist test output, logs, and other files after a job completes, making them a practical handoff format between the failing job and a later repair job. For a Maven project, the failing job can retain both console output and Surefire reports: YAML - name: Run tests run: ./mvnw test 2>&1 | tee ci.log - name: Upload failure evidence if: failure() uses: actions/upload-artifact@v4 with: name: failure-evidence path: | ci.log target/surefire-reports/ The repair process should normalize this data into a compact incident object containing the failing test, exception type, stack trace, failing command, commit SHA, and original pull-request diff. Current surveys of LLM-based software repair describe retrieval and analysis augmentation as important layers for repository-level repair, reinforcing the value of selecting failure-relevant context before generation. Before patch generation, the failure should be reproduced in an isolated checkout. Git worktrees support linked working directories and can create a detached worktree for experimental changes without disturbing the primary checkout. A repair job can create a disposable workspace and rerun only the failing test: Shell git worktree add --detach ../repair "$HEAD_SHA" cd ../repair ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test Maven Surefire supports selecting a specific test class and, for supported providers, individual test methods. If the failure cannot be reproduced, automatic patching should stop. A passing rerun provides no reliable oracle for judging a generated fix. Turn Failure Localization Into an Edit Boundary The central control is an explicit distinction between readable and editable files. A failing test, configuration file, or build script may be necessary for diagnosis without being safe for autonomous modification. A scope document can represent that boundary: YAML failure: PaymentServiceTest.shouldRetryTimeout readable: [src/test/java/com/acme/payment/**, src/main/java/com/acme/payment/**, pom.xml] editable: [src/main/java/com/acme/payment/**] protected: [src/test/**, pom.xml, .github/**, db/migration/**] The editable set can be derived from the original PR diff, stack-trace locations, and a small neighborhood of related production symbols. Git provides machine-readable changed-path output through git diff --name-only, while pathspecs can limit comparisons to selected repository regions. The model can inspect broader context, but the orchestrator should reject any patch that escapes the approved surface. A validator needs a repair baseline captured immediately before the model edits the worktree: Shell REPAIR_BASE=$(git rev-parse HEAD) git diff --name-only "$REPAIR_BASE" > repair-files.txt while read -r path; do case "$path" in src/main/java/com/acme/payment/*) ;; *) echo "Out-of-scope change: $path"; exit 1 ;; esac done < repair-files.txt This check matters because prompt instructions such as "modify only relevant code" are advisory, while a diff gate is enforceable. Git also provides exit-status behavior through git diff --exit-code, which is useful for automated policy checks. Generate a Patch, Not an Open-Ended Coding Session The repair model should receive a bounded request like normalized failure data, relevant source fragments, the original PR diff, editable paths, protected paths, and the exact verification command. The response should be structured around a patch rather than unrestricted shell execution. Java RepairRequest request = new RepairRequest( failure, relevantSource, originalDiff, editablePaths, protectedPaths, verificationCommand ); Patch patch = repairModel.generate(request); patchValidator.validate(patch); patchApplier.apply(patch); The model proposes the smallest plausible correction where the orchestrator remains responsible for permissions and validation. This separation reflects a broader trend in LLM-based program repair toward procedural and agentic workflows that combine generation with retrieval, analysis, execution feedback, and external control. For an initial implementation, tests and CI configuration should remain read-only. A failure that genuinely requires changing a test, dependency declaration, migration, or workflow can be escalated for manual review rather than allowing the agent to broaden its own permissions. Validate More Than a Green Build A candidate patch should first rerun the exact failing test. Only after that test passes should validation expand to the affected module and then the normal CI suite. Shell ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test ./mvnw -pl payment test ./mvnw verify Passing tests alone are not proof of correctness. Automated program repair research documents the overfitting problem, where generated patches satisfy the available test suite without correctly repairing the underlying behavior. Scope validation therefore acts as an additional safeguard. A successful repair should preserve protected files, avoid dependency changes, keep the patch reviewable, and retain the original failing test unchanged. A final guard can verify that tests were not modified during repair: Shell git diff --quiet "$REPAIR_BASE" -- src/test || { echo "Repair modified protected tests" exit 1 } The resulting output should be an evidence package rather than an opaque "fixed" status, but contain the root-cause summary, exact patch, changed-file list, focused-test result, module-test result, and full-CI result. That evidence makes the automated repair reviewable. To ensure a patch does not escape its intended scope, the workflow can run the agent in an isolated Git worktree and enforce explicit path-based allowlists. For example, a CI script might use git worktree add -d /tmp/agent main so the agent’s changes occur in a separate directory. After the agent generates a commit, the pipeline can run git diff --name-only origin/main...HEAD and check that all file paths match the allowlist (for instance, grep -qvE '^(src/payment/|config/)' rejects unauthorized modifications). Protected paths (e.g. .git/) would instantly be blocked by the allowlist. The pipeline then reruns the failing test class in isolation (mvn -Dtest=PaymentServiceTest test) to confirm the issue is fixed, and executes the full test suite. A quick diff or mutation-based check can detect if any test assertions were weakened or disabled. Finally, a canary merge or smoke-test job automatically undoes the change on failure. For instance, a GitHub Actions step might include the following so that any failing health check immediately triggers an automated rollback: SQL - name: Rollback if smoke tests fail if: steps.smoke.outcome == 'failure' run: ./rollback.sh These combined safeguards (worktree isolation, allowlist enforcement, retests, and rollback triggers) help guarantee the AI-generated patch is minimal, correct, and reversible. Keep the Repair Job Less Privileged Than the Code It Examines An autonomous repair workflow processes repository content and may execute code originating from a pull request, so CI permissions need careful separation. GitHub warns that privileged workflow_run and pull_request_target workflows must not blindly check out untrusted pull-request code, and recommends granting GITHUB_TOKEN only the minimum required permissions. A safer pattern is to perform diagnosis and patch generation with read-oriented credentials, store the candidate patch as an artifact, and use a separate trusted step to create a branch or pull request. Workflow artifacts should also be treated cautiously when produced by untrusted jobs. This prevents the repair agent from turning a build failure into a path toward repository write access. Conclusion A useful CI repair agent is not an LLM attached directly to a failing build. It is a constrained repair pipeline that captures structured failure evidence, reproduces the defect, narrows the editable surface, generates a minimal candidate patch, rejects out-of-scope changes, and validates the result from the failing test through the full CI suite. Industrial evidence shows that LLM-assisted repair can be effective in CI settings, but automated program repair research also shows why a green build cannot be treated as sufficient proof. Enforced edit boundaries, isolated execution, layered testing, and least-privilege automation convert model-generated code from an uncontrolled suggestion into a reviewable, evidence-backed repair mechanism.

By Uthej Mopathi CORE

As AI-generated code expands across production systems, provenance becomes essential for tracing where each change originated, which model or prompt influenced it, and what evidence supports its review, ownership, and trustworthiness. AI-assisted development changes the economics of software creation. Source code can now be produced faster than many engineering organizations can review, understand, test, secure, and maintain it. The resulting risk is not simply "bad AI code." Recent research is more nuanced, as one 2026 study of real repositories found AI-generated code to be more verbose and to exhibit different duplication patterns, while another maintenance study found lower subsequent maintenance activity for sampled AI-generated files than for comparable human-generated files. A 2025 controlled study likewise found fewer defects in some LLM-generated solutions but also structural problems on harder tasks. The operational conclusion is therefore not that generated code is inherently inferior, but that code volume can grow faster than governance capacity. The language and stack are unspecified, so the implementation patterns below use Git, Java, Python, TypeScript, and GitHub-style CI as representative examples. The central strategy is to treat code generation as an input to an engineering control system so we can record provenance, establish ownership, prevent duplicate abstractions, enforce complexity budgets, map tests to changed behavior, automate small refactorings, constrain dependencies, and make security and repository policy executable. Code Abundance Requires Traceability and Ownership Traditional repositories mostly answer who committed a change and when. AI-assisted repositories also need to answer how a change was produced, which tool or agent contributed, which task authorized the change, and which human or team accepted responsibility. Git trailers are a practical mechanism because Git supports structured key-value metadata at the end of commit messages and can parse that metadata with git interpret-trailers. Shell git commit -m "Add invoice reconciliation" \ --trailer "AI-Origin: assisted" \ --trailer "AI-Tool: coding-agent" \ --trailer "Task: PAY-431" This metadata should describe provenance rather than transfer accountability to a model. Ownership remains organizational. A CODEOWNERS file can map sensitive or high-change paths to responsible teams, and GitHub rules can require approval from those owners before merge. GitHub rulesets can also require status checks, dismiss stale approvals after new pushes, and require approval of the latest reviewable push. Plain Text /payments/ @acme/payments /auth/ @acme/security /platform/ @acme/platform The important repository metric is therefore not generated lines per week. It is generated surface area without durable ownership. A repository can absorb high code generation throughput when every merged path has an accountable maintainer, searchable provenance, and enforceable review boundaries. Quality Budgets Must Stop Net-New Entropy Code abundance makes absolute cleanup programs impractical. Quality gates should instead prevent each change from making the repository worse. Duplication is a useful example. PMD’s Copy/Paste Detector supports multiple languages, can fail CI when duplicates are detected, and can normalize identifiers for structurally similar Java or C++ fragments. Its documentation explicitly recommends removing duplication rather than attempting to keep copied blocks synchronized. Shell pmd cpd --minimum-tokens 100 \ --dir src/main/java \ --ignore-identifiers This check becomes more useful when applied differentially, as existing duplication can be baselined, while newly introduced clones fail the pull request. The same principle applies to complexity. ESLint exposes a cyclomatic-complexity rule with a configurable maximum, allowing TypeScript or JavaScript repositories to turn complexity from a review opinion into a merge condition. JavaScript export default [{ rules: { complexity: ["error", { max: 12 }] } }]; A complexity budget should be paired with automated refactoring rather than only rejection. OpenRewrite operates on semantic trees and provides recipes for framework migrations, security fixes, and consistent source transformations while preserving formatting. That model is particularly suitable for code-abundant repositories because repetitive cleanup can be expressed once and applied consistently. Recent industrial research from Bloomberg also suggests that automated maintenance is easier to review when it arrives as very small pull requests, as its experimental system targeted roughly ten line changes and merged most generated patches in a one-month deployment, although the sample was small and should not be generalized broadly. Tests and CI Need Change-Level Intelligence A large test suite does not automatically create confidence. Maintenance becomes expensive when a changed function cannot be connected to the tests that exercise it. Coverage.py supports dynamic contexts that record which test function executed each covered line, making a code-to-test map available for Python repositories. Plain Text [run] dynamic_context = test_function That mapping enables risk-based CI. Changed files can select directly related tests first, while shared libraries, authentication code, persistence layers, or other high-blast-radius modules can trigger broader suites. Coverage remains a gate rather than a vanity percentage. For Java, JaCoCo can halt a build when configured line, branch, instruction, class, or complexity thresholds are not met. A compact CI stage can combine these controls without treating AI-originated code as a separate species: YAML quality: steps: - run: npm run lint - run: pmd cpd --minimum-tokens 100 --dir src/main/java - run: pytest --cov=src --cov-context=test - run: ./mvnw verify The stronger policy is "no unexplained quality regression," not "all generated code receives maximal testing." Provenance can increase scrutiny for high-risk generated changes, but changed behavior, dependency reach, security sensitivity, and ownership should determine the actual gate. Dependency and Security Hygiene Must Scale With Generation Code-generation workflows capable of editing manifests can expand a repository’s dependency surface as quickly as application code. Dependency review therefore belongs at pull request time. GitHub’s dependency review reports direct and indirect changes from manifests and lockfiles, includes vulnerability information, and can fail a required check when vulnerable packages are introduced. This supports a repository policy in which new dependencies require an explicit purpose, lockfile changes receive review, and unused packages are removed during routine maintenance. Security analysis should operate on every merge path rather than rely on prompt instructions. GitHub code scanning can run CodeQL and other analyzers on pull requests and the default branch, and CodeQL data-flow analysis can identify unsafe flows such as dangerous arguments or sensitive data leakage. Required code-scanning checks can block merging until findings are resolved. Runtime-facing checks provide a complementary boundary. OWASP ZAP’s baseline scan spiders a deployed target, performs passive scanning, supports configurable fail conditions, and is explicitly designed for CI/CD environments. Static analysis can therefore guard source changes while dynamic scanning checks observable application behavior. The important distinction is between generation-time guidance and repository-time enforcement, as model instructions are advisory, while CI gates are reproducible controls. Repository governance ties the system together. Rulesets should require code-owner approval for protected areas, passing quality and security checks, review of dependency changes, and fresh approval after material updates. Such controls keep throughput high without allowing generated volume to bypass the same engineering standards applied to manually written code. Conclusion The maintenance problem created by AI is fundamentally a throughput mismatch. Code production can accelerate immediately, while ownership, review capacity, testing, security analysis, and architectural coherence remain finite. Current research does not justify treating AI-generated code as uniformly worse than human-written code, but it does show meaningful differences in verbosity, structure, duplication, defects, and maintenance behavior that make passive governance increasingly risky. Sustainable code abundance therefore depends on converting maintenance expectations into repository mechanics where provenance is attached to commits, ownership is attached to paths, quality budgets are enforced on new changes, tests are mapped to behavior, refactoring is delivered in small reviewable patches, dependencies are examined before merge, and security checks are made mandatory. The durable advantage will not come from generating the most code. It will come from operating a codebase that can absorb generated code without allowing maintainability to collapse under its volume.

By Uthej Mopathi CORE

Most agent projects start the same way: one model, one prompt loop, three or four tools. That's the right place to start, and for a lot of workloads it's also the right place to stop. The trouble shows up later, when the same agent is asked to inspect infrastructure, parse OpenAPI specs, write docs, answer developer questions, and call operational tools. The system prompt turns into a wall of text. Tool selection gets flaky in ways you can't reproduce. Someone tunes the prompt to fix documentation quality and quietly breaks spec analysis, and nobody notices for a week. Splitting the agent is the obvious fix. But the agents were never the hard part. The hard part is the contract between them: how one agent finds another, how it hands over work, where authentication lives, and how you trace a request that crossed three runtimes before it failed. This article builds a small multi-agent system on Amazon Bedrock AgentCore Runtime using the Agent-to-Agent (A2A) protocol: an orchestrator, an OpenAPI spec analyzer, and a documentation generator. Three agents are a toy. The decisions are not. A note on versions. AgentCore and A2A are both moving fast. Everything here was checked against the AWS documentation and AgentCore CLI 0.25.x in August 2026. Pin your CLI and Python dependencies, and re-check the docs before you deploy anything you care about. Is a Second Agent Actually Worth It? Multi-agent isn't a maturity level. Every remote handoff you add buys you latency, an extra auth boundary, and a new class of failure that only shows up under load. Keep one agent when one team owns the workflow, the tools sit inside the same security boundary, and a single prompt can plausibly cover the job. Don't split because the architecture diagram looks better with more boxes in it. Split when responsibilities genuinely diverge. The signals are usually organizational before they're technical: two teams own two capabilities, the specialists want different models, the release cadences don't match, or one capability touches data the other has no business seeing. At that point the agent stops being a prompt and starts being a service, and it needs a service's contract. The arithmetic argument for a shared protocol is the familiar one. Pairwise integrations grow with the square of the number of participants, and each one tends to invent its own payload shape, auth handling, timeout, and error format. A standard protocol doesn't make any of those problems disappear. It just means every participant expresses them in the same place, which is most of the benefit. MCP and A2A Aren't Competing These two get discussed as alternatives. They aren't: they sit on different edges of the architecture. MCP connects an agent to tools and data.A2A connects an agent (or an application) to another agent. A spec analyzer might pull an OpenAPI document out of a repo over MCP, then return its analysis to an orchestrator over A2A. Both, same request. In the code below, the OpenAPI document is passed inline so the A2A flow stays readable. One caveat about "framework-agnostic," since it gets oversold: A2A is framework-agnostic at the protocol level. That doesn't mean two arbitrary SDKs will talk to each other on the first try. Both sides still need a compatible A2A implementation, transport, protocol version, and auth configuration. Once those line up, the internals genuinely don't matter; Strands on one side, LangGraph on the other, fine. What Runtime Gives You Configure an AgentCore Runtime for A2A, and it becomes a transparent proxy in front of your container. Your side of the contract is narrow: a stateless, streamable HTTP server on 0.0.0.0:9000, mounted at the root path, speaking JSON-RPC 2.0. AgentCore passes the JSON-RPC payload straight through without touching it, and adds session isolation, authentication, scaling, and observability around it. Note the port. A2A is 9000 at /, MCP is 8000 at /mcp, plain HTTP is 8080 at /invocations. Getting this wrong is the most common reason a container deploys fine and then never answers. The A2A vocabulary you need for the rest of this: A2A client: whoever is sending work. Here is the orchestrator.A2A server: the agent endpoint receiving it.Agent Card: metadata at /.well-known/agent-card.json describing identity, endpoint, skills, capabilities, and auth requirements. This is how discovery works.Message: one turn, carrying one or more parts (text, structured data).Task and artifact: a task tracks work at one remote agent; artifacts carry its output. An orchestrator fanning out to three specialists creates three separate tasks. For inbound calls, Runtime supports IAM SigV4 or OAuth 2.0 bearer tokens (JWT). A runtime version uses one of them at a time, so pick based on who's calling and where the security boundary sits. If a client authenticates with the wrong one, you get a 403 that looks nothing like an agent error. The Shape of the System The system has a hub and spoke architecture. The orchestrator is an A2A server to the user and an A2A client to the specialists, which is worth stating explicitly because it's easy to forget when you're debugging. Hub and Spoke Achitecture The orchestrator should know what each specialist advertises. It should not know how any of them work. The test: adding a fourth agent a changelog writer, say should mean one new config entry, not edits inside the agents you already shipped. Building a Specialist Prerequisites Python 3.10+Node.js 20+ (the AgentCore CLI ships as an npm package; the A2A tutorial page still says 18, but the CLI's own docs say 20)AWS CDK installed, the CLI deploys through itAWS credentials, plus model access enabled in Bedrocknpm install -g @aws/agentcore If you still have the old Python starter toolkit installed, uninstall it first. Both provide an agentcore command, and you will lose an afternoon to that. Scaffold an A2A project and pick Strands when prompted: Shell bash agentcore create --protocol A2A The generated project gives you the config, the model loader, and the A2A server wiring. The specialist below does one thing: read an OpenAPI document and pull out a compact, honest view of its operations. Python # app/SpecAnalyzer/main.py from typing import Any import yaml from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model HTTP_METHODS = { "get", "put", "post", "delete", "options", "head", "patch", "trace", } def _auth_mode(security: Any) -> str: if security is None: return "unspecified" if security == []: return "none" if isinstance(security, list) and any(item == {} for item in security): return "optional" return "required" @tool def summarize_endpoints(openapi_yaml: str) -> list[dict[str, Any]]: """Return a compact list of operations from an OpenAPI document.""" spec = yaml.safe_load(openapi_yaml) if not isinstance(spec, dict): raise ValueError("The OpenAPI document must contain a YAML object.") paths = spec.get("paths", {}) if not isinstance(paths, dict): raise ValueError("The OpenAPI 'paths' field must be an object.") global_security = spec.get("security") operations: list[dict[str, Any]] = [] for path, path_item in paths.items(): if not isinstance(path_item, dict): continue for method, operation in path_item.items(): if method.lower() not in HTTP_METHODS: continue if not isinstance(operation, dict): continue security = operation.get("security", global_security) operations.append( { "path": path, "method": method.upper(), "operation_id": operation.get("operationId"), "summary": operation.get("summary", ""), "auth": _auth_mode(security), } ) return operations agent = Agent( model=load_model(), system_prompt=( "You are an API specification analyst. " "Call summarize_endpoints before reasoning. " "Report endpoints, authentication behavior, and concrete " "specification risks. Never invent fields that aren't present." ), tools=[summarize_endpoints], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) serve_a2a is the AgentCore SDK helper that does the boring, mandatory parts: /ping, Agent Card serving, AGENTCORE_RUNTIME_URL handling, Bedrock header propagation, and binding to port 9000. The validation inside that tool is the part I'd argue for hardest. OpenAPI path items legally contain keys that aren't HTTP methods parameters, servers, $ref, orsummary. Loop over every key as though it were an operation and you'll either crash on a string or, worse, emit a confident summary containing an endpoint called PARAMETERS. Then the doc generator writes it up, and it ships. Demos never surface this because demo specs are clean. Direct dependencies for both specialists: Shell bedrock-agentcore[a2a] strands-agents[a2a] a2a-sdk PyYAML httpx The a2a extras matter. Without them, serve_a2a and the client classes aren't there. a2a-sdk is what the orchestrator's client code imports from, so it's a direct dependency even though the server side pulls it in transitively. The documentation generator uses the same wrapper with a narrower job: Python # app/DocGenerator/main.py from strands import Agent from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model agent = Agent( model=load_model(), system_prompt=( "You write developer documentation from an API request and a " "specification analysis. Produce Markdown with an overview, " "authentication notes, request examples, error handling, and a " "short Python quickstart. Never invent endpoints, fields, or " "credentials." ), ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) That one's prompt-only on purpose. If your docs have to match a house format, this is where you add output validation or template tools instead of asking a model to remember a style guide. Test It Locally Shell agentcore dev # or, without the inspector UI: python main.py Then send it a JSON-RPC message at the root path: JSON curl -X POST http://localhost:9000/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": "req-001", "method": "message/send", "params": { "message": { "role": "user", "parts": [ { "kind": "text", "text": "Analyze the authentication on this API." } ], "messageId": "12345678-1234-1234-1234-123456789012" } } }' | jq . Pull the Agent Card too, before you deploy anything: Shell curl http://localhost:9000/.well-known/agent-card.json | jq . Thirty seconds of curl catches a missing discovery endpoint or a card whose skills describe an agent you deleted two commits ago. Card problems are miserable to debug from the client side, because what you see is a resolver failure, not a bad card. Deploy the Specialists The CLI defaults to IAM SigV4. Configure bearer-token auth when the caller can't sign requests, the AWS tutorial walks through Cognito for this, though Cognito isn't a requirement; any OIDC provider works. Deploy each specialist as its own runtime: Shell agentcore deploy The CLI packages the code, uploads the artifact to S3, creates the runtime, and hands back an ARN: Shell arn:aws:bedrock-agentcore:us-west-2:<account-id>:runtime/SpecAnalyzer-xyz123 Separate runtimes are half the point of doing this at all. Each specialist scales, releases, and rolls back on its own, without a coordinated deploy across the whole system. The URL Nobody Warns You About Here's the step that eats the most time. The invocation URL isn't the ARN, and it isn't a friendly hostname. It's the ARN, URL-encoded, embedded in a path: https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{url-encoded-arn}/invocations/ Every colon and slash in the ARN gets escaped with quote(arn, safe='') in Python, or let the SDK do it: Python from bedrock_agentcore.runtime import build_runtime_url url = build_runtime_url( "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/SpecAnalyzer-xyz123" ) The Agent Card sits under that path, at .../invocations/.well-known/agent-card.json. Keep the trailing slash on the base URL. A missing slash or an unescaped ARN both produce a 404, which reads like the runtime doesn't exist. Calling a Remote Agent This client assumes bearer auth. For an IAM runtime, sign with SigV4 or use a client that signs for you. Python # app/Orchestrator/client.py import os from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, TextPart DEFAULT_TIMEOUT = 300 def _message(text: str) -> Message: return Message( kind="message", role=Role.user, parts=[Part(TextPart(kind="text", text=text))], message_id=uuid4().hex, ) async def call_remote_agent( runtime_url: str, text: str, session_id: str | None = None, ) -> str: headers = { "Authorization": f"Bearer {os.environ['BEARER_TOKEN']}", "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": session_id or str(uuid4()), } async with httpx.AsyncClient( timeout=DEFAULT_TIMEOUT, headers=headers, ) as http: resolver = A2ACardResolver(httpx_client=http, base_url=runtime_url) card = await resolver.get_agent_card() config = ClientConfig(httpx_client=http, streaming=False) client = ClientFactory(config).create(card) async for event in client.send_message(_message(text)): if isinstance(event, Message): return event.model_dump_json(exclude_none=True) if isinstance(event, tuple) and len(event) == 2: task, _ = event return task.model_dump_json(exclude_none=True) raise RuntimeError("The remote agent returned no A2A result.") With streaming=False, you get exactly one result, which is either a Message or a (Task, UpdateEvent) tuple. Handle both. The tuple form is what trips people up on the first run. Returning raw JSON keeps the protocol visible for a walkthrough, but don't ship it. Normalize messages and artifacts into your own response type at this boundary. Otherwise, every downstream prompt has to understand the full A2A envelope, and you end up with models reasoning about kind fields instead of about your problem. Session IDs deserve a real decision, not a uuid4() you stopped thinking about. Fresh ID for an isolated one-shot task. Same ID when you're continuing a multi-turn conversation with the same remote agent. And keep a separate correlation ID of your own, because one user request may span several runtime sessions and you'll want to stitch them back together in the traces. The Orchestrator Each remote agent becomes a tool. Endpoints come from config; the client resolves the Agent Card at the endpoint before sending work. Python # app/Orchestrator/main.py import os from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model from client import call_remote_agent REMOTE_AGENTS = { "spec_analyzer": os.environ["SPEC_ANALYZER_URL"], "doc_generator": os.environ["DOC_GENERATOR_URL"], } @tool async def delegate(agent_name: str, instruction: str) -> str: """Send work to a named A2A specialist.""" runtime_url = REMOTE_AGENTS.get(agent_name) if runtime_url is None: allowed = ", ".join(sorted(REMOTE_AGENTS)) return f"Unknown agent '{agent_name}'. Available agents: {allowed}." return await call_remote_agent(runtime_url, instruction) orchestrator = Agent( model=load_model(), system_prompt=( "You coordinate two API specialists. Use spec_analyzer for OpenAPI " "parsing and risk analysis. Use doc_generator for documentation and " "SDK examples. When both are needed, analyze the specification " "first, then include that analysis in the documentation request. " "Never assign work outside an agent's advertised capability." ), tools=[delegate], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(orchestrator)) Two small things in there matter more than they look. The tool is async, which avoids wrapping the remote call in asyncio.run() that blows up when the agent is already running inside an event loop, and the traceback doesn't point anywhere near the cause. And returning a friendly string for an unknown agent, rather than raising, gives the model something it can recover from instead of a dead turn. The URLs belong in deployment config, a service catalog, or an agent registry. Not in source control. Let the Model Route, or Hard-Code the Path? The code above lets the orchestrator model decide who to call and in what order. That's genuinely useful when requests vary. It's also the default choice for the wrong reasons. If every documentation request must be analyzed before generation, always, no exceptions encode that sequence in Python and use the model only inside each step. Deterministic orchestration is easier to test, retry, and audit, and it doesn't drift when someone edits a prompt. Save model-driven routing for routes that are actually ambiguous. It shouldn't be there to make a fixed workflow feel more autonomous. Running It With all three runtimes deployed, send the user request to the orchestrator. Something like "Review this customer API spec and create a Python quickstart" should cause it to call the analyzer, feed that analysis to the doc generator, and return one answer. Should. That's expected behavior, not a guarantee the protocol hands you. Whether the right handoffs happen comes down to your orchestrator prompt, your tool descriptions, your eval cases, and your workflow code. A2A standardizes the conversation. It has no opinion about whether your orchestration logic is correct. Before You Call It Production Least privilege per runtime. Each runtime gets permissions for its own tools and data, nothing more. The failure mode to watch for is the orchestrator's role slowly accumulating every permission any specialist ever needed. Distinguish auth failures from agent failures. A2A application errors come back as JSON-RPC error objects, frequently with HTTP 200. Auth and authorization failures come back as native HTTP errors AccessDeniedException is a 403. AgentCore also maps runtime exceptions onto JSON-RPC codes: -32051 for not found (404), -32053 for throttling (429), -32054 for conflict (409), -32055 for a runtime client error (424). Inspect the JSON-RPC shape. A 200 is not a success. Retry deliberately. 429 and retryable-conflict 409 are worth retrying with backoff. A 424 usually means your container failed and the answer is in CloudWatch, not in a second attempt. Never blind-retry non-idempotent work, and pass a task or request identifier so a specialist can recognize a duplicate. Trace every handoff. AgentCore Observability gives you sessions, traces, and spans. Propagate your own business correlation data on top, or you'll have three sets of traces and no way to prove they belong to the same user request. Set lifecycle values on purpose. Defaults are a 15-minute idle timeout and an 8-hour max lifetime, both configurable (60 to 28,800 seconds). Long-running tasks may justify raising them; a bigger number should be a decision, not a leftover. Also watch that your A2A sessions actually go idle — a server that holds a connection open after the client is gone will happily keep a microVM warm on your bill. Cache Agent Cards carefully. Resolving on every call is simple and costs you a round trip per hop. Cache with a short TTL and invalidate when an agent's version or capabilities change. A stale card advertising a skill that no longer exists is a genuinely confusing bug. Keep protocol and SDK versions aligned. Pin dependencies, validate cards, and test a mixed-framework setup rather than assuming it works. Use private connectivity if you need it. Runtime can attach to a VPC for internal resources, and interface VPC endpoints via PrivateLink keep AgentCore API calls off the public internet. A Failure Test Matrix Before you call it done, break the handoffs rather than admiring the happy path: The specialist's Agent Card is unreachable or malformed.The bearer token expires between card resolution and invocation.The analyzer returns a valid task with no usable artifact.The doc generator times out after the analyzer succeeded.The orchestrator picks an agent that doesn't exist, or one it isn't allowed to call.A retry re-runs a task that already had an external side effect. Each of these teaches you something a successful demo can't. They also make the division of labor concrete: Runtime hosts and protects the endpoint, A2A defines the conversation, and workflow correctness is still entirely yours. Closing The value here isn't the agent count. It's that each capability gets a clear boundary, a narrow permission set, and its own release cycle. Runtime provides the managed execution boundary; A2A provides a common way to discover agents and pass work between them. So, start with one agent. Split it when responsibilities, ownership, or security boundaries justify a network hop, and not before. Where the sequence is fixed, keep the orchestration deterministic. Where routing is genuinely variable, let the orchestrator reason over well-described capabilities, and instrument every handoff you add. References AWS: Deploy A2A servers in AgentCore RuntimeAWS: A2A protocol contractAWS: AgentCore Runtime authentication and authorizationAWS: LifecycleConfiguration API referenceAWS: Configure AgentCore Runtime for VPC accessAgentCore CLIAgent2Agent protocol specification

As Large Language Models (LLMs) continue their rapid trajectory of development, software engineers and cloud architects regularly run into two systemic challenges: The Fixed Knowledge Cutoff: Model intelligence is inherently restricted to its training data window, making it blind to real-time changes. The "Air-Gap" Limitation: Out of the box, LLMs cannot securely interact with external systems or private APIs on their own. Historically, developers bypassed these hurdles by writing fragile, ad-hoc API wrappers or custom orchestrators. Enter the Model Context Protocol (MCP): an open standard designed to standardize how AI applications safely connect to external data sources and execution environments.In this article, we will explore the core architecture of MCP, look at why the Azure MCP Server is a game-changer for cloud engineers, and walk through a step-by-step guide to configuring it inside Visual Studio Code. The Core Architecture of MCP At its heart, MCP establishes a uniform "language" that allows AI applications (Hosts) to talk to external resources (Servers). Rather than building custom integrations for every new LLM or tool, developers can rely on a single, clean architecture: MCP Architecture The standard is built around four fundamental building blocks: MCP Host: The runtime environment or user interface where the AI agent operates (e.g., VS Code, Claude Desktop, Cursor).MCP Client: The architectural component within the host that initiates and maintains the active connection.MCP Server: A lightweight, modular helper service that exposes specific resources, prompts, and tools.Transport Layer: The underlying protocol facilitating communication, typically utilizing JSON-RPC 2.0 over standard input/output (stdio) or HTTPS. MCP Components How the MCP Handshake Works Instead of executing raw, unpredictable bash scripts, the interaction is highly structured: Initialization: The client connects to the server and queries its capabilities. Declaration: The server returns a structured schema listing the specific tools it supports. Execution Request: When an LLM determines it needs external data, the host requests a specific tool execution from the server. Context Injection: The server executes the local process, gathers the result, and returns a semantic JSON response to the host, which is then cleanly formatted for the user. MCP in Action Why Use the Azure MCP Server? If you are managing infrastructure on Microsoft Azure, the Azure MCP Server bridges the gap between your local AI assistant and your active cloud resources. Operating as a secure local process, it natively integrates with the Azure command-line context. The server supports over 40 Azure services and more than 170 tools out of the box, spanning critical cloud primitives: Compute & Containers: Azure Container Apps and Azure Kubernetes Service (AKS). Storage & Resource Management: Azure Storage (blobs and containers) and Azure Resource Groups. Infrastructure as Code: Integrated Azure Terraform Best Practices. Key Advantages Over Raw CLI Executions While you could technically let an AI agent execute arbitrary commands in a terminal, using a dedicated MCP server provides several major structural benefits: Rich Semantics: Instead of parsing messy, unstructured terminal stdout text, the MCP server passes rich semantic data blocks directly back to the LLM. Strict Governance & Scope: You can explicitly configure the server to run in read-only mode or expose only a select subset of tools, preventing the AI from accidentally deleting production infrastructure. Interactive Guardrails: The protocol requires explicit user confirmation before executing tools that touch sensitive data or perform mutative actions. Azure MCP Server Setup and Configuration Modes You can run the Azure MCP server locally across multiple development setups using stdio transport. Below are the three most common configuration schemas. NuGet Configuration For .NET teams, the server can be dynamically fetched and started using the dnx toolchain: JSON { "mcpServers": { "Azure MCP Server": { "command": "dnx", "args": [ "Azure.Mcp", "--source", "https://api.nuget.org/v3/index.json", "--yes", "--", "azmcp", "server", "start" ], "type": "stdio" } } } Node.js Configuration If you are developing in a standard JavaScript/TypeScript ecosystem, you can spin up the server dynamically using the latest npm package via npx: JSON { "mcpServers": { "azure-mcp-server": { "command": "npx", "args": [ "-y", "@azure/mcp@latest", "server", "start" ] } } } Docker Configuration For isolated development environments, you can run the server in a container. Note that you must provide a local environment file (.env) containing your Azure Service Principal credentials: JSON { "mcpServers": { "Azure MCP Server": { "comman { d": "docker", "args": [ "run", "-i", "--rm", "--env-file", "/full/path/to/.env", "mcr.microsoft.com/azure-sdk/azure-mcp:latest" ] } } } Step-by-Step Visual Studio Code Integration A great feature for those who want to work within Visual Studio Code is that they can also manage the Azure MCP Server through an exclusive extension available directly in their editor. The following steps walk you through the setup. Step 1: Install the Extension Search for and install the Azure MCP Server extension directly from the Visual Studio Code marketplace. Install Azure MCP Server Extension Step 2: Initialize and Verify Open the Command Palette (Cmd + Shift + P on macOS or Ctrl + Shift + P on Windows) and search for the extension commands to verify the server is active and running. Select an MCP Server Step 3: Configure Your Tool Accessibility Open your integrated AI chat window and select the Tools icon. Here, you will see a list of all active tools provided by the Azure MCP Server. You can toggle specific permissions on or off—for instance, disabling write operations while keeping read operations active. Select Your Tools Step 4: Interact Natively Now, your AI chat assistant can securely call Azure tools in the background. You can ask complex queries like: "Are there any inactive containers running in my resource group?""Upload our local configuration file directly to our Azure storage container blob storage." Any further interactions that reference an Azure account would use the MCP tools. Confirm Tools and Resources Real-World Engineering Use Cases To see the power of this setup, let’s look at how this changes day-to-day operations: Use Case A: Automated AKS Incident Troubleshooting When an incident occurs in an Azure Kubernetes Service (AKS) cluster, engineers typically run dozens of diagnostic commands. With the Azure MCP server connected, you can simply ask the LLM: "Investigate why the pods in our production namespace are crash-looping." The agent will call the relevant AKS tools, inspect the logs, identify the misconfiguration, and suggest the fix - all in seconds. Use Case B: Continuous Terraform and Compliance Audits Before deploying infrastructure, you can point your local AI agent to your code directory. Because the server incorporates Azure Terraform Best Practices, the agent can audit your configuration files, cross-reference them against your live Azure Resource Groups, and warn you if you are violating security compliance rules or generating drift. Conclusion The Model Context Protocol represents a major step forward in AI-assisted development. By standardizing the communication layer, the Azure MCP Server enables software engineers to transform static, isolated LLMs into active, context-aware cloud operators. Whether you are monitoring active Kubernetes clusters, auditing Terraform configurations, or automating file uploads to blob storage, MCP gives your AI assistant safe and highly scoped access to the "live" Azure ecosystem.