Microservices

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these: kubectl rollout restart deployment/my-app and eat the disruption.Install Reloader so it restarts your pods for you when a Secret changes.Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot). All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string. I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything. New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut. The Two Schemes You'll Use mamori's Kubernetes provider gives you two ref schemes: RefReads fromNotesk8s-secret://<namespace>/<name>#<key>a Secretvalue is base64-decoded for youk8s-cm://<namespace>/<name>#<key>a ConfigMapplain string values Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples: Plain Text k8s-secret://prod/db-creds#password # one key from a Secret, base64-decoded k8s-secret://prod/tls#ca.crt # a certificate as []byte k8s-cm://prod/app-config#log_level # one key from a ConfigMap k8s-cm://prod/app-config # the whole ConfigMap as a JSON map Setup The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes): Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/k8s Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori: Go import _ "github.com/xavidop/mamori/providers/k8s" In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: mamori-reader namespace: prod rules: - apiGroups: [""] resources: ["secrets", "configmaps"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: mamori-reader namespace: prod subjects: - kind: ServiceAccount name: my-app namespace: prod roleRef: kind: Role name: mamori-reader apiGroup: rbac.authorization.k8s.io The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls. Load Once, Then Watch Forever Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/k8s" ) type Config struct { DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"` LogLevel string `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"` MaxConns int `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"` CACert []byte `source:"k8s-secret://prod/tls#ca.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } _ = cfg // fully resolved, defaulted, validated } Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal(). Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts: Go w, err := mamori.Watch[Config](ctx, mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // Rebuild the pool with the rotated password — live, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // lock-free snapshot; always the last VALID config w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state. Informers, Not Polling This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds. Practically, that means: Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely. Compare that to the alternatives: ApproachHow a change is picked upRestart?kubectl rollout restartyou, manuallyyesReloader / Stakateroperator restarts the podyesSecret mounted as volumekubelet re-syncs file (~60s), if your app re-reads itno, but you must re-readmamoriinformer pushes change → validated atomic swap → callbackno Whole ConfigMaps and JSON Blobs Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout. A Note on TLS Certificates Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref: Go type TLS struct { Cert []byte `source:"k8s-secret://prod/tls#tls.crt?debounce=0"` Key secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"` } Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret. When You Don't Need This Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all. mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod. Wrapping Up Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout. Shell go get github.com/xavidop/mamori/providers/k8s Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

By Xavier Portilla Edo CORE

Distributed systems rarely fail as a single unit. More often, one dependency slows down, one shard becomes unavailable, one region loses capacity, or one pool saturates while neighboring components remain healthy. The engineering challenge is not simply availability; it is preventing a local defect from becoming a system-wide event. Google’s SRE guidance describes cascading failure as positive feedback in which lost capacity shifts load onto surviving capacity until additional components fail. Resilience design must break that loop early, cheaply, and at a boundary narrow enough to preserve useful work elsewhere. Containment Starts With Explicit Failure Domains A failure-containment framework begins by defining what is allowed to fail independently. Service names are usually too coarse. A single service may contain independent regions, tenants, shards, endpoint families, connection pools, or workload classes. Circuit breakers, concurrency limits, queues, and caches should align with those real failure domains rather than an arbitrary process boundary. Microsoft’s circuit-breaker guidance warns against combining independent resources, such as multiple data shards, behind one breaker because failure in one resource can incorrectly block healthy resources. This changes the design question from whether a service is healthy to which partition, operation, or dependency is unhealthy and what scope must be isolated. A breaker keyed by destination shard can contain a hot partition. A bulkhead keyed by traffic class can prevent batch work from consuming interactive capacity. A separate connection pool for an optional dependency can stop socket exhaustion there from starving critical calls. Resilience4j reflects this separation: circuit breakers control whether calls are permitted based on recent failures or slow calls, while bulkheads independently cap concurrent executions. Time Is the First Resource Budget Partial failure often presents as latency before an explicit error. A caller that waits indefinitely converts remote slowness into local resource retention: threads remain occupied, queues grow, memory pressure increases, and upstream deadlines begin expiring. gRPC recommends explicit realistic deadlines, and Google SRE recommends propagating the original request deadline through downstream RPCs so deeper services do not continue work after the result has become useless. A downstream timeout should fit inside the remaining end-to-end latency budget rather than being configured independently at every hop. For a request with 800 milliseconds remaining, a dependency call cannot safely claim a fresh two-second timeout. Deadline propagation preserves the original budget and cancellation propagation stops abandoned fan-out work. A concise Java boundary can make the budget explicit: Java public Inventory reserve(ReserveRequest request, long remainingMillis) { return inventoryStub .withDeadlineAfter(Math.min(remainingMillis, 250), TimeUnit.MILLISECONDS) .reserve(request); } The 250-millisecond cap is dependency-specific rather than universal. The containment rule is broader: downstream work must not outlive the request that justified it. Retries Must Consume a Bounded Failure Budget Retries are useful only for failures likely to be transient. Unbounded or synchronized retries create extra traffic precisely when capacity is impaired. Google documents how retries can amplify overload into cascading failure and recommends randomized exponential backoff, per-request retry limits, and retry budgets. AWS similarly recommends backoff for throttling, temporary network loss, and temporary service unavailability, while warning that retried operations should be idempotent to avoid duplicated state changes. A retry policy belongs inside the same time and load budget as the original call. It should distinguish transient statuses from permanent failures, leave enough deadline for another attempt, and stop when aggregate retry pressure becomes unsafe. gRPC implements configurable attempt limits, exponential backoff, jitter, retryable status codes, and retry throttling. Java @Retry(name = "catalog") @CircuitBreaker(name = "catalog", fallbackMethod = "catalogFallback") public ProductView loadProduct(String productId) { return catalogClient.fetch(productId); } private ProductView catalogFallback(String productId, Exception ex) { return ProductView.withoutCatalogEnrichment(productId); } The fallback is intentionally degraded rather than deceptive. Noncritical enrichment can disappear while the core response remains valid. A failed payment authorization, by contrast, should not become a fabricated success. Failure containment preserves service by removing optional work, not by weakening correctness. Resilience4j’s Spring integration supports composed retry, circuit-breaker, bulkhead, and fallback annotations for this boundary pattern. Admission Control Protects the Healthy Remainder Circuit breakers answer whether a dependency should receive calls; bulkheads answer how many calls may enter concurrently. Those controls solve different failure modes and should often be composed. Resilience4j notes that a closed circuit breaker does not limit concurrency, while its semaphore and fixed-thread-pool bulkheads provide explicit concurrency isolation. The next containment layer is admission control at saturation. Google SRE recommends early rejection, small queues where appropriate, load shedding, and graceful degradation because overloaded servers should preserve useful throughput rather than accept work until latency and resource exhaustion collapse the instance set. An HTTP 503 or equivalent overload signal can be healthier than a long queue that guarantees deadline misses. This creates a deliberate sequence under stress: optional work is removed, excess work is rejected, unhealthy dependencies are short-circuited, and concurrency remains partitioned. Capacity degrades in controlled steps instead of falling off a cliff. The design target is not zero errors, but a bounded error surface with stable latency and preserved capacity for higher-value operations. Recovery Behavior Matters as Much as Failure Detection Containment mechanisms can cause a second outage during recovery if suppressed traffic returns at once. The half-open circuit-breaker state probes recovery with limited traffic rather than immediately restoring full load. Microsoft describes this as protection against flooding a recovering service, while Resilience4j permits a configurable number of half-open trial calls before closing the breaker. Recovery also requires warm capacity. Google notes that restarted processes and cold caches can be more expensive than steady-state instances, so traffic should often be restored gradually. The same principle applies after regional failover, database restart, or large deployment: recovery is a controlled ramp, not a binary switch. Observability should expose breaker transitions, rejected bulkhead calls, timeout rates, retries, queue depth, saturation, and degraded responses. Microsoft recommends circuit-breaker monitoring and distributed tracing, while gRPC exposes retry-attempt metrics. Containment Is Proven Under Pressure A failure-containment framework is incomplete until overload and dependency failure are tested beyond the normal operating envelope. Google recommends load testing components until failure, observing whether excess demand produces bounded rejection instead of crashes, and testing recovery after load is reduced. It also recommends testing abrupt capacity loss and backend blackholing because gradual load tests alone can miss cascade behavior. The strongest distributed systems are not those that assume every dependency remains available; they are those that define exactly how much time, concurrency, retry traffic, and functional degradation each dependency may consume when it fails. Explicit failure domains, propagated deadlines, bounded retries, circuit breakers, bulkheads, load shedding, truthful fallbacks, controlled recovery, and failure testing form a single containment discipline. Partial failure then remains partial: damage stops at designed boundaries, healthy capacity continues doing useful work, and recovery proceeds without recreating the conditions that caused the incident.

The City Traffic Analogy Imagine a city's traffic system during rush hour. A minor accident blocks one lane on a major highway. Traffic slows down, but the road remains partially functional. Now imagine every driver, upon encountering the slowdown, decides to: Take an alternate route through residential streetsCircle back and try the highway againCall friends to warn them, who then also reroute What happens? The residential streets become congested. Side roads jam up. Intersections gridlock. The alternate routes become slower than the original highway. Drivers circle back, creating even more congestion. What started as a single blocked lane cascades into city-wide traffic paralysis. The problem isn't the initial accident. The problem is the collective response to it. This is exactly what happens in distributed systems when retry logic becomes aggressive during degradation. A slow downstream service is like a partially blocked highway. Blind retries are like drivers flooding alternate routes without coordination. The recovery mechanism — designed to help — amplifies the original problem. Traffic engineers solve this with adaptive signals, ramp metering to control highway entry, real-time routing that reflects current conditions, and occasionally closing certain routes entirely to let the system recover. Distributed systems need the same discipline: not just a mechanism to retry, but a policy for when to hold back. The Illusion of "Retry Until It Works" Distributed systems plan for failure using timeouts, circuit breakers, message queues, and dead-letter topics. Among these tools, retries are the most common because the logic seems simple: A call fails.Wait a moment.Try again. For an isolated network drop or server reboot, this works well. But when a dependency struggles under heavy load, a retry is never free. Every retry: Sends another request over the wire.Consumes worker or execution capacity while the request is in flight.Consumes connection-pool capacity while the request remains in flight.Adds another item to process in the queue. When thousands of clients hit the same hiccup and retry together, they create an accidental traffic storm. The recovery code stops being a safety net and becomes the main driver of the outage. How Small Glitches Snowball Consider a hypothetical event platform delivering millions of webhook notifications every hour. Suppose a downstream API slows down under sudden pressure. Instead of returning in 50 milliseconds, responses stretch to 4 seconds. Some calls time out; others return 503 errors. In this scenario, if 100,000 requests fail and each client is configured for 4 retries, the recovery mechanism can generate up to 400,000 additional retry attempts against a target that is already drowning. Plain Text Downstream slows down │ ▼ Requests time out │ ▼ Clients retry │ ▼ Extra inbound traffic │ ▼ Higher downstream latency │ ▼ Widespread failure │ ▼ Retry storm This creates a self-reinforcing loop: the system reacts to failure by generating more work, adding latency, triggering more timeouts, and generating even more retries. The Six-Layer Amplification Framework Most engineers view retry amplification as simple request volume—turning one failed request into several. In reality, the blast radius cuts across multiple infrastructure layers: Plain Text Request Amplification ──► 1 failed call multiplies into 3–5x outbound requests Worker Amplification ──► Saturated execution capacity reduces fresh-work capacity Connection Amplification ──► In-flight requests consume connection-pool capacity Queue Amplification ──► Repeated redelivery increases queue pressure and lag Latency Amplification ──► Shared resource contention slows healthy paths Neighbor Amplification ──► Failing targets starve unrelated platform tenants Requests: One failing transaction multiplies into multiple outgoing calls.Workers: Long-running retries tie up execution capacity, reducing throughput for fresh work.Connections: In-flight requests consume connection-pool capacity while waiting on responses.Queues: Repeated processing and redelivery increase queue pressure and can drive consumer lag.Latency: Resource contention across shared CPU and network paths slows down healthy endpoints.Neighbors: A single failing downstream target hogs shared platform resources, degrading unrelated tenants. The important insight is that retry amplification is not merely a traffic problem; it is a resource-propagation problem. Why Queues Cannot Prevent the Storm Message queues like Kafka, RabbitMQ, and AWS SQS provide durable buffering and asynchronous decoupling, helping systems preserve messages while consumers are unavailable or degraded. However, a queue answers one specific question: "Is the message saved safely in persistent storage?" It cannot answer: "Is it operationally safe to deliver this message right now?" If workers blindly pull messages off a topic and blast them against an overloaded API, the queue acts like a firehose pointed at a fragile target. Queues provide buffering; they do not determine whether delivery should happen now. Failure Is Not Just "Up" or "Down" Basic retry policies assume a binary world: a service is either working or offline. Real production outages are rarely that clean. A downstream dependency might be: Returning 200s, but running ten times slower than normal.Dropping 20% of requests because its database connection pool is full.Rejecting calls because memory crossed a critical limit.Throttling traffic because a tenant hit their API rate limit. If your system treats a slow, memory-starved server like a brief network blip, it will keep hammering that server until it crashes. Matching the Recovery to the Failure Different failure modes require completely different responses: Transient Blips: Dropped packets or DNS timeouts. Retry with exponential backoff and randomized jitter so clients do not hit the server simultaneously.Rate Limits (HTTP 429): The service asks you to slow down. Read Retry-After, back off, cut concurrency, and resume gradually.Downstream Overload: Requests time out under load. Trip circuit breakers, lower concurrency, and pause retries until latency recovers.Non-Retryable Client Errors: Invalid payloads, unsupported operations, or permanent business-rule rejections should not be retried. Route them directly to a dead-letter queue (DLQ) for inspection. Knowing When to Stop Resilient architectures know when doing nothing is safer than trying again. SituationUnsafe ResponseSafe ResponseHTTP 429 Rate LimitImmediate retry loopThrottle affected traffic and respect the limit Retry-AfterCircuit Breaker OpenKeep pinging targetFast-fail or park messages in DLQHigh Queue BacklogRetry all failed jobsDrop or delay low-priority retriesInvalid Message SchemaRetry 5 timesRoute directly to DLQ without retryingDownstream Latency > SLAFire aggressive retriesCut concurrency and shed non-critical load Adaptive Recovery: Check System State First A static policy like retries = 3, backoff = 2s has no awareness of the surrounding environment. Consider two situations handling the same error: Scenario A: Queue depth is 150, CPU is at 20%, downstream latency is 40ms. Retrying is safe.Scenario B: Queue depth is 700,000, workers are 98% busy, downstream latency is 9s. Retrying will trigger an outage. A resilient system checks platform health before firing another request: Plain Text Request Fails │ ▼ Check Failure Type ──► Permanent error? ──► Send to DLQ │ (Transient) ▼ Check Downstream Health (Circuit breaker / Latency) │ (Degraded) ▼ Check Local Load (Queue depth / Worker pool) │ ├─► System healthy ──► Exponential backoff + Jitter ──► Retry │ └─► System stressed ─► Park message / Throttle rate ─► Delay Protecting Shared Resources To keep one broken dependency from pulling down the entire platform: Enforce a Retry Budget: Limit the capacity recovery traffic can consume based on normal workload traffic and available system headroom. When exhausted, delay, throttle, or park retries.Isolate Worker Pools: Process retries on dedicated worker pools so they cannot starve fresh incoming requests.Partition by Target: Apply distinct concurrency limits per downstream endpoint so healthy routes run unaffected.Prioritize Critical Work: When queues back up, retry critical jobs (payments, security alerts) and defer low-priority ones (analytics).Warm Up Slowly: When a failed service recovers, release queued traffic in controlled batches rather than dumping the backlog all at once. What to Measure in Production Standard metrics like error rate and average CPU hide retry problems until damage is done. Track these indicators instead: Retry-to-Primary Ratio: Track how much outgoing capacity is consumed by retries. A sustained increase signals that recovery traffic is becoming a significant source of system load.Retries per Recovery: How many attempts does a message need before succeeding? Rising numbers mean wasted compute.Worker Occupancy: How much worker capacity is spent processing retries instead of fresh work?Circuit Breaker Trip Rate: How often and how long breakers stay open across each downstream target. Summary: Recovery Has a Cost Retrying is not just an error-handling convenience — it is a resource allocation decision. Every recovery attempt consumes bandwidth, threads, memory, and downstream capacity. When systems fail under load, retrying harder often becomes the primary mechanism of collapse. To build genuinely resilient distributed architectures: Treat recovery actions as operations with real infrastructure costs.Filter errors early so permanent failures are never retried.Respect downstream throttling signals and use circuit breakers when defined failure or latency thresholds are reached.Use randomized jitter and exponential backoff to prevent self-inflicted traffic spikes.Check operational system state before deciding whether another attempt will help or harm. True system resilience is not about trying indefinitely until a call succeeds. It is about recognizing that recovery consumes resources, and ensuring every recovery decision accounts for system state before taking action.

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

The project AngularAndSpring has been refactored from Spring Reactor to Spring blocking APIs using JDK 25 and virtual threads. The Old Architecture The reason AngularAndSpring was built using Spring Reactor was primarily the efficient use of operating system threads and memory under high load. Spring Reactor had a limited thread pool (usually the number of available CPU cores) and used OS threads only when the CPU had work to do. It did not let the operating system threads sleep. This used CPU resources much better and required far fewer OS threads, each of which needs around a megabyte of memory. The trade-off was a reactive programming style, which can become harder to read as the complexity of the implemented functionality grows, along with stack traces that are very difficult to read. These hard-to-read stack traces make finding and fixing errors much harder and significantly more time-consuming. How Virtual Threads Change the Trade-Off Before JDK 25, the choice was between efficient hardware use with Spring Reactor and easy-to-read code with easier debugging and readable stack traces. Spring's support for Java virtual threads offers the use of very efficient virtual threads for request processing. With virtual threads, operating system threads are used only when processing is needed; virtual threads are unmounted from the OS thread, freeing the OS thread to run a different virtual thread; and virtual threads have minimal memory requirements. That means a limited number of operating threads is able to handle a very large number of virtual threads. With JDK 25 and the virtual threads update, the choice has changed: JDK 25 with virtual threads uses hardware resources efficiently, and you can keep the code easy to read and debug with readable stack traces. There are limitations for virtual threads when native code is called, but the same limitations apply for Spring Reactor too. The main virtual threads blocker: The missing synchronized support was fixed in JDK 25. That means the main advantages of Spring Reactor are now provided by the JVM with its virtual threads. The New Architecture The basic building blocks of the new architecture can be found in the backend pom.xml. It now uses these starters: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-mongodb</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-restclient</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> The virtual threads are switched on with this in the application.properties: Properties files spring.threads.virtual.enabled=true The spring-boot-starter-webmvc is used to provide REST endpoints with blocking APIs. The spring-boot-starter-restclient is used to provide the new REST client for calling the blocking REST endpoints of other systems. The spring-boot-starter-data-mongodb is used to provide repositories and APIs for accessing MongoDB; all of these APIs are blocking. The ‘spring.threads.virtual.enabled=true’ enables the Spring Boot support for virtual threads provided by the JVM. The REST Endpoint The blocking REST endpoints now look like this in BitstampController: Java @RestController @RequestMapping("/bitstamp") public class BitstampController { private final BitstampService bitstampService; public BitstampController(BitstampService bitstampService) { this.bitstampService = bitstampService; } @GetMapping("/{currpair}/orderbook") public String getOrderbook(@PathVariable String currpair) { return this.bitstampService.getOrderbook(currpair); } ... } These endpoints do not return the reactor wrapper types. It just returns the intended return type. The ‘BitstampService’ methods now also return the intended return types without any wrappers. These endpoints just use plain Java. The Restclient The new blocking RestClient is now the Spring RestClient and looks like this in ScheduledTask: Java @Component public class ScheduledTask { private static final Logger LOG = LoggerFactory.getLogger(ScheduledTask.class); private static final String URLBS = "https://www.bitstamp.net/api"; private static final String URLCB = "https://api.coinbase.com/v2"; private static final String URLBF = "https://api.bitfinex.com"; private final BitstampService bitstampService; private final BitfinexService bitfinexService; private final CoinbaseService coinbaseService; private final MyUserService myUserService; private final RestClient.Builder restClientBuilder; public ScheduledTask(BitstampService bitstampService, MyUserService myUserService, BitfinexService bitfinexService, CoinbaseService coinbaseService, RestClient.Builder restClientBuilder) { this.bitstampService = bitstampService; this.bitfinexService = bitfinexService; this.coinbaseService = coinbaseService; this.myUserService = myUserService; this.restClientBuilder = restClientBuilder; } ... @Async("clientTaskExecutor") @Scheduled(fixedRate = 60000, initialDelay = 3000) @SchedulerLock(name = "BitstampQuoteBTC_scheduledTask", lockAtLeastFor = "PT50S", lockAtMostFor = "PT55S") public void insertBitstampQuoteBTC() { String currPair = "btceur"; insertBsQuote(currPair); } private ClientHttpRequestFactory customRequestFactory() { HttpClientSettings settings = HttpClientSettings.defaults() .withConnectTimeout(Duration.ofSeconds(4)) .withReadTimeout(Duration.ofSeconds(5)); return ClientHttpRequestFactoryBuilder.detect().build(settings); } private void insertBsQuote(String currPair) { LocalTime start = LocalTime.now(); try { QuoteBs quote = this.restClientBuilder.requestFactory(this.customRequestFactory()).build().get() .uri(String.format("%s/v2/ticker/%s/", ScheduledTask.URLBS, currPair)) .accept(MediaType.APPLICATION_JSON).retrieve().body(QuoteBs.class); quote.setPair(currPair); quote = this.limitPrecision(quote); this.bitstampService.insertQuote(quote); this.logDuration("Bitstamp", currPair, start); } catch (Exception e) { LOG.warn(String.format("Bitstamp data import failed for: %s", currPair), e); } } ... } The RestClient is created using the RestClientBuilder to allow connect and read timeouts to be configured. The timeouts are created in the customRequestFactory() method and set as a parameter of the RestClientBuilder. The RestClient uses the .uri(...) method to set the URL to call. The .accept(...) method sets the MediaType in the HTTP header that is expected from the REST client. The .retrieve() method executes the REST call, and the .body(...) method specifies the object into which the JSON response should be mapped. For debugging purposes, it can be useful to map the response body to a String first and analyze the response to ensure that it can be mapped correctly to the appropriate DTOs. The MongoDB Client For accessing the database, Spring Data MongoDB is used. The per-minute quotes are in the QuoteBs, QuoteBf, QuoteCb entities. The per-hour quotes are in the QuoteHourBs, QuoteHourBf, QuoteHourCb entities, and the per-day quotes are in the QuoteDayBs, QuoteDayBf, QuoteDayCb entities. The different entities enable Spring Data MongoDB support for each entity with its own repository. This is an example of QuoteBsMongoRepository: Java public interface QuoteBsMongoRepository extends MongoRepository<QuoteBs, ObjectId>, QuoteBsRepository { ... } The Interface ‘MongoRepository<QuoteBs, ObjectId>’ provides standard interfaces and enables the generated CRUD methods in QuoteBsRepository. Java public interface QuoteBsRepository extends QuoteRepository<QuoteBs>, QuoteMongoRepository<QuoteBs> { ... QuoteBs insert(QuoteBs quote); List<QuoteBs> findByPairAndCreatedAtAfterOrderByCreatedAtAsc(String pair, Date date, Limit limit); @Query("{ 'createdAt': { '$gt': ?0, '$lt': ?1 } }") List<QuoteBs> findByCreatedAtGreaterThanAndCreatedAtLessThan(Date from, Date to); } This interface uses the by-method-name-generated CRUD methods that Spring Data MongoDB provides and uses the ‘@Query’ annotation for a finder that can not be generated only by a method name. The entity QuoteBs looks like this: Java @Document public class QuoteBs implements Quote,QuotePairBs { @Id private ObjectId _id; @NotBlank @Indexed @JsonProperty private String pair; @NotNull @Indexed(name = "QuoteBs-createdAt") @JsonProperty private Date createdAt = new Date(); private BigDecimal high; private BigDecimal last; private Date timestamp; private BigDecimal bid; private BigDecimal vwap; private BigDecimal volume; private BigDecimal low; private BigDecimal ask; private BigDecimal open; The ‘@Document’ annotation marks the class as a MongoDB document. The ‘QuotePairBs’ interface is used to be able to access the common properties of minute, hour, and day entities of the same source. The ‘@Indexed’ annotation makes sure an index is created for this property in MongoDB for faster access and ordering. The precision of the Java BigDecimal type is higher than the precision of the MongoDB number type. Because of that, this method in ScheduledTask is used: Java private <T> T limitPrecision(T quote) { var fields = quote.getClass().getDeclaredFields(); for(Field field : fields) { if(field.getType().isAssignableFrom(BigDecimal.class)) { field.setAccessible(true); try { var value = (BigDecimal) field.get(quote); value = value.round(new MathContext(30, RoundingMode.HALF_UP)); field.set(quote, value); } catch (IllegalAccessException e) { throw new RuntimeException(e); } } } return quote; } The precision is limited to 30 decimals for all properties with the type BigDecimal. This is enough for all use cases in this application. Thread Pools If the hour and day quotes are recalculated from a large number of minute quotes, this can take several minutes. To avoid competing with the thread pool handling the user requests, a separate thread pool is created for this task in SchedulingConfig: Java @Bean(name = "clientTaskExecutor") public Executor threadPoolTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setVirtualThreads(true); executor.setThreadNamePrefix("clientTask-"); return executor; } This creates a separate TaskExecutor that can be used in the ‘@Async(“clientTaskExecutor”)’ annotations. That makes sure that no user requests wait in the queue for a free thread while long-running threads are at work. Conclusion With virtual threads in JVM 25 and Spring’s support for them, many reasons for using reactive programming in Spring have disappeared. The synchronized issue with virtual threads has been fixed in JDK 25, and the remaining causes of thread pinning are not relevant anymore for most use cases. The JVM now provides scalability and efficiency of reactive frameworks out of the box, while the blocking programming model is easier to implement and debug. Spring Data MongoDB support makes database access much easier with generated CRUD methods and the ‘@Query(…)’ support. It is time for projects still using reactive programming to move to an easier, faster programming model for developers.

Context The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP. One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed. The service used placement constraints, max replicas per node = 1, and a Traefik health check configured for /actuator/health on port 8080. This meant we relied on external health checking and strictly controlled replica placement across nodes. Swarm-level load balancing was intentionally disabled. We needed routing control at the Traefik layer, along with the ability to quickly remove unhealthy backends based on their health check status. For this production setup, that provided more flexibility than the built-in Swarm load-balancing logic. What Happened The issue was neither a widespread outage nor a failure of the entire service. One replica became unreachable to Traefik over the overlay network: a Traefik instance on one of the manager nodes could not establish a connection to that replica’s backend endpoint. The second replica continued to work normally, so the service as a whole remained available. For regular traffic, the incident did not affect service availability: Traefik continued routing requests to the healthy replica and removed the unavailable backend from routing. During targeted manual testing of the unhealthy replica’s backend endpoint through Traefik, requests ended in a timeout or HTTP 503. The container appeared healthy, but only locally. The service responded inside the container, and /actuator/health returned UP. However, other containers and Traefik instances on remote nodes could not establish a connection to that replica through its IP address in mgp_default. Initial Checks The container had the same DNS name in both networks. This ruled out name mismatch as an independent cause of the unavailability: the issue occurred specifically at the network-connectivity level to the replica IP in mgp_default. First, we identified the problematic container and inspected its state: Shell docker ps --filter "name=mgp_router-app" --q docker inspect 6bd15b62f891 Next, we checked the networks connected to the container: Shell docker inspect 6bd15b62f891 --format '{{json .NetworkSettings.Networks}' | jq The container was attached to two networks: ingress with IP 10.22.3.137/22;mgp_default with IP 10.29.72.232/13. We then entered the container and checked whether the application was listening on the expected port: Shell docker exec -it 6bd15b62f891 bash ss -tuln The health endpoint also responded correctly: Shell curl "http://10.29.72.232:8080/actuator/health" The expected response was: Shell {"status":"UP"} At this point, it was clear that the issue was not the application itself, its startup, or its bind address inside the container. The service was alive, but that alone said nothing about its reachability from outside the container. Checking Connectivity Between Nodes The next step was critical: We checked whether another node in the cluster could reach the container using its overlay IP. Shell docker exec -it 373354916bfc bash ping 10.22.3.137 This was the IP from the ingress network, and the test confirmed ICMP reachability of the address in that network. That indicated the issue did not look like a complete loss of inter-node connectivity. The situation was different for the mgp_default IP: pinging it from other nodes failed, even though this overlay network was supposed to provide inter-container communication in Docker Swarm. This distinction was important. ICMP reachability of the address in ingress remained available, while connectivity to the replica through mgp_default was broken. Therefore, the problem was not the container itself or a complete network failure; it was a specific break in the network path used to reach that replica. What Docker Logs Showed The dockerd logs on the affected node provided another useful signal. They contained repeated messages related to Swarm network synchronization: Shell Jun 02 11:08:42 node-router2 dockerd: time="2025-06-02T11:08:42.462965233Z" level=warning msg="bulk sync to node 091d01df89ae failed: failed to send a TCP message during bulk sync: dial tcp 10.170.0.145:7946: i/o timeout" Jun 02 11:17:02 node-router2 dockerd: time="2025-06-02T11:17:02.581936915Z" level=error msg="Bulk sync to node 091d01df89ae timed out" The logs also contained NetworkDB stats entries: Shell Jun 02 11:07:08 node-router2 dockerd: time="2025-06-02T11:07:08.012509772Z" level=info msg="NetworkDB stats node-router2(2642aeb5172b) - netID:q9athb3moj967oxmb3y8xw4lj leaving:false netPeers:58 entries:13563 Queue qLen:2 netMsg/s:7" These messages do not prove a specific bug or root cause, but they indicate problems with exchanging network state between nodes. Combined with the endpoint being unreachable through mgp_default, they strengthened the working hypothesis of an overlay-networking issue. This was an important diagnostic signal: the Docker daemon was running, and the container was started, but the cluster’s network state was not ideal and was likely becoming inconsistent during node-to-node state exchange. Investigating the Hypotheses Hypothesis 1: The container itself was the problem. This hypothesis was not confirmed. A curl request inside the container to http://10.29.72.232:8080/actuator/health returned {"status":"UP"}, and ss -tuln showed that the service was listening on the required port. Hypothesis 2: The problem was port publishing or binding. This hypothesis was not confirmed. Even though port 10005 was reachable externally, that did not explain why Traefik could not reach port 8080 over the overlay network. The container worked locally, while the issue occurred on the network path between nodes. Hypothesis 3: Traefik itself was the problem. Probably not. Traefik was not the source of the failure; it was the first component that detected that the backend was unavailable from its node. Hypothesis 4: The Docker Swarm overlay network was the problem. This became the primary working hypothesis. The replica was reachable from inside its container but was not properly reachable from other nodes through its overlay IP. Shell curl http://10.29.72.232:8080 nc -vz 10.29.72.232 8080 The curl request worked locally, while nc -vz 10.29.72.232 8080 failed from other nodes. This indicated that traffic was not being routed correctly through the overlay network. An MTU/VXLAN issue was also considered. This is a plausible class of problems for Docker overlay networks, especially when there is an MTU mismatch or a Path MTU Discovery issue. In this case, however, it remained only a hypothesis rather than a confirmed root cause. Recovery We restored the problematic replica by forcing Docker Swarm to recreate the task through a rolling update: Shell docker service update --force \ --update-parallelism 1 \ --update-order start-first \ --update-delay 30s \ --detach=true \ mgp_router-app --force triggered an update even though neither the image nor the service configuration had changed. Swarm recreated the task for the affected replica and reattached it to mgp_default; after that, Traefik could reach it again over the overlay network. The rollout parameters were selected to avoid affecting service availability: --update-parallelism 1 – update one replica at a time;--update-order start-first – start the new task before stopping the old one;--update-delay 30s – give the new task time to start and pass its checks. Restarting Docker on the node and placing the node into drain mode were considered fallback measures, but they were not required to resolve this incident. Why It Happened The exact upstream root cause cannot be stated without additional evidence. However, the most well-supported explanation is that a local network-state inconsistency occurred in the Docker Swarm overlay network. As a result, one replica remained operational inside its container, but became unreachable through its overlay IP. The most plausible explanations include: the overlay network on the node was not fully ready when the container started;an issue in Docker libnetwork;an outdated VXLAN routing table;an iptables issue or a race condition while updating network rules;a node restart without restarting the Docker daemon;an issue with a Swarm endpoint that was not properly advertised or connected. These situations are particularly unpleasant because Docker continues to consider the container running and its internal health check green, while the network path to the replica is already partially broken. Key Takeaways The main conclusion is simple: a healthy container does not necessarily mean a reachable replica. This is critical in production because: a local curl inside the container does not prove external reachability;an internal health check does not guarantee that Traefik can reach the backend;Swarm may remain formally healthy even when one replica has dropped out of normal network connectivity. Therefore, in such clusters, it is necessary to check not only container state but also inter-node reachability over the overlay network. What to Monitor To prevent similar incidents, it is useful to monitor not only service health but also the cluster’s network plane. The minimum set of checks includes: reachability of container overlay IPs from other nodes;health checks through Traefik;bulk sync and NetworkDB errors in Docker logs;recurring issues on the same node;consistency between a container’s local health and its external reachability. It is also useful to have a script that specifically checks connectivity between nodes through the overlay network, not just through ingress or localhost. Conclusion This case demonstrated several important points. First, a container can be healthy locally while remaining unreachable externally. This is especially important for services that handle user traffic. Second, Traefik is a good early indicator of network problems. It can reveal that a backend on one node has stopped responding. Third, Docker Swarm overlay networking can degrade locally. In that case, the cluster may appear operational, while one replica has already dropped out of normal routing. Fourth, such incidents must be diagnosed along the network path, not only through application state. First, check the container from the inside, then verify reachability from other nodes, then inspect Docker logs, and only after that draw conclusions about the root cause.

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do that, Kafka has a complex internal structure. In this part, I will try to cover an overview of the full internal structure. Overview Kafka uses a publish-subscribe model where producers produce events and consumers consume messages. Kafka producers publish events to the broker, and consumers consume the events from the broker. In the meantime, the broker has a different layer of architecture that allows Kafka to be an event streaming platform. The producer, broker, and consumer-related metadata are managed by the Kafka controller, which is elected by ZooKeeper or KRaft. Key Terms Topic: A topic is a logical name or category where Kafka stores a stream of related events/messages. Think of a topic as a database table for events or a channel where producers publish messages and consumers subscribe to read them. Partition: A partition is a physical subdivision of a Kafka topic that stores messages in an ordered, immutable sequence. It is the basic unit of parallelism and scalability in Kafka. Segment: A segment is a physical file inside a Kafka partition where messages are stored on disk. A partition is not stored as one huge file. Kafka divides it into multiple smaller segment files. Offset: An offset is a unique sequential number assigned to each message within a Kafka partition. It represents the position of a message in that partition. Metadata: Metadata is information about the Kafka cluster, not the actual message data. It describes the structure and state of the Kafka cluster. Controller: The Kafka Controller is a special broker responsible for managing cluster-level operations. In a Kafka cluster, one broker is elected as the controller. ZooKeeper (OLD): ZooKeeper is an external distributed coordination system used by older Kafka versions to manage cluster metadata and controller election. Kraft (Modern): KRaft (Kafka Raft metadata mode) is Kafka's new architecture that removes ZooKeeper and manages metadata internally using the Raft consensus algorithm. Leader: A leader is the broker that owns the primary copy of a partition and handles all read and write operations for that partition. Only one broker can be the leader for a partition at a time. Replica: A replica is a copy of a partition stored on another broker to provide fault tolerance. Internal Details With the Kafka Library At first, download Kafka from this link. After extracting, we will get the following directory structure: Here, each folder has its own functionality. The bin folder contains all the executable scripts. These scripts can be used from the command line to manage Kafka components and perform various tasks, including those listed below: Start serversStop serversTopic managementCluster managementAct as a publisher/consumer Another important folder is the config folder, where we can configure the broker and controller nodes. Initially, this folder contains the default configuration files for the Kafka components. Here we find property files named controller.properties, server.properties, broker.properties, consumer.properties, producer.properties, and some other configuration files. In a Kafka installation, these .properties files are configuration files used by different Kafka components and client applications. However, not all of them are used by the Kafka server itself. We can think of the configuration files in three major categories: Servers - server.properties - broker.properties - controller.propertiesClients - producer.properties - consumer.propertiesTools - other properties file 1. server.properties This is the main configuration file for a Kafka broker. It controls how the Kafka broker runs, including: Broker ID/node identityListener and network configurationLog directoriesNumber of network and I/O threadsDefault number of partitionsReplication settingsRetention policiesMessage size limitsController configuration in KRaft modeCluster-related settings For example: Properties files node.id=1 process.roles=broker,controller listeners=PLAINTEXT://:9092,CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 log.dirs=/tmp/kafka-logs num.partitions=1 log.retention.hours=168 In modern Kafka using KRaft, server.properties can configure both the broker and controller when a combined node is used. 2. broker.properties This is generally used as a separate broker configuration file when you want to maintain configuration for a broker independently. For example, you might create: Plain Text config/ ├── broker.properties ├── controller.properties and start Kafka with a particular configuration: Properties files kafka-server-start.sh config/broker.properties The exact files supplied with Kafka can vary depending on the Kafka version/distribution. broker.properties is not a universal mandatory filename in Kafka; Kafka mainly cares about the properties file you pass to the startup command. 3. controller.properties This is used for configuring a KRaft controller. Kafka's modern architecture doesn't require ZooKeeper. Instead, Kafka uses the KRaft consensus mechanism. A controller is responsible for managing cluster metadata, such as: Which brokers are part of the clusterTopic metadataPartition informationLeader assignmentsCluster configurationController elections A dedicated controller configuration could contain things such as: Properties files process.roles=controller node.id=1 listeners=CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 controller.listener.names=CONTROLLER A production Kafka cluster may use dedicated controller nodes rather than running broker and controller roles together. 4. producer.properties This is not a broker configuration. It contains configuration for a Kafka Producer client — the application that sends messages to Kafka. For example: Properties files bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer Other producer settings can control: acksretriesbatch.sizelinger.mscompression.typebuffer.memoryenable.idempotencehghg For example: Properties files acks=all retries=10 compression.type=snappy A producer application reads these settings when connecting to Kafka. 5. consumer.properties This is also a client configuration, not a broker configuration. It controls how a Kafka Consumer reads messages. Example: Properties files bootstrap.servers=localhost:9092 group.id=my-consumer-group key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer auto.offset.reset=earliest Important consumer settings include: bootstrap.serversKafka broker addressesgroup.idConsumer group identitykey.deserializerConverts message key bytesvalue.deserializerConverts message value bytesauto.offset.resetWhat to do when no valid offset existsenable.auto.commitControls automatic offset commitsmax.poll.recordsMaximum records returned per pollfetch.min.bytesMinimum data fetched from broker 6. Other Configuration Files Depending on your Kafka version, you may also find files such as: Plain Text config/ ├── connect-distributed.properties ├── connect-standalone.properties ├── connect-log4j.properties ├── tools-log4j.properties └── ... Some are for Kafka's additional components. For example: Kafka Connect Plain Text connect-distributed.properties connect-standalone.properties These configure Kafka Connect workers. Logging Plain Text connect-log4j.properties tools-log4j.properties These configure logging behavior for particular Kafka tools/components. So, server.properties/broker.properties/controller.properties configure Kafka infrastructure, while producer.properties and consumer.properties configure applications that communicate with Kafka. Step by Step: Test Producer and Consumer From CLI We will discuss the Kafka cluster later, but for now we will check how to run the Kafka controller, broker, and an example producer and consumer from the CLI. 1. Initialize Kafka Storage For a new Kafka installation running in KRaft mode, first generate a cluster ID. If we move to the Kafka bin directory, we can find a file called kafka-storage.sh Shell ./kafka-storage.sh random-uuid Copy the generated UUID. Then format the Kafka storage: Shell ./kafka-storage.sh format \ -t <CLUSTER_ID> \ -c ../config/server.properties For example: Shell ./kafka-storage.sh format \ -t abc123xyz \ -c ../config/server.properties You normally perform this initialization only once for a fresh Kafka storage directory. Don't repeatedly format an existing cluster unless you intentionally want to initialize/reset it. 2. Start the Kafka Server Because we are now inside the bin, specify the configuration file relative to the parent directory: Shell ./kafka-server-start.sh ../config/server.properties Or we can run it in the background: Shell ./kafka-server-start.sh -daemon ../config/server.properties The important relationship is: Plain Text bin/ └── kafka-server-start.sh config/ └── server.properties So the command is Shell ./kafka-server-start.sh ../config/server.properties 3. Verify That Kafka Is Running We can also check the Kafka port: Plain Text lsof -i :9092 4. Create a Topic Still inside the bin directory: Shell ./kafka-topics.sh \ --create \ --topic test-topic \ --bootstrap-server localhost:9092 \ --partitions 1 \ --replication-factor 1 List topics: Shell ./kafka-topics.sh \ --list \ --bootstrap-server localhost:9092 5. Start a Consumer Open another terminal and go to Kafka's bin directory: Shell ./kafka-console-consumer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 The consumer will wait for messages. 6. Start a Producer Open another terminal: Shell ./kafka-console-producer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 Now type: Plain Text Hello Kafka The message should appear in the consumer terminal. This is the introduction to this series. We will discuss more details in the upcoming articles.

The Illusion of a Single Database In a traditional monolithic application, maintaining data consistency is straightforward. If you need to create a new order and update warehouse inventory, you wrap the logic inside a single database transaction: Java @Transactional public void placeOrder(OrderRequest request) { orderRepository.save(request.toOrder()); inventoryRepository.decrementStock(request.getItemId(), request.getQuantity()); } If the inventory update throws an exception, the relational database rolls back the entire transaction. Either both operations succeed, or neither does. In a distributed microservices architecture, that safety net disappears. When your OrderService writes a record to a local PostgreSQL database and immediately publishes an event to an Apache Kafka cluster to notify the InventoryService, you are dealing with two completely independent, non-atomic systems. The Catastrophic Failure Modes When you attempt to write to a local database and publish an event within the same business method, one of two failures will inevitably occur: Scenario A (Database First, Network Fails) [Save to Database: SUCCESS] ──> [Network / Kafka Outage: FAILS] Result: Order exists in database, but downstream services are never notified. Scenario B (Publish First, Database Fails) [Publish to Kafka: SUCCESS] ──> [Database Unique Constraint Violation: FAILS] Result: Downstream services charge payment or pack inventory for an order that was never saved. Distributed two-phase commit (2PC) protocols are notoriously slow, fragile, and rarely supported across modern cloud-native message brokers. To achieve guaranteed consistency without blocking throughput, the industry-standard architecture is the Transactional Outbox Pattern. The Blueprint: The Transactional Outbox Pattern Instead of trying to speak to two external systems at once, the microservice performs all its operations within a single, local database boundary. Plain Text [Incoming Request] │ ▼ ┌───────────────────────────────────────────────────────────┐ │ Local ACID Transaction │ │ ├── 1. Insert into orders table │ │ └── 2. Insert into outbox_events table │ └───────────────────────────────────────────────────────────┘ │ ▼ [Outbox Relay / Change Data Capture (CDC)] │ ▼ [Message Broker: Kafka Topic] Atomic local write: The application saves the domain entity (orders) and a corresponding event payload into an outbox_events table inside the exact same local @Transactional boundary.Asynchronous relay: An independent background process reads the outbox table and publishes the messages to Kafka.Acknowledgment: Once Kafka acknowledges receipt, the relay marks the outbox event as published or removes the row. 1. Database Schema for the Outbox Define a dedicated outbox table designed for high-throughput polling and sequential reads: SQL CREATE TABLE outbox_events ( id UUID PRIMARY KEY, aggregate_type VARCHAR(255) NOT NULL, aggregate_id VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, processed BOOLEAN DEFAULT FALSE, processed_at TIMESTAMP WITH TIME ZONE ); CREATE INDEX idx_outbox_unprocessed ON outbox_events (created_at) WHERE processed = FALSE; 2. The Application Layer: Atomic Persistence The Spring service writes both the domain entity and the outbox event in one atomic operation: Java package com.example.outbox.service; import com.example.outbox.dto.OrderRequest; import com.example.outbox.entity.Order; import com.example.outbox.entity.OutboxEvent; import com.example.outbox.repository.OrderRepository; import com.example.outbox.repository.OutboxRepository; import com.fasterxml.jackson.databind.ObjectMapper; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import java.time.Instant; import java.util.UUID; @Service public class OrderService { private final OrderRepository orderRepository; private final OutboxRepository outboxRepository; private final ObjectMapper objectMapper; public OrderService(OrderRepository orderRepository, OutboxRepository outboxRepository, ObjectMapper objectMapper) { this.orderRepository = orderRepository; this.outboxRepository = outboxRepository; this.objectMapper = objectMapper; } @Transactional public void createOrder(OrderRequest request) { // 1. Persist domain entity Order order = new Order(UUID.randomUUID(), request.getCustomerId(), request.getTotalAmount()); orderRepository.save(order); // 2. Persist outbox event inside the exact same transaction try { String jsonPayload = objectMapper.writeValueAsString(order); OutboxEvent outbox = new OutboxEvent( UUID.randomUUID(), "ORDER", order.getId().toString(), "ORDER_CREATED", jsonPayload, Instant.now(), false ); outboxRepository.save(outbox); } catch (Exception e) { throw new RuntimeException("Failed to serialize outbox event payload", e); } } } 3. The Relay Layer: Reliable Kafka Publishing An asynchronous background worker polls the unprocessed records in the outbox and delivers them to the broker: Java package com.example.outbox.relay; import com.example.outbox.entity.OutboxEvent; import com.example.outbox.repository.OutboxRepository; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; import org.springframework.transaction.annotation.Transactional; import java.time.Instant; import java.util.List; @Component public class OutboxMessageRelay { private static final Logger log = LoggerFactory.getLogger(OutboxMessageRelay.class); private final OutboxRepository outboxRepository; private final KafkaTemplate<String, String> kafkaTemplate; public OutboxMessageRelay(OutboxRepository outboxRepository, KafkaTemplate<String, String> kafkaTemplate) { this.outboxRepository = outboxRepository; this.kafkaTemplate = kafkaTemplate; } @Scheduled(fixedDelayString = "${app.outbox.poll-interval-ms:2000}") @Transactional public void publishPendingEvents() { List<OutboxEvent> pendingEvents = outboxRepository.findTop50ByProcessedFalseOrderByCreatedAtAsc(); for (OutboxEvent event : pendingEvents) { try { // Publish using aggregateId as the Kafka partition key to preserve message ordering kafkaTemplate.send("orders-events", event.getAggregateId(), event.getPayload()) .whenComplete((result, ex) -> { if (ex == null) { event.setProcessed(true); event.setProcessedAt(Instant.now()); outboxRepository.save(event); log.info("Successfully relayed outbox event: {}", event.getId()); } else { log.error("Failed to relay event to Kafka: {}", event.getId(), ex); } }); } catch (Exception e) { log.error("Synchronous dispatch failure for outbox event: {}", event.getId(), e); break; // Halt batch progression to preserve order } } } } 4. Production Engineering Guardrails Preserve Partition Ordering Always use the aggregate_id (such as order_id) as the partition key when publishing the Kafka message. This ensures all state transitions for a single business entity land on the exact same Kafka partition and are processed in strict sequence. Polling vs. Log-Based CDC (Debezium) Scheduled polling is easy to set up and ideal for small-to-medium systems. For high-volume enterprise platforms processing thousands of writes per second, replace database polling with Change Data Capture (CDC) engines like Debezium. Debezium reads the database write-ahead log (WAL) directly, streaming changes to Kafka with zero query overhead on the operational database. Idempotency on the Consumer The Transactional Outbox pattern guarantees at-least-once delivery. If the relay publishes an event to Kafka but crashes before marking the outbox row as processed, it may re-send the message upon restart. Downstream consumers must maintain an idempotency check (e.g., tracking processed message IDs in Redis) to discard duplicates. Architectural Strategy Matrix Dimension Dual-Write (@Transactional + Kafka) Distributed 2PC Transactional Outbox Pattern Data Consistency Broken (Silent inconsistencies) Strong Strong (Eventual consistency) System Latency Low High (Blocking locks) Ultra-low local execution Broker Resilience Fragile (Network crashes drop events) Low High (Decoupled publishing) Operational Simplicity Deceptively simple Complex Straightforward Summary In distributed systems, atomicity cannot cross network boundaries. When you attempt to update a local database and publish a message to an event bus inside the same method, failure is a mathematical certainty over time. By shifting to the Transactional Outbox Pattern, you leverage the battle-tested ACID guarantees of your relational database to capture domain state and outbound events simultaneously. This eliminates the dual-write anti-pattern, guarantees at-least-once delivery, and builds a dependable bridge between relational transactions and event-driven architecture.

In my previous article, I walked through running coding agents inside Docker Sandboxes on a local machine. We installed the sbx CLI, started with a small project, and covered the commands needed to run, stop, and remove a sandbox. This time, I want to take that same workflow off the laptop. Docker added cloud sandboxes in version 0.42.0. You can now use sbx --cloud to run an agent on Docker-managed infrastructure instead of using your machine for the sandbox’s compute. The command is simple to use. The part that is worth understanding is how you get your code into that environment, work with the agent, and bring the changes back locally. That is what we will do here. Nothing complicated; we will start with a small Python project, one coding task, and a cloud sandbox. We will remove the sandbox when we are done with the work. What Changes With a Cloud Sandbox? The sbx CLI still runs in your terminal. With --cloud, supported commands target Docker’s cloud service rather than your local sandbox environment. For example: PowerShell sbx ls Lists your local sandboxes. PowerShell sbx --cloud ls Lists your cloud sandboxes. That distinction matters throughout this walkthrough. If you forget --cloud, you are not asking about the same environment. Cloud sandboxes also have separate credentials and network policies. Do not assume that an agent login or network policy you configured locally is already available in the cloud. For this example, we will copy individual files explicitly. That keeps it easy to see what we send to the sandbox and what we bring back. Before You Start You will need: An updated sbx CLI with cloud support, introduced in version 0.42.0.A Docker account with an active Docker Agentic Platform plan for cloud compute.Authentication for the coding agent you want to use. This walkthrough uses Claude.Python 3 available in the sandbox image for the example. Note: The free sbx CLI does not mean cloud compute is free. Docker bills cloud compute based on usage, and your model provider bills inference separately. Check your account’s pricing before starting. Also, use a small sample project first. Running remotely means sending code off your machine. For company repositories, make sure that is allowed before uploading anything. The host-side commands below use PowerShell. Paths inside the cloud sandbox use Linux-style paths. Step 1: Sign In and Configure the Agent First, check your installed version: PowerShell sbx version If you are still using an older version from the previous walkthrough, update it before continuing. Sign in to Docker: PowerShell sbx login For Claude, Docker documents a cloud OAuth flow: PowerShell sbx --cloud secret set anthropic --oauth Complete the provider sign-in with an account that has the required access. Notice the --cloud flag here, too. These credentials are stored for cloud use, separately from your local sandbox credentials. There is no reason to put a token in our Python files or paste it into an agent prompt. Step 2: Create a Small Project Let us give the agent something specific to fix. Create a project folder: PowerShell New-Item -ItemType Directory -Path .\cloud-sandbox-demo Set-Location .\cloud-sandbox-demo Inside it, create a file named slug.py: Python def make_slug(text): return text.lower().replace(" ", "-") This converts "Docker Sandboxes" into "docker-sandboxes". It works for that input, but it does not handle whitespace very well. Leading spaces become leading hyphens. Repeated spaces become repeated hyphens. Tabs are not handled at all. Now create test_slug.py: Python import unittest from slug import make_slug class SlugTests(unittest.TestCase): def test_two_words(self): self.assertEqual(make_slug("Docker Sandboxes"),"docker-sandboxes") if __name__ == "__main__": unittest.main() We have one passing case and a clear improvement to make. The point is not that this function needs cloud compute. It is small enough that we can focus on the sandbox workflow without spending half the article explaining an application. Step 3: Start a Cloud Sandbox Run the following command: PowerShell sbx --cloud run --detached --name cloud-demo --ttl 1h claude This creates a cloud sandbox and starts the agent without attaching your terminal to it. The flags in the above command are for doing useful things: --cloud selects the cloud environment.--detached returns control to your terminal.--name cloud-demo gives the sandbox a recognizable name.--ttl 1h requests a one-hour lifetime. Important: The documented default action when the TTL expires is deletion. Treat this as a disposable environment, and copy your work out before the deadline. The command prints a sandbox ID. You can also find it with: PowerShell sbx --cloud ls Copy that ID into a PowerShell variable: PowerShell $sandbox = "PASTE_YOUR_SANDBOX_ID_HERE" Use the real ID returned by Docker, not the placeholder above. Keep using this terminal for the remaining commands. One detail to remember is a detached cloud run creates a new sandbox. It is not the command to run repeatedly when you want to reconnect to the same one. Step 4: Copy the Project Into the Sandbox Create a directory for our example: PowerShell sbx --cloud exec $sandbox mkdir -p /workspace/demo The mkdir command runs inside the Linux sandbox, not on Windows. Now copy the two files: PowerShell sbx --cloud cp .\slug.py "${sandbox}:/workspace/demo/slug.py" sbx --cloud cp .\test_slug.py "${sandbox}:/workspace/demo/test_slug.py" The ${sandbox} syntax is intentional. In PowerShell, it separates the variable name from the colon used in Docker’s SANDBOX:PATH format. This is also why I am copying individual files rather than uploading the entire folder. We do not need a virtual environment, local configuration, or an accidentally included .env file for this task. Run the existing test inside the sandbox: PowerShell sbx --cloud exec --workdir /workspace/demo $sandbox python3 -m unittest discover -v If your selected image does not include Python 3, add it inside the sandbox before continuing. The existing test only covers two words separated by one space. Passing it does not mean the whitespace handling is correct yet. Step 5: Give the Agent a Narrow Task Attach to the running cloud sandbox: PowerShell sbx --cloud attach $sandbox Now give Claude a concrete task: Plain Text Work on the Python project in /workspace/demo. Update make_slug so that: - The output remains lowercase. - Leading and trailing whitespace is removed. - Consecutive whitespace becomes a single hyphen. - Spaces, tabs, and newlines are handled consistently. - Empty input returns an empty string. Add unit tests for these cases using unittest. Keep the existing test. Do not add third-party dependencies or modify files outside this project. Run the tests and summarize which files you changed. This is much more useful than asking the agent to “improve the project.” We have told it what the function should do, which edge cases matter, and how much freedom it has. There is no reason for it to introduce a framework or reorganize the project. The prompt is task guidance, though — not a security policy. File access, network access, and credentials still need the appropriate sandbox controls. Once the agent finishes, use Ctrl + backslash to detach and return to your local terminal. Detaching does not stop the cloud sandbox. Step 6: Run the Tests and Bring the Changes Back Run the test command again from your terminal: PowerShell sbx --cloud exec --workdir /workspace/demo $sandbox python3 -m unittest discover -v This executes inside the cloud sandbox. It is not running against your original local files. For this task, a straightforward implementation could look like: Python def make_slug(text): return "-".join(text.lower().split()) Calling split() without a separator handles consecutive whitespace and removes leading and trailing whitespace. Joining those words with a hyphen gives us the requested behavior. The agent may arrive at a different implementation. Read it rather than assuming that passing tests makes every change worth keeping. Create a separate folder for the returned files: PowerShell New-Item -ItemType Directory -Path .\review Copy the modified files into it: PowerShell sbx --cloud cp "${sandbox}:/workspace/demo/slug.py" .\review\slug.py sbx --cloud cp "${sandbox}:/workspace/demo/test_slug.py" .\review\test_slug.py Your original files are still untouched. If you have Git installed, compare the versions: PowerShell git diff --no-index -- .\slug.py .\review\slug.py git diff --no-index -- .\test_slug.py .\review\test_slug.py You can also compare them in your editor. Look at the tests as closely as the implementation. Did the agent actually add cases for tabs and newlines? Did it keep the original test? Did it add anything unrelated? For a real repository, I would bring the changes into a working branch and use the normal review process. The sandbox changes where the agent works. It does not replace code review. What About Web Applications? Our Python example does not start a server. If you use a web project instead, cloud sandboxes can expose an application through a public HTTPS URL. For an application already listening on sandbox port 3000: PowerShell sbx --cloud ports $sandbox --publish 3000 sbx --cloud ports $sandbox Use the URL returned by Docker. This is different from publishing a local port such as localhost:3000. In cloud mode, the command accepts the sandbox port, and Docker assigns the public URL. Note: Publicly reachable is not the same as private. Do not expose an unauthenticated admin page, secrets, or sensitive test data. Remove the exposure when you no longer need it: PowerShell sbx --cloud ports $sandbox --unpublish 3000 Step 7: Clean Up the Cloud Sandbox Before cleanup, make sure the files you want to keep are on your machine. If you want to pause rather than delete, Docker documents cloud stop as preserving the sandbox’s memory and disk state: PowerShell sbx --cloud stop $sandbox Do not assume that preserved resources have no cost. Check your plan’s billing terms. For this small exercise, we have already copied the results out, so we can remove the sandbox: PowerShell sbx --cloud rm $sandbox Confirm the removal when prompted, then list your cloud sandboxes: PowerShell sbx --cloud ls There is an important difference from my earlier article: sbx --cloud rm --all is intentionally disabled. Cloud cleanup requires explicit sandbox identifiers. That is a useful safeguard. A cloud credential may have access to more than the one environment you were experimenting with. A Few Things That Can Slow You Down If the agent cannot authenticate, check its cloud credentials. A successful local session does not prove that cloud authentication is configured. If it cannot reach a service, check the cloud network policy. Do not immediately open access to everything just to make an error disappear. If your local files have not changed, remember the workflow we used: we copied files into the cloud and copied the results back. Those copies are not a live synchronization mechanism. And if you are coming back to a running sandbox, use attach. Repeating the detached creation command gives you another sandbox, not another connection to the original one. Conclusion What I like about this addition is that it keeps the workflow familiar. We are still using sbx, still giving the agent a specific project, and still deciding what work to keep. The difference is where that work happens. Start small. Send only the files the agent needs, give it one clear task, and bring the results back into your normal development process. Once that feels comfortable, move on to a larger repository or a task that actually benefits from remote compute. And copy the changes back before the sandbox expires. A useful fix is not very useful if the only copy disappears with the environment.

By Naga Santhosh Reddy Vootukuri CORE

The Diagram-to-Deployment Gap Cloud security architecture often begins with a strong design, account boundaries are defined, identity federation and vending patterns are selected, centralized security services are planned, and diagrams show how telemetry, governance, and incident response should work together. The design may be reviewed by experienced architects and approved by risk stakeholders. Yet the most difficult part starts after deployment. Cloud environments are not static. New accounts are created, workloads are modernized, emergency changes are made, teams adopt new services, and temporary exceptions accumulate. Over time, the deployed environment can diverge from the approved design even when no single team intends to weaken security. A diagram captures architectural intent while the running cloud environment represents operational reality. A mature security program must continuously reconcile the two. The central challenge is therefore not only whether an organization can design a secure cloud architecture. It is whether the architecture can continuously determine that its assumptions still hold. In practice, applying the operating model in this article reduced the time to detect architecture drift from weeks to hours - because divergence from intent is checked continuously against explicit invariants rather than discovered during periodic reviews or audits. I have watched this play out on several architectures I have helped design over the years. The architecture was approved, the launch was clean, and a couple of quarters later the environment no longer matched the diagram we had signed off on. No single team broke it. It drifted organically through dozens of individually reasonable decisions, each of which looked fine on its own review. This article presents a five-stage Continuous Security Architecture Loop — Define, Prevent, Observe, Validate, and Improve — for turning architecture from a one-time deliverable into an operating system for cloud assurance. Why Traditional Security Architecture Becomes Static Traditional architecture processes are strongest at design time. Teams conduct threat modeling, select controls, review trust boundaries, approve exceptions, and publish reference patterns. Once a workload is approved, responsibility often shifts to platform engineering, application teams, security operations, compliance, and audit. The handoff creates a structural gap where architecture defines the intended state, while operations manages the actual state. This gap becomes particularly visible in multi-stakeholder environments. A central team may define a standard account structure, but application teams control many day-to-day decisions inside each account. A security service may be enabled at launch but disconnected later. A centralized log destination may exist while selected workloads stop delivering critical events. A temporary administrative role may become a permanent access path. Each change may look like a configuration issue, yet the combined effect can break the original trust model. Configuration Drift vs. Architecture Drift Configuration drift and architecture drift are different problems. Configuration drift means a resource no longer matches an expected setting. Architecture drift means a security property of the system is no longer true. Logging may still be enabled while workload administrators can now alter the evidence. Encryption may still be present while key access violates the intended separation of duties. The resources look compliant, but the architecture has quietly lost an assumption it depended on. None of this means engineering teams are careless. Drift is what you get when architectural decisions are never wired to enforceable boundaries, current evidence, and operational feedback. Defining Continuous Security Architecture Continuous security architecture is an operating model that translates security principles into testable requirements, applies those requirements through preventive and detective mechanisms, validates deployed environments against architectural intent, and feeds operational findings back into future designs. It is related to continuous compliance, policy as code, infrastructure as code, cloud security posture management, and security monitoring, but it is not equivalent to any one of them. Continuous compliance asks whether a defined requirement is satisfied; continuous security architecture asks a broader question: does the implemented environment continue to preserve the trust assumptions, control objectives, and security outcomes of the approved architecture? This distinction matters because an architecture can satisfy many individual configuration checks and still fail as a system. A useful model must therefore connect four elements: the principle being protected, the architecture requirement derived from that principle, the controls that implement the requirement, and the evidence used to determine whether the outcome remains true. How continuous security architecture relates to adjacent practices: practicecore questionrelationship to this loop Policy as code Is this rule encoded and enforced? A mechanism used inside Prevent and Validate - not the model itself. Continuous compliance Is a defined requirement satisfied? A subset: answers per-control conformance, not whether the system’s trust assumptions still hold. CSPM Are resources misconfigured against a benchmark? Feeds the Observe stage; scores configurations, not architectural invariants. Continuous security architecture Do the architecture’s trust assumptions still hold in production? The superset - connects principle, requirement, control, and evidence across all five stages. The Continuous Security Architecture Loop The Continuous Security Architecture Loop contains five stages. The stages are not a maturity sequence that an organization completes once. They form a recurring operating cycle. Each stage produces information needed by the next, and the final stage feeds learning back into the beginning. Figure 1. The Continuous Security Architecture Loop connects architectural intent with prevention, evidence, validation, and operational learning. Running example throughout this section: We follow one invariant, “Security logs cannot be modified by workload administrators” (INV-LOG-001), through all five stages, so “invariant” stops being an abstract term and becomes a single property each stage acts on. 1. Define: Convert Principles Into Testable Requirements Security principles are often written as apply least privilege, centralize visibility, minimize blast radius, protect administrative access, or encrypt sensitive data. These are directionally correct, but they do not specify what evidence proves implementation, so different teams interpret the same principle differently. The Define stage converts broad principles into testable architecture requirements. Consider centralizing security visibility. A testable requirement might state that security-relevant activity from every account must be delivered to a centrally governed logging environment. That statement decomposes into measurable conditions: required audit sources are enabled, destinations are centrally controlled, workload teams cannot delete retained evidence, delivery failures generate alerts, newly created accounts are automatically enrolled, and retention supports investigation needs. You are not trying to turn every architecture document into a long checklist. Rather, you are identifying the properties that materially support the security model. Each requirement should describe the intended outcome, identify the evidence needed to validate it, specify an owner, and state whether the implementation is preventive, detective, responsive, or compensating. The strongest requirements are technology-aware without being tool-bound: state the security property first, then map it to the chosen cloud services, so the design survives a change of implementation technology. Express the requirement as a structured, adoptable artifact rather than prose: YAML # invariant: logs-immutable-by-workload id: INV-LOG-001 principle: Centralize security visibility requirement: > Security-relevant events from every account are delivered to a centrally governed logging destination that workload administrators cannot alter or delete. outcome: Log evidence remains complete and tamper-resistant for investigation. control_type: preventive + detective owners: control: platform-engineering evidence: platform-engineering risk: security-architecture evidence_sources: - cloudtrail: organization trail delivery status - config: S3 bucket policy + KMS key policy on the log destination - scp: effective policy on workload OUs validation_frequency: near-real-time # log-delivery failure is high-consequence exception_policy: allowed: false # no standing exceptions to this invariant residual_risk_target: none Define — running example: The principle centralize security visibility becomes the spec above (INV-LOG-001). The security property is stated first, and then the AWS services are mapped underneath it. Figure 2. A security principle becomes continuously testable only when it is connected to an architecture requirement, control implementation, and current validation evidence. 2. Prevent: Enforce High-Confidence Architectural Boundaries Some architectural decisions should not depend on detection after a violation occurs. Disabling central logging, moving data into an unapproved region, disconnecting an account from governance, or creating an unmanaged administrative path can undermine the security model immediately. Preventive controls make selected decisions non-optional. In a multi-account environment, platform teams can use organizational policies, account foundations, identity boundaries, deployment controls, and protected service configurations to prevent workload accounts from changing central audit destinations, leaving the organization, disabling designated security services, or creating resources in restricted locations. The goal is to protect the boundaries that preserve the architecture. A single service control policy (SCP) makes the boundary concrete-the action is simply unavailable to workload accounts: JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "ProtectCentralLogDestination", "Effect": "Deny", "Action": [ "s3:DeleteBucket", "s3:PutBucketPolicy", "s3:PutEncryptionConfiguration", "s3:PutLifecycleConfiguration" ], "Resource": "arn:aws:s3:::org-central-security-logs*", "Condition": { "StringNotEquals": { "aws:PrincipalArn": "arn:aws:iam::*:role/PlatformLoggingAdmin" } } }, { "Sid": "PreventLeavingOrgAndDisablingAudit", "Effect": "Deny", "Action": [ "organizations:LeaveOrganization", "cloudtrail:StopLogging", "cloudtrail:DeleteTrail" ], "Resource": "*" } ] } The design challenge is selectivity. Excessive preventive control makes the platform brittle, blocks legitimate engineering work, and creates pressure that leads to bypasses. Not every deviation has the same consequence. A useful rule: prevent actions that would materially break the security architecture, and detect-and-remediate lower-risk deviations where flexibility is necessary. Only a small number of actions, like those above, warrant a hard Deny. Before enforcing a preventive control, evaluate service behavior, failure modes, exception needs, and recovery paths. A guardrail that cannot be safely changed during an incident introduces a different form of risk. Prevent — running example: The SCP above, applied to all workload OUs, denies StopLogging, DeleteTrail, and mutating actions on the central bucket for every principal except the platform logging role. INV-LOG-001’s boundary is now unavailable, not merely monitored. 3. Observe: Collect Evidence About the Deployed Architecture Architecture cannot be validated using resource configuration alone. The evidence layer may need configuration state, identity activity, network flows, deployment events, security findings, data-access records, control exceptions, and account-lifecycle events. Observation turns a running environment into evidence that can be compared with architectural intent. Consider an approved privileged-access model requiring federation, strong authentication, time-limited role assumption, and centralized activity logging. A configuration scan will happily confirm that the administrative role exists. What it will not show you is the engineer who skips that path entirely - using a long-lived key, an alternate role, or direct access from an unmanaged identity. That only shows up in behavioral evidence. So the questions worth asking are practical ones: does the property still hold, what proves it, how fresh is that proof, and who gets paged when it goes missing? Treat absence of telemetry as a finding in its own right — a gap in the logs is a gap in your ability to say anything true about that account. Centralization should not eliminate local ownership. Workload teams still need visibility into their findings and operational context, while the organization needs a protected evidence plane that cannot be selectively altered by the systems being observed. Observe — running example: For INV-LOG-001, the platform collects trail delivery status, bucket and key policy state, and every S3 mutation event against the log destination. Absent delivery telemetry is itself recorded as a finding. 4. Validate: Compare Deployed Reality With Architectural Intent Security programs often report findings as isolated events: one missing log source, one excessive role, one unmanaged connection, or one failed service enrollment. Continuous architecture validation asks whether those findings indicate that a larger architectural property is no longer true. Validation can be organized around architectural invariants which represent properties that must remain true regardless of application changes. Examples include: security logs cannot be modified by workload administrators; production identities originate only from approved sources; all production accounts inherit baseline governance; privileged access is time-bound and attributable; and external connectivity passes through approved control points. An invariant is machine-checkable. A query for INV-LOG-001, where a zero-row result is the passing state: MS SQL -- Did any non-platform principal touch the central log destination? SELECT eventtime, useridentity.arn AS principal, eventname, requestparameters FROM security_events WHERE eventsource = 's3.amazonaws.com' AND eventname IN ('PutBucketPolicy','DeleteBucket', 'PutEncryptionConfiguration','PutLifecycleConfiguration') AND element_at(requestparameters, 'bucketName') LIKE 'org-central-security-logs%' AND useridentity.arn NOT LIKE '%role/PlatformLoggingAdmin' AND eventtime > date_add('day', -1, now()); A returned row is an architecture-conformance failure rather than a lone configuration finding. Even when the SCP already blocked the action, the attempt itself is a signal the Improve stage should see. Each invariant should be connected to a validation record containing the expected state, evidence sources, current state, owner, validation frequency, exception status, residual risk, and remediation target. The record is not simply an audit artifact; it provides a shared language for architecture, platform, operations, and workload teams. A single failing record communicates the thesis better than a page of prose: fieldexample value Invariant Security logs cannot be modified by workload administrators (INV-LOG-001) Expected state Central log bucket + KMS key policies deny workload principals; org trail delivering Evidence sources Config rule s3-log-bucket-policy; org CloudTrail delivery status; effective SCP Current state FAIL — Account 1234 delivering, but bucket policy drifted to allow WorkloadAdmin Owner Platform engineering (control) / Security architecture (risk) Validation frequency Near-real-time Residual risk High until remediated — evidence integrity not guaranteed Remediation target 4 hours Validation frequency should reflect risk and rate of change. Public exposure, privileged access, and log-delivery failures may require near-real-time evaluation. Account ownership may be checked daily. Exception reviews may occur monthly or quarterly. Reference architectures should also be reviewed when new services, threats, incidents, or business models invalidate prior assumptions. Validate — running example: A daily and event-driven check finds Account 1234’s bucket policy has drifted to grant a workload role write access. It is recorded as an architecture-conformance failure against INV-LOG-001 - with owner, residual risk, and a 4-hour remediation target - not as a standalone S3 finding. 5. Improve: Feed Operational Learning Back Into Architecture Security incidents and recurring findings are frequently remediated at the workload level. The immediate problem is fixed, but the reusable architecture remains unchanged, allowing the same design weakness to appear in other environments. The Improve stage treats operational events as feedback about the architecture itself. Incidents, near misses, recurring findings, control bypasses, exception patterns, deployment failures, new threat intelligence, and cloud-service changes should all influence future design decisions. Suppose an incident reveals that a workload role could redirect security logs. Correcting the role is necessary but not sufficient. The organization should ask whether the reference architecture clearly separates log ownership, whether organizational controls should protect the destination, whether other accounts share the condition, whether new validation logic is required, and whether the account-vending process should be updated. A practical rule: every significant incident should produce both a workload-level corrective action and an architecture-level learning decision—a revised reference pattern, a new preventive control, an additional validation test, clearer ownership, an improved deployment template, or an explicit acceptance of residual risk. I remember a stretch where three newly vended accounts failed the same detection-enrollment step inside a single month. Fixing them one at a time felt like progress until the pattern became impossible to ignore. It revealed that the account vending pipeline was the defect, not the accounts. Once we corrected the pipeline, the failure class stopped appearing. Improve — running example: Root cause of the Account 1234 drift: the account-vending template applied the bucket policy once but did not protect it against later edits. The fix is not just repairing Account 1234 - it is (a) adding the mutating actions to the SCP, (b) adding a Config rule to catch policy drift, and (c) updating the vending template so future accounts start protected. The invariant, not the incident, drives the change. From Control Deployment to Control Effectiveness A control being deployed does not prove that it is effective. Logging may be enabled while important events are excluded. Encryption may be enabled while key access is broader than intended. Threat detection may be active while findings have no response owner. Backup policies may exist while restoration has never been tested. Organizational guardrails may be present while alternative paths bypass the intended restriction. Continuous assurance therefore needs more than a binary deployed/not-deployed status. A practical evaluation can examine five dimensions: dimensionquestion it answers Coverage How much of the intended environment is protected. Correctness Whether the implementation matches the requirement. Resilience Whether the control can be bypassed, altered, or disabled. Responsiveness Whether failure produces timely action. Outcome Whether the control measurably reduces the intended risk. These dimensions should not be collapsed into a universal score without context. A high coverage percentage can conceal a critical gap, while a small number of exceptions may carry disproportionate risk. The architecture team should define what effective means for each important control objective and how that effectiveness will be demonstrated. The specific dimension I have seen fail most quietly is Resilience. A control can be present, correct, and even alerting, and a workload role can still disable it or route around it. Coverage and Correctness are the easy numbers to put on a slide, while Resilience is the one that decides whether those numbers mean anything. Central Governance With Distributed Ownership No central team can manage every workload configuration, and no workload team can set enterprise-wide requirements on its own. The shared responsibility that works in practice is that architecture owns the principles, reference patterns, invariants, and the hard exception calls. Platform turns those into the account foundations, guardrails, enrollment workflows, and evidence collection everyone else inherits. Workload teams own what is specific to their application — the controls, the context behind a finding, and their own exceptions. Security operations watches for threats and control failures and feeds what it learns back into the architecture. Risk and compliance tie the evidence to obligations and to whatever risk has been formally accepted. The model must distinguish among control ownership, evidence ownership, and risk ownership. The platform team may operate centralized logging, while the workload owner remains accountable for producing the application events needed for investigation. Security operations may own an alerting process, while architecture owns the invariant that the process is meant to protect. Ambiguity at these boundaries is a common cause of unaddressed findings. Figure 3. Continuous assurance requires explicit coordination among architecture, platform, security operations, and workload teams. End-to-End Scenario: Creating a New Production Account Consider the creation of a new production account. The organization defines several requirements: the account must join the production governance hierarchy, use approved identity federation, deliver security logs to a protected destination, enroll in centralized detection, and restrict deployment to approved regions. During Prevent, organizational policies enforce these boundaries with concrete mechanisms. SCPs on the production OU deny organizations:LeaveOrganization, deny mutating actions on the central log bucket, and deny non-approved regions via an aws:RequestedRegion condition. Account Factory (or an equivalent vending pipeline) provisions the account directly into the production OU so the guardrails apply from creation, not after. During Observe, the security platform collects the literal events: CloudTrail CreateAccount and the account’s move into the OU, the AWS Config recorder status, GuardDuty/Security Hub enrollment state, identity-federation configuration, and organization-trail delivery status. Validation compares the account with the production architecture invariants. Suppose the account is delivering logs but has not enrolled in centralized detection. The check returns a failing record: YAML invariant: all-prod-accounts-inherit-baseline-governance (INV-GOV-002) account: 1234 expected_state: GuardDuty + Security Hub enrolled via delegated admin current_state: FAIL - Security Hub not enrolled (Config recorder ON, trail OK) owner: platform-engineering (control) / security-architecture (risk) residual_risk: medium - threat findings not aggregated for this account remediation_by: 24h This is recorded as an architecture-conformance issue rather than an isolated configuration finding. The record names the account owner, the missing evidence, the remediation target, and any approved exception. The Improve stage examines patterns across accounts. If several new accounts fail at the same enrollment step, the organization does not continue fixing them individually. It treats the repeated failure as evidence that the account-vending or enrollment architecture is incomplete, and the reusable process is corrected so future accounts begin in the expected state. This example illustrates the essential shift where continuous architecture is not a larger collection of controls; it is a system that connects design intent, platform implementation, operational evidence, validation, and learning. Common Anti-Patterns Architecture by diagram. There is a beautiful target-state picture on the wiki, and no way to answer the only question that matters: does the running environment still look like it?Guardrail accumulation. Controls pile up over years. Nobody removes them, nobody re-checks whether they still fire, and eventually the platform is so encrusted that engineers route around it - which is its own risk.Dashboard assurance. The board is green, so leadership feels safe - even though the dashboard only measures the handful of configurations someone remembered to wire up.Permanent “temporary” exceptions. The exception was granted for two weeks in 2023. It has no expiry, no owner, no compensating control, and no evidence the original risk still exists. It is now load-bearing.Finding-by-finding remediation. Teams close tickets faster than the architecture produces them, treating each symptom as new while the weakness that generates them stays untouched.Tool-defined architecture. The security model quietly shrinks to whatever the chosen product happens to detect. The tool should serve an architecture you defined independently - not the other way around. Adopting the Loop Incrementally Continuous security architecture does not require building all five stages at once. A workable sequence can be: Start with 3-5 invariants, not a catalog. Pick the properties whose failure would most damage the trust model, such as log integrity, production identity origin, privileged-access time-bounding, network egress control, baseline governance inheritance. Write each as a spec like INV-LOG-001.Validate before you observe everything. You do not need a complete evidence lake to begin. For each invariant, identify the single signal that proves it and check that. Breadth of telemetry comes later.Prevent only the highest-consequence actions first. A small set of well-chosen Deny guardrails, like leaving the org, disabling logging, or altering the log destination, protects more than a large, brittle policy set. Add preventive controls where a violation is irreversible; detect-and-remediate everywhere else.Wire the feedback loop early, even if manual. A monthly review that turns recurring findings into architecture changes delivers most of the Improve-stage value before any automation exists.Automate by consequence and rate of change. Move the highest-risk, fastest-changing invariants to near-real-time checks first. Slower-moving properties can stay on a daily or weekly cadence. A team can reach a useful state with five invariants, a handful of SCPs, one validation query per invariant, and a recurring review, and then expand coverage as the model proves its value. This is also how the weeks-to-hours improvement in drift detection is realized in practice: near-real-time validation of a few high-consequence invariants, not full automation on day one. When I have taken teams through this, the first few invariants we picked mattered far more than the tooling around them. Choose the properties whose failure would genuinely hurt, prove those, and resist the pull to boil the ocean on day one. Conclusion: Architecture as an Operating System A reference architecture captures intended security design. Continuous security architecture is how you find out whether that intent still holds as systems, teams, threats, and cloud services change. The loop gives you a practical model: Define testable requirements, Prevent the actions that would break critical boundaries, Observe the evidence you need to understand the environment, Validate reality against intent, and Improve the design from what operations teaches you. The value of a security architecture is not determined by the quality of its diagram. It is determined by how reliably the organization preserves its security assumptions in production and how quickly it learns when those assumptions no longer hold. Applied to a multi-account environment, this model cut architecture-drift detection from weeks to hours, turning drift from a condition found in periodic reviews into one that is continuously observed.