DevOps and CI/CD

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text <actor-name>.<atespace>.actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

Executive Summary The Ampere® PMU Profiler (APP) is a Python-based tool designed to provide deep insight into the microarchitectural behavior of applications running on Ampere CPUs (e.g., Ampere® Altra® and AmpereOne®). Unlike standard profilers that identify where time is spent (e.g., which functions consume CPU time), the PMU Profiler explains why time is being spent by measuring low-level hardware events associated with the CPU pipeline and execution behavior. A key outcome of APP is that it enables performance engineers to move from coarse symptoms to actionable causes. For example, while application-level profiling can show an expensive code path, APP can help identify whether the expense stems from inefficient instruction fetching, data cache misses, or other microarchitectural factors that are difficult or impossible to isolate using application-level tools alone. The document outlines a top-down performance analysis methodology and positions APP as an essential final step for expert-level tuning, particularly on Ampere platforms, where you must understand hardware-level bottlenecks and then apply targeted code optimizations. APP is intended to complement system-level analysis rather than replace it. System-level profilers are useful for identifying high-level bottlenecks such as resource saturation or contention, but APP is focused on microarchitecture-level analysis by collecting hardware events. This makes APP especially valuable after system bottlenecks have been eliminated or ruled out, leaving “microarchitecture inefficiency” as the remaining likely cause of slowdowns. What Is the Ampere PMU Profiler? The Ampere PMU Profiler uses Linux perf utility with validated PMUs and metrics on Ampere CPUs. Its purpose is to capture microarchitectural performance indicators through hardware event measurement. In practice, this means APP collects events measured by perf stat that relate to the CPU pipeline and execution mechanisms, allowing engineers to determine what is slow and the underlying microarchitectural reason. A central distinction between APP and application profiling tools is the level of visibility. Tools that sample stack traces (or count function invocations) typically answer the question, “Which functions are active during the slow period?” APP answers a more hardware-specific question: “Which microarchitectural mechanisms are consuming cycles, and what stalls or inefficiencies are present?” The APP workflow assumes that developers can form a hypothesis about where the bottleneck likely originates, such as a particular loop or data access pattern, and then rely on PMU event measurements to confirm or refute those hypotheses at the microarchitectural level. Why Do We Need APP? Performance problems are frequently multi-layered. Even after system-level bottlenecks are addressed (for example, ensuring that CPU is not idling due to I/O, ensuring there is sufficient memory, and verifying resource utilization), some workloads still perform poorly because the CPU spends cycles in inefficient pipeline states. APP helps solve this class of problems by measuring hardware-level behavior. For example, APP can identify microarchitectural bottlenecks such as: Inefficient instruction fetchingData cache missesBranch-related pipeline effectsOther pipeline-level stall sources that manifest as lost cycles This capability is important because microarchitectural causes often do not map cleanly to application symptoms. Code can appear “hot” in a profiler, but the reason it is slow might be due to how it interacts with cache hierarchies, how it causes translation or fetch inefficiencies, or how the processor recovers from pipeline disruptions. Those details are what PMU-based measurement aims to expose. APP also links investigation to “unlocking the full performance potential” of the hardware. By understanding CPU-level bottlenecks, engineers can choose targeted optimizations that application-level tools alone cannot determine with confidence. This ultimately leads to more efficient software and better utilization of Ampere hardware for competitive workloads. When Do We Use the Ampere PMU Profiler? Understanding the APEX Framework Performance tuning is a process of systematic investigation, moving from a broad, system-wide view down to the specific interactions between code and hardware.  Fig. 1: APEX Benchmarking and Optimization Funnel Performance optimization is as much art as it is science. The APEX (Adaptive Profiling and Execution) framework uses tools and methodologies to add structure and rigor to the process and can bridge the gap between creative intuition and empirical fact. We propose applying the APEX methodology to enable root cause analysis for solving performance problems. Follow the funnel above from top to bottom to effectively use the procedure. The methodology recommends starting with assessing platform health as a first step to ensure that the platform used for performance analysis is set up well as an unhealthy platform may mislead the performance analysis. Consider capturing initial performance metrics before tuning any system or application settings. This establishes a clear understanding of the current workload and identifies key scalability knobs. We recommend using Ampere’s PerfKit Benchmarker (APB), which supports many open-source applications, to create a reliable baseline for further analysis and tuning. Next is to assess system performance and any hardware or system bottlenecks— this is where Ampere System Profiler (ASP) is useful to eliminate any system or resource bottlenecks. ASP can also be used to right-size the instance shape and ensure the compute resources are efficiently consumed by the workload. One method that may be used is to leverage APB’s automated benchmarking framework to start and stop ASP’s collectors during the run phase of a given APB benchmark. This ensures that profile is collected while critical code paths are executed and a clear report profile is generated. Once system and resource bottlenecks are eliminated, if the performance issue persists and points to CPU cycles not being used efficiently, we propose going to the next step in the pyramid and using the Ampere PMU Profiler to root-cause the issue further. Finally, system benchmarking should be done after all bottlenecks are resolved or analyzed to effectively measure the system’s performance for the workload. Following this systematic APEX methodology ensures that we eliminate possible issues as a part of a structured process to efficiently conduct root-cause analysis. System-Level Analysis At the microarchitecture level, performance is shaped by how the CPU pipeline handles instruction delivery, execution, and memory access. APP leverages PMU measurements to identify pipeline behavior and stall sources. Memory Hierarchy and Performance Loss APP emphasizes the performance significance of the memory hierarchy. As data access moves from registers to L1 cache, to L2 cache, to L3 cache, and finally to DRAM, access becomes exponentially slower. Because of this, cache misses are a primary cause of performance loss. This provides a conceptual foundation for many APP investigations: If a workload touches large working sets or accesses data in a non-contiguous pattern, it may trigger cache misses that increase effective latency and reduce throughput. Microarchitectural Bottleneck Identification APP can be used at the microarchitecture level to understand where stalls might be in the pipeline. The APP role is to collect hardware events related to pipeline stall behavior and to use those events to characterize the workload’s execution profile. This capability matters because pipeline stalls and inefficiencies can dominate runtime even when application-level profiling points to a “hot” function without explaining the root cause. Key Questions APP is structured around answering questions that cannot be fully resolved with application-level profiling alone. Based on the described APP workflow and report interpretation strategy, APP can help you answer: Where are cycles going at the microarchitectural level? The APP HTML report and TDA sunburst charts are used to broadly characterize whether time is dominated by categories such as instruction retirement behavior, front-end bound behavior, or back-end bound behavior.Which stall or inefficiency class is consistent with the hot code path? Once you hypothesize a bottleneck mechanism (e.g., cache misses from non-contiguous access), APP measurements can confirm whether the observed behavior aligns with that mechanism.What microarchitectural reason explains a hot function’s cost? APP’s purpose is explicitly to explain why time is spent by measuring hardware events. This allows developers to translate hot functions into hardware interactions that can be optimized.Is the workload limited by instruction delivery vs execution/memory? By inspecting broad characterization categories (front-end vs back-end bound) in the APP HTML report, engineers can determine which side of the pipeline is more likely to be limiting performance. Example Usage and Output The below example command attempts to collect: PMU profiling samples for 120sWith a sampling interval of 1sProfiles on cores 1 and 2TopDown metrics and render TDA sunburst chartPMU profiles while running the workload affinitized to cores 1 and 2 Shell app -n 120 -c 1,2 -i 1 –tda -o <folder> -j “taskset -c1,2 <workload> Metrics reported by APP: Metric NameDescriptionIPCInstructions retired per CPU cycle across user and kernel execution unless separatedIPC_kernelInstructions retired per CPU cycle while executing in kernel/EL1cpu_freqAverage core frequency during the measurement interval, typically in GHz or MHzCycle Accounting Metricsfrontend_boundShare of cycles in which retirement is limited by front-end activity (e.g., fetch, branch prediction, decode, ICache, ITLB, queueing)backend_boundShare of cycles in which retirement is limited by back-end resources, cache or memory latency/bandwidth, or ROB/LSQ pressureBranch Effectiveness Metricsbranch_mispredict%Percentage of retired branch instructions that were mispredictedbranch_mpkiBranch mispredictions per 1,000 retired instructionsDTLB Effectiveness Metricsdtlb_mpkiData TLB misses per 1,000 retired instructions requiring translation refill or a walk beyond L1 DTLBdtlb_walk%Percentage of DTLB misses that trigger a page-table walk rather than being resolved by another TLB levell1d_tlb_miss%L1 DTLB miss rate relative to DTLB accessesl1d_tlb_mpkiL1 DTLB misses per 1,000 retired instructionsl2_tlb_miss%L2 or second-level DTLB miss rate relative to L2 TLB accessesl2_tlb_mpkiL2 or second-level DTLB misses per 1,000 retired instructionsITLB Effectiveness Metricsitlb_mpkiInstruction TLB misses per 1,000 retired instructionsitlb_walk%Percentage of ITLB misses that trigger a page-table walk instead of hitting in a next-level TLBl1i_tlb_miss%L1 ITLB miss rate relative to ITLB accessesl1i_tlb_mpkiL1 ITLB misses per 1,000 retired instructionsL1 Cache Effectiveness Metricsl1i_mpkiL1 instruction-cache misses per 1,000 retired instructionsl1d_mpkiL1 data-cache misses per 1,000 retired instructionsl1i_miss%L1 instruction-cache miss ratel1d_miss%L1 data-cache miss rateL2 Cache Effectiveness Metrics l2_mpkiL2 cache misses per 1,000 retired instructions; exact scope depends on event mappingl2_miss%L2 cache miss rate relative to L2 accessesl2d_inv_pkiL2 data-cache invalidations per 1,000 instructionsl2_snoops_pkiL2 snoop transactions per 1,000 instructionsl2d_inv_per_snoopAverage number of invalidations generated per snoopOperation Mix Metricsbranch_percentagePercentage of retired instructions that are branch instructionscrypto_percentagePercentage of retired instructions that are crypto, CRC, or hash-class instructionsinteger_dp_percentagePercentage of retired instructions that are integer data-processing operationsload_percentagePercentage of retired instructions that are loadsstore_percentagePercentage of retired instructions that are storesscalar_fp_percentagePercentage of retired instructions that are scalar floating-point operationssimd_percentagePercentage of retired instructions that are SIMD/NEON vector operationsPipeline Stall Frontendstall_frontend_cache_rateShare of cycles stalled because of instruction-side cache or fetch-delivery issuesstall_frontend_tlb_rateShare of cycles stalled because of ITLB or translation-related front-end issuesstall_recovery_rateShare of cycles spent recovering from pipeline flushes (e.g., branch-misprediction recovery)stall_fronetend_bob_rateShare of cycles stalled because the front-end buffer or queue is full or blockedPipeline Stall Backendstall_backend_cache_rateShare of cycles stalled because of data-side cache-hierarchy latencystall_backend_tlb_rateShare of cycles stalled because DTLB misses or page walks delay loads and storesstall_backend_mem_rateShare of cycles stalled because of main-memory/DRAM latency or bandwidth limitsstall_backend_core_rateShare of cycles stalled because of core execution limits (e.g., dependency chains, execution-unit throughput)stall_backend_resource_rateShare of cycles stalled because of internal resource pressure (e.g., queues, buffers, credits)stall_rob_id_rateShare of cycles in which progress is limited by reorder-buffer or in-flight instruction capacitystall_ixu_sched_rateShare of cycles stalled because of integer execution scheduler or issue-queue pressurestall_fsu_sched_rateShare of cycles stalled because of FP/SIMD execution scheduler or issue-queue pressurestall_lob_id_rateShare of cycles stalled because the load buffer or queue is full or blockedstall_sob_id_rateShare of cycles stalled because the store buffer or queue is full or blockedUncore Metricsslc_miss%System-level cache (SLC/LLC) miss rate for requests reaching the SLCmc_retry_rate%Percentage of memory-controller transactions that are retried, indicating fabric or memory-controller pressurememrd_bw_GBpsEstimated DRAM read bandwidth consumed in GB/smemwr_bw_GBpsEstimated DRAM write bandwidth consumed in GB/sccix_in_bw_MBpsCCIX coherent-interconnect inbound bandwidth to the socket/system in MB/sccix_out_bw_MBpsCCIX coherent-interconnect outbound bandwidth from the socket/system in MB/s Refer to a detailed tuning guide here. Conclusion The APP enables PMU hardware event measurement to provide microarchitecture-level performance insight on Ampere CPUs. It is designed to answer the “why” behind performance problems by identifying microarchitectural causes such as inefficient instruction fetching and data cache misses, which are difficult to detect through application-level profiling alone. The APP workflow is top-down and hypothesis-driven: Form a hypothesis from the hot function, measure with APP profiles, and then analyze using the APP HTML report with TDA sunburst charts to characterize where cycles are being spent (instruction retirement, front-end bound, back-end bound). APP is most valuable when system-level bottlenecks have been characterized or ruled out and microarchitecture-level explanation is required for expert tuning. Check out the full Ampere article collection here.

Change data capture (CDC) pipelines look straightforward on paper: capture database changes, publish them to Kafka, and update downstream systems. The difficulty starts when events are duplicated, consumers restart, projections drift, or a team needs to replay months of history without corrupting the state it is trying to recover. A reliable CDC design has to account for those failure modes from the beginning. That means combining Kafka and Debezium with idempotent writes, deterministic projections, controlled replay workflows, reconciliation checks, and enough recovery evidence to explain what happened when something goes wrong. The architecture: The goal is not only to move inventory changes quickly. The goal is to make replay safe enough that operators can rebuild and explain the derived state after failure. This article builds one concrete pattern: The important detail is that replay safety is not a single feature. It is the result of several boring decisions lining up correctly. Data Model The data model should separate the aggregate state, the classification state, and the transaction history. PLSQL CREATE TABLE inventory_stock_on_hand ( sku VARCHAR(64) PRIMARY KEY, stock_on_hand BIGINT NOT NULL, updated_at TIMESTAMP NOT NULL ); CREATE TABLE inventory_bucket ( sku VARCHAR(64) NOT NULL, bucket_type VARCHAR(32) NOT NULL, location_id VARCHAR(64) NOT NULL, quantity BIGINT NOT NULL, updated_at TIMESTAMP NOT NULL, PRIMARY KEY (sku, bucket_type, location_id) ); CREATE TABLE inventory_transaction ( event_id VARCHAR(128) PRIMARY KEY, sku VARCHAR(64) NOT NULL, seller_id VARCHAR(64) NOT NULL, delta_quantity BIGINT NOT NULL, event_time TIMESTAMP NOT NULL, accepted_at TIMESTAMP NOT NULL ); CREATE INDEX idx_inventory_transaction_sku_time ON inventory_transaction (sku, event_time); CREATE INDEX idx_inventory_bucket_sku_bucket ON inventory_bucket (sku, bucket_type); The transaction table is the recovery anchor. If the availability projection drifts, the system needs a history to explain the projection. Do not rely only on the mutable aggregate table. inventory_stock_on_hand is useful for fast reads, but it is not enough for recovery. If the aggregate is wrong, it cannot explain how it became wrong. The accepted transaction history gives replay something durable to reason from. Ingestion Event Use an event ID that can survive retries and replay. JSON { "event_id": "mkt-evt-8f11a", "sku": "1231241", "quantity": 100, "operation": "I", "event_time": "2026-06-19T18:23:11Z", "seller_id": "seller-42" } The consumer should perform an idempotent write. One pattern is to insert the transaction first using event_id as the primary key. If the insert fails because the event already exists, skip the duplicate and emit a duplicate-suppression metric. Java public InventoryWriteResult apply(InventoryEvent event) { try { transactionRepository.insert(event.toTransactionRow()); } catch (DuplicateKeyException duplicate) { metrics.increment("inventory.duplicate_event"); return InventoryWriteResult.duplicate(event.eventId()); } stockRepository.incrementStockOnHand(event.sku(), event.quantity()); bucketRepository.incrementBucket(event.sku(), "SELLABLE", event.quantity()); return InventoryWriteResult.accepted(event.eventId()); } In production, the accepted transaction insert and the aggregate updates should be part of the same database transaction. A useful shape is: PLSQL BEGIN; WITH accepted AS ( INSERT INTO inventory_transaction ( event_id, sku, seller_id, delta_quantity, event_time, accepted_at ) VALUES ( :event_id, :sku, :seller_id, :delta_quantity, :event_time, now() ) ON CONFLICT (event_id) DO NOTHING RETURNING sku, delta_quantity ) INSERT INTO inventory_stock_on_hand (sku, stock_on_hand, updated_at) SELECT sku, delta_quantity, now() FROM accepted ON CONFLICT (sku) DO UPDATE SET stock_on_hand = inventory_stock_on_hand.stock_on_hand + EXCLUDED.stock_on_hand, updated_at = now(); COMMIT; That ON CONFLICT clause is not just a database convenience. It is part of the replay contract. It ensures that retrying the same business event does not apply the same inventory delta twice. Debezium Configuration Enable PostgreSQL logical decoding and configure Debezium to emit CDC topics for the inventory tables. JSON { "name": "postgres-inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<POSTGRES_HOSTNAME>", "database.port": "5432", "database.user": "<POSTGRES_USER>", "database.password": "<POSTGRES_PASSWORD>", "database.dbname": "<POSTGRES_DBNAME>", "topic.prefix": "inventory_source", "plugin.name": "pgoutput", "slot.name": "debezium_inventory_slot", "publication.autocreate.mode": "filtered", "table.include.list": "public.inventory_stock_on_hand,public.inventory_bucket,public.inventory_transaction", "snapshot.mode": "initial", "heartbeat.interval.ms": "10000", "tombstones.on.delete": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "true", "value.converter.schemas.enable": "true" } } Debezium gives you history, but not recovery confidence. The confidence comes from how you key, project, replay, and reconcile that history. For replay work, track these connector facts in your runbook: Connector name and versionReplication slot namePublication name and included tablesSnapshot mode used for initial loadTopic prefixLast processed LSNConnector lagSchema history topic When a connector interruption happens, those details tell you whether you can resume normally, need a bounded replay, or need a new snapshot plus downstream reconciliation. Partition-Aware Routing The partition key should be chosen from the business ordering boundary. Java public class SkuPartitioner implements Partitioner { @Override public int partition( String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { InventoryEvent event = (InventoryEvent) value; String orderingKey = event.getSku(); int partitionCount = cluster.partitionCountForTopic(topic); return Math.floorMod(orderingKey.hashCode(), partitionCount); } } Partitioning is not merely a throughput setting. If the projection depends on entity-local ordering, the entity belongs in the key. Kafka Streams Topology A simplified topology might rekey CDC records by SKU, materialize source tables, and compute availability. Java StreamsBuilder builder = new StreamsBuilder(); KTable<String, StockOnHand> stock = builder.table("inventory_source.public.inventory_stock_on_hand", Consumed.with(Serdes.String(), stockSerde)); KTable<String, InventoryBuckets> buckets = builder.table("inventory_source.public.inventory_bucket", Consumed.with(Serdes.String(), bucketSerde)); KTable<String, AvailabilityProjection> availability = stock.join( buckets, (stockRow, bucketRows) -> AvailabilityProjection.compute(stockRow, bucketRows), Materialized.<String, AvailabilityProjection, KeyValueStore<Bytes, byte[]>>as("availability-store") .withKeySerde(Serdes.String()) .withValueSerde(availabilitySerde) ); availability .toStream() .filter((sku, projection) -> projection.isPublishable()) .to("inventory.availability.v2", Produced.with(Serdes.String(), availabilitySerde)); The projection function should be deterministic. If replaying the same accepted history does not produce the same projection, the topology is not replay-safe. Recovery Contract Attach a Recovery Contract to the flow. YAML recovery_contract: flow: inventory-availability-projection tuple: "<H, O, I, F, S, Q, E>" history: source: - inventory_transaction - debezium.inventory_transaction order: key: sku idempotency: key: event_id duplicate_policy: skip_and_report function: name: compute_sellable_availability deterministic: true scope: supported: - by_sku - by_time_window - by_partition checks: - stock_on_hand_matches_transactions - sellable_quantity_non_negative - projection_event_time_valid evidence: - replay_scope - events_processed - duplicates_skipped - projections_changed - reconciliation_failures - confidence_status Treat this file as executable architecture documentation. A service should fail fast if the contract is incomplete for a critical flow. Java public final class RecoveryContractValidator { public void validate(RecoveryContract contract) { requireNonEmpty(contract.flow(), "flow"); requireNonEmpty(contract.history().source(), "history.source"); requireNonEmpty(contract.order().key(), "order.key"); requireNonEmpty(contract.idempotency().key(), "idempotency.key"); requireNonEmpty(contract.function().name(), "function.name"); requireTrue(contract.function().deterministic(), "projection must be deterministic"); requireNonEmpty(contract.scope().supported(), "scope.supported"); requireNonEmpty(contract.checks(), "checks"); requireNonEmpty(contract.evidence(), "evidence"); } private void requireNonEmpty(Object value, String field) { if (value == null || value.toString().isBlank()) { throw new IllegalArgumentException("Missing recovery contract field: " + field); } } private void requireTrue(boolean value, String message) { if (!value) { throw new IllegalArgumentException(message); } } } That validator does not make the system correct by itself. It prevents a more common failure: discovering during an incident that nobody defined the replay scope, idempotency key, or reconciliation checks. Replay Workflow Replay should be treated as a controlled workflow. Plain Text 1. Identify incident scope. 2. Select replay scope by SKU, time window, or partition. 3. Read authoritative history. 4. Rebuild deterministic projection. 5. Run reconciliation checks. 6. Emit recovery evidence. 7. Republish only if checks pass. The output should be an evidence report. JSON { "recovery_id": "rec-2026-06-19-001", "flow": "inventory-availability-projection", "events_processed": 1842, "duplicates_skipped": 17, "projection_rows_changed": 11, "reconciliation": { "stock_on_hand_matches_transactions": true, "sellable_quantity_non_negative": true, "projection_event_time_valid": true }, "confidence_status": "trusted" } A replay runner can keep the workflow explicit: Java public RecoveryEvidence replay(ReplayRequest request) { RecoveryContract contract = contracts.load(request.flow()); validator.validate(contract); ReplayScope scope = scopeResolver.resolve(request, contract); List<InventoryEvent> history = historyReader.read(contract.history(), scope); ReplayResult result = projector.rebuild(history, contract.function()); ReconciliationResult reconciliation = reconciliationRunner.run(contract.checks(), scope, result); RecoveryEvidence evidence = RecoveryEvidence.builder() .recoveryId(UUID.randomUUID().toString()) .flow(request.flow()) .scope(scope) .eventsProcessed(history.size()) .duplicatesSkipped(result.duplicatesSkipped()) .projectionsChanged(result.changedRows()) .reconciliation(reconciliation) .confidenceStatus(reconciliation.passed() ? "trusted" : "review_required") .build(); evidenceStore.write(evidence); if (request.publish() && reconciliation.passed()) { publisher.publish(result.projections()); } return evidence; } The replay runner should support dry runs. Dry runs let operators answer "What would change?" before republishing availability, billing, or detection outputs. Operational Metrics Track ordinary health and recovery confidence separately. Ordinary health: Consumer lagConnector lagTask restartsDLQ countEnd-to-end latency Recovery confidence: Replay durationReplay scope sizeDuplicate suppression countProjection rows changedReconciliation failuresConfidence status Example metric names: Plain Text inventory_ingest_events_total{result="accepted|duplicate|rejected"} inventory_cdc_connector_lag_seconds{connector="postgres-inventory-connector"} inventory_stream_projection_lag_seconds{topology="availability"} inventory_replay_duration_seconds{flow="inventory-availability-projection"} inventory_replay_events_processed_total{flow="inventory-availability-projection"} inventory_replay_duplicates_skipped_total{flow="inventory-availability-projection"} inventory_reconciliation_failures_total{check="stock_on_hand_matches_transactions"} inventory_recovery_confidence_status{status="trusted|review_required|failed"} Alert on disagreement, not only lag. A good pipeline can be caught up and still be wrong. YAML alerts: - name: InventoryProjectionReconciliationFailure expr: inventory_reconciliation_failures_total > 0 severity: page - name: InventoryReplayRequiresReview expr: inventory_recovery_confidence_status{status="review_required"} > 0 severity: ticket - name: InventoryConnectorLagHigh expr: inventory_cdc_connector_lag_seconds > 300 severity: ticket Reconciliation Queries Reconciliation should be executable, not just a diagram in a runbook. Start with invariants that are simple enough to automate. Example: Stock-on-hand should match accepted transaction deltas for a replay window. PLSQL WITH accepted_delta AS ( SELECT sku, SUM(delta_quantity) AS expected_delta FROM inventory_transaction WHERE accepted_at BETWEEN :from_time AND :to_time GROUP BY sku ), actual_delta AS ( SELECT sku, stock_on_hand - :baseline_stock_on_hand AS observed_delta FROM inventory_stock_on_hand WHERE sku = :sku ) SELECT a.sku, a.expected_delta, b.observed_delta, (a.expected_delta = b.observed_delta) AS matches FROM accepted_delta a JOIN actual_delta b ON a.sku = b.sku; Example: Sellable inventory should never be negative. PLSQL SELECT sku, location_id, quantity FROM inventory_bucket WHERE bucket_type = 'SELLABLE' AND quantity < 0; These queries are not academically exciting, but they are operationally powerful. They turn "the replay finished" into "the replay finished and the invariants passed." Replay Endpoint Sketch A replay workflow should be explicit and permissioned. One possible internal API: HTTP POST /internal/recovery/replay Content-Type: application/json { "flow": "inventory-availability-projection", "scope": { "type": "sku_and_time_window", "sku": "1231241", "from_event_time": "2026-06-19T18:00:00Z", "to_event_time": "2026-06-19T19:00:00Z" }, "dry_run": false, "requested_by": "sre-oncall", "reason": "projection drift after stream task restart" } The response should not just say 200 OK. JSON { "recovery_id": "rec-2026-06-19-001", "status": "trusted", "events_processed": 1842, "duplicates_skipped": 17, "projections_changed": 11, "reconciliation_failures": 0, "evidence_uri": "<RECOVERY_EVIDENCE_URI>" } The response is the operational artifact. It gives the team something to attach to an incident timeline and something to compare against later recovery runs. Tests for Replay Safety Replay safety should be tested before production incidents. Java @Test void replayingSameHistoryDoesNotChangeProjectionTwice() { List<InventoryEvent> history = List.of( event("evt-1", "SKU-1", 10), event("evt-2", "SKU-1", -2), event("evt-1", "SKU-1", 10) // duplicate ); AvailabilityProjection first = projector.replay(history); AvailabilityProjection second = projector.replay(history); assertThat(first).isEqualTo(second); assertThat(first.sellableQuantity()).isEqualTo(8); assertThat(first.duplicatesSkipped()).isEqualTo(1); } Also test late events, schema versions, partition rebalance, connector restart, and partial replay by entity. If replay is part of your recovery model, it deserves the same test discipline as the happy-path pipeline. Add failure injection tests that mirror production recovery: Java @Test void lateEventTriggersReviewWhenItChangesPublishedAvailability() { ReplayScope scope = ReplayScope.forSkuAndWindow( "SKU-1", Instant.parse("2026-06-19T18:00:00Z"), Instant.parse("2026-06-19T19:00:00Z") ); history.append(event("evt-1", "SKU-1", 10, "2026-06-19T18:01:00Z")); history.append(event("evt-2", "SKU-1", -3, "2026-06-19T18:59:00Z")); history.appendLate(event("evt-3", "SKU-1", -2, "2026-06-19T18:30:00Z")); RecoveryEvidence evidence = replayRunner.replay( ReplayRequest.dryRun("inventory-availability-projection", scope) ); assertThat(evidence.eventsProcessed()).isEqualTo(3); assertThat(evidence.projectionsChanged()).isGreaterThan(0); assertThat(evidence.confidenceStatus()).isEqualTo("review_required"); } Failure Injection Matrix Use a small matrix before every major release of the pipeline. Duplicate Event Injection: Send the same event_id twice.Expected evidence: duplicates_skipped > 0; no double-counted stock.Late Event Injection: Delay event arrival until after the projection has already published output.Expected evidence: late event count, changed projections, and review status if the output changes.Connector Pause Injection: Stop the Debezium connector for several minutes.Expected evidence: connector lag, replay scope, and reconciliation status.Offset Rewind Injection: Reprocess a known event range.Expected evidence: deterministic replay agreement.Schema Change Injection: Replay old and new schema versions.Expected evidence: schema versions recorded in the recovery evidence.Bad projection deploy Injection: Publish an incorrect derived state, then replay.Expected evidence: projections changed; reconciliation passes after rebuild. The point is not to create chaos for its own sake. The point is to practice the exact recovery motion before a real incident. Production Hardening Checklist Before relying on replay in production, confirm: The authoritative history has retention longer than the largest expected recovery window.The idempotency key is stable across producer retries.The Kafka partition key matches the business ordering boundary.The projection function is deterministic for the supported replay scope.The contract names every source topic, source table, check, and evidence field.The replay endpoint supports dry runs.Republish requires reconciliation success.Evidence is written to durable storage.Evidence records include schema versions and replay input bounds.Operators can find the runbook from the alert.The DLQ is treated as an input to recovery, not as the recovery plan itself. For high-value flows, make this checklist part of the architecture review. It is much cheaper to define replay semantics while designing the pipeline than to invent them under pressure. Common Mistakes Treating CDC topics as transient integration messages instead of durable recovery history.Choosing partition keys for infrastructure convenience rather than business ordering.Allowing stream processors to perform hidden non-idempotent side effects.Measuring lag but not correctness.Resetting offsets without a reconciliation plan.Assuming exactly-once semantics removes the need for recovery evidence. Conclusion Replay-safe CDC pipelines require more than Kafka, Debezium, and stream processing. They require explicit recovery semantics. Recovery Contracts give teams a compact way to define those semantics. Confidence-carrying replay gives operators evidence that the recovered state can be trusted. That is the difference between a pipeline that resumes and a platform that actually recovers.

I needed a job to run once a day, remember what it did yesterday, and cost nothing to operate. The obvious answer is a small VM with cron, or a Lambda plus DynamoDB. I did not want to pay for either, and I did not want a server to patch. So I pushed the whole thing onto GitHub Actions and used a JSON file committed back to the repo as the database. It has now run 139 times in production on the free tier, tracking just over 1,000 records, and the operating bill is still zero. Here is the part that took the most thought: keeping state across runs that are, by design, completely stateless. "The daily digest the pipeline sends, with new postings badged." The Constraint That Shapes Everything GitHub Actions gives you a cron trigger for free: Shell on: schedule: - cron: "0 16 * * *" # 09:00 EST daily workflow_dispatch: # manual button That solves scheduling. It does not solve memory. Every run starts on a fresh ubuntu-latest runner with a clean checkout. Anything you write to disk during the run is gone when the job ends. For my use case (a daily digest that must not re-send jobs it already sent), that is the entire problem. The script has to know what it saw yesterday. The standard fix is an external store. But for a workload that writes a few kilobytes once a day, standing up a database is more operational surface than the actual task. The repo is already there, the runner already has a checkout, and the workflow already has a token. So the store is the repo. Git as the Database The pattern is three lines at the end of the workflow: stage the state files, commit if they changed, push. Shell permissions: contents: write # the default token is read-only; you must opt in # ... run the script, which writes seen_links.json and job_history.json ... - name: Commit updated history files run: | git config user.name "GitHub Actions Bot" git config user.email "[email protected]" git add seen_links.json job_history.json 2>/dev/null || true git diff --staged --quiet || git commit -m "Update job history [skip ci]" git push One automated commit per day. The repo's own history is the database, and the audit log comes for free. Two details here are not optional, and I learned both the slow way. First, permissions: contents: write. The GITHUB_TOKEN handed to a workflow is read-only by default. Without this block, the git push fails with a 403, and the failure is at the very end of the run, after the real work succeeded, so it looks like everything worked until you check tomorrow and the state never persisted. Second, git diff --staged --quiet || git commit. This commits only when something actually changed. Committing an unchanged tree is an error, and a daily job that finds nothing new is a normal Tuesday. The || makes "nothing to commit" a no-op instead of a red X. The result is that the database lives in git history. Every state change is a commit. I can read yesterday's seen_links.json by checking out yesterday's commit. That is free audit logging I did not have to build. The Infinite-Loop Trap Here is the gotcha that will bite anyone who copies this pattern: a workflow that pushes a commit can trigger a workflow that runs on push, which pushes a commit, which triggers the workflow. The guard is the [skip ci] token in the commit message: git commit -m "Update job history [skip ci]" GitHub treats [skip ci] in a commit message as "do not start workflows for this commit." My scheduled workflow uses it. I also had a second, older workflow file in the repo whose commit message was a plain "Update seen links" with no skip token. Because that workflow only ran on schedule (not on push), it never actually looped, but it was one: push line away from a runaway. If your state-committing workflow has any push trigger, the skip token is the difference between a daily job and a billing incident. Put it in from the start. Decoupling "New" From "Still Worth Showing" The other decision I am glad I made early was separating two ideas that look like one: a record being new today, and a record being relevant today. A naive version sends only what is new since the last run. That breaks the moment a run finds nothing, or the moment the user skips a day. So state is two files with two jobs. seen_links.json is a flat set of every URL ever processed, used purely for deduplication. job_history.json is a rolling window: each entry carries a first_seen timestamp, and a record stays in the window for ten days regardless of how many runs happen in between. Shell def cleanup_old_jobs(history, max_days): today = datetime.now().date() cleaned = {} for category, jobs in history.items(): cleaned[category] = [] for job in jobs: first_seen = job.get("first_seen") seen_date = datetime.fromisoformat(first_seen).date() if (today - seen_date).days <= max_days: cleaned[category].append(job) return cleaned So "new" is computed per run (anything not in seen_links.json), and "relevant" is the trailing ten-day window. The daily output is never empty, nothing is ever sent twice, and a record ages out on a fixed schedule instead of vanishing the first quiet day. Two files, two responsibilities. Trying to make one structure do both is where this kind of project usually rots. The Dependency I Refused to Add The source data is two different table formats from upstream pages: one uses GitHub-flavored markdown tables, the other uses raw HTML tables inside the same document. The clean answer is a parsing library. I chose regex and the standard library instead, and I want to be honest about why and what it costs. The script tries markdown first, then falls back to HTML: Shell parsed_jobs = parse_markdown_table(text) if len(parsed_jobs) == 0: parsed_jobs = parse_html_table(text) # SimplifyJobs uses HTML The upside is a requirements.txt with exactly one line (requests), which means the install step on a cold runner is near-instant, and there is no transitive dependency that can break a 9 a.m. job. The downside is real, and I will not pretend otherwise: regex table parsing is brittle. When an upstream source changed its column layout, my parser silently returned zero rows for that source. It did not crash. It just quietly stopped finding jobs from one feed, which is the worst failure mode because nothing alerts you. For a personal tool with one user, that trade is fine: I notice within a day and patch a regex. For anything with real users, I would add a parser and, more importantly, a "parsed zero rows from a source that normally returns dozens" alarm. The lesson is not "regex bad." It is that a zero-result parse should be treated as a failure signal, not a valid empty result. Cheap Correctness Wins Two small filters do more work than their size suggests. Deduplication is a set membership check, which makes the whole pipeline idempotent. Running the workflow twice in one day produces the same output as running it once, because the second pass finds everything already in seen_links.json. For a cron job that you will inevitably trigger manually while debugging, idempotency is what lets you mash the button without consequences. Link quality is an allowlist of known applicant-tracking domains (Greenhouse, Lever, Workday, Ashby, and friends). Upstream rows mix real application links with company homepages and image badges. Filtering to known ATS hosts drops the noise without trying to validate every URL: Shell JOB_HOST_HINTS = ("greenhouse.io", "lever.co", "myworkdayjobs.com", "ashbyhq.com", "smartrecruiters.com", "icims.com", ...) def looks_like_job_link(url): return any(h in url.lower() for h in JOB_HOST_HINTS) An allowlist is the right default here because the failure mode is asymmetric. Letting through a dead homepage link wastes a click; an allowlist that occasionally drops a valid but unusual ATS is a one-line addition when I notice it. I would rather under-include than ship dead links. What it Actually Costs The numbers from production: 139 scheduled runs committed back to the repo, 1,062 unique links tracked in the dedupe set, three Python files, one runtime dependency, and one YAML workflow. Infrastructure cost is zero, because GitHub Actions' free tier covers a once-a-day job comfortably and Gmail's SMTP handles the delivery. There is no server, no database, no secret rotation beyond an app password, and nothing to wake up to at 3 a.m. When is This Pattern the Right Call? Reach for git-as-a-database when the write volume is low (you are committing on a human timescale, not a request timescale), the state is small and serializable, a single writer is doing the writing (the scheduled job), and you actively want the change history. A daily digest, a status snapshot, a slowly-changing config, a scoreboard: all good fits. Do not reach for it when you have concurrent writers (two runs racing to push will collide and one will fail the non-fast-forward push), when the state is large enough to bloat the repo, or when you need sub-minute reads or transactions. At that point you have outgrown the trick and a real datastore earns its keep. For everything in the first bucket, the calculus is hard to beat: the scheduler, the runtime, the storage, and the audit log are all things you already have for free. The only code you write is the part that does the work.

Analytics teams do not get too upset about small errors. If a product dashboard is off by half a percent on a Tuesday, nobody files a ticket. If your marketing funnel counts some web sessions twice, the overall trend is still okay. Everyone moves on. I spent a part of my early career in that world. It is a place to learn how to move fast, ship features, and use data to get a general idea. Then I started building pipelines that fed automated billing and revenue recognition systems. The rules changed completely. Financial-grade data is different. When a number goes on a customer invoice, drives a usage-based billing meter, or gets repeated by an executive to the board of directors, "roughly right" becomes a problem. The pipeline is not just informing a business decision - it is the decision. If it fails, someone has to answer for it to an external auditor. That change moving from analytics to shipping numbers people stake their reputations on — made me scrap my old way of doing things and rethink how I design data infrastructure. If you are building lakehouse platforms that have to scale out and remain completely defensible under scrutiny, here is what actually matters. The Reconciliation Gap Nobody Warns You About Here is the first painful lesson: correctness and scale do not work well together, and billing data is right in the middle. Usage-based billing means you are dealing with huge, high-volume event streams, API hits, compute-seconds, database operations, and converting those numbers into actual cash. The volume forces you toward distributed systems. The money demands accuracy. You cannot ship an infrastructure that's very fast but drops some events, and you cannot ship a framework that is perfectly consistent but takes a long time to close out a daily ledger. The place where this trade-off is hardest is late-arriving or out-of-order data. Imagine a streaming meter where an event happens at 11:58 PM. It does not hit your ingestion engine until 12:03 AM the next morning. If your daily aggregation pipeline already completed at midnight, that customer usage falls into the wrong billing month or disappears. Multiply that event by many transactions, and you have a massive reconciliation gap that your finance team will catch. Because of this, my absolute baseline rule for any pipeline touching revenue is that it must be 100% idempotent and completely reprocessable from source. I mean reprocessable in the sense that I can replay a raw event window from three weeks ago and land on the exact same decimal point. To do that, your transformation logic has to be completely deterministic and keyed entirely on business identifiers rather than system arrival times. In production, that usually looks like a merge statement driven by event and entity IDs: SQL MERGE INTO billing_usage_gold AS target USING staged_events AS source ON target.event_id = source.event_id WHEN MATCHED AND source.ingested_at > target.ingested_at THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * The SQL looks simple. The actual engineering discipline is ensuring that event_id remains stable, unique, and uncorrupted all the way back to the source application code. If you lock down that data contract, your downstream reconciliation nightmares mostly go away. Layering for Defensiveness, Not Aesthetics I am a pragmatist when it comes to the classic layered lakehouse. Many data teams adopt this setup just because it looks tidy in a slide deck. When you are dealing with financial pipelines, those layers serve a functional, defensive purpose. The raw layer needs to be entirely immutable and append-only. Think of it as a ledger of exactly what the world looked like when the event happened, timestamped, raw, and completely untouched. Never let transformation logic touch or rewrite this layer. When an auditor asks, "What exactly did the system report on November 14th?" this table holds the answer. It should not change just because you refactored a downstream SQL model six months later. The refined layer is where you handle the reality of data engineering: deduplication, type casting, schema enforcement, and core business rules. This is also where you have to build structural data-quality checkpoints. For architectures, that means ditching passive logs or soft warnings and leaning into automated testing frameworks like dbt to physically break things when they go wrong. If a data point turns into an invoice line item, a bad value should not log an error; it needs to kill the process. We handle this by setting our dbt data assertions to a hard error severity level: YAML # models/staging/staged_events.yml version: 2 models: - name: billing_usage_silver columns: - name: event_id tests: - unique: config: severity: error - not_null: config: severity: error - name: compute_seconds tests: - dbt_utils.expression_is_true: expression: ">= 0" config: severity: error By explicitly setting severity: error, a single duplicate event ID or a bizarre negative usage value will not just trigger a warning. It will kill the execution DAG instantly. Is it annoying to debug a stopped pipeline at 2:00 AM? Yes. I would much rather explain a delayed operational dashboard to an internal stakeholder than explain a fraudulent or inaccurate charge to a paying enterprise customer. The serving layer is your business-facing interface. It features grains, locked-down definitions, and the exact tables that feed your downstream billing engines, margin tools, and executive reporting. By the time any row hits this layer, it has survived every quality gate you can throw at it. Your analysts and finance partners can build on top of it safely, without rewriting core logic five different ways and coming up with five different answers. If It Isn't Observable, It Isn't Auditable People in data engineering tend to talk about observability like it's a nice-to-have optimization trick or a post-launch polish item. For financial systems, observability is literally the entire game. When you sit down with auditors or finance directors, they do not care if your Apache Spark clusters are running at peak efficiency. They want to know two things: How do you know this final number is correct, and can you prove it to me right now? Answering that honestly requires three things built directly into your infrastructure: Freshness monitoring that actually wakes you up. Silence does not mean everything is working. If a key serving table misses its scheduled data drop, you should not find out because a finance manager pings you on Slack. You need to wire freshness monitoring into a high-priority on-call rotation like PagerDuty. You have to catch the delay before the downstream billing window closes out.Lineage a human can trace. When a revenue metric looks weird on a summary, you need to be able to trace that specific number back through every single SQL transformation, join, and filter to the original raw event in minutes. Relying on "trust me I wrote the code" does not work. Automated, column-level data lineage maps turn an afternoon of code review into a two-minute look.Continuous data quality logging. Treat data quality metrics as a first-class production output. We track row-count variations, null rates, and distribution drifts on every run, logging them out to monitoring tables or platforms like Elementary. If your system ingestion drops out of nowhere, you need to know whether your customers actually stopped using the product or an upstream webhook silently broke. [Raw Event Ingestion] ⬇ Flows into:[Silver Layer] ➡ (Runs dbt Hard Schema & Unique Tests ➡ Fails? HALT & ALERT) ⬇ Flows into:[Gold Serving] ➡ (Triggers Continuous DQ & Freshness Monitoring ➡ PagerDuty / Slack Alerts) Compliance Is Just a Feature Wearing a Suit If you have ever been through a pre-IPO sprint or a standard Sarbanes-Oxley (SOX) audit, you know how exhausting it feels. The biggest mental shift is realizing that compliance guidelines are really just standard system requirements written in legal language. Auditors care about controls, lineage, reproducibility, and separation of duties. If you translate that into engineering terms, it means: your transformation code must be version-controlled and peer-reviewed, production deployments should happen via automated CI/CD pipelines instead of a local laptop terminal, data access needs to be tightly permissioned and logged, and you must be able to reproduce historical numbers on demand. Infrastructure-as-Code (IaC) handles all of this heavy lifting for you. When your cloud environments, access roles, and pipeline configurations live inside a Git repository, the question of "Who changed this permission, and when did they do it?" always has an unalterable answer. Teams that treat compliance as a chore end up panicking every single quarter. Teams that build these automated checks directly into their deployment workflow barely even notice the audit happening. It is the same amount of work either way; doing it continuously is just significantly cheaper. Unlocking Self-Service Without the Chaos The real reward for dealing with all this architecture is that you can finally let other teams get their own data without causing problems. "Self-service analytics" usually gets a bad name because companies often give raw, messy tables to a lot of people. As you would expect, everyone comes up with their own definition of what "gross margin" or "active user" means, and you end up with big arguments inside the company about whose spreadsheet is correct. A controlled and reliable serving layer completely changes this situation. When your definitions are fixed, consistent, and easy to see, your finance team can look at margins by market segment, your marketing teams can build expansion models, and your product managers can look at consumption trends. Everyone is getting their data from the same place. That is the moment your data engineering team stops being a bottleneck for the whole organization. Instead of spending your week answering special requests or running manual data extractions, you get to focus on building infrastructure that can handle a lot of work. Faster decision-making and clear visibility into operations do not come from a magic machine learning model. They happen because your underlying numbers are finally stable enough to act on without needing to check. A Few Things I Wish I Knew Earlier If you are currently moving from building product analytics to managing data that has real financial importance, remember that while your technical skills are still useful, your standards for engineering are not good enough. Design your systems so that you can repeat everything exactly, not just handle a lot of work. Make your data quality tools stop the pipeline if there is a problem instead of just giving a warning. Treat data history, system updates, and automated alerts as parts of your infrastructure rather than things you will do later. And stop thinking of compliance as a rule. A well-built pipeline is already mostly ready for audits anyway. The logic of distributed systems is hard. That is what we all talk about and study. The harder thing is accepting that when your data represents real money, "close enough" is not good enough.

A CI Runner That Shouldn't Have Died If you deploy AWS Lambdas through Terraform, you almost certainly use archive_file. With enough lambdas, a single terraform apply can kill the CI runner with OOM. The trickiest part is that you will not see any errors in Terraform output and have no clue what just happened. I noticed this when my lambdas started failing — every first terraform apply after a routine change. SIGKILL from the kernel OOM killer and nothing in Terraform logs. The strange part is that reapply sometimes worked — not always on the first try, but eventually it went through. I've named the ticket "Flaky CI," and two weeks of investigation was focused on the CI itself: runner memory, parallel jobs, Docker leaks. terraform apply was the last suspect — from my perspective, there was no way or reason for it to consume so much memory. If you've never wondered how Terraform providers work, it's actually pretty simple. Most of them are just API wrappers. They send HTTP requests, parse responses, and update state. archive_file is one of the exceptions — it works with real files on disk. This means that its memory usage is actually determined not by the number of defined resources, but by the total size of the data it should process. That's why the pattern went unnoticed for years — without knowing about the provider's insides, the issue looks like some CI flakiness. When I finally reached the source code, the answer was found in a few lines in zip_archiver.go file. What archive_file Actually Does archive_file data source creates a zip or tar archive from a directory or file. This is a standard pattern for lambdas: you point source_dir at the function code and pass the resulting archive to aws_lambda_function. YAML data "archive_file" "lambda" { type = "zip" source_dir = "${path.module}/src" output_path = "${path.module}/lambda.zip" } Nothing suspicious at first glance, but behind these lines is a call chain, which is worth a deeper look. When Terraform processes this data source, the provider calls archiveFile — it creates a ZipArchiver and iterates over files in source_dir. For each file, it calls the ArchiveFile method, which does the following: Go content, err := os.ReadFile(fname) // ... f, err := a.writer.Create(name) // ... _, err = f.Write(content) os.ReadFile reads the entire file into a []byte — one contiguous buffer in memory. Then that buffer is passed to the zip writer via Write. After the write, the buffer becomes garbage. This was a design choice from 2016, and at the time, it was reasonable. Terraform configurations archived small files — configs, scripts, and templates. A typical source_dir weighed something like kilobytes, so there was nothing to optimize at this point. That's why the simplest way to read a file was chosen — os.ReadFile. The code looks like a textbook example. But the context changed. Lambda zips today are 50-250 MB uncompressed. ML models, large dependencies (numpy, pandas, puppeteer), bundled assets. And teams deploy not one lambda but five, ten, or twenty through a single Terraform workspace. The code from 2016 didn't change. The scale of the data did. Why Can't the Garbage Collector Help The natural and reasonable question: doesn't Go's garbage collector reclaim memory between files? GC runs indeed — it just has nothing to reclaim. All ten archive_file data sources are independent — they have different source directories and no shared references (if you do not specify them directly). Terraform's graph walker places them at the same level and evaluates them concurrently. This is usually a good thing timewise, but not in this case, as all 10 buffers are alive at the same time. Each goroutine holds its 50 MB until zip write completes. The garbage collector scans the heap and identifies every buffer as still in use, so it reclaims nothing. Meanwhile, peak heap hits 10 x 50 MB = 500 MB (measured: 508 MB). If the model is right, peak memory should scale linearly with parallelism. Your CI runner's memory limit doesn't. Measuring the Pattern I've chosen two ways of measurement: a standard Go benchmark for precision (isolating the archiver) and a Terraform integration test for realism (a real provider during terraform plan). The headline: for 10x50 MB concurrent archives, peak heap drops from 508 MB to 8 MB -- a 98% reduction. Full results, heap growth during archiving, buffered versus streaming: 1x50MB: 50.8 → 0.8 MB (98% reduction)10x10MB: 108 → 8.1 MB (92% reduction)10x50MB: 508 → 8.1 MB (98% reduction) Real Terraform under terraform plan with parallelism matrix, peak RSS in MB: Implp=1p=2p=5p=10Buffered1232765791034Streaming173275384533 Buffered RSS scales linearly with parallelism. Streaming flattens the curve. One anomaly you could've noticed: at p=1, streaming shows a higher RSS than buffered. I'm fairly sure it's just noise. Single-archive runs finish fast, and sampling RSS every 100ms is too coarse to catch what's really happening in that window. The number that matters is p>=2, and that's where the pattern holds. On speed: Go benchmark wall time stays within about 3% across every scenario. So the streaming fix isn't quietly buying memory savings with a performance hit. You get the memory back for free. All measurements are reproducible: https://github.com/olegmmv/terraform-archive-memory-research. Putting these measurements together gives a three-stage picture of the memory cost: StagePeak Heap (10x50MB, p=10)StatusBaseline (current provider)1034 MBMeasuredWith input-side streaming533 MBMeasuredWith full pipeline streaming~320 KBArithmetic projection The third row isn't measured, but is arithmetic. I'll describe later why, but for now, just keep in mind that it shows what we'd see if a second os.ReadFile in the output path is also streamed. The Tar Archiver Already Streams The fix isn't speculative; just open a neighboring file in the same provider. In tar_archiver.go, addFile opens the file, defers close, and copies via io.Copy into tarWriter. No buffering — streaming by default. Go file, err := os.Open(filePath) // ... defer file.Close() // ... _, err = io.Copy(a.tarWriter, file) The zip_archiver.go path, though, chose the buffered approach: Go content, err := os.ReadFile(infilename) // ... _, err = f.Write(content) Same codebase and job to be done, but two different choices. archive/zip.Writer.Create returns an io.Writer that streams, with CRC-32 computed during the write via crc32.NewIEEE. There was never a technical barrier. The only thing needed for the fix now is applying the same pattern. The Streaming Fix Here is the diff: replace os.ReadFile with os.Open and Write with io.Copy: diff - content, err := os.ReadFile(infilename) + file, err := os.Open(infilename) if err != nil { return err } + defer file.Close() if err := a.open(); err != nil { ... - _, err = f.Write(content) + _, err = io.Copy(f, file) Everything else stays the same; the only thing that's different is the read-write pattern. This is the actual implementation behind the streaming numbers in the previous section. The streaming version does still allocate memory, of course — you can't get to zero. But it's way down: my benchmark put it at around 0.8 MB. This is due to archive/zip internal buffering: the io.Copy buffer, the deflate compressor state, and small zip metadata structures. One caveat worth flagging: this is the input side only. On the output path, the provider uses its own ReadFile function to compute checksums on the completed zip archive. The Second ReadFile: Output Checksums The Go benchmark showed a 98% reduction, but terraform plan with parallelism=10 only drops from 1034 MB to 533 MB -- about 50%. Where's the missing 48%? Once the zip lands on disk, the provider turns around and reads it straight back. That's what genFileChecksums does: it opens the output file and computes four hashes -- md5, sha1, sha256, sha512 -- for Terraform state. And each one of those hashes wants the full file content. So the provider pulls the entire output zip into memory, using the same os.ReadFile we've been dealing with all along. In my benchmark, the output zip comes out roughly the size of the input. The test data is random bytes, and Deflate can't do much with those. Real Lambda packages compress a lot better, but the pattern remains: the provider reads whatever the output size is back into memory. Run ten of these in parallel at 50 MB a pop, and you're already 500 MB deep, purely on checksums. The PR goes after the input side. It removes the os.ReadFile allocation during archive creation, and the effect is big. In straight Go benchmarks, heap usage drops by 98%, from 508 MB to 8 MB. Real Terraform runs are tamer, about half: peak RSS falls from 1034 MB to 533 MB. So where's that remaining 533 MB coming from? It's the second os.ReadFile, the one inside genFileChecksums, still reading the finished zip back into memory so it can hash it for Terraform state. Technically, you can stream the checksums too. hash.Hash already satisfies io.Writer, so nothing stops you from wrapping all four hashes in an io.MultiWriter and feeding them while the zip is being written. One pass, no second read. The catch is that it's a very different patch from the input-side one. genFileChecksums is structured around post-hoc reading. Making it streaming means restructuring how the provider integrates checksum computation with archive creation. That's state-management territory, not plain I/O. If both sides streamed, the only thing left to allocate would be io.Copy's default buffer. Ten goroutines, 32 KB each, and you land at 320 KB total. Throw in a sliver of zip writer state per goroutine, and that's basically it. The theoretical floor. What the PR actually does is the first half: input streaming, leaving that 533 MB residual behind. The output half, streaming through MultiWriter, is written down as future work. So one PR cuts the problem in half. Closing it out takes two. What It Costs in Practice At the Lambda deployment limit of 250 MB, ten concurrent archives push peak heap to roughly 5 GB -- well past most CI runner allocations. There are workarounds, each with a price tag. Dial parallelism down, and you trade throughput for memory. Spin up beefier CI runners, and you trade dollars for memory. Both get you unstuck, but neither addresses the root cause. The PR is up at https://github.com/hashicorp/terraform-provider-archive/pull/501. The fix is under ten lines of Go, so the investigation took much longer than the implementation. Some design choices age well, but some scale with your infrastructure.

A scheduled job that needs ninety to one hundred eighty seconds to produce a single output file looks harmless until the day you ship a new build while it is still running. The deployment controller drains the old task and starts a replacement. For a window of two or three minutes, both replicas are alive, both read the same input snapshot, and both intend to write the same logical output. Without idempotent output keying, they write it twice, and the second write has no obligation to agree with the first. Any consumer that reads during that window can pick up state assembled from two different runs. This is not a theoretical race. It shows up in any system where a long-running task publishes to shared storage, and the orchestrator uses rolling deployments, which is to say most production batch pipelines. The failure is quiet. Nothing crashes. Logs show two successful task completions. The corruption lives entirely in the output, and it surfaces later as a downstream decision made on data that never existed as a coherent snapshot. Why Rolling Deployments Break Long-Running Tasks The root cause is a mismatch between two time scales. A rolling deployment is designed around request handlers that finish in milliseconds, so a few seconds of overlap between old and new replicas is invisible. A task that runs for minutes does not fit that assumption. When the controller starts the new replica, the old one is often most of the way through its work, holding partial results in memory and heading toward the same destination key. The orchestrator considers both healthy. It has no concept of the work each task is doing, only of the process lifecycle. Most teams reach first for at-least-once scheduling with a fixed output path. The task computes its result and writes to a known location; the newest write wins. That model is fine when only one task ever runs. Under deployment overlap, it produces last-writer-wins on a destination that two writers reached through different code paths or different partial reads. If the new build changed how a field is aggregated, the surviving file depends on which replica finished last, which is nondeterministic. Distributed locks are the next instinct, and they trade one failure mode for another. A lease in a coordination service such as etcd or ZooKeeper can stop two tasks from writing at once, but a task that holds a lease for three minutes and then suffers a stop-the-world pause or a network partition forces a choice. Either the lease expires and a second task proceeds, which is the exact duplication you wanted to prevent, or the lease is held conservatively, and a crashed task blocks all progress until an operator intervenes. Locks move the problem; they do not remove it. The durable fix does not try to prevent overlap. It makes overlap harmless. Detecting Divergent Writes Before They Reach Downstream Consumers You cannot fix what you cannot see, and duplicate writes are close to invisible by default. On a store that keeps only the latest object, the second write erases the evidence of the first. The first instrumentation step is to turn on object versioning for the output prefix, which costs storage but converts a silent overwrite into an inspectable history. With versioning on, a duplicate write is detectable as more than one version of the same key inside a single scheduled window. That alone is not a defect: an idempotent rewrite of identical bytes is benign. The real signal is divergence: two versions of the same logical output whose checksums differ. The scan below walks every version under a window prefix, groups by key, and reports only keys whose versions carry more than one distinct entity tag (ETag), the marker that two runs produced different bytes for the same window. Plain Text #!/usr/bin/env bash # Scans an object store for duplicate, DIVERGENT writes to the same logical # output window: the signature of two task replicas racing during a deploy. # Works against any S3-compatible store (AWS S3, MinIO, Ceph RGW). It only # reports, so it is safe to run against production. set -euo pipefail BUCKET="${1:?usage: detect_divergence.sh <bucket> <prefix>}" PREFIX="${2:?usage: detect_divergence.sh <bucket> <prefix>}" # Object versioning is what makes a duplicate write visible at all: without it, # the second write silently overwrites the first and you lose the evidence. versions_json="$(aws s3api list-object-versions \ --bucket "$BUCKET" --prefix "$PREFIX" \ --query 'Versions[].{Key:Key,ETag:ETag,Time:LastModified}' \ --output json)" # A key with one version, or several versions sharing an ETag, is benign. A key # with MULTIPLE DISTINCT ETags means two runs produced different bytes for the # same window: a real correctness defect, not a cosmetic duplicate. echo "$versions_json" | jq -r ' group_by(.Key)[] | {key: .[0].Key, etags: ([.[].ETag] | unique), writes: length} | select((.etags | length) > 1) | "DIVERGENT \(.key) writes=\(.writes) payloads=\(.etags | length)"' # Exit non-zero if any divergence was found, so a deploy gate can block. divergent="$(echo "$versions_json" | jq ' [ group_by(.Key)[] | select(([.[].ETag] | unique | length) > 1) ] | length')" echo "scanned prefix=$PREFIX divergent_keys=$divergent" test "$divergent" -eq 0 Run this on a schedule and wire the exit code into a deployment gate. A nonzero result during or just after a rollout is a direct measurement of the bug, not an inference from downstream symptoms. The divergence rate climbs sharply with task duration. A job under thirty seconds rarely overlaps a rollout, while a job in the two- to three-minute range will overlap nearly every deployment that lands during its run. Idempotent Output Keying and Atomic Publish The structural fix has two parts. First, derive the output key from the inputs rather than from wall-clock time or a process identifier. Two replicas working the same scheduled window must compute the same key, so that duplication targets one object instead of two. Second, publish that object atomically, so a reader never sees a partial write and a duplicate publish becomes a no-op rather than a second racing write. Start with the key. Build it from the fields that define the unit of work: the pipeline name, the closed time window being summarized, and a schema version that you bump only when the output format changes. The schema version earns its place during exactly the moment under discussion. A new binary mid-deploy that emits a new format gets a different key, so it does not collide with the old binary's output. Rust use sha2::{Digest, Sha256}; // Two task replicas that pick up the same scheduled window build the SAME // RunSpec. That property is what the whole scheme relies on. #[derive(Clone)] struct RunSpec { pipeline: String, window_start_epoch: u64, // closed window, deterministic per schedule tick window_len_secs: u64, schema_version: u32, // bump only when the OUTPUT FORMAT changes } impl RunSpec { // Content key derived purely from inputs. Identical inputs -> identical key, // which is what lets two overlapping runs target one object, not two. fn output_key(&self) -> String { let mut h = Sha256::new(); h.update(self.pipeline.as_bytes()); h.update(self.window_start_epoch.to_be_bytes()); h.update(self.window_len_secs.to_be_bytes()); h.update(self.schema_version.to_be_bytes()); let digest = h.finalize(); format!("{}/{}/state-{:x}", self.pipeline, self.window_start_epoch, digest) } } The key is content-derived, so identical inputs yield an identical key, and a changed format yields a new one. The second piece is publishing without a destructive overwrite. The pattern that holds up is to write to a unique temporary object, flush it to durable storage, then promote it into the final key with an operation that is atomic at the storage layer. On a single filesystem, that promotion is a rename. On an object store it is a conditional put that fails if the key already exists, or a multipart completion. Rust use std::fs; use std::io::Write; // Atomic publish: write to a unique temp object, fsync, then promote into the // final key with an operation that is atomic at the storage layer. On one // filesystem that is rename(2). On an object store it maps to a conditional // PutObject (If-None-Match) or a multipart completion, NOT a streamed append. fn atomic_publish(key: &str, payload: &[u8], writer_id: &str) -> std::io::Result<bool> { let final_path = store_root().join(key); fs::create_dir_all(final_path.parent().unwrap())?; // Skip-if-exists: a duplicate run that finds the object already there does // no work and produces no second write. Handles the common finish-early case. if final_path.exists() { return Ok(false); } let tmp = store_root().join(format!(".tmp-{}-{}", key.replace('/', "_"), writer_id)); let mut f = fs::File::create(&tmp)?; f.write_all(payload)?; f.sync_all()?; // durable before it becomes visible // Two writers can both pass the exists() check; rename is still atomic, so // the object is whole, and the payloads are byte-identical because the key // is content-derived. It does not matter which one lands. fs::rename(&tmp, &final_path)?; Ok(true) } Skip-if-exists handles the common case where one replica finishes well ahead of the other. The harder case is two writers that both pass the existence check before either commits. Atomicity at the promotion step is what saves you: the object is always whole, and because the key is content-derived, both candidate payloads are byte-identical, so it does not matter which one lands. Readers need one more guarantee. They should never have to guess which key is current. Publish each generation under its own immutable key, then advance a single pointer with a compare-and-swap (CAS), so consumers follow the pointer and always read a complete generation. A losing writer detects the conflict and backs off instead of regressing the pointer to an older or duplicated generation. Rust use std::fs; // Readers follow a single pointer, so they always observe one COMPLETE // generation, never a partially written one. fn publish_generation(key: &str, payload: &[u8]) -> std::io::Result<()> { let p = store_root().join(key); fs::create_dir_all(p.parent().unwrap())?; fs::write(p, payload) // immutable, content-addressed } // Optimistic compare-and-swap: only advance the pointer if it still holds the // value the writer last observed. A losing writer (a duplicate from the deploy) // detects the conflict and backs off instead of regressing to an older or // duplicated generation. Maps to a conditional write (If-Match on an ETag) in a // real object store or a small consistent key-value store. fn cas_pointer(expected: Option<&str>, next: &str) -> std::io::Result<bool> { let ptr = store_root().join("latest"); let current = fs::read_to_string(&ptr).ok(); let matches = match (current.as_deref(), expected) { (None, None) => true, (Some(c), Some(e)) => c == e, _ => false, }; if !matches { return Ok(false); // someone else moved it; do not clobber } fs::write(&ptr, next)?; Ok(true) } Trade-Offs: Content Keys vs. Locks, and What Teams Pay Content-derived keys with atomic publish cost more storage and more writes than a single fixed path. Every generation is retained until a lifecycle policy expires it, and versioning multiplies object count during the overlap windows you are now able to observe. For a pipeline producing one object per minute, the added cost is small, a few percent of the storage line in most setups, and it buys an output history you can audit and roll back. Against distributed locks, the comparison is starker. A lock-based design adds a hard dependency on a coordination service in the write path, which means its availability becomes your availability and its tail latency becomes your tail latency. The keying approach has no such dependency at write time. Its correctness comes from determinism and atomic promotion, both properties of code and storage you already run. The cost is discipline: every input that affects the output must be folded into the key, or two genuinely different results can collide under one key, and you reintroduce silent corruption from a new direction. The methodology that makes this safe to adopt is incremental rollout validated by the detection scan. Deploy the keyed publish path to a single region first, then run the divergence scan across a full deployment cycle before widening. A clean scan across one rollout is strong evidence the keying covers every input that matters. The verification below runs two overlapping replicas of the same task and asserts that exactly one object results and its contents match what either replica intended. Rust // Verification: two replicas of the SAME logical task, as happens when an old // pod and a new pod both fire during a rolling deploy. Exactly one object must // result, and its bytes must match what either replica intended. fn overlapping_runs_converge() { let spec = RunSpec { pipeline: "border-state".into(), window_start_epoch: 1_726_000_000, window_len_secs: 60, schema_version: 3, }; let key = spec.output_key(); let payload = build_payload(&spec); let wrote_old = atomic_publish(&key, &payload, "old-replica").unwrap(); let wrote_new = atomic_publish(&key, &payload, "new-replica").unwrap(); assert!(wrote_old ^ wrote_new, "exactly one replica writes the object"); assert_eq!(walk(&store_root()).len(), 1, "overlap converges to one export"); } #[test] fn identical_inputs_yield_identical_keys() { let a = RunSpec { pipeline: "p".into(), window_start_epoch: 100, window_len_secs: 60, schema_version: 1 }; assert_eq!(a.output_key(), a.clone().output_key()); } Teams that skip this work do not see failures immediately, which is what makes the omission dangerous. The pipeline runs clean for weeks, then a deployment lands during a long task and a single corrupted generation flows downstream. By the time anyone traces the bad decision back to its source, the offending object has been overwritten, and the logs show two clean completions. The keying and atomic publish pattern turns that entire class of incident into a no-op, and the detection scan turns the residual risk into a number you can watch.

Feature flags are widely used in modern software delivery to control how and when functionality is exposed to users. They allow teams to deploy code independently of releasing features, reducing the risk associated with large or tightly coupled releases. But feature flags are not limited to simple on/off switches. They can support gradual rollouts, experimentation, access control, operational safeguards, and runtime configuration. Each of these use cases has a different purpose and requires a different way of designing and managing flags. This is where feature flag patterns become useful. Instead of treating every flag the same way, teams can classify them based on the problem they are intended to solve. Feature Flags as a Runtime Control Plane Feature flags can be viewed as more than switches embedded in application code. Collectively, they form a lightweight runtime control plane that allows teams to influence application behavior without changing or redeploying the underlying software. In a traditional deployment model, changing application behavior usually requires modifying code, rebuilding the application, and deploying a new version. Feature flags introduce a layer of indirection between the deployed code and the behavior that users experience. The code may already be running in production, while the flag determines whether a particular capability is enabled, who can access it, or under what conditions it should execute. This separation creates two distinct concerns: Deployment plane: Controls what code and artifacts are deployed into an environment.Feature control plane: Controls how the deployed application behaves at runtime. For example, the same deployed version of an application could expose a new feature to internal users, 5% of production traffic, customers in a specific region, or no users at all — simply by changing flag configuration. This makes feature flags useful control points for several software delivery decisions, including release management, progressive delivery, experimentation, operational protection, access control, and runtime configuration. However, these controls do not all serve the same purpose. A flag controlling a canary rollout has different characteristics and lifecycle requirements from an emergency kill switch or an experimentation flag. Understanding these differences provides the basis for organizing feature flags into distinct patterns. A Taxonomy of Feature Flag Patterns Feature flags are used for different purposes across the software delivery lifecycle. Grouping them into patterns helps teams understand why a flag exists, how long it should live, who owns it, and what risks it introduces. A practical taxonomy can organize feature flag patterns into five broad categories. These categories often overlap in implementation, but their intent and lifecycle are different. An operational kill switch may need strict access controls and rapid propagation, whereas an experimentation flag may prioritize accurate audience segmentation and metric collection. Release Management Patterns Release management flags separate code deployment from feature release. Teams can deploy code safely while deciding independently when and to whom the new functionality becomes available. Characteristics of Release Management Flags Release management flags are designed to separate deployment from feature availability. Their main characteristics include: Usually temporary: Most release flags should be removed after the feature reaches full production availability. Progressive exposure: Features can be introduced gradually by percentage, release ring, environment, tenant, or user group. Rapid rollback: A problematic feature or implementation can be disabled without rebuilding or redeploying the application. Stable targeting: Users should consistently receive the same experience during a staged rollout. Production validation: Teams can evaluate new functionality under real-world workloads before complete release. Deployment independence: Code can be deployed even when the associated functionality is not yet ready for users. Short lifecycle: Each flag should have an owner, release criteria, expiration date, and removal plan. Controlled permissions: Only authorized release owners or operators should be able to change production rollout settings. Low-latency evaluation: Flag evaluation should not introduce noticeable latency into the application request path. Release management flags should have clearly defined rollout stages and rollback thresholds. Once the feature is stable and available to its intended population, the flag and obsolete code paths should be removed. Release Toggle A release toggle hides incomplete or unapproved functionality while allowing the underlying code to be deployed to production. For example, a new checkout workflow may be included in the production build but remain disabled until testing and business approval are complete. Once the feature is ready, the flag is enabled without requiring another deployment. Dark Launch A dark launch deploys a new capability into production while keeping it invisible to end users. The system may execute the new functionality in the background to validate its performance, scalability, and integration behavior using real production traffic. For example, requests may be sent to both an existing recommendation engine and a new engine, while only the existing engine’s response is returned to the customer. The new engine’s results and performance can then be evaluated safely. Dark launches are especially useful for validating infrastructure-intensive services, machine-learning models, search engines, and new backend architectures. Percentage or Gradual Rollout A percentage rollout enables a feature for a controlled percentage of the user population. Exposure can gradually increase—for example, from 1% to 5%, 25%, 50%, and finally 100%. The rollout may be based on users, sessions, devices, tenants, or requests. Stable targeting is important: the same user should normally receive the same flag variation throughout the rollout. This pattern limits the impact of defects and provides an opportunity to monitor errors, latency, customer behavior, and business metrics before wider adoption. Ring-Based Rollout A ring-based rollout releases functionality to predefined groups in increasing order of risk. A typical sequence may include: Development and test users Internal employees Selected beta customers Low-risk production tenants The general customer population Unlike a purely percentage-based rollout, rings are defined by user or organizational characteristics. Each ring acts as a validation stage, and promotion to the next ring occurs only after the required technical and business criteria are satisfied. Canary Release Toggle A canary release toggle directs a small amount of production traffic to a new application version or implementation. The behavior of the canary is compared with the stable version before the rollout expands. This pattern is commonly used with microservices, Kubernetes deployments, API gateways, and service mesh. Although it resembles a gradual rollout, the focus of a canary release is typically the validation of a new software version or deployment rather than the exposure of an individual user-facing feature. If the canary shows elevated latency, errors, or resource consumption, the flag can immediately redirect traffic to the stable version. Environment-Based Toggle An environment-based toggle enables different functionality across development, testing, staging, and production environments. For example, diagnostic features may be enabled in development but disabled in production, while a new integration may be enabled only in staging until certification is complete. Environment flags are useful when deployment environments require different behavior, but they should not become a substitute for proper environment configuration. Security-sensitive settings such as secrets, access policies, and credentials should remain in dedicated configuration and secret-management systems. Experimentation Patterns Experimentation flags help teams evaluate product ideas using measurable evidence. Unlike release flags, their primary purpose is not simply to control availability but to compare outcomes across different user groups or system variations. Characteristics of Experimentation Flags Experimentation flags are intended to generate evidence about user behavior, product decisions, or technical alternatives. Their main characteristics include: Hypothesis-driven: Every experiment should begin with a clear and testable assumption. Multiple variations: The flag commonly returns values such as control, treatment A, or treatment B rather than a simple Boolean result.Consistent assignment: A participant should remain in the same experiment group throughout the experiment. Randomized allocation: Where appropriate, participants should be assigned randomly to minimize selection bias. Measurable outcomes: Each experiment should define primary metrics, secondary metrics, and guardrail metrics. Time-bound execution: The experiment should have specified start and end dates or statistically justified stopping conditions. Statistical evaluation: Results should be assessed using appropriate statistical methods rather than informal observation. Mutual-exclusion awareness: Overlapping experiments should be controlled when they could influence one another. Privacy-conscious: Experiment attributes and behavioral data should be collected and processed according to privacy requirements. Decision-oriented: The experiment should conclude with a decision to adopt, modify, reject, or investigate the variation further. Temporary lifecycle: Once the experiment concludes, the winning variation should become the default, and the flag should normally be retired. An experimentation flag is not simply a mechanism for showing different experiences. It should be connected to experiment metadata, participant assignment, telemetry collection, statistical analysis, and a documented final decision. A/B Testing An A/B testing flag divides users into two groups. The control group receives the existing experience, while the treatment group receives a new variation. For example, an online platform may compare two registration pages and measure their completion rates. Users must be assigned consistently to avoid switching between variations during the experiment. A/B tests should be associated with a defined hypothesis, target population, success metric, experiment duration, and stopping criteria. Without these elements, a feature flag only creates different experiences—it does not constitute a controlled experiment. Multivariate Experimentation Multivariate experimentation evaluates several variations or combinations of variables simultaneously. For example, a page may test different combinations of headings, button colors, and recommendation layouts. This can reveal not only which individual variation performs well but also how different variables interact. Because the number of possible combinations can grow quickly, multivariate experiments require sufficient traffic and careful statistical design. They are therefore best suited to platforms with mature experimentation capabilities. Cohort-Based Flags A cohort-based flag provides different functionality to groups that share defined characteristics. Cohorts may be based on account age, usage behavior, industry, geography, device type, or participation in a previous experiment. For example, a simplified onboarding flow may be shown only to first-time users, while existing customers continue to use the established process. Cohort flags are useful for both product learning and targeted delivery. However, cohort definitions should be documented and governed to prevent unintended discrimination or inconsistent customer experiences. Hypothesis or Experiment Toggle A hypothesis toggle represents a specific product or technical assumption that the organization wants to validate. For example: Providing automated remediation recommendations will reduce the average time required to resolve an incident. The flag enables the proposed capability for the selected treatment group, while telemetry measures resolution time, adoption, accuracy, and user feedback. This pattern connects flag configuration to the broader experiment lifecycle. The flag should record the hypothesis, owner, metrics, start and end dates, and final decision. Once the hypothesis has been accepted or rejected, the experiment flag should be retired. Operational and Reliability Patterns Operational flags allow teams to change system behavior quickly without modifying or redeploying code. They are particularly valuable during incidents, traffic spikes, dependency failures, and other production events. Characteristics of Operational and Reliability Flags Operational and reliability flags allow teams to alter production behavior quickly in response to incidents, dependency failures, capacity constraints, or changing operating conditions. Their main characteristics include: Immediate effect: Changes should propagate quickly enough to support incident response. Safe defaults: The default and fallback values should preserve critical services and minimize potential harm. High availability: Flag evaluation should continue working even when the central flag-management service is unavailable. Fail-safe behavior: The application should use a predefined safe value when it cannot retrieve the latest configuration. Restricted access: Only authorized operational personnel should be able to modify high-impact flags. Strong auditability: Every change should record who changed the flag, when it changed, why it changed, and its previous value. Runtime control: Operators can change system behavior without modifying code or initiating a deployment. Incident readiness: Flags should be documented in operational runbooks and tested before an actual emergency. Observability integration: Changes should be correlated with service-level indicators, logs, traces, alerts, and incident timelines. Dependency awareness: Teams must understand which services, workflows, and customer capabilities will be affected. Reversibility: Operators should be able to restore normal behavior safely when the incident is resolved. Variable lifetime: Some operational flags, such as kill switches, may remain permanently available, while incident-specific flags should be retired. These flags are part of the production control plane and should be treated with the same care as other operational mechanisms. An incorrectly configured reliability flag can itself become a source of widespread failure. Kill Switch A kill switch immediately disables a feature or operation that is causing serious problems. For example, if a newly introduced payment integration begins creating duplicate transactions, operators can disable it while leaving the rest of the application available. Kill switches must be easy to find, fast to evaluate, and restricted to authorized personnel. Their safe state should be determined in advance, and the switch should be tested regularly. A kill switch that has never been exercised may fail when it is most urgently needed. Circuit-Breaker Flag A circuit-breaker flag prevents calls to a failing or unstable dependency. It allows operators to open or close the circuit manually or override an automated circuit breaker. For example, if an external credit-check service becomes slow, the flag can temporarily stop outgoing calls and redirect requests to an alternative workflow. This flag should complement — not replace — automatic timeout, retry, and circuit-breaker mechanisms. It provides an operational override for situations that automated policies do not handle correctly. Degraded-Mode Toggle A degraded-mode toggle moves the application into a reduced-functionality state so that essential services remain available. For example, an e-commerce system may disable personalized recommendations and advanced search filters while continuing to support product browsing and checkout. A monitoring platform may suspend historical analytics while preserving real-time alerting. This pattern supports graceful degradation. Teams should define which functions are essential, which can be temporarily disabled, and what users should see when degraded mode is active. Dependency Isolation Flag A dependency isolation flag disconnects a specific internal or external dependency without shutting down the entire feature. For example, an application may isolate a failing notification provider while continuing to process the underlying business transaction. Notifications can be queued and delivered after the dependency recovers. This pattern limits cascading failures and is especially useful in microservice architectures, where a problem in one service can otherwise propagate across the system. Load-Shedding or Capacity Flag A load-shedding flag reduces non-essential work when the system approaches its capacity limits. It may reject, delay, sample, or deprioritize selected requests. For example, during a traffic surge, a platform might disable report generation, reduce recommendation depth, limit expensive queries, or accept only high-priority requests. Load shedding differs from general degraded mode because it is directly concerned with protecting finite resources such as CPU, memory, database connections, thread pools, and inference capacity. It should be connected to clearly defined capacity signals and service-level objectives. Entitlement and Access-Control Patterns Entitlement flags determine which users, organizations, or regions can access a capability. Unlike short-lived release flags, these flags may remain in the system for an extended period because they represent business rules or access policies. Characteristics of Entitlement and Access-Control Flags Entitlement and access-control flags determine whether a capability is available to a particular user, role, customer, subscription, tenant, or jurisdiction. Their main characteristics include: Identity-aware evaluation: Decisions depend on trusted attributes such as user identity, role, tenant, subscription, or contractual region. Fine-grained targeting: Access may vary across users, organizations, plans, regions, or memberships. Potentially long-lived: Unlike release flags, entitlement flags may represent permanent product or contractual rules. Deterministic behavior: The same valid identity and entitlement context should produce a consistent decision. Backend enforcement: Server-side authorization must enforce access even when the user interface hides a feature. Integration with authoritative systems: Subscription and entitlement decisions should use reliable sources such as identity, billing, licensing, and policy systems. Security-sensitive configuration: Changes require strong authentication, role-based access control, and separation of duties where necessary. Auditable decisions: Organizations should be able to determine why access was granted or denied. Privacy-conscious targeting: Only necessary attributes should be used, stored, and transmitted during evaluation. Regulatory awareness: Geographic or compliance rules should be reviewed and approved by appropriate legal and compliance stakeholders. Correct revocation: Access should be removed promptly when a role, subscription, consent status, or contractual condition changes. Failure-safe behavior: If the entitlement cannot be verified, security-sensitive features should normally remain inaccessible. Feature flags can support entitlement decisions, but they should not replace a dedicated authentication and authorization system. They determine feature availability, whereas security controls must protect the underlying data and operations. Permission Toggle A permission toggle enables functionality according to a user’s role or authorized actions. For example, only administrators may be allowed to delete resources, view audit logs, or change organization-wide settings. Feature flags can help expose or hide the relevant user interface, but they must not be the only security control. The backend must independently enforce authentication and authorization. Hiding a button does not prevent an unauthorized user from calling the underlying API. Subscription or Plan-Based Feature A subscription-based flag enables functionality according to a customer’s purchased plan. For example, advanced analytics may be available only in an enterprise tier, while basic reporting is available to all customers. The flag evaluation may use attributes such as product edition, subscription status, licensed capacity, or purchased add-ons. Because these flags affect billing and contractual obligations, their configuration should be integrated with the organization’s entitlement system and protected by strong audit controls. Tenant-Specific Toggle A tenant-specific toggle enables or disables a capability for an individual customer organization. This pattern is valuable in multi-tenant platforms where customers may have different configurations, integration requirements, or adoption schedules. For example, a new data-retention workflow may be enabled for one enterprise tenant after its administrators complete the necessary migration. Tenant-specific flags should be managed carefully. Many ad hoc exceptions can create configuration sprawl and make system behavior difficult to understand. Internal or Beta User Flag An internal or beta-user flag makes early functionality available to employees, testers, design partners, or customers enrolled in a preview program. This allows the organization to collect feedback and identify problems before general release. Beta targeting may use user IDs, email domains, account attributes, or explicit programmed membership. The beta experience should be clearly identified, and users should understand that the feature may change or be withdrawn. Sensitive or unstable functionality may also require explicit consent. Geographic or Regulatory Flag A geographic or regulatory flag controls functionality according to a user’s country, region, legal jurisdiction, or data-residency requirement. For example, biometric authentication may be disabled in regions where regulatory approval has not been obtained. A data-processing feature may be enabled only when the required regional infrastructure is available. Location must be determined using reliable attributes such as the customer’s contractual region or account configuration. IP-based geolocation alone may be inaccurate. Because regulatory decisions carry legal risk, the rules should be reviewed by the appropriate compliance and legal teams. Migration and Architecture Patterns Migration flags allow teams to introduce large technical changes incrementally. They support coexistence between old and new implementations, making it possible to validate behavior, limit risk, and reverse the transition when necessary. Characteristics of Migration and Architecture Flags Migration and architecture flags support the controlled transition between implementations, services, data stores, APIs, infrastructure components, or system architectures. Their main characteristics include: Coexistence of implementations: Old and new components may operate simultaneously during the migration period. Incremental cutover: Traffic, users, tenants, reads, or writes can move gradually to the new implementation. Reversible routing: Workloads can be returned to the previous implementation if the new component fails. Compatibility requirements: Both paths may need to support compatible interfaces, schemas, and operational behavior. State-awareness: Data migrations must account for consistency, ordering, synchronization, and the authoritative source of truth. Comparison capability: Shadow execution, dual writes, or result comparison may be used to validate the new implementation. Strong observability: Teams should compare errors, latency, output correctness, resource consumption, and business results across both paths. Idempotency and reconciliation: Data operations must tolerate retries, duplicates, partial failures, and divergence between systems. Longer but finite lifecycle: Architectural migrations may take months, but their flags should still have completion criteria and removal plans. Broader impact: These flags can affect several services, data flows, or infrastructure components simultaneously. Carefully controlled changes: Flag updates should be reviewed, authorized, audited, and coordinated across responsible teams. Explicit rollback limits: Teams must identify the point after which rollback is unsafe — for example, after an irreversible schema or data-format change. Technical-debt risk: Leaving old and new paths active indefinitely increases maintenance, testing, and operational complexity. Migration flags should be supported by a defined transition plan covering validation, reconciliation, rollback, ownership, cutover criteria, and eventual removal of the legacy implementation. Branch-by-Abstraction Branch-by-abstraction introduces an abstraction layer between the application and an implementation that needs to change. A feature flag selects either the old or new implementation behind that abstraction. For example, an application may define a common storage interface implemented by both a legacy database and a new cloud-native data store. The flag decides which implementation handles a request. This pattern allows teams to perform long-running architectural work in the main codebase without maintaining a separate development branch. After the new implementation is fully adopted, the flag and legacy implementation should be removed. Legacy-to-New-System Migration This pattern routes selected users, tenants, or transactions from a legacy system to its replacement. Migration can proceed incrementally, beginning with internal users or low-risk tenants and expanding after validation. If problems occur, traffic can be returned to the legacy system. Unlike branch-by-abstraction, which describes a code-structuring technique, this pattern describes the operational transition between two complete systems or services. Dual-Write Toggle A dual-write toggle sends updates to both the existing data store and the new one during a migration. For example, when moving customer profiles to a new database, the application may continue writing to the legacy database while also writing the same changes to the new database. The outputs can then be compared for consistency. Dual writes introduce risks such as partial failure, ordering differences, retries, and duplicate operations. The design should include idempotency, reconciliation, observability, and a clearly defined source of truth. Read-Path Switching A read-path flag determines whether data is retrieved from the old system or the new system. The migration may initially write to both systems while continuing to read from the old one. After the new store has been validated and reconciled, a small portion of read traffic can be directed to it. The percentage can then increase gradually. Read switching should account for differences in data freshness, schema, caching, consistency, and error handling. Shadow reads may also be used to compare results without returning the new system’s response to users. API Version Migration An API version migration flag routes requests between different versions of an API, protocol, or service contract. For example, selected clients may be routed from version 1 to version 2 while other consumers remain on the original version. This supports progressive compatibility testing and reduces the risk of a single cutover. The flag should not hide permanent incompatibilities indefinitely. API ownership, deprecation deadlines, consumer migration, and contract testing are still required. Infrastructure or Configuration Toggle An infrastructure or configuration toggle controls the adoption of a new infrastructure component or runtime configuration. Examples include switching between message brokers, selecting a new cache cluster, enabling a new autoscaling policy, changing an observability pipeline, or routing workloads to a different cloud region. These flags require stronger governance than ordinary user-interface flags because an incorrect change can affect the entire platform. Access should be restricted, changes audited, dependencies validated, and rollback behavior tested before production use. Choosing the Appropriate Pattern The correct pattern depends on the intent of the flag: Primary objective Suitable pattern Hide unfinished functionality Release toggle Validate a backend capability invisibly Dark launch Limit initial user exposure Percentage rollout Release through controlled user groups Ring-based rollout Compare a new deployment with a stable version Canary release toggle Test a product hypothesis A/B or experiment toggle Stop harmful functionality during an incident Kill switch Preserve essential functionality during failure Degraded-mode toggle Protect the system during excess demand Load-shedding flag Control commercial availability Subscription-based flag Enable functionality for selected customers Tenant-specific toggle Move safely between implementations Branch-by-abstraction Validate a new data store Dual-write and read-path flags Transition consumers to a new contract API version migration The most important distinction is not how a flag is implemented, but why it exists. Its purpose determines its owner, expected lifetime, targeting rules, monitoring requirements, security controls, and retirement process. Treating every flag as the same kind of Boolean switch leads to unmanaged dependencies and technical debt. Treating flags as explicit architectural and operational patterns makes them safer and easier to govern. Feature Flag Lifecycle A feature flag should be managed from creation to removal. Without a defined lifecycle, temporary flags can remain in the codebase, increase complexity, and create technical debt. Feature Flag Anti-Patterns Feature flags provide flexibility and reduce deployment risk, but poor implementation can introduce technical debt, inconsistent behavior, security vulnerabilities, and operational failures. The following anti-patterns should be avoided. Permanent temporary flags: Release, experiment, and migration flags remain in the system long after their purpose has been completed. These stale flags increase conditional logic, complicate testing, and make the codebase harder to understand. Avoidance: Assign every temporary flag an owner, expiration date, and removal criteria when it is created.Excessive flag dependencies: One flag’s behavior depends on several other flags, creating complex combinations and unexpected outcomes. Developers and testers may be unable to determine which feature state is active. Avoidance: Keep flags independent where possible. Document unavoidable dependencies and validate permitted combinations.Deeply nested flag logic: Multiple flag checks are nested throughout the code, producing difficult-to-follow execution paths. Avoidance: Centralize flag decisions, use clear abstractions, and select the required implementation near the system boundary.Reusing a flag for multiple purposes: A single flag is reused across unrelated features, experiments, or operational controls. Changing it for one reason may unintentionally affect another part of the system. Avoidance: Each flag should have one clearly defined purpose, owner, and lifecycle.Using flags as a substitute for configuration: Feature flags are used to manage every application setting, including database connections, credentials, and static environment properties. Avoidance: Use feature flags for runtime behavioral decisions. Store secrets in secret-management systems and stable settings in appropriate configuration systems.Treating flags as security controls: A feature is hidden in the user interface through a flag, but its backend API remains accessible. An unauthorized user may bypass the interface and call the API directly. Avoidance: Enforce authentication and authorization independently on the server. Feature flags may control availability, but they must not replace security controls.Unsafe default or fallback values: The application uses an arbitrary value when the flag service is unavailable. This can expose unfinished features, block critical operations, or amplify an incident. Avoidance: Define and test a safe fallback for every flag based on its purpose and risk.Remote evaluation on every request: The application contacts the flag-management service synchronously for every evaluation. Network latency or a service outage can then affect the application’s availability. Avoidance: Use local evaluation, cached configurations, asynchronous updates, and predefined fallback values where appropriate.Unstable user assignment: Users move between enabled and disabled variations across sessions or requests. This creates an inconsistent experience and invalidates experiment results. Avoidance: Use deterministic targeting based on stable identifiers and consistent hashing.Uncontrolled percentage rollouts: Traffic exposure is increased without health checks, approval gates, rollback thresholds, or sufficient observation time. Avoidance: Define staged rollout steps and measurable promotion and rollback criteria before activation.Missing ownership and documentation: No team or individual is responsible for a flag, and its purpose, dependencies, or expected lifetime are unclear. Avoidance: Record the flag’s owner, category, description, creation date, affected services, and review or expiration date.Inadequate testing of flag states: Only the default flag value is tested. The alternate path — or interactions with other important flags — may fail when enabled in production. Avoidance: Test enabled, disabled, fallback, and transition behavior. Test critical supported combinations without attempting every theoretical combination.Direct production changes without governance: Anyone can change a high-impact flag in production without approval, audit records, or change validation. Avoidance: Apply role-based access control, audit logging, peer approval, and separation of duties according to the flag’s risk.Missing observability: A flag is enabled without recording evaluation results or correlating the change with application and business metrics. Teams may not recognize when the rollout causes harm. Avoidance: Track flag changes and variations alongside errors, latency, resource usage, user outcomes, and service-level indicators.Flag naming and semantic confusion: Names such as disable_new_flow=false use negative logic and make the effective behavior difficult to interpret. Avoidance: Use clear, positive, purpose-specific names such as new_checkout_enabled, together with documented variation meanings.Flags at the wrong granularity: A flag controls too much functionality, making rollback disruptive, or controls tiny implementation details, causing flag proliferation. Avoidance: Choose boundaries that represent independently releasable, operable, or measurable capabilities. Indefinite dual paths: Old and new implementations continue running long after migration or release. Both paths must then be maintained, secured, and tested indefinitely. Avoidance: Define completion criteria, a cutover date, and tasks for removing the legacy path and associated flag.Emergency flags that are never tested: Kill switches and degraded-mode flags exist but have never been exercised. During an incident, they may fail, propagate too slowly, or cause unexpected side effects. Avoidance: Test operational flags through scheduled drills and include their activation and recovery procedures in runbooks.Sensitive data in targeting rules: Personally identifiable or confidential data is embedded directly in flag rules, logs, or evaluation contexts. Avoidance: Minimize targeting attributes, use opaque identifiers where possible, restrict access, and apply appropriate retention and privacy controls.Making irreversible operations reversible in appearance only: A flag suggests that a change can be rolled back even after irreversible actions — such as destructive schema changes or incompatible data writes — have occurred. Avoidance: Define the rollback boundary before activation and use staged migrations, compatibility layers, backups, reconciliation, and forward-recovery plans. A sound feature-flag practice therefore requires more than adding conditional statements. Flags should be purpose-specific, observable, securely governed, thoroughly tested, and removed when they no longer provide value. Conclusion Feature flags are more than on/off switches. When applied through the right patterns, they enable safer releases, controlled experimentation, rapid incident response, targeted access, and gradual system migrations. Their value depends on disciplined management. Every flag should have a clear purpose, owner, safe default, monitoring strategy, and retirement plan. The goal is not to create more flags, but to use the right flag pattern for the right problem. Deploy with confidence, release with control, and let feature flags make the difference.

By Josephine Eskaline Joyce, Ph.D CORE

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.