Testing, Deployment, and Maintenance

Abstract Modern distributed systems rarely fail in isolation — they degrade across multiple execution steps. This article presents a control-loop-based architecture for building self-healing systems that detect anomalies early, precisely isolate failures, and automatically recover using context-aware decisions. Introduction Modern distributed systems are large-scale platforms built on service-oriented architecture. In such systems, an individual request — the unit of execution — typically flows through multiple services, including clients (request initiators), orchestrators, enrichment layers, validation or policy-evaluation systems, routing layers, downstream dependencies, state management systems, reconciliation processes, and notification systems. Each service in this chain introduces latency, retries, dependencies, and failure modes. Because of this, failures in distributed systems rarely appear as clean, isolated events. Instead, they emerge as a sequence of interacting issues that create a cascading effect across the system. For example, a downstream dependency may become slow in a specific region. This increases retries, which in turn increases queue depth. The growing queue depth puts pressure on the orchestrator, eventually causing it to fail unrelated requests due to resource saturation. What initially was a local dependency problem rapidly turned into a widespread degradation of workflow. This problem is particularly difficult in asynchronous systems, where failures are not always instantly visible. A request may not fail instantly — it may remain pending, miss its expected execution window, be delayed in execution, get stuck in an intermediate state, or lose coordination between system components. When the operator detects the issue, the impact could already be large enough. However, traditional protection mechanisms such as fixed failure thresholds, static alerts, and global circuit breakers are often too coarse-grained for these scenarios. A localized dependency failure should not halt the entire system. At the same time, localized issues must not be allowed to trigger storms or cascade into otherwise healthy execution paths. The goal, therefore, is to build a self-healing control system that can detect anomalies at the level of individual requests, aggregate signals across execution and system dimensions, isolate only the affected scope, and recover gradually based on real-time evidence. This post presents such a system. It is designed to provide predictive anomaly detection, hierarchical aggregation, scoped and global kill switches, adaptive leaky-bucket flow control, observability, and AI-assisted investigation and escalation. featurestatic thresholds (old way)context-aware loops (new way)DetectionStatic ThresholdingPredictive Anomaly DetectionContainmentGlobalScopedControlBinary ShutdownAdaptive Flow ControlRecoveryManualEvidence-Based Self-Healing Why Traditional Systems With Static Thresholds Won’t Work Most distributed systems rely on mechanisms like retries, dead-letter queues, alerts, and circuit breakers. These are useful but not enough for complex async workflows as they depend on static thresholds, which are context-blind by nature. A rule like “trigger an alert when failures exceed X%” cannot distinguish between fundamentally different types of failures: Logical failures, where a request completes but produces an incorrect result due to issues in input, configuration, or application logic Execution failures, where a request produces no result due to delays, retries, or loss of coordination across system components For example, in an AI inference system, a request may return an incorrect response due to model configuration issues (logical failure), or it may be accepted but never complete due to stalled execution in downstream components (execution failure). Static thresholds treat both cases uniformly, even though they require very different responses. As a result, systems either overreact to expected failures or miss critical anomalies such as stuck or silently failing requests. Failure volume alone is also a weak signal. A small number of failures could be highly significant if those requests were anticipated to be successful. For instance, if requests following the same execution path have historically resulted in high reliability, even a few failures in that cohort can imply a serious issue. Static thresholds also lack scope awareness. A local failure example, requests routed through a particular execution path, dependency, or region, should not cause a global shutdown. However, a pattern of small anomalies across different paths, regions, or request classes could indicate a larger systemic problem, even if no single threshold is crossed. For instance, in an inference system, requests served by a specific model variant may observe increased latency or degraded outputs due to recent changes to configurations or parameters, while other models and request paths continue to function normally. These limitations are amplified in asynchronous systems, where failures are not always specific. Coordination gaps can cause requests to be stuck, delayed, retried multiple times, or enter into inconsistent states. This leads to higher latency, missed completion signals, or repeated retries with no progress. These weaknesses are further revealed during recovery. AI Agents or operators have to manually inspect logs and dashboards to determine when to resume traffic, resulting in inconsistent performance, slowness, and reactive recovery. In summary, these challenges demonstrate that static thresholding is not sufficient for modern distributed systems. What is needed is a system that understands request context, expected behavior, and the scope of the anomaly. This leads to a fundamental shift in system design: Static thresholding → Predictive anomaly detection Global containment → Scoped containment Binary shutdown → Adaptive flow control Manual recovery → Evidence-based self-healing Instead of asking: Are requests failing? The system should ask: Are requests behaving as expected within their defined SLA, given their execution context and expected outcomes? System Architecture as a Control Loop The system functions as a control loop during request execution. It does not replace the execution path. Instead, it constantly monitors the system's behavior, predicts expected outcomes, identifies deviations, and makes control decisions based on real-time signals. Orchestrated Execution With Continuous Monitoring A primary orchestrator drives the system. It executes each request through a series of steps. At each step, the orchestrator calls on one or more downstream systems, either synchronously or asynchronously. These downstream systems may have their own dependencies. As the request moves forward, it carries contextual metadata like tenant class, region, request type, execution path, and routing decisions. This context defines how the request should behave at each step or at a specific point. While the orchestrator manages execution, anomaly detection serves as a continuous control layer throughout these steps. It tracks the outcome of each phase to ensure that the request moves forward as expected and that the contextual integrity remains intact. Context Preservation and Signal Collection At every step, the system captures signals such as latency, retries, routing decisions, execution status, and downstream responses. It also augments the request with derived attributes such as execution path identifiers and historical behavior patterns. This ensures that each request is evaluated relative to similar cohorts, and more importantly, allows the system to identify where deviations occur within the execution flow — not just whether the request ultimately fails. Success Prediction Engine Intuition: The system learns what 'normal' looks like for similar requests and uses that to estimate expected outcomes. The system estimates how likely a request is to succeed based on its context and historical behavior. For each request i, the expected success is computed as: Plain Text P_i = P(success | x_i) Where: x_i = request features (context, routing path, system state) P_i = expected probability of success This establishes what should happen at different stages of execution, allowing the system to detect deviations between expected and actual outcomes throughout the request lifecycle. Step-Level Anomaly Detection Unlike traditional systems that evaluate only final success or failure, this system continuously monitors each critical step of execution. A request may: Be accepted but delayed Be routed to an unexpected path Experience retries at a specific step Produce degraded output Fail to progress beyond a step By evaluating these signals against expected behavior for that request’s context, the system can detect anomalies early and pinpoint the exact step where deviation occurs. Inference Example (Grounding) For example, in an inference system, the orchestrator can direct a request from a certain tenant class to a summarization model in a certain subnet of a region. If that subnet/region experiences network latency, requests may still be accepted and processed, but exhibit higher latency or delayed responses. In this case, the orchestrator continues execution, but a specific step — model execution in that region — is deviating from expected behavior. Other models or regions may continue to function normally. Hierarchical Roll-up Counters The hierarchical roll-up model aggregates anomalies across multiple contextual dimensions. When a request deviates from expected behavior at any step, the system updates counters across relevant dimensions such as dependency, execution path, tenant class, and region. Example roll-ups: Plain Text (dependency, request_type) (dependency,request_type, tenant_class) (dependency, region) (execution_path, request_type) (global) A single anomalous request may update multiple roll-ups simultaneously. For example, a request routed to a summarization model in a latency-affected region may update: Plain Text (summarizer_model, tenant_class_A, region_us_west) (summarizer_model, region_us_west) (summarizer_model, tenant_class_A) (global) This multi-dimensional view allows the system to isolate issues precisely while still capturing broader systemic patterns. Roll-Up Configuration Model Each roll-up is independently configurable, allowing the system to adapt thresholds and behavior based on the criticality of different execution paths and request classes. Example configuration: JSON { "roll-up_id": "dependency_request_type_region", "dimensions": ["dependency", "request_type", "region"], "threshold": 25, "tumbling_window": "30m", "parent_roll-up_ids": [ "dependency_region", "dependency_request_type", "dependency", "global" ], "control_action": "HOLD_AND_PROBE" } Key Fields dimensions → define how the rollup key is constructed threshold → anomaly count required to trigger tumbling_window → fixed evaluation window (e.g., 30 minutes) parent_rollup_ids → defines relationships across rollups control_action → action applied when this rollup becomes the resolved scope Hierarchical Rollup Model (DAG) The hierarchy is modeled as a directed acyclic graph (DAG). This allows a granular rollup to contribute to multiple parent views. For example: Plain Text (dependency=D1, request_type=TYPE_A, region=EU) → (dependency=D1, region=EU) → (dependency=D1, request_type=TYPE_A) → (dependency=D1) → (global) A single anomalous request may update multiple rollups simultaneously, including both child and parent scopes. Rollup Runtime State At runtime, each rollup key maintains its own state within a tumbling window: Plain Text Rollup: (dependency, region) Key: D1:EU Window: 30 mins Anomaly Count: 35 Threshold: 25 → FIRED Each rollup evaluates independently: A child rollup may fire without the parent firing A parent rollup may fire when anomalies are distributed across multiple children Parent Roll-up Escalation Guard Since parent roll-ups aggregate signals, the system must prevent escalation caused by a single noisy child. Instead of maintaining a full child-level state, each parent tracks lightweight signals: parent_anomaly_countimpacted_child_countmax_child_contribution_ratio A parent roll-up is considered impacted only when: Plain Text parent_anomaly_count >= parent_threshold AND impacted_child_count >= min_required_children AND max_child_contribution_ratio <= max_allowed_ratio Example: Do not escalate at the parent level if only the request Type_A is failing. Plain Text TYPE_A = 100 anomalies TYPE_B = 0 TYPE_C = 0 Parent count = 100 Impacted children = 1 → Keep control at child level Example: Escalate. Plain Text TYPE_A = 40 TYPE_B = 35 TYPE_C = 25 Parent count = 100 Impacted children = 3 → Escalate to parent scope Why This Matters This ensures: Localized issues remain scoped Distributed anomalies are escalated correctly. Noisy signals do not trigger unnecessary global actions Anomaly Detection Engine The anomaly detection engine identifies unexpected deviations by comparing predicted outcomes and actual results and propagates these signals to rollup counters. A request is marked anomalous only if it was expected to succeed but deviates from expected behavior: Plain Text Anomaly_i = 1 if P_i ≥ τ AND Y_i deviates from expected outcome Where: Pi = predicted success probability Yi = observed outcome (failure, delay, degraded output, etc.) Each anomalous request updates multiple rollups across dimensions such as dependency, region, request type, and tenant class. The system evaluates all rollups that breach their thresholds and resolves the appropriate control scope. It then: Deduplicates overlapping signals Selects the highest meaningful level in the hierarchy Avoids redundant or conflicting controls This ensures: Localized issues remain scoped Correlated anomalies are elevated appropriately Duplicate control actions are avoided Kill Switch Controller The kill switch controller enforces control actions at the resolved anomaly scope. Based on severity and scope, it determines whether to: Stop new incoming requests within the scope Hold in-progress requests before critical downstream steps Allow controlled traffic via throttling or probing Control Actions Plain Text ALLOW → continue processing HOLD → pause new and in-progress requests THROTTLE → limit request rate PROBE → allow controlled traffic REROUTE → send via alternate path ESCALATE → trigger alerts / human intervention The controller applies actions consistently across the resolved scope, ensuring full containment without partial or conflicting behavior. Adaptive Recovery Strategy Once a control action is applied, the system does not immediately resume normal traffic. Instead, it gradually reintroduces traffic using a probing strategy. For example: Plain Text Step 1: allow 1 request Step 2: if successful (actual outcome == predicted outcome, allow 2 Step 3: if stable, allow 5 Step 4: gradually increase Step 5: if failures reappear, reduce or stop Recovery is guided by: Plain Text Recovery_G = Successful_G / Released_G Where: G = impacted roll-up scope This ensures: Safe and gradual recovery Avoidance of sudden failure spikes Validation of real system behavior Observability and Audit Layer The system captures all signals across execution: Predicted outcome Actual outcome Anomaly classification Impacted rollups Resolved scope Control action Recovery state These signals provide visibility into: Anomaly trends Active control scopes Held vs released requests Recovery progress This ensures full transparency, debuggability, and auditability. AI Control Plane The AI control plane operates outside the execution path and complements deterministic control logic. It consumes: Anomaly signals Roll-ups Deployment changes System health Control decisions It performs: Investigation → correlates anomalies with systems or changes Automated remediation → triggers safe rollback Escalation → notifies relevant teams Summarization → generates incident insights Key Separation Plain Text Decision Plane → deterministic (prediction, anomaly detection, control) AI Control Plane → intelligent (analysis, remediation, escalation) Conclusion Modern distributed systems cannot rely on static thresholds and reactive controls. Failures are often contextual, asynchronous, and distributed across multiple execution paths. This architecture introduces a fundamental shift: From failure counting → context-aware detection From global shutdown → scoped containment From reactive response → adaptive, evidence-based recovery By combining prediction, hierarchical rollups, scoped control, and adaptive recovery, the system can precisely isolate deviations, minimize impact, and restore stability safely. The core idea is simple but powerful: Systems should not just detect failures — they should continuously understand system behavior, localize deviations in context, and adapt in real time to maintain reliability. What’s Next: From Architecture to Code Designing the architecture is only the first step. In the next post, we move from the blueprint to the technical implementation, diving deep into: The State Machine: Managing high-cardinality counters without latency and affecting execution path.The Escalation Guard: Pseudo-code to prevent "noisy neighbor" failures.Adaptive Recovery: The logarithmic logic for safe traffic re-introduction. Stay tuned for the implementation deep-dive. Case Study: Applying the Control Loop to a Multi-Region Inference System End-to-end Example: Inference system with scoped control and adaptive recovery This example illustrates how anomalies propagate, how scope is resolved, and how control and recovery are applied in an inference system. Step 1: Incoming Requests Requests are routed by the orchestrator to model services in the DUB region: Plain Text (model=summarizer_v2, tenant_class=A, region=DUB) (model=translator_v1, tenant_class=A, region=DUB) (model=qa_model_v3, tenant_class=A, region=DUB) Predicted success: Pi≈0.95+ Step 2: Deviations → Anomalies Due to network degradation in DUB, requests begin to show: increased latency delayed responses occasional degraded outputs Yi deviates and Pi≥τ⇒Anomalyi=1Y_i \text{ deviates and } P_i \geq \tau \Rightarrow Anomaly_i = 1. Step 3: Roll-up Updates Each anomalous request updates multiple rollups: Plain Text (summarizer_v2, tenant=A, DUB) → 40 (translator_v1, tenant=A, DUB) → 35 (qa_model_v3, tenant=A, DUB) → 25 (region=DUB) → 100 Step 4: Parent Escalation Guard Plain Text parent_count = 100 impacted_child_count = 3 max_child_ratio ≈ 40% Since anomalies are distributed across multiple models, not concentrated in one: Plain Text → Escalate to (region=DUB) Step 5: Impact Resolution Fired roll-ups: Plain Text (summarizer_v2, tenant=A, DUB) (translator_v1, tenant=A, DUB) (qa_model_v3, tenant=A, DUB) (region=DUB) Resolved scope: Plain Text (region=DUB) Child rollups are de-duplicated and consolidated under the parent scope. Step 6: Control (Scoped Isolation + Reroute + Local Probing) Action: Plain Text HOLD_AND_PROBE + REROUTE Effect: Throttle or hold most requests routed to DUB Reroute the majority of traffic to FRA only after verifying that the region has sufficient available capacity and is operating within stable limits.Allow a small number of low-impact requests to continue via DUB as probes These probe requests validate whether the issue is transient or persistent without exposing the system to large-scale risk. Step 7: Adaptive Recovery Traffic is managed dynamically: Plain Text DUB (probe path): 1 → 2 → 5 → gradual increase FRA (rerouted path): handles majority of traffic Recovery signal: RecoveryG = SuccessfulGReleasedGRecovery_G = \frac{Successful_G}{Released_G} If probe requests via DUB succeed → gradually restore DUB traffic If failures persist → continue routing to FRA and reduce DUB probes Step 8: AI Control Plane Based on observed signals: Regional network issue → continue routing to FRA Model deployment issue → rollback model version Infrastructure saturation → rebalance across regions Transient degradation → generate summary without escalation Key Takeaways Failures are localized but distributed across modelsControl is applied at the correct scope (region-level)System avoids global shutdownRecovery is validated through controlled probingTraffic is dynamically rerouted and restored The system does not simply stop traffic-it isolates the impacted scope, reroutes intelligently, and verifies recovery through controlled probing before storing normal behavior.

Overview This article explores whether dependency injection (DI) can exist in Rust without sacrificing the language’s core philosophy of zero-cost abstractions. We will approach the question from three angles: Why dependency injection still matters in Rust, even for systems built with zero-sized types and compile-time guarantees.How DI evolved in other ecosystems, using Java as a reference point.A practical Rust-oriented approach to implementing DI with compile-time guarantees. We’ll also show how Rust traits enable DI patterns that scale across crates, preserving zero-cost guarantees. All Rust source code used in this article is available in this repository. Rust DI: The Problem Rust Hasn’t Solved Yet Rust has solved problems most languages haven’t even dared to touch: memory safety without a garbage collector, fearless concurrency, and powerful zero-cost abstractions. But there is a class of problems Rust hasn’t fully confronted yet. Not because Rust is incapable — but because these problems exist above the machine level. They are not about memory safety or performance. They are about composition, modularity, and architectural correctness in large systems. Managing dependencies between dozens or hundreds of components is fundamentally different from managing memory or threads. Rust gives us powerful primitives, but the question remains: How do we scale composition safely and maintainably? What “Enterprise” Really Means in Rust Terms When Rust developers hear enterprise, they often think slow, over-engineered, and bloated. But that perception is misleading. Enterprise systems are not bloated by accident. They are complex because composition eventually stops being trivial. The complexity comes from business requirements, not from the technology stack. Enterprise: The Burden We Can’t Avoid When a company reaches a certain scale, several things inevitably happen: Products serve thousands or millions of usersSystems integrate with vendors, partners, and third-party servicesTeams work independently on modules and featuresSoftware must evolve continuously without stopping the business These realities create architectural pressure. From a technical perspective, systems must support: Scalability: At multiple levels — both in terms of users and data, including hundreds, thousands, or millions, or even up to billions of concurrent users, as well as functional modules interacting across teams.Reliability: Systems run 24/7. Services must handle failures because dependencies on vendors, partners, or third-party services mean that failures are inevitable, and the system must continue operating despite them.Modularity: Independent teams need to work on isolated components without breaking other parts of the system.Flexibility: Infrastructure choices may change. Databases, messaging systems, or integrations might need to be swapped without rewriting the entire application.Observability: To detect and respond to performance bottlenecks, integration failures, or unexpected behaviors quickly.Extensibility: New products, markets, and regulations require systems to evolve incrementally rather than being rebuilt from scratch.Maintainable: Every business decision introduces new dependencies. And every dependency increases the complexity of the system’s composition. Ensuring that the system doesn’t become so convoluted that small changes introduce cascading errors. Even with Rust’s ownership model and strong type system, manually managing this dependency graph eventually becomes impractical. These pressures are not theoretical — they define the daily reality of enterprise software engineering. Every design decision must balance immediate business needs with long-term sustainability, especially under high concurrent load. Where Dependency Injection Becomes Relevant This is exactly where dependency injection becomes useful. DI allows systems to manage complexity by separating what components need from how those dependencies are created and connected. In practice, this means: Components declare their dependencies without constructing them directlyDependencies are provided externally, keeping components isolatedSystems evolve gradually without breaking existing modulesOptional features and plugins can be integrated without tightly coupling the system DI is not just a convenience. It is a structured approach to handling inevitable architectural complexity. Enterprise Isn’t Just Complexity — It’s Heterogeneity Large systems are rarely uniform. They typically contain: Independent components with their own dependency treesStateful infrastructure such as databases, caches, and message brokersOptional features and plugin-style modulesMultiple implementations of the same interface This heterogeneity appears naturally over time. Systems accumulate tools built years apart, libraries maintained by different teams, and components that survive long after their original authors have moved on. Enterprise systems grow gradually, and they rarely get the chance to start over. Rust does not eliminate these pressures. Any real system eventually faces them. Java’s Historical Perspective: DI Was Inevitable Java did not adopt dependency injection because it was fashionable. It adopted DI because large systems were becoming impossible to manage without it. Without DI, developers quickly ran into familiar problems: Tight coupling between componentsFragile initialization orderHard-coded dependencies scattered across the codebaseChanges in one module unexpectedly breaking another Dependency injection emerged as a discipline for managing complexity. Components declare what they depend on, and the system provides those dependencies when constructing the application. This separation allows systems to evolve without collapsing under their own architecture. DI in a Nutshell You can think of dependency injection as a kind of runtime composition system. If your application contains many services, modules, plugins, or optional components, something must assemble them and ensure they are wired correctly, and that role belongs to the DI system. DI is conceptually similar to package managers such as Cargo or Maven, but it operates at a different level: Package managers resolve dependencies between libraries at build time.Dependency injection resolves dependencies between components at runtime. Loading executable code into memory is easy — the operating system handles that. What is harder is creating objects, initializing them correctly, and ensuring that all components interact with the right dependencies. This becomes increasingly difficult as systems grow. Dependency injection addresses this problem directly. How Dependency Injection Is Typically Solved in Java Java provides one of the most mature ecosystems for dependency injection. Frameworks such as Spring or Guice automate object creation and dependency wiring almost entirely. Let’s revisit the same example from the previous section: a simple User Management API. We have two controllers: ReadController — retrieves users from a databaseWriteController — creates users and publishes events to a message broker Both controllers depend on infrastructure services that must be created and wired correctly. Without Dependency Injection In a traditional manual setup, object creation and wiring might look like this: public class Application { public static void main(String[] args) { Database database = new PostgresDatabase(); MessageBroker broker = new KafkaBroker(); ReadController readController = new ReadController(database); WriteController writeController = new WriteController(database, broker); // start application } } At first glance, this appears manageable. But as the application grows, the initialization code expands rapidly: Multiple infrastructure servicesOptional modulesConfiguration logicConditional wiring depending on the environment The main method eventually becomes responsible for constructing the entire dependency graph of the application. This approach becomes difficult to maintain and extremely fragile as the system evolves. Dependency Injection With Spring Dependency injection frameworks solve this by moving the responsibility of object creation and wiring to a container. Components simply declare what they need. @Service public class Database { } @Service public class KafkaBroker implements MessageBroker { } @RestController public class ReadController { private final Database database; @Autowired public ReadController(Database database) { this.database = database; } } Dependencies are declared in constructors, and the DI container automatically provides the correct instances. The application no longer manually constructs the object graph. Instead, the framework scans components and resolves dependencies automatically. Polymorphism in Java DI Java DI frameworks also support multiple implementations of the same interface. For example, an application may support several message brokers simultaneously: @Service public class KafkaBroker implements MessageBroker { } @Service public class RabbitBroker implements MessageBroker { } A controller can receive all implementations at once: @RestController public class WriteController { private final List<MessageBroker> brokers; @Autowired public WriteController(List<MessageBroker> brokers) { this.brokers = brokers; } } The DI container automatically collects all implementations of MessageBroker and injects them into the controller. This makes the system highly extensible: New brokers can be addedExisting ones can be removedThe controller remains unchanged The Cost of Traditional DI Java DI frameworks provide powerful capabilities, but they come with trade-offs: Dependency resolution happens at runtimeReflection is heavily usedErrors may only appear during application startupDependency graphs are not always fully visible to the compiler This runtime flexibility works well for the Java ecosystem, but it introduces overhead and reduces compile-time guarantees. Rust, on the other hand, encourages a different philosophy: If something can be verified at compile time, it should be. This raises an interesting question: Can Rust achieve the same flexibility of dependency injection while preserving compile-time guarantees and zero runtime cost? Journey into Rust Coding Let’s try to build a dependency injection approach in Rust gradually. We will follow the same conceptual example used in the Java section: A ReadControllerA WriteControllerMultiple implementations of a MessageBrokerAn abstraction for database connectivity Rust Without Dependency Injection In the first example, we will implement a small Rust application without dependency injection. However, we will introduce use-traits, which will later allow us to transition naturally to a dependency injection model. 1. Defining Database Interfaces First, let’s define the interface used to access the database. 1.1 DatabaseConnection Trait This trait represents an abstraction for database connectivity that can support multiple implementations (Postgres, MySQL, etc.). trait DatabaseConnection { fn read_query(&self, query: &str); fn write_query(&self, query: &str); } 1.2 UseDatabaseConnection Trait Next, we define a trait that allows components to request a database connection from a context. trait UseDatabaseConnection { type T: DatabaseConnection; fn database_connection(&self) -> &Self::T; } This trait will later be used as the foundation of dependency resolution. Instead of components knowing the entire application context, they simply declare that they require a DatabaseConnection. This keeps components decoupled from the full application structure. 2. Database Implementation Now we provide a concrete implementation of DatabaseConnection. #[derive(Default)] struct PostgresDatabaseConnection {} impl DatabaseConnection for PostgresDatabaseConnection { fn read_query(&self, query: &str) { println!("Reading from Postgres DB: {}", query) } fn write_query(&self, query: &str) { println!("Writing into Postgres DB: {}", query) } } For simplicity, this example only prints messages instead of connecting to a real database. In a real system, this could be implemented using any production database library. 3. Controllers Now we define the controllers responsible for performing application logic. 3.1 Controller Structs #[derive(Default)] struct ReadController {} #[derive(Default)] struct WriteController {} Rust allows structs with no fields. These zero-sized types have no runtime cost, but they still represent concrete types at compile time and can participate in abstractions. 3.2 Controller Use Traits Next, we define traits that expose controllers to other components. trait UseReadController { fn read_controller(&self) -> &ReadController; } trait UseWriteController { fn write_controller(&self) -> &WriteController; } These traits allow components to access controllers without knowing anything about the application context. 3.3 Controller Context Now we combine the previously defined traits into a context trait. trait ControllerContext: UseDatabaseConnection + UseReadController + UseWriteController {} This context describes the minimal environment required for controllers to function. Controllers will depend only on this trait instead of the full application context. 3.4 Controller Implementation Now we implement the controller logic. impl ReadController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection() .read_query(format!("SELECT * FROM table WHERE id = '{}'", argument).as_str()); } } impl WriteController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection().write_query( format!("UPDATE table SET value = 'new' WHERE id = '{}'", argument).as_str(), ); } } Notice something important here: The controllers do not know about the full application context. They only know about the traits they depend on. This means the controller and database code could already be extracted into separate crates, reusable by any application implementing the required use-traits. 4. Wiring the Application Now we wire all components together. 4.1 Application Context We define a struct that holds all application components. #[derive(Default)] struct ApplicationContext { read_controller: ReadController, write_controller: WriteController, postgres_database_connection: PostgresDatabaseConnection, } This struct acts as the composition root of the application. 4.2 Implement Use Traits Next we implement the previously defined traits. impl UseReadController for ApplicationContext { fn read_controller(&self) -> &ReadController { &self.read_controller } } impl UseWriteController for ApplicationContext { fn write_controller(&self) -> &WriteController { &self.write_controller } } impl UseDatabaseConnection for ApplicationContext { type T = PostgresDatabaseConnection; fn database_connection(&self) -> &Self::T { &self.postgres_database_connection } } By implementing these traits, ApplicationContext becomes capable of providing dependencies to components. 4.3 Controller Context Implementation impl ControllerContext for ApplicationContext {} Since ApplicationContext already implements the required traits, it automatically satisfies ControllerContext. 5. Running the Application Finally we run the application. pub fn run() { let ctx = ApplicationContext::default(); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } Key characteristics of this approach: No dyn traitsNo Arc or RcNo runtime dependency container All wiring is resolved at compile time through generics and monomorphization. Multi-Threading An attentive reader may ask: Will this approach work in multi-threaded environments? In Rust, thread safety is typically ensured using the Send and Sync traits. These traits are automatically implemented by the compiler if all fields of a struct are also Send + Sync. We can verify thread safety with a compile-time assertion: const _: () = { const fn assert_send_sync<T: Send + Sync>() {} assert_send_sync::<ApplicationContext>(); }; If this compiles, the entire application context can safely be shared between threads. In real systems, some components (such as database connections) may not be inherently thread-safe. In such cases, a connection pool or synchronization mechanisms such as Mutex are required. This limitation is not related to the dependency injection approach itself, but rather to shared resource management in concurrent systems. What the Compiler Actually Generates If we inspect the compiled output with: cargo asm rust_di_example::main ... 26 │ lea rbx, [rsp, +, 32] 27 │ mov rdx, rbx 28 │ call qword, ptr, [rip, +, _ZN5alloc3fmt6format12format_inner17he42ed4cf3cdc276bE@GOTPCREL] 29 │ movups xmm0, xmmword, ptr, [rsp] 30 │ mov rax, qword, ptr, [rsp, +, 16] 31 │ movups xmmword, ptr, [rsp, +, 48], xmm0 32 │ mov qword, ptr, [rsp, +, 64], rax 33 │ mov qword, ptr, [rsp, +, 32], r14 34 │ mov qword, ptr, [rsp, +, 40], 18 35 │ mov qword, ptr, [rsp], rbx 36 │ mov qword, ptr, [rsp, +, 8], r13 37 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.26] 38 │ mov rsi, rsp 39 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] 40 │ lea rax, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.27] 41 │ mov qword, ptr, [rsp, +, 32], rax 42 │ mov qword, ptr, [rsp, +, 40], 19 43 │ mov qword, ptr, [rsp], rbx 44 │ mov qword, ptr, [rsp, +, 8], r13 45 │ lea r14, [rsp, +, 48] 46 │ mov qword, ptr, [rsp, +, 16], r14 47 │ lea r15, [rip, +, _ZN60_$LT$alloc..string..String$u20$as$u20$core..fmt..Display$GT$3fmt17h9d11f1d81b352ac8E] 48 │ mov qword, ptr, [rsp, +, 24], r15 49 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.7] 50 │ mov rsi, rsp 51 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] 52 │ lea rax, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.28] 53 │ mov qword, ptr, [rsp, +, 32], rax 54 │ mov qword, ptr, [rsp, +, 40], 21 55 │ mov qword, ptr, [rsp], rbx 56 │ mov qword, ptr, [rsp, +, 8], r13 57 │ mov qword, ptr, [rsp, +, 16], r14 58 │ mov qword, ptr, [rsp, +, 24], r15 59 │ lea rdi, [rip, +, .Lanon.63c02f0152e6743e61fdeaf76f1d4051.8] 60 │ mov rsi, rsp 61 │ call qword, ptr, [rip, +, _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL] ... We see extremely flat assembly code with series of invocation to _ZN3std2io5stdio6_print17hba8f5eda1e4e495eE@GOTPCREL that is just printing subroutine in rust runtime. There are no runtime dependency resolution mechanisms, no dynamic dispatch, and no container logic. The generated code mostly contains calls to standard library functions such as printing. This demonstrates that the abstractions introduced here do not introduce runtime overhead. Why Use-Traits Matter At first glance, the use-trait might look like unnecessary indirection. Why not simply pass ApplicationContext directly to every component? The reason is crate-level decoupling. Enterprise applications often grow into multiple crates. Controllers, database access layers, messaging integrations, and domain logic are very often implemented as reusable libraries. For example, a Spring Boot actuator–style module may contain all layers inside the DB, provide REST API endpoints, and integrate with a monitoring aggregator service — it acts as a standalone sub-program. However, if a component directly depends on ApplicationContext, it becomes tied to the executable crate that defines it. That creates an architectural problem: Libraries would depend on the application crateThe application crate would depend on the libraries This circular dependency makes reuse impossible. Use-trait solve this by defining capability-based interfaces. Instead of depending on the application context, components depend only on the capabilities they require. Example: trait UseDatabaseConnection { type T: DatabaseConnection; fn database_connection(&self) -> &Self::T; } A controller does not know anything about the application structure. It simply requires that the context provides access to a database connection. impl ReadController { fn do_something<C: ControllerContext>(&self, ctx: &C, argument: &str) { ctx.database_connection() .read_query(format!("SELECT * FROM table WHERE id = '{}'", argument).as_str()); } } Because of this design: ReadController can live in its own crateThe crate only exports traits describing the capabilities it needsAny application can use the controller by implementing those traits The application context becomes an adapter, wiring together independent components. Application ├── implements UseDatabaseConnection ├── implements UseReadController └── implements UseWriteController This pattern enables a powerful architectural property: Components become fully reusable libraries, while the application remains responsible only for wiring them together. In other words, use-traits allow dependency injection to cross crate boundaries while preserving Rust’s compile-time guarantees. Without this indirection, the system collapses into a monolithic application context that cannot be decomposed into reusable modules. Limitations of This Approach Although this example demonstrates many useful properties, it is not yet a complete dependency injection system. The main limitation is that ApplicationContext still has too much knowledge about component internals. In real DI frameworks, modules often contain many components, initialization logic, and internal dependencies. For example, consider a Spring Boot module such as Spring Data. When you add the dependency to your project, it automatically provides: Database driver integrationConnection poolingRepository interfacesTransaction managementEntity scanningMetrics integrationHealth check integration All of this functionality is assembled automatically by the DI framework. From the application developer’s perspective, only minimal configuration is required. Real dependency injection modules therefore consist of entire subgraphs of components, not just individual services. In our example we intentionally introduced two controllers to demonstrate that even a simple module may contain multiple cooperating components. A complete dependency injection framework must also manage: Module compositionInitialization lifecycleDependency resolutionOptional componentsMultiple implementations This is where the real challenge begins. Rust With Dependency Injection To implement dependency injection in Rust, we will build iteratively. We start from the previous “no DI” approach and gradually close the gap toward a complete DI system. The good news is that we already have use-traits, and our components are decoupled. We can extract certain code into reusable modules. What’s missing for a true dependency injection system: ApplicationContext still has too much knowledge about the components it uses.Some wiring and initialization steps are still manual. Our goal is to move the wiring into DI modules, giving each component full control over how it is connected. Because we are still targeting compile-time injection, we cannot rely on runtime reflection (like Java DI frameworks do). Instead, we will push this logic into Rust macros, allowing compile-time wiring while preserving zero-cost abstractions. 1. Registering Components in ApplicationContext In traditional DI, the application knows which modules it depends on (like Spring Data). But modules themselves should control which components they export. In our previous example, ApplicationContext was a struct, and registering a component meant adding a field manually. This ties the application to module internals. We need a way to add fields to ApplicationContext automatically, without putting module-specific code into the executable. We can achieve this using the combine-structs crate, which provides macros to embed multiple structs into one. Each module defines an embeddable struct as a context extension. When imported, ApplicationContext automatically merges all fields from these extensions. 1.1 Context Extension for PostgreSQL #[allow(dead_code)] #[derive(Fields)] struct PostgresDatabaseContextExtension { postgres_database_connection: PostgresDatabaseConnection, } The Fields derive macro allows this struct to be merged into ApplicationContext. 1.2 Context Extension for Controllers #[allow(dead_code)] #[derive(Fields)] struct ControllerContextExtension { read_controller: ReadController, write_controller: WriteController, } The controller module exports two controllers. More components can be added without touching the main executable. 1.3 Embedding Context Extensions #[combine_fields(PostgresDatabaseContextExtension, ControllerContextExtension)] #[derive(Default)] struct ApplicationContext {} The combine_fields macro merges all fields from the context extensions. ApplicationContext now has all components automatically wired. 2. Providing Use-Traits Previously, wiring was done via use-traits. Now that ApplicationContext doesn’t know which components exist, modules must export use-trait implementations via macros. 2.1 Macro for Database Connectivity macro_rules! inject_postgres_impl { () => { impl UseDatabaseConnection for ApplicationContext { type T = PostgresDatabaseConnection; fn database_connection(&self) -> &Self::T { &self.postgres_database_connection } } }; } 2.2 Macro for Controllers macro_rules! inject_controller_impl { () => { impl UseReadController for ApplicationContext { fn read_controller(&self) -> &ReadController { &self.read_controller } } impl UseWriteController for ApplicationContext { fn write_controller(&self) -> &WriteController { &self.write_controller } } impl ControllerContext for ApplicationContext {} }; } 2.3 Injecting Components #[combine_fields(PostgresDatabaseContextExtension, ControllerContextExtension)] #[derive(Default)] struct ApplicationContext {} inject_postgres_impl!(); inject_controller_impl!(); The executable only calls these macros. Components remain isolated from the main application, and the wiring happens automatically. 3. Intermediate Conclusion At this stage: No component code has been changed.Modules can add or remove components freely.Components are decoupled from each other and from the container.Wiring happens automatically through macros and use-traits. This gives us a bare-minimum dependency injection system: application components are decoupled, wiring is automatic, and no single component needs full knowledge of the application. 4. Limitations Even though we now have a working DI mechanism, it isn’t fully production-ready: Initialization: Components may require setup before wiring.Lifecycle Management: Controlling initialization order, cleanup, or optional components can be challenging. Next, we will explore a Rust DI framework capable of automating component initialization and lifecycle management, moving closer to a complete solution. Dependency Injection and Initialization Cycle in Rust So far, we have built a dependency injection (DI) container where all components are stored as fields in ApplicationContext. The next challenge is initializing these components. Press enter or click to view image in full size The goal is to: Enumerate the fields of ApplicationContext.Identify which fields require initialization.Call an initialization method for each such component. Since we want everything to happen at compile time, we need a macro to generate a Rust method that calls init() on every tagged component without runtime loops or collections. I could not find an existing macro for this, so I implemented one myself. If you want the details, check the implementation here: di_macro/src/lib.rs. We will focus on how to use this macro, not how it works internally. Macro Example: Enumerating Tagged Fields Full example code: struct_enumerator.rs 1. Define a Struct with Tagged Fields #[allow(dead_code)] #[derive(Debug, FieldEnumerator, Default)] pub struct MyStruct { #[tag(init_listener)] field_1: i32, #[tag(init_listener)] #[tag(start_listener)] field_1_2: i32, field_2: i32, #[tag(start_listener)] field_3: i32, } FieldEnumerator is our custom derive macro.Fields can have one or more tags (init_listener, start_listener). 2. Define a Callback Macro macro_rules! my_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { println!( "struct = {}, field = {}, type = {}", stringify!($struct_name), stringify!($field_name), stringify!($listener_type), ) }; } For every tagged field, the callback macro is called at compile time.Arguments passed: struct_name, field_name, and listener_type. 3. Invoke the Field Enumerator pub fn run() { let my_struct = MyStruct::default(); println!("my_struct = {:?}", my_struct); enumerate_tags_MyStruct_init_listener!(my_callback); enumerate_tags_MyStruct_start_listener!(my_callback); } enumerate_tags_MyStruct_init_listener! and enumerate_tags_MyStruct_start_listener! are generated automatically by the FieldEnumerator macro.The macro expands into a flat sequence of println!() calls. Macro Example Output: // enumerate_tags_MyStruct_init_listener!(my_callback); // my_callback!(MyStruct, field_1, init_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1", "init_listener") // my_callback!(MyStruct, field_1_2, init_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1_2", "init_listener") //enumerate_tags_MyStruct_start_listener!(my_callback) // my_callback!(MyStruct, field_1_2, start_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_1_2", "start_listener") // my_callback!(MyStruct, field_3, start_listener) println!("struct = {}, field = {}, type = {}", "MyStruct", "field_3", "start_listener") Notice: No vectors, arrays, loops, or runtime collections — everything happens at compile time. Rust Dependency Injection with Initialization We can now use the same macro to enumerate all fields in ApplicationContext and initialize them. Code reference: di_init.rs We introduce a Configuration component to demonstrate how initialization can depend on runtime data. 1. Configuration Module #[derive(Default)] struct Configuration { run_arguments: &'static str, } #[allow(dead_code)] #[derive(Fields, Default)] struct ConfigurationContextExtension { configuration: Configuration, } trait UseConfiguration { fn configuration(&self) -> &Configuration; fn configuration_mut(&mut self) -> &mut Configuration; } macro_rules! inject_configuration_impl { () => { impl UseConfiguration for ApplicationContext { fn configuration(&self) -> &Configuration { &self.configuration } fn configuration_mut(&mut self) -> &mut Configuration { &mut self.configuration } } }; } Steps: Define the component struct (Configuration).Define a context extension for ApplicationContext.Define a use-trait (UseConfiguration) for wiring.Provide a macro to implement the trait on ApplicationContext. Note: Configuration is no longer zero-sized—it contains runtime data (run_arguments). 2. Database Connection Initialization 2.1 Update PostgresDatabaseConnection #[derive(Default)] struct PostgresDatabaseConnection { connection_string: String, } Now contains runtime data.Initialization depends on configuration. 2.2 Tag Component for Initialization #[allow(dead_code)] #[derive(Fields, ContextExtension)] struct PostgresDatabaseContextExtension { #[tag(init_listener)] postgres_database_connection: PostgresDatabaseConnection, } init_listener signals that the component requires initialization. 2.3 Define Initializable Trait trait Initializable<C> { fn init(ctx: &mut C); } Components implementing this trait can be initialized automatically. 2.4 Implement Initialization impl<C: UseConfiguration + UsePostgresDatabaseConnection> Initializable<C> for PostgresDatabaseConnection { fn init(ctx: &mut C) { println!("Init sequence = {}", ctx.configuration().run_arguments); ctx.postgres_database_connection_mut().connection_string = format!("Postgres DB on {}", ctx.configuration().run_arguments); } } Accesses ApplicationContext mutably for initialization of any of component. 2.5 Prepare ApplicationContext #[combine_fields( ConfigurationContextExtension, PostgresDatabaseContextExtension, ControllerContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationContext {} inject_postgres_impl!(); inject_controller_impl!(); inject_configuration_impl!(); Added FieldEnumerator for tag enumeration.Configuration module bindings included. 2.6 Initialization Sequence impl ApplicationContext { fn init(&mut self) { fn call_init<T: Initializable<ApplicationContext>, F: Fn(ApplicationContext) -> T>( ctx: &mut ApplicationContext, _closure: F, ) { T::init(ctx); } macro_rules! init_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { call_init(self, |x| x.$field_name); }; } enumerate_tags_ApplicationContext_init_listener!(init_callback); } } How it works: call_initfunction This helper function takes a generic type T that implements Initializable<ApplicationContext>.It also takes a closure _closure of type Fn(ApplicationContext) -> T.The trick here: the Rust compiler monomorphizes the closure to the actual type of the field passed in, so T::init(ctx) is called with the concrete type.init_callback!macro The macro expands for each field tagged with init_listener.It calls call_init with the correct field from self, ensuring the proper Initializable implementation is invoked.enumerate_tags_ApplicationContext_init_listener!macro This macro iterates over all fields in ApplicationContext that are marked with #[init_listener].For each field, it invokes init_callback!, which triggers Initializable::init for that specific component. Key rust trick: By using the Fn trait and generics in call_init, the compiler resolves the actual type of the field at compile time. This avoids any runtime type checks and ensures zero-cost initialization while keeping strong type safety. 2.7 Running the Application pub fn run() { let mut ctx = ApplicationContext::default(); ctx.configuration_mut().run_arguments = "DB_URL=127.0.0.1:5555"; ctx.init(); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } Sample Output: Init sequence = DB_URL=127.0.0.1:5555 Reading from Postgres DB on DB_URL=127.0.0.1:5555: SELECT * FROM table WHERE id = 'argument' Writing into Postgres DB on DB_URL=127.0.0.1:5555: UPDATE table SET value = 'new' WHERE id = 'argument' run_arguments successfully propagated into runtime data. Performance Considerations In this demo, some structs now hold runtime data — but this is intentional. It’s added to demonstrate initialization, just like in real applications where components manage runtime state. The wiring mechanism itself remains zero-cost: All bindings are resolved at compile time through monomorphization. Even with the initialization sequence broadcasting multiple init calls, the compiler generates a flat sequence of calls: no loops, no runtime collections, no dynamic dispatch — everything happens at compile time, efficiently. Limitations This approach is now mature and production-ready for wiring, decoupling, and initialization.Next steps can explore advanced topics, such as polymorphism and more complex runtime behaviors. Dependency Injection and Polymorphism This is the final example of the article and introduces what I would consider an advanced topic for the core engine of any dependency injection framework: polymorphism. Press enter or click to view image in full size Many DI frameworks handle basic dependency wiring well. For example, Java Spring Boot provides a very mature implementation. However, in many other DI implementations, one important capability is often missing — the ability to handle multiple implementations of the same abstraction in a flexible and compile-time-safe way. Let’s extend our example with a new requirement. New Requirement Our application should support multiple message brokers, for example: KafkaRabbitMQ After writing data to the database, the controller should publish a message to one or more brokers. However: The component does not know which brokers existThe container may contain multiple brokersThe DI framework must maintain this one-to-many relationship One component should be able to call many broker implementations without knowing which ones exist. To make things even more interesting, we introduce the concept of profiles. Each profile represents a different configuration of the application context. Example: Profile1 PostgreSQL databaseKafka brokerRabbitMQ broker Profile2 Oracle databaseRabbitMQ broker only See the complete example. Injection Macros and Profiles First, we slightly modify our injection macros so they accept the application context type as an argument. macro_rules! inject_configuration_impl { ($ctx:ident) => { impl UseConfiguration for $ctx { fn configuration(&self) -> &Configuration { &self.configuration } fn configuration_mut(&mut self) -> &mut Configuration { &mut self.configuration } } }; } This change is necessary because the DI module does not know which profile will be used. Each executable can choose a different application context profile, and the macros must work with whichever profile is selected. Oracle Database Component Now we introduce a new database implementation. #[allow(dead_code)] #[derive(Fields, ContextExtension)] struct OracleDatabaseContextExtension {} And the injection macro: macro_rules! inject_oracle_impl { ($ctx: ident) => { impl DatabaseConnection for $ctx { fn read_query(&self, query: &str) { println!("Reading from Oracle DB: {}", query) } fn write_query(&self, query: &str) { println!("Writing into Oracle DB {}", query) } } impl UseDatabaseConnection for $ctx { type T = $ctx; fn database_connection(&self) -> &Self::T { self } } }; } Here we apply a small trick. Instead of defining a separate struct for the database connection, we implement the trait directly on the application context. This approach avoids additional boilerplate and works well when we know there will only be one database implementation per profile. Defining Message Brokers Now we define the abstraction for message brokers. Broker Interface trait BrokerSender { fn send_to_broker(&self, value: &str); } RabbitMQ Broker #[allow(dead_code)] #[derive(Default, Fields, ContextExtension)] struct RabbitMqContextExtension { #[tag(broker)] rabbit_mq: RabbitMq, } #[derive(Default)] struct RabbitMq; impl BrokerSender for RabbitMq { fn send_to_broker(&self, value: &str) { println!("{} sent to RabbitMq", value); } } Notice the important detail: #[tag(broker)] This tag allows the DI framework to enumerate all brokers automatically using the same mechanism we previously used for initialization. Kafka Broker Kafka is implemented in exactly the same way. #[allow(dead_code)] #[derive(Default, Fields, ContextExtension)] struct KafkaContextExtension { #[tag(broker)] kafka: Kafka, } #[derive(Default)] struct Kafka; impl BrokerSender for Kafka { fn send_to_broker(&self, value: &str) { println!("{} sent to Kafka", value); } } Publisher — Compile-Time Polymorphism Now comes the most interesting part. We define a Publisher component that sends messages to all available brokers. trait Publisher { fn publish(&self, value: &str); } Injection macro: macro_rules! inject_publisher_impl { ($ctx:ident) => { impl Publisher for $ctx { fn publish(&self, value: &str) { macro_rules! broker_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { self.$field_name.send_to_broker(value); }; } enumerate_tags!($ctx, broker, broker_callback); } } impl UsePublisher for $ctx { type T = $ctx; fn publisher(&self) -> &Self::T { self } } }; } The key idea: the publisher does not know which brokers exist. Instead, the FieldEnumerator macro generates code that calls send_to_broker for each tagged broker. This gives us: One-to-many relationshipCompile-time wiringNo dynamic dispatchNo runtime overhead Helper Macro for Tag Enumeration macro_rules! enumerate_tags { ($ctx:ident, $tag:ident, $callback:ident) => { paste! { [<enumerate_tags_ $ctx _ $tag >]!($callback) } }; } This macro simply dispatches to the procedural macro generated earlier. Application Profiles Now we define two different application contexts. Profile 1 #[combine_fields( ConfigurationContextExtension, PostgresDatabaseContextExtension, ControllerContextExtension, PublisherExtension, RabbitMqContextExtension, KafkaContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationProfile1 {} Profile1 includes: PostgreSQLRabbitMQKafka Profile 2 #[combine_fields( ConfigurationContextExtension, OracleDatabaseContextExtension, ControllerContextExtension, PublisherExtension, RabbitMqContextExtension )] #[derive(Default, FieldEnumerator)] struct ApplicationProfile2 {} Profile2 includes: Oracle databaseRabbitMQ brokerno Kafka Initialization Macro for Context We move the previously used initialization logic into a reusable macro: macro_rules! application_context { ($ctx: ident) => { const _: () = { const fn assert_send_sync<T: Send + Sync>() {} assert_send_sync::<$ctx>(); }; impl Initializable<$ctx> for $ctx { fn init(ctx: &mut $ctx) { fn call_init<T: Initializable<$ctx>, F: Fn($ctx) -> T>( ctx: &mut $ctx, _closure: F, ) { T::init(ctx); } macro_rules! init_callback { ($struct_name:ident, $field_name:ident, $listener_type:ident) => { call_init(ctx, |x| x.$field_name); }; } enumerate_tags!($ctx, init_listener, init_callback); } } }; } Wiring Profiles Profile1 application_context!(ApplicationProfile1); inject_postgres_impl!(ApplicationProfile1); inject_controller_impl!(ApplicationProfile1); inject_configuration_impl!(ApplicationProfile1); inject_publisher_impl!(ApplicationProfile1); inject_rabbit_mq_impl!(ApplicationProfile1); inject_kafka_impl!(ApplicationProfile1); Profile2 application_context!(ApplicationProfile2); inject_oracle_impl!(ApplicationProfile2); inject_controller_impl!(ApplicationProfile2); inject_configuration_impl!(ApplicationProfile2); inject_publisher_impl!(ApplicationProfile2); inject_rabbit_mq_impl!(ApplicationProfile2); Running the Example fn do_run<T: Initializable<T> + Default + UseConfiguration + ControllerContext>() { let mut ctx = T::default(); ctx.configuration_mut().run_arguments = "DB_URL=127.0.0.1:5555"; T::init(&mut ctx); ctx.read_controller().do_something(&ctx, "argument"); ctx.write_controller().do_something(&ctx, "argument"); } pub fn run() { println!("Running Profile1"); do_run::<ApplicationProfile1>(); println!(); println!("Running Profile2"); do_run::<ApplicationProfile2>(); } Example Output Running Profile1 Configuration = DB_URL=127.0.0.1:5555 PostgresDB connection init sequence = DB_URL=127.0.0.1:5555 Reading from Postgres DB... Writing into Postgres DB... WriteController 'argument' sent to RabbitMq WriteController 'argument' sent to Kafka Running Profile2 Configuration = DB_URL=127.0.0.1:5555 Reading from Oracle DB... Writing into Oracle DB... WriteController 'argument' sent to RabbitMq Final Result With this approach we achieved: Compile-time polymorphismOne-to-many dependency injectionProfile-based application configurationNo dynamic dispatchNo runtime containerFully monomorphized wiring Everything is resolved at compile time while still supporting flexible application configurations. Conclusion: Can Rust Have Zero-Cost Dependency Injection? Throughout this article we explored whether Dependency Injection can exist in Rust without introducing runtime overhead. Traditional DI frameworks in languages such as Java rely heavily on reflection, runtime containers, dynamic dispatch, and runtime graph construction. These features make frameworks like Spring Boot extremely flexible, but they also introduce runtime complexity and performance costs. Rust approaches the problem differently. Instead of relying on runtime containers, the examples in this article demonstrate how compile-time composition can be used to build a dependency injection system. Using traits, generics, procedural macros, and compile-time code generation, we can construct an application context where: Component wiring happens at compile timeDependencies are resolved through traits and genericsInitialization logic can be generated staticallyPolymorphism can be implemented without dynamic dispatch Because Rust performs monomorphization during compilation, every dependency binding is resolved into concrete function calls. This means the final binary contains no reflection, no dynamic lookup tables, and no runtime dependency container. In other words, dependency injection becomes a compile-time architectural pattern rather than a runtime framework. We also demonstrated several important features typically expected from mature DI systems: Modular component composition through context extensionsControlled initialization sequencesOne-to-many polymorphism for components such as brokersConfigurable application profiles And all of this without introducing runtime cost or dynamic dispatch The result is a system where flexibility and performance are not in conflict. Rust’s type system and macro system allow us to design architectures that remain fully decoupled, while still producing simple, predictable, zero-cost binaries. This raises an interesting conclusion. Rust may never have a DI framework that looks like Spring Boot — and it probably shouldn’t. But Rust does allow dependency injection to exist in a different form, one that embraces the language’s philosophy: compile-time guarantees, explicit composition, and zero-cost abstractions. Future Directions The examples in this article intentionally keep the framework small in order to focus on the core ideas. However, a production-ready system would likely evolve further. For example, initialization often requires explicit ordering between components, where some services must be initialized before others. The current example also contains a fair amount of boilerplate, which could be significantly reduced with a more advanced procedural macro design. Heavier use of derive and attribute macros could also improve IDE code completion and developer ergonomics while keeping the system fully type-safe. Beyond the core container mechanics, several practical features naturally follow from this model: improved testing support, built-in mechanisms for mocking and stubbing components, and the ability to override components in derived profiles — a common requirement when building test environments or specialized deployments. Finally, dependency injection frameworks rarely exist in isolation. Systems such as Spring Boot succeeded not only because of their DI container, but because they provided a standard foundation for an ecosystem of reusable modules. A similar approach in Rust could allow libraries to integrate around a shared compile-time DI model, enabling a broader ecosystem of interoperable components while preserving Rust’s philosophy of explicit composition and zero-cost abstractions.

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

Picture this: two features are being developed in parallel. One has already been tested in lower environments, but is still awaiting business approvalThe other is fully validated and ready to go live Naturally, you want to release the second feature to production. But you can’t, because your deployment model forces you to release everything together. If you’ve worked with Azure Data Factory (ADF), this situation probably sounds familiar. Azure Data Factory (ADF) is a cloud-based data integration service from Microsoft that helps you build and orchestrate data pipelines across systems. It works extremely well for managing data workflows — but when it comes to deployments at scale, things get tricky. As our ADF usage grew across multiple teams and environments, we started running into a recurring problem: We had control over development — but very little control over what actually got deployedA simple pipeline fix could unintentionally introduce unrelated changesParallel feature development became harder to manageProduction releases became riskier than they needed to be That’s when we realized: The issue wasn’t ADF itself — it was the deployment model we were relying on. The issue wasn’t ADF itself — it was the deployment model we were relying on. This article walks through how we addressed that challenge by implementing a selective deployment pattern, allowing us to promote only intended changes without impacting everything else. The Real Problem: Parallel Feature Releases in ADF Before diving into the solution, let’s look at a scenario that frequently occurs in real-world teams. What This Diagram Represents This diagram shows two features progressing across environments: Feature 100 Developed earlier, successfully deployed to Dev and TestCurrently in UAT (User Acceptance Testing)Still awaiting business approval before production Feature 200 Developed later, successfully completed across Dev → Test → UATFully validated and ready for production Expected Behavior At this stage, the expectation is straightforward: “Let’s release Feature 200 to production.” Feature 100 is still under testing, so it should remain in UAT. What Actually Happens in ADF Azure Data Factory follows a full-state deployment model. That means when you deploy, you are not deploying a feature; you are deploying the entire factory state. So when you attempt to release Feature 200: Feature 100 gets included automaticallyYou cannot isolate Feature 200You lose control over what reaches production Why This Becomes a Real Problem This isn’t an edge case; it becomes a recurring pattern in larger environments. You’ll encounter this when: Multiple teams are working in parallelFeatures move at different speedsUAT cycles varyProduction fixes need to be released quickly It becomes even more complex when: Existing production pipelines are modifiedPartial updates are requiredDependencies overlap across features The Core Limitation: ADF promotes state, not intent. It does not differentiate between what is ready for production and what is still under testing. Why We Had to Rethink Deployment This limitation introduced real risks: Accidental promotion of incomplete featuresDelayed production releasesIncreased coordination overheadHigher chances of breaking stable pipelines We needed a way to: Promote only Feature 200Keep Feature 100 in UATAvoid impacting unrelated artifactsReduce production risk Architecture Overview To address this challenge, we introduced a selective packaging layer between build and deployment. Flow Feature Branch → PR → Validate → Selective Packaging → ARM Export → Incremental Deploy → Trigger Control Key Idea: Instead of exporting ARM templates from the full ADF repository, we export from a filtered staging folder containing only the required artifacts. Understanding Default ADF Deployment Behavior Before implementing selective deployment, it’s important to understand how Azure Data Factory works by default. ADF follows a full-state deployment model. How Default ADF Deployment Works When you use ADF with Git integration: Developers work in a collaboration branch (typically main)Changes are committed and merged via pull requestsADF provides a Publish button in the UI When you click Publish, ADF generates ARM templates representing the entire factory state. These templates are stored in the adf_publish branch: In modern setups, instead of clicking Publish manually, teams often use @microsoft/azure-data-factory-utilities (npm-based export). This allows pipelines to validate ADF resources and export ARM templates programmatically. YAML - name: Validate ADF resources run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.SOURCE_FACTORY_NAME }" npm run build validate "${{ github.workspace }" "$FACTORY_ID" YAML - name: Export ARM templates (CI publish) run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.DEV_FACTORY_NAME }" npm run build export "${{ github.workspace }" "$FACTORY_ID" "${{ env.ARM_OUTPUT_DIR }" Whether you click Publish manually or use npm export in CI/CD, the outcome is the same: Full factory deploymentNo control over individual featuresAll changes get bundled together Selective Deployment Layer (Core Design) We can address this requirement and the associated challenges by introducing a workflow driven by a manifest to define the deployment scope, and a program to identify all necessary ADF dependencies for each manifest file. As a developer, I can now control which release is promoted to production, without worrying about releasing any other features that are not ready. The manifest controls which pipelines to deploy and which optional categories to include. Below is an example of a manifest file JSON { "pipelines": ["pl_ingest_population_selective"], "includeTriggers": false, "includeIntegrationRuntimes": false, "includeAllGlobalParameters": true, "includeLinkedServices": true, "validateLinkedServicesExist": true, "includeManagedVirtualNetwork": false, "includeManagedPrivateEndpoints": false } Workflow Explanation Let's understand the crux of the selective deployment workflow now. I am working in the release branch on my feature branch directly in ADF Studio. Since ADF Studio is integrated with Git, my development changes will be saved to my branch. Here are the steps I can take to promote my change to a higher environment. 1) Validation of ADF on PR validation This is an early validation step and a guardrail: if the PR fails, it's because objects are invalid and misaligned. This is equivalent to the "validation all" button in the ADF ui, here is this workflow Trigger: Pull requests targeting the branch selective_deployment. Purpose: Validate that the ADF JSON in the PR is valid in the context of the target factory. Main steps: CheckoutSet up Node.js 20npm installAzure login using OIDC (azure/login@v2)Validate with ADF Utilities: YAML FACTORY_ID="/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${DEV_FACTORY_NAME}" npm run build validate "$GITHUB_WORKSPACE" "$FACTORY_ID" 2) Release build + selective deploy to DEV adf-release-build-selective-deploy.yml Triggers: Push to selective_deploymentManual run (workflow_dispatch) with optional manifest inputDefault: deploy/manifests/release.json This workflow has two jobs: Job A: adf-build (staging + export + sanitize + artifacts) Checkout (full history)Azure login using OIDCSet up Node.js 20Install build dependencies inside build/ (npm install in build)Stage selective subset python scripts/select_adf_subset.py <manifest>, a code snippet below for the complete script, refer to the GitHub repository link given Python import json import re import shutil import sys from pathlib import Path from typing import Dict, Set, Tuple, List from collections import defaultdict # Your repo layout has pipeline/, dataset/, linkedService/ at ROOT. REPO_ROOT = Path(".") STAGE_ROOT = Path("build/adf_subset") RESOURCE_DIRS = { "pipeline": REPO_ROOT / "pipeline", "dataset": REPO_ROOT / "dataset", "linkedService": REPO_ROOT / "linkedService", "dataflow": REPO_ROOT / "dataflow", "trigger": REPO_ROOT / "trigger", "integrationRuntime": REPO_ROOT / "integrationRuntime", "credential": REPO_ROOT / "credential", "managedVirtualNetwork": REPO_ROOT / "managedVirtualNetwork", } # Copy these if present so ADF utilities behave the same on staged subset. ROOT_FILES_TO_COPY = [ "publish_config.json", "arm-template-parameters-definition.json", "arm_template_parameters-definition.json", "package.json", "package-lock.json", ] Produces: build/adf_subset/ (staged tree)build/adf_subset_report.json (dependency report)Refer to logs below (showing output of stage selective subset and debug to view output generated after select_adf_subset.py )Export ARM templates from the staged subset via ADF Utilities: npm --prefix build run build -- export "adf_subset" "$FACTORY_ID" "ArmTemplate"Produces: build/ArmTemplate/ARMTemplateForFactory.jsonbuild/ArmTemplate/ARMTemplateParametersForFactory.jsonStrip infra-owned resources scripts/strip_arm_resources.py to produce a safe template: build/ArmTemplate/ARMTemplateForFactory.safe.json⚠️ Note on Infrastructure Components (Refer to the “Future Work & Next Steps” section for follow-up topics in this series) The step above intentionally strips infrastructure-dependent components from the generated subset to avoid overwriting existing shared resources such as linked services. This implementation focuses on developer-owned artifacts (pipelines, datasets, and triggers) and assumes that infrastructure components — such as Integration Runtimes, managed private endpoints, and linked services — are pre-provisioned and managed outside of this deployment workflow.Upload artifacts: ARM templates (adf-arm)metadata (adf-release-meta)subset report (adf-subset-report) Job B: deploy_dev (deploy safe template) Download ARM artifactAzure login using OIDCEnsure az Data Factory extension is installedValidate JSON files exist/parseDeploy via azure/arm-deploy@v2(Incremental) to DEV RG/factory: Template: ARMTemplateForFactory.safe.jsonParameters: ARMTemplateParametersForFactory.json + factoryName=<DEV_FACTORY_NAME> Lesson Learned Setting up selective deployment in ADF was more than a technical task. It made us rethink our approach to deployments, ownership, and CI/CD design. Here are the main things we learned: 1. The Problem Is Not Tooling; It’s Deployment Granularity At first, we thought the limitation came from the tools we used, like UI publish or npm export. However, both methods yielded the same result: full factory templates. The real problem was that we couldn’t control the scope of deployments, not how the templates were made. 2. Dependency Awareness Is Critical Selective deployment only works when every dependency is found and included. We learned that: Pipelines often reference multiple datasets and linked services. Missing even one dependency results in deployment failure You must automate dependency discovery. 3. “Incremental” Is Often Misunderstood Incremental deployment is important, but it doesn’t work like a patch. It reapplies the full configuration for all included resources. This means: Your generated templates need to be complete for all the artifacts you include. If you use partial definitions, deployments can fail. 4. Separation of Concerns Is Key Not all ADF artifacts are the same. We began to separate them into different groups: Application-owned artifacts: pipelines, datasets, triggers Infrastructure-owned artifacts: linked service, managed virtual networks, managed private endpoints, and integration-runtime, among others. This separation proved crucial for safe, scalable deployments. 5. Selective Deployment Adds Complexity, But It’s Worth It It’s true that implementing this approach brings in additional scripts, manifest management, and CI/CD complexity. But in exchange, we gained precise control over releases, reduced production risk, and faster hotfix deployments. Future Work and Next Steps While selective deployment solved a major gap in ADF CI/CD, it also opened up new areas for improvement and standardization. 1. Defining Infrastructure vs Application Ownership One of the biggest follow-up areas is clearly defining ownership boundaries. In our experience: Application teams should own pipelines, datasets, and triggers Platform or infrastructure teams should own linked services, managed virtual networks, and managed private endpoints, among other things. Future work can focus on: Enforcing this separation in CI/CD. Preventing accidental deployment of infrastructure components Integrating Terraform or platform pipelines for infrastructure provisioning 2. Governance Around Linked Services Linked services are often shared across multiple pipelines and teams. Future improvements include: Centralizing linked service management Using Key Vault and Managed Identity consistently Preventing direct modifications through application pipelines

The 3:00 AM Incident That Changed Everything It was a Tuesday morning when the alerts started firing. Our recommendation engine, the one that drives 30% of our revenue, had tanked. Accuracy dropped from 94% to 58% overnight. The data science team immediately blamed the model. They started tweaking hyperparameters, re-training on new data, and running diagnostics. Nothing worked. I got pulled into the war room at 3:00 AM. The first thing I asked wasn't "What's wrong with the model?" It was "What changed in the data pipeline?" Turns out, everything. A vendor had pushed a schema change upstream. A field that used to be required became optional. Null values started flowing through our pipeline. Our feature engineering code didn't handle nulls gracefully; it just propagated them downstream. By the time the data reached the model, 40% of our feature vectors were corrupted. The model wasn't broken. The data was. We spent six hours manually rolling back the schema change, re-running the pipeline, and restoring service. The incident report was brutal: "Lack of data validation caught a breaking change too late." That's when I realized we needed observability in our data pipeline, not just in our models. The Problem: Data Quality is Invisible Until It Breaks Here's the uncomfortable truth about data pipelines: they fail silently. Your ETL job completes successfully. Your Spark cluster finishes transformations. Your data warehouse loads without errors. Everything looks green in the monitoring dashboard. But the data itself? Garbage in, garbage out. There are three categories of failures that break AI models in production: Missing Values: A source system stops populating a field. Your pipeline doesn't validate it. The model gets NaN values it never saw during training. Predictions become random noise. Schema Changes: An upstream team adds a new column, renames an existing one, or changes data types. Your pipeline doesn't expect these changes. Either it crashes, or worse, it silently maps data to the wrong columns. Distribution Shifts: The statistical properties of your data change. A field that was always between 0 and 100 suddenly has values of 50,000. Your model's scaling assumptions break. Predictions become nonsensical. None of these show up in traditional infrastructure monitoring. Your CPU is fine. Memory is fine. Network is fine. But your data is on fire. The Solution: Observability at Every Layer I started building a three-layer observability framework using dbt, Great Expectations, and custom validation logic. The goal was simple: catch data quality issues before they reach the model. Layer 1: dbt Tests (The First Line of Defense) dbt tests are your cheapest, fastest way to catch obvious data quality issues. They run after every transformation and fail the entire pipeline if something's wrong. Here's what we implemented: SQL -- models/staging/stg_user_events.yml version: 2 models: - name: stg_user_events columns: - name: user_id tests: - not_null - unique - name: event_timestamp tests: - not_null - dbt_utils.expression_is_true: expression: "event_timestamp <= current_timestamp()" - name: event_value tests: - not_null - dbt_utils.expression_is_true: expression: "event_value > 0" These tests are simple but powerful. They catch: Missing required fields (not_null)Duplicate records (unique)Impossible values (event_timestamp in the future)Out-of-range values (negative prices) We run these tests on every dbt run. If any test fails, the pipeline stops. No data reaches the model. No silent corruption. The beauty of dbt tests is that they're version-controlled, documented, and part of your transformation code. When a schema change happens, you update the test, commit it, and everyone knows what changed. Layer 2: Great Expectations (The Statistical Validator) dbt tests catch structural issues. Great Expectations catches statistical anomalies, the subtle shifts that break models. Here's a real scenario: our user_age column had a distribution of 18-65 for two years. Then one day, we started getting ages of 200, 500, 1000. A data entry bug upstream. dbt tests wouldn't catch this because the values are technically valid integers. But Great Expectations would. Python # great_expectations/expectations/user_events_expectations.py from great_expectations.core.batch import RuntimeBatchRequest from great_expectations.data_context import DataContext context = DataContext() suite = context.create_expectation_suite( expectation_suite_name="user_events_suite", overwrite_existing=True ) validator = context.get_validator( batch_request=RuntimeBatchRequest( datasource_name="my_spark_datasource", data_connector_name="default_runtime_data_connector", data_asset_name="user_events" ), expectation_suite_name="user_events_suite" ) # Expect user_age to be between 18 and 120 validator.expect_column_values_to_be_between( column="user_age", min_value=18, max_value=120 ) # Expect event_value to have a mean between 50 and 200 validator.expect_column_mean_to_be_between( column="event_value", min_value=50, max_value=200 ) # Expect less than 5% missing values in critical columns validator.expect_column_values_to_not_be_null( column="user_id", mostly=0.95 ) # Expect the distribution to match historical patterns validator.expect_column_kl_divergence_from_list( column="event_type", partition_object={"event_type": ["click", "view", "purchase"]}, threshold=0.1 ) validator.save_expectation_suite(discard_failed_expectations=False) Great Expectations runs after dbt tests. It validates: Value ranges (age between 18 and 120)Statistical properties (mean event value between 50 and 200)Null rates (less than 5% missing in critical columns)Distribution shifts (event_type distribution matches historical patterns) If Great Expectations detects an anomaly, it alerts us. We investigate before the data reaches the model. Layer 3: Custom Validation (The Domain Expert) dbt and Great Expectations are generic. Your domain is specific. We added custom validation logic that understands our business. Python # pipelines/validation/custom_validators.py import pandas as pd from datetime import datetime, timedelta def validate_feature_engineering(df: pd.DataFrame) -> dict: """ Custom validation for features before they reach the model. Returns a dict of validation results. """ results = {} # Validate 1: Feature completeness # We need at least 95% of features populated feature_cols = [col for col in df.columns if col.startswith('feature_')] null_rate = df[feature_cols].isnull().sum().sum() / (len(df) * len(feature_cols)) results['feature_completeness'] = { 'passed': null_rate < 0.05, 'null_rate': null_rate, 'threshold': 0.05 } # Validate 2: Feature scaling # After normalization, features should be roughly between -3 and 3 (3 sigma) for col in feature_cols: max_val = df[col].max() min_val = df[col].min() results[f'{col}_scaling'] = { 'passed': max_val < 10 and min_val > -10, 'max': max_val, 'min': min_val } # Validate 3: Temporal consistency # Events should be recent (within last 30 days) if 'event_date' in df.columns: df['event_date'] = pd.to_datetime(df['event_date']) days_old = (datetime.now() - df['event_date'].max()).days results['temporal_freshness'] = { 'passed': days_old < 30, 'days_old': days_old, 'threshold_days': 30 } # Validate 4: Business logic # Revenue should always be positive if 'revenue' in df.columns: negative_revenue = (df['revenue'] < 0).sum() results['business_logic_revenue'] = { 'passed': negative_revenue == 0, 'negative_count': negative_revenue } return results def validate_and_alert(df: pd.DataFrame, validation_results: dict) -> bool: """ Check all validations and alert if any fail. Returns True if all pass, False otherwise. """ all_passed = True for check_name, check_result in validation_results.items(): if not check_result['passed']: all_passed = False print(f"ALERT: {check_name} failed") print(f"Details: {check_result}") # Send to monitoring system (Datadog, New Relic, etc.) # send_alert(check_name, check_result) return all_passed This custom validation runs after Great Expectations. It checks: Feature completeness (95% of features populated)Feature scaling (normalized features in the expected range)Temporal freshness (data is recent)Business logic (revenue is positive) If any check fails, we block the pipeline and alert the team. The Real-World Gotchas We Discovered Gotcha 1: Validation Overhead Running dbt tests, Great Expectations, and custom validation on every pipeline run adds latency. We went from 15-minute runs to 25-minute runs. The trade-off was worth it (catching one data quality issue saved us more time than we lost), but you need to plan for it. Gotcha 2: False Positives Great Expectations' distribution shift detection is sensitive. Legitimate business changes (a marketing campaign causing a spike in user_age distribution) triggered false alerts. We had to tune thresholds carefully and add context to alerts. Gotcha 3: Schema Changes Are Sneaky A vendor added a new column to an upstream table. Our pipeline didn't break; it just ignored the new column. But the data science team expected it. We added schema validation to catch new columns and alert us. Gotcha 4: Null Handling Varies Python treats null as None. SQL treats it as NULL. Spark treats it as null. When data flows between systems, nulls get lost or misinterpreted. We had to standardize null handling across the entire pipeline. The Framework: A Decision Matrix Here's how we decide which validation layer to use: Issue TypeCaught ByExampleActionMissing required fielddbt testsuser_id is nullFail pipeline immediatelyDuplicate recordsdbt testsSame user_id appears twiceFail pipeline immediatelyImpossible valuesdbt testsevent_timestamp in futureFail pipeline immediatelyOut-of-range valuesGreat Expectationsage > 150Alert, investigate, fail if severeDistribution shiftGreat Expectationsevent_value mean changes 50%Alert, investigate, continue if acceptableBusiness logic violationCustom validationrevenue is negativeAlert, investigate, failSchema changeCustom validationNew column added upstreamAlert, investigate, update tests The Results: From Chaos to Confidence After implementing this three-layer framework: Incident reduction: We went from 2-3 data quality incidents per month to 0 in six months.Time to resolution: When issues do occur, we catch them within minutes instead of hours.Model stability: Model accuracy stopped fluctuating. It's now consistently 93-95%.Team confidence: Data scientists trust the data. Engineers trust the pipeline. The best part? We caught the schema change incident before it happened. Great Expectations detected the distribution shift, we investigated, found the upstream change, and coordinated with the vendor team before any data reached production. Getting Started: The Minimal Viable Observability You don't need to implement everything at once. Start here: Week 1: Add dbt tests for not_null and unique on critical columns.Week 1: Add dbt tests for not_null and unique on critical columns.Week 1: Add dbt tests for not_null and unique on critical columns.Week 4: Set up alerting so you're notified when validations fail. That's it. You now have observability in your data pipeline. Conclusion: Observability Saves Models Your AI model isn't failing because it's bad. It's failing because the data feeding it is bad. And you won't know the data is bad until you look. The best models in the world can't save you from garbage data. But good observability can. dbt tests, Great Expectations, and custom validation aren't fun. They don't make it into conference talks. But they'll save your production system at 3:00 AM. Start small. Test early. Validate often.

When I decided to move into AI infrastructure, nobody warned me that I had to relearn how to think about compute. I proceeded with the usual steps, such as spinning up VMs, configuring networking, and managing costs. But then a moment came, and I watched, slightly horrified. I misconfigured the inter-node networking. The result was that an eight-node GPU ran a training job at just 11% GPU utilization. It was a wake-up call for me. AI workloads aren’t just different in a marketing sense. They’re different where it counts, i.e., in the architecture — how you build and run things. The ML engineers on that project immediately assumed the model was the problem. They decided to redesign the model and spent a couple of days tweaking the architecture, like chasing a ghost. The real issue resurfaced only when someone checked the network telemetry — the cluster nodes were using standard Ethernet, not InfiniBand. The model had no issues. The infrastructure configuration was incorrect. After years of working with Azure and a period on AWS before that, I wish someone had given me a cheat sheet before starting that project. Compute: Breaking Down the Model Many cloud engineers assume that AI infrastructure requires larger VMs: more cores and more memory, and the workload will run. This approach is insufficient. While right-sizing CPUs remains relevant, it now accounts for only about 20% of considerations. The remaining 80% is driven by GPUs, which operate fundamentally differently from CPUs and significantly impact the infrastructure. A GPU isn’t just a faster CPU; it's a collection of thousands of smaller cores working together to handle large datasets. If any part of your system—such as storage speed, network bandwidth, or data preprocessing—can't keep up, the GPU remains idle, incurring huge unwanted costs. On Azure, idle GPUs cost as much as active ones. Usually, the main limitation in AI infrastructure isn't the GPU itself, but the upstream systems that supply data to it. When working with Azure, you'll mostly use two main GPU families. The NC-series gives you a single A100 per VM at about $3.60 per hour on demand, making it the go-to choice for fine-tuning and inference tasks. The ND-series has eight A100S that are connected through NVLink and InfiniBand, which is perfect for distributed training. If your cluster uses regular Ethernet instead of InfiniBand between nodes, inter-GPU bandwidth can drop by 60 to 70 percent, and Azure may not warn you about this. It’s smart to double-check that your cluster is set up with InfiniBand before starting a multi-node run and to make sure your GPU quota is ready ahead of time. Storage: Where Training Jobs Are Exhausted When you’re training a language model, expect to chew through the dataset over and over — think of it as laps around a track, not a sprint. If you try to pipe 500GB of text straight from regular Azure Blob Storage, you’ll quickly find yourself staring at a progress bar that barely budges. Each blob tops out at about 60 megabytes per second, but an A100 GPU can eat data for breakfast at several gigabytes per second. There’s a massive mismatch. If you want to keep your GPUs busy (and not just waiting around), you’ll need something beefier — Azure Managed Lustre fits the bill, since it can dish out data to your training jobs at speeds regular storage can’t dream of. I’ll admit, the first time I ran into this, I wasted hours on model tweaks before realizing the bottleneck was staring me in the face the whole time. Model checkpoints are a cost trap that is often overlooked. A single checkpoint for a 7B parameter model is around 28GB. Saving checkpoints every 30 minutes over 72 hours generates more than 4TB of data. Configure a Blob lifecycle policy before you start to avoid unexpected storage costs. Networking: Two Problems, One Person Responsible During training, each GPU shares gradient updates with the others in the cluster via AllReduce. The efficiency of the cluster is directly determined by the bandwidth and latency of this communication. If this communication is disrupted, GPU utilization drops. Machine Learning teams often attribute this to model architecture issues, such as an excessive number of parameters or an incorrect batch size, but the network is usually the cause. First, assess network performance and address any issues before the job runs to avoid unnecessary model design, as ML engineers may not consider this when monitoring loss curves. The second networking problem is well known among cloud engineers. Many enterprise clients in financial services and healthcare require AI services that avoid the public internet. Azure AI services, such as Azure OpenAI, Azure ML, and Azure AI Search, all support Private Link, and the configuration process is identical to that of other PaaS services. The key consideration is to integrate private endpoint DNS zones with existing private DNS or manage them manually. ML engineers may interpret a generic “connection refused” error caused by an incorrect DNS configuration as an API issue. Both inter-GPU bandwidth and private network isolation — critical infrastructure concerns — typically fall under the same person’s responsibility. The Azure AI Services Stack: Known Infrastructure, Unknown Branding Recent Azure services such as OpenAI Service, Machine Learning, and AKS with GPU node pools might sound new, but for most infrastructure teams, the actual work remains familiar. The phrase “managed service” sometimes suggests that everything is taken care of, but in reality, only the AI model is managed. Everyday responsibilities like network security, permissions, cost tracking, and system monitoring still rest with your team, no matter how polished the portal looks. Azure OpenAI Service works much like other managed API endpoints, supporting private connections, role-based access, managed identities, and API Management for controlling usage rates. The main distinction is its use of Provisioned Throughput Units (PTUs) — these reserve GPU resources to guarantee performance. If you see HTTP 429 errors, it’s almost always a sign of resource bottlenecks rather than issues in your code, although the latter is a common assumption. Azure Machine Learning sits on top of other infrastructure stacks, such as Blob Storage, ACR, Key Vault, and compute, which you already manage. The failure mode is unique to Azure ML: the compute cluster lifecycle. Ensure clusters auto-scale to zero when idle. Unfortunately, this is not the default setting. When a bill arrives with huge costs due to a cluster running overnight because of an unset idle timeout, everyone looks to the cloud engineer first. While it’s tempting to go with Azure Container Apps for their apparent simplicity, most real-world inference workloads ultimately end up on AKS with GPU node pools. The reason? Container Apps are easy—that is, until you’re hit with cold start lag during actual user traffic and realize spinning up a GPU container on the fly just isn’t fast enough to meet your SLA. With AKS, you get far more say over things like keeping node pools warm, tuning autoscaling, and controlling scheduling—options that simply aren’t available with Container Apps. Costs: Higher Stakes, Faster Exposure Eight GPUs on an ND-series cluster aren’t cheap — about $27 an hour adds up quickly. A few long training runs and you’re already close to $2,000, and if you’re running a batch of experiments, $20,000 can disappear before anything launches. The price tag often slips by until accounting points it out. When models underperform, it’s easy to blame the architecture, but I’ve learned to glance at GPU usage first. If you’re seeing less than 60% during distributed runs, chances are the bottleneck is in the infrastructure, not the model itself. If you want to slash costs, spot VMs can drop your bill by as much as 90%. The catch? Your training jobs must be able to handle abrupt interruptions—so regular checkpointing and clean restarts are a must. If that’s not in place, spot isn’t the way to go—sort it out with your ML team before finance starts asking questions. Reserving GPU resources is a whole different equation than CPUs: GPU supply changes from region to region, and with how quickly AI hardware evolves, locking in a three-year reservation on today’s gear is a real gamble. Security: Same Toolkit, New Attack Surface For AI projects, you still need the basics like private networks, Managed Identity, strong RBAC, and encryption. But now there’s a twist: prompt injection. It’s like the old trick with SQL injection, but for language models. Someone might simply ask a chatbot to show its system prompt. If you haven’t set up protections, it could actually answer. Firewalls won’t help here. Azure Content Safety can block some of these risky requests, but most teams don’t use it until after trouble starts. If you’re in a regulated industry, logging every inference is a must. In finance or healthcare, you need to record inputs, outputs, who did what, and when, so auditors have all the details they need. Decide on your schema and retention policy before going live, because adding it later, after compliance comes calling, is always a headache. The ML engineers on these teams know the models well. But when infrastructure acts up, causing higher costs, slowdowns, or new risks, they're often the last to spot the cause. Closing that gap is the real challenge. For cloud engineers, "architecturally different" isn’t a red flag; it’s a chance to improve.

Most infrastructure teams have a moment where someone says “we should build a platform.” The motivation is real: teams are duplicating work, the current setup is hard to use consistently, and a more structured approach would help. A few months later, the platform is a Terraform module collection, a GitLab CI template, a shared repository of scripts, and a README that several people have tried to keep current. That is a useful thing. It is not a platform. The distinction is worth being clear about, not to dismiss the work, but because the word “platform” creates expectations. When internal teams hear “we have a platform,” they assume stability, a usable interface, a versioning model, and some mechanism for raising problems when things break. A toolchain with documentation does not deliver those things by default. What Makes Something a Platform A platform is defined by its contract, not its technology. The contract describes what the consumer can expect: what they call, what parameters they provide, what outputs they receive, and what stability guarantees apply to that interface. A Terraform module with a published interface is closer to a platform primitive than a pipeline that provisions the same resources through environment variables, undocumented flags, and positional arguments. The module has a contract. The pipeline has a process. The contract does not have to be formal. It needs three things. A stable surface. Consumers should be able to call the same interface next month and receive the same type of result. Internal changes to how it works do not break consumers.A versioning model. When the interface changes, that change is communicated, and consumers are not silently broken. A git tag is enough to start with. Semantic versioning is better.A feedback path. Consumers can report when the contract is violated or the interface does not behave as documented. Someone is responsible for responding. A Terraform module with these three properties is a platform primitive. A set of modules with a shared versioning model, a stable registry entry, and a team responsible for maintaining the contract is starting to look like a platform. What Teams Actually Experience The gap between a toolchain and a platform shows up in how teams actually use it. With a toolchain, onboarding a new team means pointing them at the repository and telling them to read the README. Anything not in the README requires asking someone who has been around for a while. Changes to the toolchain break existing consumers silently because there is no versioning model. The team that maintains the toolchain treats every consumer as having kept up with the latest state of the repository. With a platform, onboarding means pointing teams at interface documentation with a working example. Changes go through a version increment. Consuming teams that pin to a version are not broken by changes they did not ask for. Plain Text # Consuming a module with a pinned version module "vm" { source = "registry.example.com/hybridops/vm/proxmox" version = "~> 2.1" name = "web-01" cores = 2 memory = 4096 } This looks like a small detail. For teams consuming infrastructure modules across a growing estate, it is the difference between a managed dependency and a shared folder everyone is afraid to touch. When a Toolchain Is the Right Call Not every infrastructure system needs to be a platform. A toolchain is appropriate when the team is small and holds the full mental model, the surface area is limited, and the rate of change is low enough that everyone stays current without a formal versioning model. When those conditions hold, the overhead of maintaining a platform contract is not justified. The problem is not having a toolchain. The problem is calling it a platform when it is not, and then finding that the expectations it created are not being met. Teams told they have a stable platform, then hit with a broken workflow from an unannounced change, lose confidence quickly. That confidence is hard to rebuild. HybridOps has been working in this space: publishing Terraform modules to a registry, versioning releases, and treating module interfaces as contracts. It is not a finished platform. It is a direction, and being explicit about that direction changes how the work gets done. A Simple Test If a consuming team pins to the current version of your toolchain today, will it still work in three months without any changes on their side? If you cannot answer yes with confidence, you have a toolchain, not a platform. Both are useful. Only one creates the kind of trust that makes a growing engineering organisation move faster rather than slower. Knowing which one you have is the first step toward building the right one.

When working on API automation projects, one of the first things that becomes repetitive is configuring the same settings for every test. The base URL, content type, request logging, and common response validations often appear in multiple test classes. As the number of tests increases, maintaining these repeated configurations becomes difficult. REST Assured provides specifications to solve this problem. Instead of defining the same settings in every test, common configurations and specifications can be created once and reused throughout the test suite. This article demonstrates a simple approach to configuring REST Assured using a Base Test class along with Request and Response Specification. What Are REST-Assured Specifications? A specification is a reusable configuration object that contains common request or response settings. So, instead of repeatedly writing: Java given() .baseUri("https://api.example.com") .header("Authorization", "Bearer token") .contentType(ContentType.JSON) The configuration can be defined once and reused across multiple tests. Similarly, the common validations can also be written using the specifications. Specifications help in: Reduce code duplicationImprove test readabilityCentralize API configurationsSimplify maintenanceStandardize request and response validations Why Use Specifications? Consider an API test that retrieves user details. Java @Test public void getUserDetails() { given() .baseUri("https://api.example.com") .when() .get("/orders/2") .then() .statusCode(200); } The test works correctly, but the base URI and common validations, such as status code, will need to be repeated in every test. A better approach is to move these common settings into reusable specifications. What Problem Does It Solve? In many API automation projects, test cases often contain repeated configuration code. The same base URL, content type, authentication details, headers, and response validations are repetitive across multiple test classes. While this may not seem like a problem when there are only a few tests, maintaining the test suite becomes difficult as the project grows. Consider a scenario where the API base URL changes from a QA environment to a Staging environment. Without a centralized configuration, every test containing the old URL would need to be updated. Similarly, if a common header or authentication mechanism changes, modifications would be required in multiple places. Request and Response Specifications solve this problem by moving common configurations into reusable objects. Instead of repeating the same setup in every test, the configuration is defined once and reused wherever required. This reduces code duplication, improves readability, and makes the test suite easier to maintain. As a result, test methods can focus on validating business functionality rather than configuring API requests and responses. This leads to cleaner and more maintainable automation code. Creating a SetupSpecification Class The most common configurations should be placed in a separate class. This allows all test classes to inherit the same setup. The following example creates a Request and Response Specification in a separate class using the @BeforeClass annotation. Java public class SetupSpecification { @BeforeClass public void setup () { final RequestSpecification request = new RequestSpecBuilder () .addHeader ("Content-Type", "application/json") .setBaseUri ("http://localhost:3004") .addFilter (new RequestLoggingFilter ()) .addFilter (new ResponseLoggingFilter ()) .build (); final ResponseSpecification response = new ResponseSpecBuilder () .expectResponseTime (lessThan (10000L)) .build (); RestAssured.requestSpecification = request; RestAssured.responseSpecification = response; } } This setup method runs before the test class execution. The Request Specification contains the base URI, content type, and logging configuration. Any configuration defined in a Request Specification will be applied to every API request that uses that specification. For example, if the specification includes a common header, authentication token, content type, or query parameter, those values will automatically be sent with all requests that reference the specification. While this promotes reusability and reduces duplication, care should be taken when adding request-specific details to a shared specification. Not all APIs may require the same headers, authentication mechanisms, query parameters, or request bodies. Including such configurations in a common specification can lead to unintended behavior and make tests more difficult to maintain. The Response Specification contains the common validations that are expected from the API response. The expectResponseTime() method validates that the API responds within the specified time limit. Additionally, we can also add the validations for: Status CodeHeadersContent-TypeCookieBody However, it is important to understand that any validation defined in a Response Specification will be applied to every API test that uses that specification. For example, if the specification includes a validation for a 200 status code, all tests using that specification will automatically expect a 200 response. This may not be appropriate for APIs that are expected to return different status codes, such as 201, 204, 400, or 404. The same consideration applies to validations related to headers, content type, cookies, and response body content. Including endpoint-specific validations in a shared specification can reduce flexibility and make tests harder to maintain. A good practice is to keep only the truly common validations in a shared Response Specification and add endpoint-specific assertions within the individual test methods. The statement below makes the Request Specification available globally for the test execution. Java RestAssured.requestSpecification = request; RestAssured.responseSpecification = response; As a result, the base URI and header(Content-Type), and validation to check the response time do not need to be specified in every test. Writing a Test Using the Specifications Once the setup is complete, test classes can extend the SetupSpecification class. Java public class TestGetRequestWithRestAssuredSpecs extends SetupSpecification { @Test public void getRequestTestWithRestAssuredConfig () { final int orderId = 3; given ().when () .queryParam ("id", orderId) .get ("/getOrder") .then () .statusCode (200) .and () .assertThat () .body ("orders[0].id", equalTo (orderId), "orders[0].product_name", equalTo ("USB-C Charger")); } } The Request Specification is automatically applied because it was configured in the SetupSpecification class. It means all the common request configurations, such as the base URI, headers, content type, and logging settings, are automatically applied to the request. Similarly, the common response validations configured for expected response time in the SetupSpecification class are reused during test execution. The test itself focuses only on endpoint-specific details by passing the id query parameter, invoking the /getOrder endpoint. This approach keeps the test concise and improves maintainability by separating common configuration from test-specific assertions. Adding Additional Assertions The Response Specification can handle common validations, while endpoint-specific assertions can still be added in the test. Java public class TestGetRequestWithRestAssuredSpecs extends SetupSpecification { @Test public void getRequestTestWithRestAssuredConfig () { final int orderId = 3; given ().when () .queryParam ("id", orderId) .get ("/getOrder") .then () .statusCode (200) .and () .assertThat () .body ("orders[0].id", equalTo (orderId), "orders[0].product_name", equalTo ("USB-C Charger")); } } In this example, the response body validations for order ID and product name remain inside the test because they are specific to this API endpoint. Why This Approach Is Useful As the test suite grows, hundreds of API tests may use the same base URL, content type, authentication, and response validations. Maintaining these configurations in every test class can quickly become difficult. Keeping the Request and Response Specifications in a separate class provides a centralized location for managing common settings. If the API URL changes or additional configurations need to be added, only a single file needs to be updated. This approach also improves readability because the test methods contain only the business validations relevant to the API being tested. Using Request and Response Specifications Directly in the Test Class While many automation projects prefer keeping specifications in a separate class, there are situations where creating specifications directly inside the test class makes sense. This approach is useful for smaller projects, proof-of-concept implementations, or when a test class requires its own configuration that is not shared with other tests. In this approach, the Request and Response Specifications are created using the @BeforeClass annotation and are available only within the current test class. Java public class StringRelatedAssertionTests { private static ResponseSpecification responseSpecification; private static RequestSpecification requestSpecification; @BeforeClass public void setupSpecBuilder () { final RequestSpecBuilder requestSpecBuilder = new RequestSpecBuilder ().setBaseUri ( "https://api.restful-api.dev/objects") .addQueryParam ("id", 3) .addFilter (new RequestLoggingFilter ()) .addFilter (new ResponseLoggingFilter ()); final ResponseSpecBuilder responseSpecBuilder = new ResponseSpecBuilder ().expectStatusCode (200); responseSpecification = responseSpecBuilder.build (); requestSpecification = requestSpecBuilder.build (); } @Test public void testStringAssertions () { given ().spec (requestSpecification) .get () .then () .spec (responseSpecification) .assertThat () .body ("[0].name", equalTo ("Apple iPhone 12 Pro Max")) } } In this example, the Request and Response Specifications are created once in the @BeforeClass method and stored in static variables. The Request Specification contains common request details such as the base URI, query parameters, and logging filters, while the Response Specification defines the expected status code. During test execution, the Request Specification is applied using the spec(requestSpecification) method before sending the request. After the response is received, the Response Specification is applied using spec(responseSpecification) to validate the common response expectations before performing additional assertions on the response body. Keeping the specifications and test logic within the same class makes the example easy to follow, as both the setup and test execution are located in a single file. However, as the test suite grows and multiple test classes require the same configurations, duplicating specifications across classes can become difficult to maintain. In such situations, moving the common Request and Response Specifications to a separate class provides better reusability and reduces code duplication. For smaller projects or learning purposes, defining the specifications directly within the test class remains a simple and effective approach. Summary Rest-Assured Specifications help create cleaner and more maintainable API automation tests. A best practice is to define Request and Response Specification in a separate class and initialize them using the @BeforeClass annotation. The Request Specification manages settings such as the base URI, content type, and logging, while the Response Specification handles common response validations. By centralizing these configurations, test classes become shorter, easier to read, and simpler to maintain. For API automation frameworks built with REST Assured and TestNG, this pattern provides a clean foundation that scales well as the number of tests increases.

By Faisal Khatri CORE

OpenTofu is an open-source infrastructure as code (IaC) tool maintained by the Linux Foundation. It lets you define cloud infrastructure in configuration files and deploy it with a single command-line tool called tofu. This tutorial explains how to deploy infrastructure with OpenTofu, from installing the CLI to provisioning and destroying a real cloud resource. What You Need Before You Start You need three things to follow along: An AWS account with credentials configured locally (the AWS command-line interface reads them from ~/.aws/credentials or standard environment variables).Basic comfort in a terminal.About 15 minutes. The resources in this tutorial cost almost nothing, and the final step deletes everything you create. Pick a region you are happy to work in, such as us-east-1. A Quick Word on OpenTofu OpenTofu is a fork of Terraform, created in 2023 after Terraform moved to the Business Source License (BSL), a source-available license that is not OSI-approved open source. OpenTofu is a Linux Foundation project and was accepted into the Cloud Native Computing Foundation (CNCF) as a sandbox project in 2025. The configuration language is the same HashiCorp Configuration Language (HCL) you may already know, every provider works the same way, and the command-line interface is tofu instead of terraform. If you have written Terraform before, you already know most of this. Step 1: Install OpenTofu Install OpenTofu using whichever method works best for your machine. On macOS or Linux with Homebrew: Shell brew install opentofu On Linux or macOS without Homebrew, use the official installer script: Shell curl --proto '=https' --tlsv1.2 -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh chmod +x install-opentofu.sh ./install-opentofu.sh --install-method standalone rm install-opentofu.sh The standalone installer verifies the integrity of what it downloads, so it expects cosign or GnuPG to be available. If you don't have either and just want to try it quickly, add --skip-verify to the install command. On Windows, use winget: Shell winget install --exact --id=OpenTofu.Tofu Confirm the install worked: Shell tofu --version You should see OpenTofu v1.12.0 or later. Step 2: Write Your First Configuration Create a new directory and a single file inside it called main.tf: Shell mkdir tofu-demo && cd tofu-demo Open main.tf and add the following. Each block is explained right after. Shell terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } random = { source = "hashicorp/random" version = "~> 3.0" } } } provider "aws" { region = "us-east-1" } resource "random_pet" "suffix" { length = 2 } resource "aws_s3_bucket" "demo" { bucket = "tofu-demo-${random_pet.suffix.id}" } output "bucket_name" { value = aws_s3_bucket.demo.bucket } A few things worth understanding here. The terraform block declares which providers your configuration depends on and where to download them. OpenTofu keeps this block name for backward compatibility, so the same configuration runs on either tool. The provider "aws" block sets the region you deploy into. The random_pet resource generates a short, readable suffix such as clever-mongoose, which keeps your bucket name globally unique without you having to invent one. The aws_s3_bucket resource is the infrastructure you are actually creating, and it references the random suffix, so OpenTofu knows to create the suffix first. The output block prints the final bucket name once everything is deployed. Step 3: Initialize the Project Run tofu init from inside your project directory: Shell tofu init This reads your terraform block, downloads the AWS and random providers from the OpenTofu registry, and sets up the working directory. You only rerun it when you add a new provider or module. You should see a message confirming OpenTofu has been initialized. Step 4: Preview the Changes Before OpenTofu touches your account, ask it what it intends to do: Shell tofu plan The plan is the most important habit in IaC. It shows you exactly what will be created, changed, or destroyed before anything happens. For this configuration, the plan shows two resources to add: the random pet and the S3 bucket. Read it. Make sure it matches what you expect. A clean plan-and-review step is what stops a one-line config change from accidentally deleting a database. Step 5: Deploy When the plan looks right, apply it: Shell tofu apply OpenTofu shows you the plan one more time and waits for you to type yes. Confirm, and it provisions the bucket. When it finishes, you see your bucket_name output. Open the S3 console in AWS and your new bucket is there, created entirely from code. You now have real infrastructure under version control. Change the configuration, run tofu plan to see the diff, and tofu apply to roll it out. That loop, edit, then plan, then apply, is the whole job. Step 6: Understand State After your first apply, OpenTofu creates a file called terraform.tfstate in your directory. This is the state file, which OpenTofu uses to map the resources in your configuration to the actual resources in your account. When you run a plan, OpenTofu compares your configuration, the state file, and the actual infrastructure to work out what changed. On your laptop, with one person and one project, a local state file is fine. It stops being fine the moment a second engineer needs to run a deployment. Two people with two copies of the state file will overwrite each other's work. The state file also holds resource metadata you do not want sitting in a Git repository or on a shared drive. This is the problem every team hits once IaC moves beyond a single person. Step 7: Clean Up Tear down everything you created so it costs you nothing: Shell tofu destroy OpenTofu shows you what it will delete and waits for a yes. Confirm, and your bucket and the random suffix are gone. The destroy command is the counterpart to apply, and it reads the same state file to know what to remove. Where This Goes Next Running tofu from your laptop is the right way to learn. It is the wrong way to run infrastructure for a team. Once more than one engineer is involved, you need shared remote state with locking so two applies cannot collide, a record of who changed what and when, policy checks that run before an apply rather than after an incident, and a way to catch drift when someone makes a manual change in the console. You can assemble these pieces yourself with a remote state backend, a continuous integration pipeline, and a set of scripts. Many teams start there. As the number of stacks and engineers grows, that homegrown setup becomes its own maintenance burden, which is the point where teams adopt an infrastructure orchestration platform. A platform like Spacelift manages OpenTofu runs against shared remote state, gates changes with policy as code, and detects drift between your configuration and what is actually deployed, with an audit trail across every change. It is one option in a category of tooling built to solve the team-scale problems this tutorial only hints at. The decision of whether and when to adopt one depends on how many people and environments you are managing. For now, you have the foundation: install, write, init, plan, apply, and destroy. Every OpenTofu project, from a single bucket to a fleet of production environments, runs on the same loop you just learned. Next, read the OpenTofu documentation on modules and remote backends to see how that loop scales from one file to a real codebase.

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.