Monitoring and Observability

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed. Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet. That script eventually became four different experiments, then a collection of small scanners, and eventually the dashboard shown in this article. Over the next eight months, that script evolved into OpsCart Watcher — an open-source operational triage dashboard for Kubernetes. This article is about what the journey taught me, and what I think is still missing from most Kubernetes environments. OpsCart Watcher — operational triage for Kubernetes (6 minutes) The Problem the Script Revealed The script did one thing well: it looked at an entire cluster and listed what was broken. Engineers who ran it kept telling me the same thing — "I had no idea this was there." That response was the important signal. These engineers had Grafana, Prometheus, and kubectl. Visibility was not their problem. The problem was that nothing told them to look at this specific namespace, this specific pod, this specific storage volume — before it became an incident. Consider a pod in CrashLoopBackOff for 19 days with 5,000+ restarts. To a metrics dashboard, that deployment looks healthy: replica count satisfied, a pod exists in Running state between crashes, CPU and memory flat because the container barely lives long enough to consume anything. The dashboard is answering the question it was built to answer — is the cluster meeting its SLOs? — and the answer is yes. The question nobody built tooling for: what deserves attention right now? LayerWhat It AnswersToolsMetricsIs the cluster meeting its SLOs?Prometheus, Grafana, DatadogPer-resource stateWhat is this specific pod doing?kubectl, k9s, LensOperational triageWhat deserves attention right now?Prioritizing operational work across cluster state What Triage Looks Like in Practice Overview page — Incident Score 41/100, KPI bar, Top 5, War Room panel The first time I ran the rebuilt dashboard against a cluster with real failures, the top of the screen didn't show me a CrashLoopBackOff pod. It showed me four CrashLoopBackOff pods spread across three namespaces, collapsed into a single operational problem: Plain Text 1. 4 pods crash-looping CRITICAL payments/fraud-detection (1810 restarts) → kubectl logs fraud-detection-... -n payments --previous That collapsing is the entire idea. Instead of inspecting every deployment individually, I was looking at a ranked list of operational problems — each with a severity, a location, and the exact kubectl command to start investigating. The full output for this environment: Plain Text Incident Score: 41/100 (Degraded) Top 5 Things to Fix: 1. 4 pods crash-looping CRITICAL 4 pods 2. 3 image_pull_backoff issues CRITICAL 3 items 3. 1 privileged_container issue CRITICAL 1 item 4. 1 namespace missing NetworkPolicy HIGH 1 ns 5. 3 orphaned PVCs wasting money MEDIUM 80 GB None of these had triggered an alert. All were present and accumulating before the scan. The Incident Score — a composite 0–100 across reliability, security, and waste — exists for one reason. Engineers fix incidents. Managers remember numbers. "We moved the Incident Score from 41 to 67" is a sentence that sticks. The crash loops and NetworkPolicies are the work behind it. The Step After Detection Finding problems was never the hard part. Knowing where to begin was. The most common feedback on the original bash script was some version of: "I found the problem, but I still didn't know what to do next." In March, I wrote about finding a container with 24,069 restarts that had been accumulating undetected. Finding it took sixty seconds. The next hour was the actual work: what do I run first? Is this configuration or code? Is it customer-facing? The investigation page is my answer to that hour. Investigation page — OpsCart Assessment, Evidence, Recommended Investigation One click from any triage finding opens a dedicated investigation view: Plain Text OpsCart Assessment This workload has restarted 1810 times over 6 days. The restart rate appears stable, suggesting a deterministic configuration or application failure rather than an intermittent infrastructure issue. No referenced ConfigMaps or Secrets were detected in the pod spec — missing configuration is unlikely to be the root cause. Investigation should begin with previous container logs. Estimated time: 5–10 minutes. Evidence [1810 Restarts] [CrashLoopBackOff] [6d] [Deployment/fraud-detection] Recommended Investigation HIGH CONFIDENCE Check previous container logs MEDIUM Verify ConfigMaps and Secrets exist LOW Check for OOMKill in events The assessment is rules-based — no AI. It reads restart count, failure pattern (stable vs accelerating), and referenced configuration objects, then produces a deterministic, auditable summary. The confidence levels reflect how a senior engineer actually reasons: previous logs are almost always the right first move for a crash loop; OOMKill is worth checking but less likely. This is the part kubectl doesn't give you. Neither does Lens, k9s, or Headlamp. From "What Is Broken?" to "What Changed?" The biggest architectural change came when the dashboard gained memory. The first version of the tool answered: "what is broken?" The current version — backed by a small embedded database recording every scan — answers "what changed?" That sounds like a minor distinction. Operationally, it changes everything. An incident that has existed for three days deserves different attention than one that appeared five minutes ago. A cluster whose Incident Score dropped eight points overnight is telling you something that no single scan can. War Room — critical issues with visual differentiation per type Every KPI now carries a trend arrow — critical issues up three since the last scan, waste down one — and the Incident Score shows a seven-point sparkline. Each incident is tracked with first-seen and last-seen timestamps and an active/resolved status, so "CrashLoopBackOff — first detected 6 days ago, still active" replaces "CrashLoopBackOff." Operational memory changed the tool from a scanner into something that remembers the history of a cluster. What This Is Not The triage pattern does not answer when an issue started at the metrics level, why an application is slow, or whether last Tuesday's deployment caused a regression. Prometheus, APM tooling, and deployment audit logs remain the right tools for those questions. The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability stack. The Biggest Lesson When I started, I thought Kubernetes debugging was about collecting more information. It wasn't. Kubernetes already exposes almost everything an operator needs through its API. The difficult part is deciding what deserves attention first. Over eight months, I found myself spending less time searching for failures and more time ranking them. That is ultimately what OpsCart became — not another dashboard, but a prioritization engine for cluster operations. Why Open Source I considered keeping the dashboard private. Instead, I open-sourced it because operational patterns only become useful when they're tested across different clusters. Every environment fails differently, and I wanted the prioritization model to evolve from real-world feedback rather than a single infrastructure. The Remaining Gap The conclusion from my March article is still true: the question worth asking of your environment is not whether these conditions exist — they almost certainly do — but whether your current observability layer would surface them before they become incident preconditions. Eight months of building has only made that conclusion more specific. The gap is not data. The gap is attention: knowing which five things, out of hundreds of resources, deserve a human's time right now. Eight months ago I thought I was building a better debugging script. I wasn't. I was building something that helps operators decide where to spend the next ten minutes. About the environment: The scenarios shown in this article — CrashLoopBackOff pods, orphaned PVCs, missing NetworkPolicies, privileged containers — are representative of what OpsCart finds on real production clusters. The environment shown is a dedicated demonstration cluster configured with realistic failure scenarios. No production data was used. About the tool: OpsCart Watcher is open-source at github.com/opscart/opscart-k8s-watcher. It deploys as a single read-only container: Shell kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80

By Shamsher Khan CORE

AWS Glue makes it easy to get a PySpark pipeline running quickly. It is significantly harder to build one that stays maintainable as logic grows, performs reliably at scale, and does not quietly accumulate operational debt over time. Most Glue pipelines start simple and become difficult to manage gradually — formulas get hardcoded, modules grow without boundaries, output files proliferate, and before long a single job is doing too many things in ways that are hard to test, hard to debug, and expensive to change. This article presents a set of design principles drawn from production Glue ETL pipelines processing billions of rows. Each principle is independent — you do not need to adopt all of them to benefit from any one. But together they form a coherent approach to building Glue pipelines that are modular, observable, cost-efficient, and built to last. Principle 1: Externalize Logic Into Config, Not Code The single most impactful structural decision in a Glue pipeline is where business logic lives. When formulas, dataset references, column selections, and filter conditions are hardcoded in PySpark, every change requires modifying job code, redeploying, and re-validating the full pipeline. A one-line formula change carries the same deployment risk as a structural refactor. Over time, this creates a strong disincentive to make changes, and the pipeline calcifies. The better pattern is to treat the Spark job as a generic executor and externalize all business-specific declarations into configuration. Formulas are declared as config entries with operands, rounding rules, and output names. Dataset loading behavior — which table, which columns, which filters, whether to cache — is declared per source rather than scripted per job. Schema shapes for complex types are declared explicitly rather than inlined. JSON { "source_table": "headcount_actuals", "database": "finance_db", "select_columns": ["site", "badge_type", "headcount", "fiscal_week"], "filters": [{"column": "is_active", "value": "Y"}], "rename": {"hc_count": "headcount"}, "cache": true } When a new dataset is needed, a new config entry is added — no Spark code changes. When a formula changes, the config entry is updated — no job redeployment required. The job itself becomes stable and generic; only config changes as business requirements evolve. This principle pays increasing dividends over time. Pipelines with externalized logic are faster to modify, safer to deploy, and easier to hand off because the business rules are readable independently of the execution engine. Principle 2: Design Modules With Explicit Boundaries A Glue job that does everything in one place is easy to write and hard to maintain. As pipelines grow, the instinct to add more logic to an existing job accelerates technical debt faster than almost any other decision. The more durable pattern is to decompose computation into modules with explicit input and output contracts. Each module receives one or more DataFrames, applies a focused set of transformations, and produces a named output DataFrame. Modules communicate exclusively through in-memory DataFrame references — there is no disk I/O between stages, no shared mutable state, and no implicit dependency on execution order beyond what the data flow itself requires. Utilities follow the same boundary principle, organized into two layers. Generic pipeline utilities handle cross-cutting concerns — file writing, dataset loading, filtering, deduplication, pivot operations — and are shared across all modules. Module-specific utilities implement transformation logic scoped to a single module and are never invoked outside it. This structure means adding a new module requires only writing its scoped utilities and wiring it into the pipeline. The generic layer is never touched. Existing modules are never at risk from new module development. The downstream benefit is testability. Each module with clean boundaries can be validated independently using mocked PySpark DataFrames with no Glue environment required. Engineers can run pytest locally against individual modules, iterate quickly, and deploy only after local validation passes. Principle 3: Choose Your Job Topology Deliberately A common default in complex pipelines is to split computation across multiple Glue jobs, using S3 as the handoff layer between stages. This is sometimes the right choice — but it should be a deliberate decision, not an instinct. Multi-job topologies make sense when stages have genuinely different compute profiles, when intermediate outputs need to be reused independently by other consumers, or when a stage failure should not force a full recompute from the beginning. In these cases, job separation gives you independent retry boundaries, independent DPU sizing, and the ability to schedule stages on different cadences. Single-job topologies — where the full pipeline runs within one Spark session — make sense when all computation is tightly coupled, modules share the same input datasets, and intermediate outputs have no standalone value. Running everything in one session eliminates cold start overhead for intermediate stages, avoids the cost of serializing data to S3 and deserializing it back between jobs, and keeps the execution model simple to reason about: one trigger, one job, one result. The question to ask is whether the stages truly need to be independent. If intermediate S3 persistence adds coordination complexity without adding value — no independent consumers, no differential retry requirements, no meaningful DPU difference between stages — then collapsing to a single job is usually faster, simpler, and cheaper. If stages have real independence requirements, splitting them is the right call and the operational overhead is justified. Neither topology is inherently superior. The mistake is defaulting to one without evaluating the trade-offs for the specific pipeline at hand. Principle 4: Overlap Writes With Computation When Latency Matters Overlapping writes with computation is a well-established technique in high-performance computing, deep learning training, and heavy database operations. The core idea is to hide the slow latency of I/O operations by running them in the background while the CPU or GPU continues processing data. Rather than waiting for a write to complete before starting the next computation, both proceed simultaneously — I/O latency is absorbed into computation time rather than added on top of it. In Glue ETL pipelines, the same principle applies directly. In a pipeline where multiple output DataFrames are produced, the naive write strategy — complete all computation, then write all outputs sequentially — has two compounding problems. First, it creates a peak memory spike: all computed results are held in memory simultaneously while writes proceed one by one. Second, it serializes work that does not need to be serial: every millisecond spent waiting for S3 acknowledgment is a millisecond the Spark executors are idle. This is worth addressing only when latency is a meaningful constraint. For low-frequency batch jobs running overnight with no user-facing SLA, sequential writes are perfectly adequate. But for pipelines where users or downstream systems are waiting on results — or where job duration directly affects infrastructure cost — overlapping writes with computation delivers measurable wall-clock reduction. The two-phase write strategy implements this directly. Outputs from early modules are written to S3 in background threads immediately after those modules complete, running in parallel with later computation stages. By the time all computation finishes, a significant portion of the output data has already landed in S3. Remaining outputs are then flushed concurrently in a second phase. The implementation leans on Python's concurrent.futures.ThreadPoolExecutor to manage background write threads while the main Spark session continues computation on the driver. A generic write orchestration utility can wrap this pattern so individual modules never need to manage thread lifecycle directly — they simply declare their output and the utility handles scheduling, thread management, and error propagation. Python from concurrent.futures import ThreadPoolExecutor, as_completed def write_phase_a(write_tasks): with ThreadPoolExecutor(max_workers=len(write_tasks)) as executor: futures = {executor.submit(task["fn"], task["df"], task["path"]): task["name"] for task in write_tasks} for future in as_completed(futures): name = futures[future] future.result() logger.info(f"[Phase A] Write complete: {name}") The practical effect is that peak memory pressure is distributed over the job's lifetime rather than concentrated at the end, and total wall-clock time is reduced by the overlap between I/O and CPU-bound computation. For pipelines with many output datasets and a latency SLA to meet, the savings compound significantly. Principle 5: Right-Size Output Files With a Reusable Writer Utility Right-sizing output files is the practice of tuning file sizes to balance disk I/O performance, network transfer speeds, and downstream processing efficiency. Too many small files and downstream readers spend more time on metadata operations and S3 API calls than on actual data reads. Too few large files and parallelism suffers — readers cannot split work efficiently across threads or nodes. The target is consolidated, evenly sized files that match the read patterns of downstream consumers. Spark's default output behavior writes one file per partition, and partition counts are typically tuned for computation throughput rather than output shape. A job optimized for shuffle performance might produce hundreds of partitions, each containing a few megabytes of output data — perfectly reasonable for Spark internals, but harmful for any reader that comes after. This small file problem compounds over time as output partitions accumulate in S3 and the Glue Catalog metadata grows with them. The fix is a reusable writer utility that decouples output file sizing from Spark's internal partition count. Rather than accepting the default, the utility estimates the DataFrame's actual size, calculates the appropriate number of output files for a target file size — typically 128MB to 256MB per file — and coalesces partitions before writing. Python def write_optimized(df, output_path, partition_cols, target_file_size_mb=128): estimated_size_mb = df.rdd.map(lambda row: len(str(row))).sum() / (1024 * 1024) optimal_partitions = max(1, int(estimated_size_mb / target_file_size_mb)) df.coalesce(optimal_partitions) \ .write \ .partitionBy(*partition_cols) \ .parquet(output_path, mode="overwrite") Making this a shared generic utility rather than inline logic in each module has two practical benefits. First, it enforces consistent file sizing behavior across all outputs in the pipeline — no module accidentally writes thousands of tiny files because an engineer forgot to coalesce. Second, it centralizes the tuning knob: when the target file size needs to change — because downstream query patterns shift or a new consumer has different read characteristics — it changes in one place and applies everywhere. Right-sized output files improve Athena scan performance, reduce per-query S3 API costs, keep Glue Catalog partition metadata manageable, and make the output data easier to consume for any downstream system reading from S3. This is a low-effort, high-payoff improvement that applies to virtually every Glue pipeline writing to S3. Principle 6: Use Complex Types to Defer Denormalization SQL-based pipelines are constrained to flat, fully denormalized row structures at every intermediate stage because SQL has no native complex type support. This forces denormalization to happen early, inflating data volume at every subsequent join and aggregation. PySpark has native support for structs, maps, and arrays. Using these types at intermediate stages allows related values to be grouped logically without inflating row counts. A row that would require five denormalized rows in SQL can be represented as a single row with a struct or array column in Spark. Denormalization is then deferred to the final output layer only — applied once, at write time, for consumers that require flat structures. Everything upstream of the final write benefits from reduced volume, fewer shuffles, and faster joins. This principle is particularly impactful in pipelines with multi-level aggregations or wide schemas where dozens of metrics attach to the same dimensional key. Keeping those metrics grouped in a struct until the final output stage reduces the effective row count and join complexity throughout the pipeline. Principle 7: Build Observability Into Every Stage Glue jobs that fail silently or surface errors as opaque stack traces at the end of a long execution are expensive to debug. The investment in step-level observability pays back quickly the first time something goes wrong in production. The minimum viable observability pattern is row count logging at every materialization point. After each module completes and after each write, log the output row count with a descriptive label. This gives a running picture of data volume through the pipeline and makes it immediately obvious when a transformation has dropped rows unexpectedly or produced more rows than expected. Python def log_step(df, step_name): count = df.count() logger.info(f"[{step_name}] Row count: {count:,}") return df Pair this with a try/except/finally pattern at the job level that ensures spark.catalog.clearCache() is always called on exit — whether the job succeeds or fails — to release cached DataFrames and avoid memory leaks across retries. Python try: run_pipeline() except Exception as e: logger.error(f"Pipeline failed: {e}") raise finally: spark.catalog.clearCache() CloudWatch captures all logs automatically. When a job fails, the row count trail shows exactly where in the pipeline the problem occurred, making triage faster and reducing the time between failure and fix. Principle 8: Isolate Executions for Concurrency Pipelines that share compute resources across simultaneous executions create contention that is difficult to predict and expensive to manage. The common response — queue-based serialization — adds operational complexity without solving the underlying resource constraint. AWS Glue's execution model eliminates this problem structurally. Each job execution gets its own isolated DPU allocation. There is no shared compute pool. Ten simultaneous executions consume ten independent DPU allocations and do not interfere with each other in any way. Designing for this means treating each execution as fully independent: no shared state, no cross-execution coordination, no assumption about what other executions are running. Combined with idempotent writes — using overwrite mode so a retry produces the same result as the original execution — the pipeline becomes safe to run concurrently at any scale without additional coordination logic. The cost model reinforces this. Glue bills per DPU-second of actual compute consumed. An execution that takes eight minutes on 240 DPUs costs the same whether it runs alone or alongside a hundred other executions. There is no premium for concurrency and no shared pool to provision for peak load. Putting It Together These eight principles are independent but complementary. A pipeline that applies all of them is modular enough to develop in parallel, observable enough to debug quickly, cost-efficient enough to run at scale, and stable enough to maintain over time without accumulating structural debt. The quickest wins for most existing pipelines are Principles 1, 5, and 7 — externalizing logic into config, right-sizing output files with a shared utility, and adding row count logging at every stage. Each can be applied incrementally without restructuring the full pipeline. The remaining principles become more valuable as pipeline complexity grows and concurrency requirements increase. The underlying thesis is simple: a well-designed Glue pipeline should be easy to change, easy to test, easy to debug, and cheap to run. None of those properties require exotic infrastructure. They require deliberate design decisions applied consistently from the start.

By Janani Annur Thiruvengadam CORE

For years, service organizations measured operational efficiency through response time. A machine failed, a ticket dropped, a technician arrived on-site, and the diagnosis and repair resolved the issue. Industries dependent on physical assets accepted this framework because they believed that it was not possible to avoid downtime. The benchmark for operational excellence depended on how quickly teams reacted after disruption occurred. That definition of service reliability has changed dramatically. Across industries such as ATM infrastructure, elevator systems, industrial manufacturing, HVAC networks, utilities, and connected buildings, uptime has evolved from a technical KPI into a direct business expectation. A malfunctioning elevator inside a commercial tower immediately affects tenant experience. An unavailable ATM network during a transaction spike escalates into a customer-service issue within minutes. In sectors where Service Level Agreements (SLAs) define accountability, even short-lived disruption can simultaneously create financial penalties, reputational damage, and customer churn. This growing pressure explains why organizations are restructuring service operations around predictive intelligence, telemetry ecosystems, and AI-driven operational visibility. Businesses targeting 99.9% uptime, commonly referred to as “three nines” availability, now operate within extremely narrow tolerance margins. Operationally, that benchmark allows for less than nine hours of annual downtime across distributed infrastructure environments involving connected assets, IoT systems, APIs, cloud platforms, and field-service networks. Connected Assets Are Reshaping Service Delivery The most significant transformation inside the service industry is happening beyond customer-facing applications. Machines themselves are becoming active participants in operational decision-making. Modern industrial assets continuously transmit telemetry related to vibration intensity, thermal behavior, airflow fluctuations, voltage variation, load cycles, and component stress. Earlier maintenance environments depended heavily on scheduled inspections and manual servicing intervals. Predictive ecosystems now analyze live operational behavior continuously, allowing organizations to identify abnormal machine patterns before a visible breakdown occurs. Large elevator manufacturers increasingly rely on telemetry-driven systems that can identify brake-pressure instability and motor stress, even before shutdown occurs inside high-footfall commercial environments. Similarly, ATM infrastructure providers now use transaction telemetry and demand analytics to forecast cash replenishment cycles proactively during high-volume periods. According to McKinsey & Company, predictive maintenance typically reduces machine downtime by 30 to 50% and increases machine life by 20 to 40%. IBM has also estimated that such predictive maintenance frameworks can improve labor productivity while helping organizations reduce downtime and improve asset reliability. Why Predictive Maintenance Is Replacing Reactive Service Models Traditional field-service environments created inefficiencies that organizations quietly accepted for years. Once a machine failed, there was a simultaneous trigger effect on multiple disconnected workflows. Service teams logged tickets, identified technicians, diagnosed faults, verified spare-part availability, and scheduled follow-up visits. Very often, engineers reached the site without the required replacement component, forcing additional visits and extending downtime unnecessarily. Predictive service ecosystems reduce that operational friction. Modern AI-enabled maintenance systems increasingly integrate telemetry platforms directly with workforce management tools, inventory systems, and service histories. Instead of merely identifying faults, these environments support operational decision-making before engineers physically engage with the asset. operational eventconventional workflowpredictive ai-led workflow ATM cash depletion Shortage identified after customer disruption AI forecasts replenishment needs proactively Elevator motor instability Technician dispatched after operational failure Telemetry predicts degradation before shutdown HVAC compressor fluctuation Complaint-driven escalation Continuous monitoring detects abnormal pressure patterns Industrial equipment fault Manual diagnosis during site visit AI identifies component failure in advance Modern industrial-service providers use AI-led technician orchestration systems that evaluate technician expertise, asset familiarity, certification levels, and spare-part availability before dispatch approval occurs. The objective is not faster repair cycles anymore. Organizations are now trying to prevent customer-facing disruption before it begins. Observability Is Replacing Conventional Monitoring Earlier, the designs of monitoring systems ensured they could primarily identify if the infrastructure was functioning properly. Modern service ecosystems require deeper operational visibility because enterprises no longer operate in isolated environments. Most organizations now manage interconnected systems spanning IoT networks, enterprise applications, APIs, operational technology environments, cloud platforms, and legacy infrastructure. In such environments, isolated alerts provide limited value because operational disruption often emerges from cascading dependencies rather than a single infrastructure failure. Observability platforms address this challenge by correlating telemetry, metrics, traces, logs, and behavioral anomalies into unified operational intelligence layers. Instead of simply reporting that a service has failed, these systems analyze why the disruption occurred, which systems contributed to it, and how the issue may spread across dependent environments. Platforms such as Datadog, New Relic, and Dynatrace have become central to enterprises attempting to maintain high-availability infrastructure environments. Agentic Observability Is Introducing Autonomous Operations The latest evolution in observability is moving beyond monitoring toward autonomous operational investigation. Dynatrace’s Davis AI engine, for example, maps infrastructure dependencies continuously across cloud and on-premises ecosystems. Instead of overwhelming operations teams with fragmented alerts, the platform isolates probable root causes and predicts which infrastructure layers may destabilize next. Several enterprises are now moving toward what technology leaders describe as “agentic observability,” where AI systems autonomously investigate operational anomalies, correlate dependencies, recommend corrective action, and reduce the likelihood of SLA breaches before customers experience visible disruption. External observability platforms such as Site24x7 and UptimeRobot further strengthen operational assurance by validating customer-facing service availability across regions continuously. According to Gartner, as predictive root-cause analysis becomes more mature across enterprise infrastructure ecosystems, enterprises adopting AI-led operational intelligence frameworks help to reduce incident-resolution timelines. Why Incident Response Speed Has Become a Competitive Differentiator Even the most advanced predictive ecosystems cannot eliminate every operational incident. What increasingly separates high-performing service organizations from reactive operators is the speed and coordination of their response environments once disruption begins. Modern incident-management platforms are now heavily automated. Enterprises increasingly use AI-enabled response systems that identify affected services, create incident channels automatically, notify relevant engineers, and coordinate escalation processes in real time. Several operational capabilities now determine how effectively organizations respond to high-severity incidents in modern uptime environments. These include: Faster escalation reduces Mean Time to Resolution (MTTR) and minimizes SLA impact.Automated response coordination that prevents communication delays during outagesIntelligent alert routing to ensure that the right teams engage immediately.Slack-native response environments to improve collaboration across distributed teams.AI-driven incident workflows that reduce operational confusion during high-severity failures. Platforms such as PagerDuty, Rootly, FireHydrant, and incident.io are helping enterprises streamline incident coordination significantly across distributed operational environments. Uptime Architecture Is Becoming a Strategic Business Decision Many enterprises still approach disaster recovery as a secondary IT function rather than a central business-continuity strategy. That approach is becoming increasingly risky in sectors where even brief disruption can affect customer trust and SLA commitments. Modern uptime environments now depend heavily on resilience architecture designed to absorb disruption without affecting customer operations. Enterprises are therefore investing aggressively in multi-region infrastructure, failover environments, and redundancy frameworks intended to eliminate single points of failure. Several financial services firms and industrial infrastructure providers now operate active-active environments where workloads distribute simultaneously across multiple operational regions. If one region experiences instability, remaining infrastructure absorbs traffic automatically with minimal disruption. Recovery-as-Code Is Changing Disaster Recovery Planning Other organizations rely on active-passive models where secondary standby environments activate rapidly during outages. Large enterprises have also started adopting hybrid multi-cloud strategies involving combinations of AWS, Azure, and Google Cloud to reduce dependency on a single provider. Disaster recovery itself has evolved significantly over the last few years. Earlier recovery frameworks depended heavily on manual restoration processes, isolated backups, and infrastructure rebuilding exercises that often stretched across several hours. Modern recovery environments increasingly rely on software-driven replication and automated restoration systems. Infrastructure-as-Code frameworks such as Terraform and Pulumi now allow enterprises to recreate infrastructure environments programmatically. Platforms such as AWS Elastic Disaster Recovery and ControlMonkey are helping organizations replicate workloads, restore cloud configurations, and improve recovery consistency during failover scenarios. Enterprises increasingly design systems capable of functioning effectively even while failure conditions occur. Why Data Availability Has Become as Critical as Infrastructure Availability As service ecosystems become more dependent on real-time operational intelligence, enterprises are also discovering that uptime extends far beyond infrastructure resilience alone. Data availability now plays a key role in maintaining service continuity. In asset-intensive industries, operational environments depend heavily on uninterrupted access to telemetry streams, maintenance histories, customer records, compliance data, and software supply chains. A ransomware incident or corrupted recovery environment can affect service operations as severely as infrastructure failure itself. This explains why organizations are investing heavily in platforms such as Cohesity and Rubrik, which focus on rapid recovery, immutable backup environments, and zero-trust data resilience strategies. Similarly, JFrog has increasingly positioned software supply-chain availability as a critical reliability layer for enterprises managing continuous deployment environments. Chaos Engineering Is Moving into the Mainstream For years, organizations assumed failover systems would function correctly during outages simply because backup infrastructure existed architecturally. Recovery environments often failed under real-world pressure because teams had never tested them comprehensively. Chaos engineering emerged as a direct response to that gap. Platforms such as Gremlin and LitmusChaos deliberately simulate disruption scenarios inside controlled environments. Teams intentionally interrupt APIs, overload infrastructure layers, disable databases, and simulate cloud-region failures to evaluate whether resilience mechanisms function correctly under operational stress. Organizations operating large-scale digital infrastructure increasingly use controlled-failure testing to understand how systems behave during real outages rather than relying solely on theoretical resilience assumptions. The Operational Disciplines Separating Mature Reliability Teams from Reactive Service Organizations Organizations that consistently maintain high uptime rarely depend on infrastructure investment alone. Most high-performing service environments combine technology modernization with disciplined operational governance frameworks designed to reduce preventable disruption. Error Budgets Are Forcing Teams to Balance Innovation with Stability Modern Site Reliability Engineering (SRE) environments no longer chase unrealistic zero-downtime goals. Organizations define acceptable downtime thresholds and pause feature deployment if operational instability crosses predefined limits. Progressive Deployment Models Are Reducing Large-Scale Service Failures Many enterprises now use canary deployment strategies that release updates gradually across smaller user environments before full-scale deployment occurs. This allows organizations to isolate instability before broader infrastructure disruption affects customers. Blameless Post-Mortems Are Improving Long-Term Operational Maturity Several organizations have shifted away from punitive outage-review cultures because delayed escalation often worsens downtime impact. Blameless review frameworks encourage teams to identify missing safeguards and process weaknesses more transparently. Change-Freeze Windows Are Becoming Standard Across High-Risk Operations Industries operating under strict SLA commitments increasingly enforce no-change windows during high-volume transaction periods, financial closings, infrastructure migrations, or critical production cycles. Incident Command Structures Are Accelerating Crisis Coordination High-availability environments increasingly rely on predefined incident-response hierarchies involving technical leads, communication owners, escalation managers, and operational coordinators. Enterprises that consistently maintain high uptime typically treat governance maturity as seriously as infrastructure resilience. Operational discipline often determines whether advanced technology investments really deliver measurable reliability outcomes. Technologies Driving Predictive SLA Management The service industry is moving steadily toward operational environments where organizations can forecast SLA risk before customer disruption occurs. This transition is accelerating because enterprises now recognize that service continuity directly influences revenue stability, retention, and operational trust. Telemetry Analytics Is Helping Enterprises Detect Early-Stage Operational Instability Connected infrastructure environments continuously generate operational intelligence related to machine performance, infrastructure stress, transaction behavior, and service degradation patterns. AI-Led Anomaly Detection Is Improving Failure Prediction Accuracy Platforms such as Dynatrace, IBM Maximo Application Suite, and C3 AI now combine anomaly detection with machine-learning models capable of forecasting operational degradation across industrial systems. SLA Risk Scoring Models Are Changing Operational Decision-Making Solutions such as Sirion and Nobl9 increasingly combine telemetry analytics, infrastructure dependencies, incident history, and contractual thresholds to generate SLA breach probability scores. Predictive environments can now identify rising compliance risks a week to two before a potential SLA breach occurs. Workforce Orchestration Systems Are Improving First-Time Resolution Rates Modern field-service environments increasingly integrate AI-led dispatch intelligence with technician certification data, inventory systems, and asset history. This allows organizations to assign the most suitable technician with the right replacement components before service disruption expands further. The broader transition toward predictive SLA intelligence reflects a larger shift across the service industry. Organizations are gradually moving away from response-driven operations toward environments capable of identifying operational instability before customers experience visible disruption. The Future of Service Operations Will Depend on Prevention The digital transformation of the service industry extends far beyond automation or cloud migration. Organizations leading this transition increasingly combine connected telemetry ecosystems, AI-driven observability, predictive asset intelligence, resilient infrastructure architecture, workforce orchestration platforms, and operational governance frameworks into unified service environments designed around prevention rather than response. Historically, service organizations optimized for repair efficiency. The next generation of operational leaders is optimizing for disruption avoidance. Predictive intelligence, connected telemetry, and AI-led service orchestration are steadily becoming foundational requirements for enterprises operating large-scale asset-driven service ecosystems. Over the next few years, the competitive gap between service organizations will no longer depend solely on who resolves incidents faster. It will depend on which enterprises can predict operational instability earlier, coordinate response systems more intelligently, and prevent disruption before customers experience its impact. In industries where uptime increasingly shapes customer trust, contractual performance, and operational continuity simultaneously, prevention is steadily becoming the new benchmark for service excellence.

In modern application development, feature flags are the guardrails that keep experiments controlled and rollbacks safe when conditions shift. If feature flags act as the guardrails, observability provides the visibility: the headlights (traces), mirrors (logs), and dashboard instruments (metrics) that reveal what’s happening in the environment and how well a feature is performing. Together, feature flags and observability unlock powerful insights by correlating code changes with real-time system behavior. This combination reduces time-to-diagnosis and builds greater confidence when rolling out new features. In this post, we’ll walk through just how to add observability to a React Native application using LaunchDarkly’s observability SDK. To demonstrate the process, we’ll build on the PlusOne app, a simple counter app that includes increment (+1), reset, and error-triggering buttons. This lightweight demo provides a clean foundation to showcase how logs, traces, and errors can seamlessly flow into LaunchDarkly for monitoring and debugging. Prerequisites LaunchDarkly account. Sign up for a free one here.Visual Studio or another code editor of choice. All code from this tutorial can be found on GitHub. Setting Up Your Environment Before running a React Native app, make sure your development environment is set up correctly. You can find the full setup instructions for both Android and iOS here. In this tutorial, we'll be running iOS, but keep in mind Expo Orbit, the platform we'll be using to run our iOS simulator, requires both Xcode and Android Studio to be installed. After going through the instructions, you should have the following installed: Node JS (preferably via nvm)Watchman for file monitoringJDK via zulu package managerAndroid Studio. Don’t forget to set your Android_Home environment variablesXcode for the iOS simulatorCocoapods for iOS dependency managementExpo Orbit for running Expo apps on Android or iOS If you're using Android, don't forget to add your environment variables to bash or zsh profile. JavaScript export ANDROID_HOME=$HOME/Library/Android/sdk export PATH=$PATH:$ANDROID_HOME/emulator export PATH=$PATH:$ANDROID_HOME/platform-tools Starting Up the PlusOne App To get started, let’s clone the repo for the PlusOne app and run npm install to ensure the proper dependencies are present in our node_modules file. Clone the repo. JavaScript git clone https://github.com/arober39/PlusOne Install dependencies using npm. JavaScript cd PlusOne npm install We’ll also need to run both the prebuild command to generate the iOS file and the expo run command to run the iOS simulator. Prebuild for iOS. JavaScript npx expo prebuild Run expo app. JavaScript npm expo run:ios Now we can view the iOS app in the iPhone simulator using npm. JavaScript # iOS npm run ios # Android npm run android The app should look something like this: Feel free to interact with the app to ensure all is working as expected. As you can see in the code, we have three buttons: one that adds one to the displayed number, one to bring the count back to zero, and an intentional Error button to test error monitoring within the LaunchDarkly UI. JavaScript // app/index.tsx import { useState } from "react"; import { StyleSheet, Text, TouchableOpacity, View } from "react-native"; export default function Index() { const [count, setCount] = useState(0); const handleReset = () => setCount(0); const handleIncrement = () => setCount((prev) => prev + 1); const triggerRecordedError = () => { try { throw new Error("Simulated controlled error from Plus One app") } catch (e) { alert("You intentionally threw an error") } }; return ( <View style={styles.container}> <View style={styles.header}> <Text style={styles.headerText}>Plus One</Text> </View> <View style={styles.counterWrapper}> <Text style={styles.counterText}>{count}</Text> </View> <View style={styles.actionsRow}> <ButtonBox label="Reset" onPress={handleReset} /> <ButtonBox label="+1" onPress={handleIncrement} /> <ButtonBox label="Error" onPress={triggerRecordedError} /> </View> </View> ); } type ButtonBoxProps = { label: string; onPress: () => void; }; function ButtonBox({ label, onPress }: ButtonBoxProps) { return ( <TouchableOpacity onPress={onPress} style={styles.button} activeOpacity={0.8}> <Text style={styles.buttonText}>{label}</Text> </TouchableOpacity> ); } /* The rest of the application code */ Now that we have verified a working app, we can add observability support by downloading the observability React Native SDK. Install LaunchDarkly SDK dependencies. JavaScript npm install @launchdarkly/react-native-client-sdk npm install @launchdarkly/observability-react-native Next, you’ll need to initialize the React Native LD client in the app/_layout file. Replace the in the layout file by pasting the following code. JavaScript // app/_layout.tsx import { Observability } from '@launchdarkly/observability-react-native'; import { AutoEnvAttributes, LDOptions, LDProvider, ReactNativeLDClient } from '@launchdarkly/react-native-client-sdk'; import { Stack } from 'expo-router'; import { useEffect, useState } from 'react'; const options: LDOptions = { applicationInfo: { id: 'Plus-One', name: 'Sample Application', version: '1.0.0', versionName: 'v1', }, debug: true, plugins: [ new Observability({ serviceName: 'my-react-native-app', serviceVersion: '1.0.0', }) ], }; const userContext = { kind: 'user', key: 'test-hello' }; export default function RootLayout() { const [client, setClient] = useState<ReactNativeLDClient | null>(null); useEffect(() => { // Initialize client const featureClient = new ReactNativeLDClient( 'mob-abc123', AutoEnvAttributes.Enabled, options, ); featureClient.identify(userContext).catch((e: any) => console.log(e)); setClient(featureClient); // Cleanup function that runs when component unmounts return () => { featureClient.close(); }; }, []); if (!client) { return null; } return ( <LDProvider client={client}> <Stack /> </LDProvider> ); } First, we’re importing the Observability SDK as well as a few LD libraries to add options and attributes to the LD client. Initialized the SDK and plugin options.Defined the user context.Lastly, you initialized the client. Now that you have defined your LD React Native client, you can implement different observability methods within your application logic. We can do this by importing the LDObserve library in the app/_layout.tsx file. JavaScript import { LDObserve } from '@launchdarkly/observability-react-native'; Then, add the recordError() method within the triggerRecordedError function inside the app/_layout.tsx file. This will allow for error messages to be sent back to the LD UI. JavaScript const triggerRecordedError = () => { try { throw new Error("Simulated controlled error from Plus One app") } catch (e) { LDObserve.recordError(e as Error, {feature: "test-button"}) alert("You intentionally threw an error") } }; Before being able to receive data in the LD UI, you’ll need to add your mobile key to the React Native LD client, which can be found by logging in to the LD UI. Once logged in, tap the settings button at the bottom left. Navigate to the Projects page and click Create to create a new project. Define the new Project and click Create Project. Then, define the environment where you would like your data to be sent. Now, grab the mobile key by pressing the three dots for the environment and selecting the mobile key, which will copy the key to your keyboard. Then, add it to the app/_layout file. JavaScript const featureClient = new ReactNativeLDClient( ‘mob-abc123’, AutoEnvAttributes.Enabled, options, ); Finally, you can generate data by interacting with your app in the iOS app simulator. Feel free to restart the app to ensure data is displaying in real time. JavaScript npm expo run:ios Once you navigate back to the LD UI, you should be able to see the logs, traces, and errors under the Monitor section. Logs Traces Errors Conclusion In just a few minutes, we’ve taken the PlusOne React Native app from a simple counter to a fully observable application connected to LaunchDarkly. By setting up the SDK, initializing observability plugins, and recording errors, we now have a live feedback loop where application behavior is visible in the LaunchDarkly UI. This makes it far easier to diagnose issues, validate feature flag rollouts, and ensure smooth user experiences. Next Steps Looking ahead, there are many ways to expand on what we’ve built by including features like recording custom metrics and session replay, which provide even deeper insights into app behavior. By integrating observability at the foundation of your React Native projects, you equip your team with the clarity needed to debug faster, ship features more confidently, and deliver reliable experiences to your users. You can also read this article to learn more about observability and guarded releases.

Raw data doesn't win model competitions. Features do. And when your raw data is tens of billions of rows sitting across multiple sources, you can't afford to run pandas in a notebook and call it a day. In this tutorial, I'll walk through building a production-grade feature engineering pipeline on Azure Databricks using: Apache Spark for distributed transformation at scaleDelta Lake for reliable, versioned feature storage with ACID guaranteesMLflow for tracking feature pipeline runs, parameters, and the models trained on top of them The use case is a customer churn prediction system, but the patterns apply to any ML feature pipeline. Architecture Overview The pipeline follows the Medallion Architecture — a layered approach where data gets progressively cleaner and more feature-ready as it moves from Bronze to Silver to Gold. MLflow sits across all three layers, tracking every run. Pipeline Flow Layer Breakdown LayerDelta TableWhat happens hereTypical latencyBronzechurn.bronze.eventsRaw ingest, no transforms, append onlyMinutesSilverchurn.silver.customersDeduplication, null handling, schema enforcementMinutesGoldchurn.gold.featuresAggregations, window functions, encodingMinutes to hoursMLflow RunN/ATraining, metric logging, artifact storageHoursRegistryN/AVersioned model store, stage promotionOn demand Step 1 — Bronze Layer: Raw Ingest The Bronze layer is append-only. No transforms. No business logic. Just get the data in and preserve it exactly as it arrived so you can always replay from source. Python from pyspark.sql import SparkSession from pyspark.sql.functions import current_timestamp, lit from delta.tables import DeltaTable spark = SparkSession.builder.getOrCreate() # Read raw events from ADLS Gen2 / Event Hub / source of choice raw_events = spark.read.format('json').load('abfss://[email protected]/events/') # Add ingestion metadata — never mutate source columns bronze_df = raw_events.withColumn('_ingested_at', current_timestamp()) \ .withColumn('_source', lit('events_api')) # Write to Bronze Delta table — append only, no overwrites bronze_df.write \ .format('delta') \ .mode('append') \ .option('mergeSchema', 'true') \ .saveAsTable('churn.bronze.events') print(f"Bronze rows written: {bronze_df.count()}") Why append-only? If your downstream pipeline produces bad features, you want to replay from Bronze without re-ingesting from source. Overwriting Bronze breaks that ability. Step 2 — Silver Layer: Clean and Validate Silver is where you enforce schema, handle nulls, deduplicate, and standardize. Think of it as your canonical, trusted dataset. Python from pyspark.sql.functions import col, to_timestamp, when, trim, upper from delta.tables import DeltaTable bronze = spark.table('churn.bronze.events') silver_df = bronze \ .filter(col('customer_id').isNotNull()) \ .filter(col('event_type').isNotNull()) \ .dropDuplicates(['customer_id', 'event_id']) \ .withColumn('event_ts', to_timestamp(col('event_timestamp'))) \ .withColumn('event_type', upper(trim(col('event_type')))) \ .withColumn('country_code', when(col('country').isNull(), lit('UNKNOWN')) .otherwise(upper(col('country')))) \ .select( 'customer_id', 'event_id', 'event_type', 'event_ts', 'country_code', 'product_id', 'session_id', '_ingested_at', ) # Upsert into Silver using Delta MERGE — idempotent on re-runs if DeltaTable.isDeltaTable(spark, 'churn.silver.customers'): silver_table = DeltaTable.forName(spark, 'churn.silver.customers') silver_table.alias('tgt').merge( silver_df.alias('src'), 'tgt.customer_id = src.customer_id AND tgt.event_id = src.event_id' ).whenNotMatchedInsertAll().execute() else: silver_df.write.format('delta').saveAsTable('churn.silver.customers') print(f"Silver table updated. Total rows: {spark.table('churn.silver.customers').count()}") Step 3 — Gold Layer: Feature Engineering This is the heart of the pipeline. We compute aggregated, windowed, and encoded features that the model will actually train on. Python from pyspark.sql.functions import ( col, count, countDistinct, sum as _sum, avg, datediff, max as _max, min as _min, current_date, expr, when ) from pyspark.sql.window import Window silver = spark.table('churn.silver.customers') # ------------------------------------------------------------------ # 1. Aggregate features per customer over 30 / 90 day windows # ------------------------------------------------------------------ today = current_date() agg_features = silver \ .withColumn('days_since_event', datediff(today, col('event_ts'))) \ .groupBy('customer_id') \ .agg( count('event_id') .alias('total_events'), countDistinct('session_id') .alias('total_sessions'), countDistinct('product_id') .alias('distinct_products'), _sum(when(col('days_since_event') <= 30, 1).otherwise(0)) .alias('events_last_30d'), _sum(when(col('days_since_event') <= 90, 1).otherwise(0)) .alias('events_last_90d'), _max('event_ts') .alias('last_event_ts'), _min('event_ts') .alias('first_event_ts'), ) \ .withColumn('days_since_last_event', datediff(today, col('last_event_ts'))) \ .withColumn('customer_tenure_days', datediff(today, col('first_event_ts'))) \ .withColumn('avg_events_per_day', col('total_events') / (col('customer_tenure_days') + 1)) # ------------------------------------------------------------------ # 2. Encode churn risk tier as ordinal feature # ------------------------------------------------------------------ feature_df = agg_features \ .withColumn('recency_tier', when(col('days_since_last_event') <= 7, lit(3)) # active .when(col('days_since_last_event') <= 30, lit(2)) # at risk .otherwise(lit(1)) # churned ) \ .withColumn('engagement_score', (col('events_last_30d') * 0.6 + col('events_last_90d') * 0.4) / (col('customer_tenure_days') + 1) ) # ------------------------------------------------------------------ # 3. Write to Gold feature store — overwrite with partition by date # ------------------------------------------------------------------ feature_df \ .withColumn('feature_date', current_date()) \ .write \ .format('delta') \ .mode('overwrite') \ .option('replaceWhere', f"feature_date = '{today}'") \ .saveAsTable('churn.gold.features') print(f"Gold features written: {feature_df.count()} customers") Step 4 — MLflow: Track the Training Run With features in Gold, we hand off to MLflow to train, track, and register the model. Notice we log the Delta table version so we can always reproduce exactly which feature snapshot trained which model. Python import mlflow import mlflow.sklearn from mlflow.models.signature import infer_signature from sklearn.ensemble import GradientBoostingClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import roc_auc_score, f1_score import pandas as pd mlflow.set_experiment('/churn-prediction/feature-pipeline') # Read Gold features — capture Delta version for reproducibility gold_table = DeltaTable.forName(spark, 'churn.gold.features') delta_version = gold_table.history(1).select('version').collect()[0][0] features_pdf = spark.table('churn.gold.features').toPandas() FEATURE_COLS = [ 'total_events', 'total_sessions', 'distinct_products', 'events_last_30d', 'events_last_90d', 'days_since_last_event', 'customer_tenure_days', 'avg_events_per_day', 'recency_tier', 'engagement_score', ] TARGET = 'churned' X = features_pdf[FEATURE_COLS] y = features_pdf[TARGET] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) with mlflow.start_run(run_name=f'gbm-features-v{delta_version}') as run: params = {'n_estimators': 200, 'max_depth': 5, 'learning_rate': 0.05} model = GradientBoostingClassifier(**params, random_state=42) model.fit(X_train, y_train) y_pred = model.predict(X_test) y_prob = model.predict_proba(X_test)[:, 1] # Log everything mlflow.log_params(params) mlflow.log_metric('roc_auc', roc_auc_score(y_test, y_prob)) mlflow.log_metric('f1_score', f1_score(y_test, y_pred)) mlflow.log_param('delta_feature_version', delta_version) mlflow.log_param('feature_columns', FEATURE_COLS) mlflow.log_param('training_rows', len(X_train)) # Log model with signature signature = infer_signature(X_train, y_pred) mlflow.sklearn.log_model( model, artifact_path='churn-gbm', signature=signature, registered_model_name='churn-prediction-gbm', ) print(f"Run ID: {run.info.run_id}") print(f"ROC-AUC: {roc_auc_score(y_test, y_prob):.4f}") print(f"Feature Delta version logged: {delta_version}") Bonus: Delta Lake Time Travel for Feature Reproducibility One of the best things about Delta Lake is time travel. If a model behaves unexpectedly in production, you can reload the exact feature snapshot it was trained on. Python # Reload the exact feature version that trained a specific model run import mlflow run = mlflow.get_run('your-run-id-here') feature_version = int(run.data.params['delta_feature_version']) # Rehydrate that exact feature snapshot historical_features = spark.read \ .format('delta') \ .option('versionAsOf', feature_version) \ .table('churn.gold.features') print(f"Loaded feature snapshot from Delta version {feature_version}") print(f"Row count: {historical_features.count()}") # You can now retrain on the exact same data to reproduce the result Service Comparison ToolRole in pipelineWhy not the alternativeApache SparkDistributed feature computationPandas (single node, OOM at scale), Dask (less native Databricks integration)Delta LakeFeature storage with versioningParquet (no ACID, no time travel), Hive tables (no merge support)MLflow TrackingExperiment and param loggingManual logging (not reproducible), W&B (extra cost, less native on Databricks)MLflow RegistryModel versioning and promotionCustom model store (more ops overhead)Medallion ArchitecturePipeline layer separationFlat pipelines (hard to debug, no replay capability)Delta MERGEIdempotent Silver upsertsOverwrite (destroys history), append (creates duplicates) Things to Watch in Production Shuffle partitions matter. Spark defaults to 200 shuffle partitions, which is fine for small data but will bottleneck at scale. Set spark.conf.set("spark.sql.shuffle.partitions", "auto") on Databricks Runtime 10+ or tune it manually to 2-3x your core count. Z-ordering on Gold features. If you're querying Gold by customer_id frequently, add OPTIMIZE churn.gold.features ZORDER BY (customer_id) after the write. This co-locates related data and cuts query times dramatically on large tables. Log Delta version in every MLflow run. This is non-negotiable for reproducibility. Without it you can't prove which feature snapshot trained which model, which becomes a compliance problem in regulated industries. Cluster autoscaling for feature jobs. Feature engineering jobs tend to have spiky resource needs (big during aggregation, small during writes). Enable autoscaling on your Databricks cluster and set a min/max node count rather than a fixed size. Wrapping Up The combination of Spark, Delta Lake, and MLflow on Databricks gives you a feature engineering pipeline that is reproducible (Delta time travel + MLflow param logging), scalable (Spark handles billions of rows), and auditable (every run is tracked, every feature version is stored). The Medallion Architecture keeps the pipeline modular — you can rerun just the Gold layer if you change a feature definition without touching Bronze or Silver, and MLflow ties model performance back to the exact feature version that produced it. References Azure Databricks DocumentationDelta Lake — The Definitive GuideApache Spark SQL — Window FunctionsMLflow Tracking DocumentationMLflow Model RegistryMedallion Architecture on DatabricksDelta Lake Time TravelDatabricks Feature Store Overview

By Jubin Soni, FBCS CORE

Abstract Modern distributed systems rarely fail in isolation — they degrade across multiple execution steps. This article presents a control-loop-based architecture for building self-healing systems that detect anomalies early, precisely isolate failures, and automatically recover using context-aware decisions. Introduction Modern distributed systems are large-scale platforms built on service-oriented architecture. In such systems, an individual request — the unit of execution — typically flows through multiple services, including clients (request initiators), orchestrators, enrichment layers, validation or policy-evaluation systems, routing layers, downstream dependencies, state management systems, reconciliation processes, and notification systems. Each service in this chain introduces latency, retries, dependencies, and failure modes. Because of this, failures in distributed systems rarely appear as clean, isolated events. Instead, they emerge as a sequence of interacting issues that create a cascading effect across the system. For example, a downstream dependency may become slow in a specific region. This increases retries, which in turn increases queue depth. The growing queue depth puts pressure on the orchestrator, eventually causing it to fail unrelated requests due to resource saturation. What initially was a local dependency problem rapidly turned into a widespread degradation of workflow. This problem is particularly difficult in asynchronous systems, where failures are not always instantly visible. A request may not fail instantly — it may remain pending, miss its expected execution window, be delayed in execution, get stuck in an intermediate state, or lose coordination between system components. When the operator detects the issue, the impact could already be large enough. However, traditional protection mechanisms such as fixed failure thresholds, static alerts, and global circuit breakers are often too coarse-grained for these scenarios. A localized dependency failure should not halt the entire system. At the same time, localized issues must not be allowed to trigger storms or cascade into otherwise healthy execution paths. The goal, therefore, is to build a self-healing control system that can detect anomalies at the level of individual requests, aggregate signals across execution and system dimensions, isolate only the affected scope, and recover gradually based on real-time evidence. This post presents such a system. It is designed to provide predictive anomaly detection, hierarchical aggregation, scoped and global kill switches, adaptive leaky-bucket flow control, observability, and AI-assisted investigation and escalation. featurestatic thresholds (old way)context-aware loops (new way)DetectionStatic ThresholdingPredictive Anomaly DetectionContainmentGlobalScopedControlBinary ShutdownAdaptive Flow ControlRecoveryManualEvidence-Based Self-Healing Why Traditional Systems With Static Thresholds Won’t Work Most distributed systems rely on mechanisms like retries, dead-letter queues, alerts, and circuit breakers. These are useful but not enough for complex async workflows as they depend on static thresholds, which are context-blind by nature. A rule like “trigger an alert when failures exceed X%” cannot distinguish between fundamentally different types of failures: Logical failures, where a request completes but produces an incorrect result due to issues in input, configuration, or application logic Execution failures, where a request produces no result due to delays, retries, or loss of coordination across system components For example, in an AI inference system, a request may return an incorrect response due to model configuration issues (logical failure), or it may be accepted but never complete due to stalled execution in downstream components (execution failure). Static thresholds treat both cases uniformly, even though they require very different responses. As a result, systems either overreact to expected failures or miss critical anomalies such as stuck or silently failing requests. Failure volume alone is also a weak signal. A small number of failures could be highly significant if those requests were anticipated to be successful. For instance, if requests following the same execution path have historically resulted in high reliability, even a few failures in that cohort can imply a serious issue. Static thresholds also lack scope awareness. A local failure example, requests routed through a particular execution path, dependency, or region, should not cause a global shutdown. However, a pattern of small anomalies across different paths, regions, or request classes could indicate a larger systemic problem, even if no single threshold is crossed. For instance, in an inference system, requests served by a specific model variant may observe increased latency or degraded outputs due to recent changes to configurations or parameters, while other models and request paths continue to function normally. These limitations are amplified in asynchronous systems, where failures are not always specific. Coordination gaps can cause requests to be stuck, delayed, retried multiple times, or enter into inconsistent states. This leads to higher latency, missed completion signals, or repeated retries with no progress. These weaknesses are further revealed during recovery. AI Agents or operators have to manually inspect logs and dashboards to determine when to resume traffic, resulting in inconsistent performance, slowness, and reactive recovery. In summary, these challenges demonstrate that static thresholding is not sufficient for modern distributed systems. What is needed is a system that understands request context, expected behavior, and the scope of the anomaly. This leads to a fundamental shift in system design: Static thresholding → Predictive anomaly detection Global containment → Scoped containment Binary shutdown → Adaptive flow control Manual recovery → Evidence-based self-healing Instead of asking: Are requests failing? The system should ask: Are requests behaving as expected within their defined SLA, given their execution context and expected outcomes? System Architecture as a Control Loop The system functions as a control loop during request execution. It does not replace the execution path. Instead, it constantly monitors the system's behavior, predicts expected outcomes, identifies deviations, and makes control decisions based on real-time signals. Orchestrated Execution With Continuous Monitoring A primary orchestrator drives the system. It executes each request through a series of steps. At each step, the orchestrator calls on one or more downstream systems, either synchronously or asynchronously. These downstream systems may have their own dependencies. As the request moves forward, it carries contextual metadata like tenant class, region, request type, execution path, and routing decisions. This context defines how the request should behave at each step or at a specific point. While the orchestrator manages execution, anomaly detection serves as a continuous control layer throughout these steps. It tracks the outcome of each phase to ensure that the request moves forward as expected and that the contextual integrity remains intact. Context Preservation and Signal Collection At every step, the system captures signals such as latency, retries, routing decisions, execution status, and downstream responses. It also augments the request with derived attributes such as execution path identifiers and historical behavior patterns. This ensures that each request is evaluated relative to similar cohorts, and more importantly, allows the system to identify where deviations occur within the execution flow — not just whether the request ultimately fails. Success Prediction Engine Intuition: The system learns what 'normal' looks like for similar requests and uses that to estimate expected outcomes. The system estimates how likely a request is to succeed based on its context and historical behavior. For each request i, the expected success is computed as: Plain Text P_i = P(success | x_i) Where: x_i = request features (context, routing path, system state) P_i = expected probability of success This establishes what should happen at different stages of execution, allowing the system to detect deviations between expected and actual outcomes throughout the request lifecycle. Step-Level Anomaly Detection Unlike traditional systems that evaluate only final success or failure, this system continuously monitors each critical step of execution. A request may: Be accepted but delayed Be routed to an unexpected path Experience retries at a specific step Produce degraded output Fail to progress beyond a step By evaluating these signals against expected behavior for that request’s context, the system can detect anomalies early and pinpoint the exact step where deviation occurs. Inference Example (Grounding) For example, in an inference system, the orchestrator can direct a request from a certain tenant class to a summarization model in a certain subnet of a region. If that subnet/region experiences network latency, requests may still be accepted and processed, but exhibit higher latency or delayed responses. In this case, the orchestrator continues execution, but a specific step — model execution in that region — is deviating from expected behavior. Other models or regions may continue to function normally. Hierarchical Roll-up Counters The hierarchical roll-up model aggregates anomalies across multiple contextual dimensions. When a request deviates from expected behavior at any step, the system updates counters across relevant dimensions such as dependency, execution path, tenant class, and region. Example roll-ups: Plain Text (dependency, request_type) (dependency,request_type, tenant_class) (dependency, region) (execution_path, request_type) (global) A single anomalous request may update multiple roll-ups simultaneously. For example, a request routed to a summarization model in a latency-affected region may update: Plain Text (summarizer_model, tenant_class_A, region_us_west) (summarizer_model, region_us_west) (summarizer_model, tenant_class_A) (global) This multi-dimensional view allows the system to isolate issues precisely while still capturing broader systemic patterns. Roll-Up Configuration Model Each roll-up is independently configurable, allowing the system to adapt thresholds and behavior based on the criticality of different execution paths and request classes. Example configuration: JSON { "roll-up_id": "dependency_request_type_region", "dimensions": ["dependency", "request_type", "region"], "threshold": 25, "tumbling_window": "30m", "parent_roll-up_ids": [ "dependency_region", "dependency_request_type", "dependency", "global" ], "control_action": "HOLD_AND_PROBE" } Key Fields dimensions → define how the rollup key is constructed threshold → anomaly count required to trigger tumbling_window → fixed evaluation window (e.g., 30 minutes) parent_rollup_ids → defines relationships across rollups control_action → action applied when this rollup becomes the resolved scope Hierarchical Rollup Model (DAG) The hierarchy is modeled as a directed acyclic graph (DAG). This allows a granular rollup to contribute to multiple parent views. For example: Plain Text (dependency=D1, request_type=TYPE_A, region=EU) → (dependency=D1, region=EU) → (dependency=D1, request_type=TYPE_A) → (dependency=D1) → (global) A single anomalous request may update multiple rollups simultaneously, including both child and parent scopes. Rollup Runtime State At runtime, each rollup key maintains its own state within a tumbling window: Plain Text Rollup: (dependency, region) Key: D1:EU Window: 30 mins Anomaly Count: 35 Threshold: 25 → FIRED Each rollup evaluates independently: A child rollup may fire without the parent firing A parent rollup may fire when anomalies are distributed across multiple children Parent Roll-up Escalation Guard Since parent roll-ups aggregate signals, the system must prevent escalation caused by a single noisy child. Instead of maintaining a full child-level state, each parent tracks lightweight signals: parent_anomaly_countimpacted_child_countmax_child_contribution_ratio A parent roll-up is considered impacted only when: Plain Text parent_anomaly_count >= parent_threshold AND impacted_child_count >= min_required_children AND max_child_contribution_ratio <= max_allowed_ratio Example: Do not escalate at the parent level if only the request Type_A is failing. Plain Text TYPE_A = 100 anomalies TYPE_B = 0 TYPE_C = 0 Parent count = 100 Impacted children = 1 → Keep control at child level Example: Escalate. Plain Text TYPE_A = 40 TYPE_B = 35 TYPE_C = 25 Parent count = 100 Impacted children = 3 → Escalate to parent scope Why This Matters This ensures: Localized issues remain scoped Distributed anomalies are escalated correctly. Noisy signals do not trigger unnecessary global actions Anomaly Detection Engine The anomaly detection engine identifies unexpected deviations by comparing predicted outcomes and actual results and propagates these signals to rollup counters. A request is marked anomalous only if it was expected to succeed but deviates from expected behavior: Plain Text Anomaly_i = 1 if P_i ≥ τ AND Y_i deviates from expected outcome Where: Pi = predicted success probability Yi = observed outcome (failure, delay, degraded output, etc.) Each anomalous request updates multiple rollups across dimensions such as dependency, region, request type, and tenant class. The system evaluates all rollups that breach their thresholds and resolves the appropriate control scope. It then: Deduplicates overlapping signals Selects the highest meaningful level in the hierarchy Avoids redundant or conflicting controls This ensures: Localized issues remain scoped Correlated anomalies are elevated appropriately Duplicate control actions are avoided Kill Switch Controller The kill switch controller enforces control actions at the resolved anomaly scope. Based on severity and scope, it determines whether to: Stop new incoming requests within the scope Hold in-progress requests before critical downstream steps Allow controlled traffic via throttling or probing Control Actions Plain Text ALLOW → continue processing HOLD → pause new and in-progress requests THROTTLE → limit request rate PROBE → allow controlled traffic REROUTE → send via alternate path ESCALATE → trigger alerts / human intervention The controller applies actions consistently across the resolved scope, ensuring full containment without partial or conflicting behavior. Adaptive Recovery Strategy Once a control action is applied, the system does not immediately resume normal traffic. Instead, it gradually reintroduces traffic using a probing strategy. For example: Plain Text Step 1: allow 1 request Step 2: if successful (actual outcome == predicted outcome, allow 2 Step 3: if stable, allow 5 Step 4: gradually increase Step 5: if failures reappear, reduce or stop Recovery is guided by: Plain Text Recovery_G = Successful_G / Released_G Where: G = impacted roll-up scope This ensures: Safe and gradual recovery Avoidance of sudden failure spikes Validation of real system behavior Observability and Audit Layer The system captures all signals across execution: Predicted outcome Actual outcome Anomaly classification Impacted rollups Resolved scope Control action Recovery state These signals provide visibility into: Anomaly trends Active control scopes Held vs released requests Recovery progress This ensures full transparency, debuggability, and auditability. AI Control Plane The AI control plane operates outside the execution path and complements deterministic control logic. It consumes: Anomaly signals Roll-ups Deployment changes System health Control decisions It performs: Investigation → correlates anomalies with systems or changes Automated remediation → triggers safe rollback Escalation → notifies relevant teams Summarization → generates incident insights Key Separation Plain Text Decision Plane → deterministic (prediction, anomaly detection, control) AI Control Plane → intelligent (analysis, remediation, escalation) Conclusion Modern distributed systems cannot rely on static thresholds and reactive controls. Failures are often contextual, asynchronous, and distributed across multiple execution paths. This architecture introduces a fundamental shift: From failure counting → context-aware detection From global shutdown → scoped containment From reactive response → adaptive, evidence-based recovery By combining prediction, hierarchical rollups, scoped control, and adaptive recovery, the system can precisely isolate deviations, minimize impact, and restore stability safely. The core idea is simple but powerful: Systems should not just detect failures — they should continuously understand system behavior, localize deviations in context, and adapt in real time to maintain reliability. What’s Next: From Architecture to Code Designing the architecture is only the first step. In the next post, we move from the blueprint to the technical implementation, diving deep into: The State Machine: Managing high-cardinality counters without latency and affecting execution path.The Escalation Guard: Pseudo-code to prevent "noisy neighbor" failures.Adaptive Recovery: The logarithmic logic for safe traffic re-introduction. Stay tuned for the implementation deep-dive. Case Study: Applying the Control Loop to a Multi-Region Inference System End-to-end Example: Inference system with scoped control and adaptive recovery This example illustrates how anomalies propagate, how scope is resolved, and how control and recovery are applied in an inference system. Step 1: Incoming Requests Requests are routed by the orchestrator to model services in the DUB region: Plain Text (model=summarizer_v2, tenant_class=A, region=DUB) (model=translator_v1, tenant_class=A, region=DUB) (model=qa_model_v3, tenant_class=A, region=DUB) Predicted success: Pi≈0.95+ Step 2: Deviations → Anomalies Due to network degradation in DUB, requests begin to show: increased latency delayed responses occasional degraded outputs Yi deviates and Pi≥τ⇒Anomalyi=1Y_i \text{ deviates and } P_i \geq \tau \Rightarrow Anomaly_i = 1. Step 3: Roll-up Updates Each anomalous request updates multiple rollups: Plain Text (summarizer_v2, tenant=A, DUB) → 40 (translator_v1, tenant=A, DUB) → 35 (qa_model_v3, tenant=A, DUB) → 25 (region=DUB) → 100 Step 4: Parent Escalation Guard Plain Text parent_count = 100 impacted_child_count = 3 max_child_ratio ≈ 40% Since anomalies are distributed across multiple models, not concentrated in one: Plain Text → Escalate to (region=DUB) Step 5: Impact Resolution Fired roll-ups: Plain Text (summarizer_v2, tenant=A, DUB) (translator_v1, tenant=A, DUB) (qa_model_v3, tenant=A, DUB) (region=DUB) Resolved scope: Plain Text (region=DUB) Child rollups are de-duplicated and consolidated under the parent scope. Step 6: Control (Scoped Isolation + Reroute + Local Probing) Action: Plain Text HOLD_AND_PROBE + REROUTE Effect: Throttle or hold most requests routed to DUB Reroute the majority of traffic to FRA only after verifying that the region has sufficient available capacity and is operating within stable limits.Allow a small number of low-impact requests to continue via DUB as probes These probe requests validate whether the issue is transient or persistent without exposing the system to large-scale risk. Step 7: Adaptive Recovery Traffic is managed dynamically: Plain Text DUB (probe path): 1 → 2 → 5 → gradual increase FRA (rerouted path): handles majority of traffic Recovery signal: RecoveryG = SuccessfulGReleasedGRecovery_G = \frac{Successful_G}{Released_G} If probe requests via DUB succeed → gradually restore DUB traffic If failures persist → continue routing to FRA and reduce DUB probes Step 8: AI Control Plane Based on observed signals: Regional network issue → continue routing to FRA Model deployment issue → rollback model version Infrastructure saturation → rebalance across regions Transient degradation → generate summary without escalation Key Takeaways Failures are localized but distributed across modelsControl is applied at the correct scope (region-level)System avoids global shutdownRecovery is validated through controlled probingTraffic is dynamically rerouted and restored The system does not simply stop traffic-it isolates the impacted scope, reroutes intelligently, and verifies recovery through controlled probing before storing normal behavior.

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.

Picture this: two features are being developed in parallel. One has already been tested in lower environments, but is still awaiting business approvalThe other is fully validated and ready to go live Naturally, you want to release the second feature to production. But you can’t, because your deployment model forces you to release everything together. If you’ve worked with Azure Data Factory (ADF), this situation probably sounds familiar. Azure Data Factory (ADF) is a cloud-based data integration service from Microsoft that helps you build and orchestrate data pipelines across systems. It works extremely well for managing data workflows — but when it comes to deployments at scale, things get tricky. As our ADF usage grew across multiple teams and environments, we started running into a recurring problem: We had control over development — but very little control over what actually got deployedA simple pipeline fix could unintentionally introduce unrelated changesParallel feature development became harder to manageProduction releases became riskier than they needed to be That’s when we realized: The issue wasn’t ADF itself — it was the deployment model we were relying on. The issue wasn’t ADF itself — it was the deployment model we were relying on. This article walks through how we addressed that challenge by implementing a selective deployment pattern, allowing us to promote only intended changes without impacting everything else. The Real Problem: Parallel Feature Releases in ADF Before diving into the solution, let’s look at a scenario that frequently occurs in real-world teams. What This Diagram Represents This diagram shows two features progressing across environments: Feature 100 Developed earlier, successfully deployed to Dev and TestCurrently in UAT (User Acceptance Testing)Still awaiting business approval before production Feature 200 Developed later, successfully completed across Dev → Test → UATFully validated and ready for production Expected Behavior At this stage, the expectation is straightforward: “Let’s release Feature 200 to production.” Feature 100 is still under testing, so it should remain in UAT. What Actually Happens in ADF Azure Data Factory follows a full-state deployment model. That means when you deploy, you are not deploying a feature; you are deploying the entire factory state. So when you attempt to release Feature 200: Feature 100 gets included automaticallyYou cannot isolate Feature 200You lose control over what reaches production Why This Becomes a Real Problem This isn’t an edge case; it becomes a recurring pattern in larger environments. You’ll encounter this when: Multiple teams are working in parallelFeatures move at different speedsUAT cycles varyProduction fixes need to be released quickly It becomes even more complex when: Existing production pipelines are modifiedPartial updates are requiredDependencies overlap across features The Core Limitation: ADF promotes state, not intent. It does not differentiate between what is ready for production and what is still under testing. Why We Had to Rethink Deployment This limitation introduced real risks: Accidental promotion of incomplete featuresDelayed production releasesIncreased coordination overheadHigher chances of breaking stable pipelines We needed a way to: Promote only Feature 200Keep Feature 100 in UATAvoid impacting unrelated artifactsReduce production risk Architecture Overview To address this challenge, we introduced a selective packaging layer between build and deployment. Flow Feature Branch → PR → Validate → Selective Packaging → ARM Export → Incremental Deploy → Trigger Control Key Idea: Instead of exporting ARM templates from the full ADF repository, we export from a filtered staging folder containing only the required artifacts. Understanding Default ADF Deployment Behavior Before implementing selective deployment, it’s important to understand how Azure Data Factory works by default. ADF follows a full-state deployment model. How Default ADF Deployment Works When you use ADF with Git integration: Developers work in a collaboration branch (typically main)Changes are committed and merged via pull requestsADF provides a Publish button in the UI When you click Publish, ADF generates ARM templates representing the entire factory state. These templates are stored in the adf_publish branch: In modern setups, instead of clicking Publish manually, teams often use @microsoft/azure-data-factory-utilities (npm-based export). This allows pipelines to validate ADF resources and export ARM templates programmatically. YAML - name: Validate ADF resources run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.SOURCE_FACTORY_NAME }" npm run build validate "${{ github.workspace }" "$FACTORY_ID" YAML - name: Export ARM templates (CI publish) run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.DEV_FACTORY_NAME }" npm run build export "${{ github.workspace }" "$FACTORY_ID" "${{ env.ARM_OUTPUT_DIR }" Whether you click Publish manually or use npm export in CI/CD, the outcome is the same: Full factory deploymentNo control over individual featuresAll changes get bundled together Selective Deployment Layer (Core Design) We can address this requirement and the associated challenges by introducing a workflow driven by a manifest to define the deployment scope, and a program to identify all necessary ADF dependencies for each manifest file. As a developer, I can now control which release is promoted to production, without worrying about releasing any other features that are not ready. The manifest controls which pipelines to deploy and which optional categories to include. Below is an example of a manifest file JSON { "pipelines": ["pl_ingest_population_selective"], "includeTriggers": false, "includeIntegrationRuntimes": false, "includeAllGlobalParameters": true, "includeLinkedServices": true, "validateLinkedServicesExist": true, "includeManagedVirtualNetwork": false, "includeManagedPrivateEndpoints": false } Workflow Explanation Let's understand the crux of the selective deployment workflow now. I am working in the release branch on my feature branch directly in ADF Studio. Since ADF Studio is integrated with Git, my development changes will be saved to my branch. Here are the steps I can take to promote my change to a higher environment. 1) Validation of ADF on PR validation This is an early validation step and a guardrail: if the PR fails, it's because objects are invalid and misaligned. This is equivalent to the "validation all" button in the ADF ui, here is this workflow Trigger: Pull requests targeting the branch selective_deployment. Purpose: Validate that the ADF JSON in the PR is valid in the context of the target factory. Main steps: CheckoutSet up Node.js 20npm installAzure login using OIDC (azure/login@v2)Validate with ADF Utilities: YAML FACTORY_ID="/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${DEV_FACTORY_NAME}" npm run build validate "$GITHUB_WORKSPACE" "$FACTORY_ID" 2) Release build + selective deploy to DEV adf-release-build-selective-deploy.yml Triggers: Push to selective_deploymentManual run (workflow_dispatch) with optional manifest inputDefault: deploy/manifests/release.json This workflow has two jobs: Job A: adf-build (staging + export + sanitize + artifacts) Checkout (full history)Azure login using OIDCSet up Node.js 20Install build dependencies inside build/ (npm install in build)Stage selective subset python scripts/select_adf_subset.py <manifest>, a code snippet below for the complete script, refer to the GitHub repository link given Python import json import re import shutil import sys from pathlib import Path from typing import Dict, Set, Tuple, List from collections import defaultdict # Your repo layout has pipeline/, dataset/, linkedService/ at ROOT. REPO_ROOT = Path(".") STAGE_ROOT = Path("build/adf_subset") RESOURCE_DIRS = { "pipeline": REPO_ROOT / "pipeline", "dataset": REPO_ROOT / "dataset", "linkedService": REPO_ROOT / "linkedService", "dataflow": REPO_ROOT / "dataflow", "trigger": REPO_ROOT / "trigger", "integrationRuntime": REPO_ROOT / "integrationRuntime", "credential": REPO_ROOT / "credential", "managedVirtualNetwork": REPO_ROOT / "managedVirtualNetwork", } # Copy these if present so ADF utilities behave the same on staged subset. ROOT_FILES_TO_COPY = [ "publish_config.json", "arm-template-parameters-definition.json", "arm_template_parameters-definition.json", "package.json", "package-lock.json", ] Produces: build/adf_subset/ (staged tree)build/adf_subset_report.json (dependency report)Refer to logs below (showing output of stage selective subset and debug to view output generated after select_adf_subset.py )Export ARM templates from the staged subset via ADF Utilities: npm --prefix build run build -- export "adf_subset" "$FACTORY_ID" "ArmTemplate"Produces: build/ArmTemplate/ARMTemplateForFactory.jsonbuild/ArmTemplate/ARMTemplateParametersForFactory.jsonStrip infra-owned resources scripts/strip_arm_resources.py to produce a safe template: build/ArmTemplate/ARMTemplateForFactory.safe.json⚠️ Note on Infrastructure Components (Refer to the “Future Work & Next Steps” section for follow-up topics in this series) The step above intentionally strips infrastructure-dependent components from the generated subset to avoid overwriting existing shared resources such as linked services. This implementation focuses on developer-owned artifacts (pipelines, datasets, and triggers) and assumes that infrastructure components — such as Integration Runtimes, managed private endpoints, and linked services — are pre-provisioned and managed outside of this deployment workflow.Upload artifacts: ARM templates (adf-arm)metadata (adf-release-meta)subset report (adf-subset-report) Job B: deploy_dev (deploy safe template) Download ARM artifactAzure login using OIDCEnsure az Data Factory extension is installedValidate JSON files exist/parseDeploy via azure/arm-deploy@v2(Incremental) to DEV RG/factory: Template: ARMTemplateForFactory.safe.jsonParameters: ARMTemplateParametersForFactory.json + factoryName=<DEV_FACTORY_NAME> Lesson Learned Setting up selective deployment in ADF was more than a technical task. It made us rethink our approach to deployments, ownership, and CI/CD design. Here are the main things we learned: 1. The Problem Is Not Tooling; It’s Deployment Granularity At first, we thought the limitation came from the tools we used, like UI publish or npm export. However, both methods yielded the same result: full factory templates. The real problem was that we couldn’t control the scope of deployments, not how the templates were made. 2. Dependency Awareness Is Critical Selective deployment only works when every dependency is found and included. We learned that: Pipelines often reference multiple datasets and linked services. Missing even one dependency results in deployment failure You must automate dependency discovery. 3. “Incremental” Is Often Misunderstood Incremental deployment is important, but it doesn’t work like a patch. It reapplies the full configuration for all included resources. This means: Your generated templates need to be complete for all the artifacts you include. If you use partial definitions, deployments can fail. 4. Separation of Concerns Is Key Not all ADF artifacts are the same. We began to separate them into different groups: Application-owned artifacts: pipelines, datasets, triggers Infrastructure-owned artifacts: linked service, managed virtual networks, managed private endpoints, and integration-runtime, among others. This separation proved crucial for safe, scalable deployments. 5. Selective Deployment Adds Complexity, But It’s Worth It It’s true that implementing this approach brings in additional scripts, manifest management, and CI/CD complexity. But in exchange, we gained precise control over releases, reduced production risk, and faster hotfix deployments. Future Work and Next Steps While selective deployment solved a major gap in ADF CI/CD, it also opened up new areas for improvement and standardization. 1. Defining Infrastructure vs Application Ownership One of the biggest follow-up areas is clearly defining ownership boundaries. In our experience: Application teams should own pipelines, datasets, and triggers Platform or infrastructure teams should own linked services, managed virtual networks, and managed private endpoints, among other things. Future work can focus on: Enforcing this separation in CI/CD. Preventing accidental deployment of infrastructure components Integrating Terraform or platform pipelines for infrastructure provisioning 2. Governance Around Linked Services Linked services are often shared across multiple pipelines and teams. Future improvements include: Centralizing linked service management Using Key Vault and Managed Identity consistently Preventing direct modifications through application pipelines

The Problem Azure AI Foundry has a genuinely great portal. You can see your agent runs, the tools it calls, the messages it sends and receives, and even a breakdown of token usage — all in a clean UI. But here's what actually happens when you're building an agent locally: Write some code, trigger a runSwitch to the browser, open the Foundry portalNavigate to your project → your agent → Traces tabFind the right runClick through to see what happenedSwitch back to VS Code to make a fixRepeat That context switch sounds minor. But when you're iterating fast — tweaking a system prompt, adjusting tool call logic, debugging why an agent handed off to the wrong sub-agent — it adds up. You're constantly pulling your attention out of your editor and into the browser and back again. What I wanted was simple: see the trace right where I'm working. What Foundry Trace Inspector Does The extension connects to your Azure AI Foundry project and gives you three views for every agent run, all inside a VS Code panel: Trajectories: The Full Span Tree A Gantt-style collapsible tree showing the full execution: Session → Invoke Agent → Chat turns → Tool calls. Every span shows duration, token counts, and cost. Click any span to open a detail drawer with the model, status, token breakdown, and raw input/output. Duration Per-span timing bars — see exactly how long each step took. Tokens Input vs output token breakdown per span. This is the view I use most during debugging. At a glance, I can see: did the tool call happen? How long did it take? What did the LLM actually receive as input? User View: Readable Conversation Replay A chat-bubble timeline of the full conversation: user messages and assistant replies rendered the way a human reads them, with the agent name and model on each assistant turn. Each assistant bubble has a "View Trace" button that jumps directly to the corresponding response in the sidebar — so you can go from "something looked off in this reply" to the raw span in one click. Token and Cost Chart A stacked bar chart (input vs output tokens per LLM turn) so you can instantly spot which turns are burning the most tokens — useful when you're trying to understand why a multi-turn conversation is getting expensive. Per span cost breakdown for both input and output tokens consumed. How It Works Under the Hood Azure AI Foundry agents use the OpenAI Responses API internally. Every agent reply produces a resp_... response ID that's visible in the Foundry portal's Traces tab. The extension fetches those responses directly via the same API and reconstructs the full conversation timeline locally. When a session spans multiple turns, each response links to the previous one via previous_response_id. Load any response in the chain and the extension walks the chain automatically — you don't need to manually track down every ID. Conversation IDs (conv_...) are discovered automatically from your saved responses, so once you track one response, the whole conversation surfaces. No intermediate server. The extension makes API calls only to the Azure endpoint you configure. Your API key is stored in VS Code's encrypted SecretStorage — it never touches settings.json and never leaves your machine. Setting It Up You need two things: An Azure AI Foundry project endpoint URL (found in the Foundry portal under your project → Overview)Either an API key or Azure CLI auth (az login) via DefaultAzureCredential Once configured, grab a conv_... conversation ID from the portal's Traces tab, paste it into the sidebar, and the extension fetches all responses in that conversation automatically. What's Next A few things I want to add in v0.2: Auto-discovery of recent runs – instead of pasting IDs manually, list recent conversations directly from the panelSide-by-side diff – compare two runs of the same agent to see what changed between runsExport to Markdown – generate a readable trace report you can paste into a PR or incident note Further Reading What is Foundry Agent Service? – official overview of the service this extension connects toUse the Azure OpenAI Responses API – the underlying API the extension fetches trace data fromMicrosoft Foundry Pricing – understand what your agents actually cost to runVS Code Webview API – how the timeline panel is builtVS Code Extension API – full reference if you want to contribute or build on top of this

By Jubin Soni, FBCS CORE

Every observability vendor's roadmap right now includes some version of "AI-powered insights." Smarter dashboards, with an assistant bolted on, to help you make sense of the data faster. That's not what developers are asking for. Nobody opens a laptop hoping for a better dashboard. What they're actually hoping for is a system that goes from bug to fix on its own, so their job shifts from digging through logs at 3 a.m. to something that actually uses their judgment: governing outcomes, managing risk, deciding which fixes get shipped and which need a second look. That idea of self-healing software isn't new. IBM coined the term in 2001 with the vision formalized into a loop: monitor, analyze, plan, execute. For two decades, only the first and last steps were actually automated. Analyzing why something broke and planning a fix for it requires judgment, and that's always been a human job. AI coding agents are the first real candidates to take it on. This article looks at what that actually means in practice and what has to change before AI agents can close the bug-to-fix loop for real. The Self-Healing Loop, Only Half Solved Infrastructure heals itself constantly. Kubernetes restarts what crashes. Autoscalers add capacity. Circuit breakers fail over. Nobody gets paged for any of it. Graceful degradation, resilient architecture, automated failover: these ideas are so built into how we expect distributed systems to behave that it's easy to forget they weren't always there. "Self-healing" was formally introduced by IBM in 2001, when Paul Horn proposed systems that could regulate themselves the way the human autonomic nervous system does: automatically, without conscious thought. “An autonomic computing system must perform something akin to healing — it must be able to recover from routine and extraordinary events that might cause some of its parts to malfunction. It must be able to discover problems or potential problems, then find an alternate way of using resources or reconfiguring the system to keep functioning smoothly.” The paper itself acknowledges that the easy part was already solved even in 2001: "certain types of 'healing' have been a part of computing for some time. Error checking and correction […] and redundant storage systems like RAID allow data to be recovered even when parts of the storage system fail." In other words, IBM already knew the hard part would be root cause analysis: figuring out what actually broke and why, not just recovering from the fact that something did. These principles were eventually formalized into the MAPE-K loop: Monitor, Analyze, Plan, Execute, all running against a shared Knowledge base. It became the reference model for how a self-managing system should behave. Two and a half decades later, half of that loop is a solved problem. Monitoring and executing are largely mechanical: detect a deviation, run a predefined response. The infrastructure layer works so well that we've stopped calling it "self-healing" at all. It's just how systems behave now. The other half was, and still is, the hard part. Analyzing why something broke and planning what to do about it requires reasoning about what a system is supposed to do, not just whether it's currently running. For application-level bugs, that reasoning has always required a person. No amount of infrastructure automation changes the fact that someone still has to figure out why the checkout flow is returning the wrong total. With AI coding agents, we finally have the first credible candidate to take on the analysis and the planning. Closing the Loop Here's what fixing a bug actually looks like for most developers today. An alert fires in PagerDuty or Slack. You open your APM (Datadog, New Relic, ...) and start hunting for the error. Once you find it, you switch to logs, search for the request ID, and start piecing together what happened. From there, it's traces: open Tempo or Jaeger, scroll through 200+ spans looking for the one that matters. By now, you've switched tools four times, and you still don't have a fix. You move to your IDE, run git blame to figure out who touched this code last and why, form a theory about what's actually wrong, and finally try something. Five tools. Eleven steps. Four hours. And that's the good outcome, where the fix on the first attempt is the right one. This is the loop that "AI-powered observability" claims to close. Bolt an agent onto the APM, give it access to the logs and traces, and, in principle, the agent does steps 2 through 10, and a developer just reviews step 11. In practice, this doesn't close the loop. It automates a workflow that was designed for humans and not agents (and that matters greatly). Every step in that staircase exists because the data needed for the next step lives somewhere else, in a different tool, with a different data model, often with no shared identifier connecting them. A human bridges these gaps with intuition: they know, roughly, what an error in the APM probably looks like in the logs, and what a slow span in the trace probably means for the code. An agent doing the same walk doesn't have that intuition. It has to either guess at the same correlations a human guesses at, usually with less context, or be given a stack where those correlations already exist before it starts. Closing the loop, for real, means the agent's starting point isn't step 1. It's closer to step 11, already holding the unsampled, full-stack session data, pre-correlated, deduplicated, with the relevant code already identified, before it ever opens a single tool. Systems That Watch and Heal Themselves Developers don't want more dashboards to stare at or more alerts to triage. They want the thing that broke to fix itself, the way a bruise heals without you having to think about it. Getting there starts with the telemetry layer. Today it's a passive record: data gets written somewhere, and someone (human or agent) comes along later to dig through it. An architecture built for this new consumer, AI coding agents, works differently. It captures full-fidelity, pre-correlated session data at the source, so an agent isn't reconstructing a failure from sampled traces or scattered tools. The data arrives ready to reason about. A few concrete shifts follow from that. Random sampling fades, replaced by systems that cache locally and decide, in the moment, what's worth keeping when something goes wrong. Observability stops being a separate product bolted onto the side of a system and becomes part of how the system runs: less a storage bucket, more an active participant. And the whole model flips from pull to push: instead of someone opening a dashboard to go looking for a problem, the system surfaces what happened, pre-correlated by user, session, and deployment, the moment it happens. What changes for developers is the shape of the work itself. Less time spent reconstructing what broke from fragments across five tools. More time spent on the things that actually require judgment: deciding which fixes are safe to ship automatically, which ones need a second look, and what the system should and shouldn't be allowed to do on its own. That's the goal "self-healing" has pointed at since IBM coined the term in 2001, modeled on a nervous system that handles the details so you can think about the things that matter. Paul Horn put it simply: the best measure of success is when people think about the functioning of computing systems "about as often as they think about the beating of their hearts." Twenty-five years later, that's finally within reach.

By Thomas Johnson CORE