Maintenance

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale. After a few years of operating a shared Kubernetes environment, the shift in the center of gravity becomes clear. Cluster provisioning, container scheduling, and upgrades become routine, yet releases still stall over ownership, access, telemetry, and cost allocation. Consider a hypothetical product team adding a stateful order-processing service to a shared platform. The service has an API, a worker, database migrations, and a data store backed by cluster-managed persistent storage, and it must run in staging and production. We will follow that service through its delivery path to examine where mature infrastructure stops helping, how local workflow differences compound, and which operating model decisions restore consistency without stripping teams of useful autonomy. When Infrastructure Maturity Stops Solving the Hard Part At first glance, onboarding the service should be routine. The cluster exists, the CI system can build an image, and infrastructure as code can create the namespace. Then the service reaches production and encounters a different StorageClass, quota profile, network policy, or service account configuration from staging. Each difference may be valid, but the delivery workflow didn’t surface the environment contract early enough. This is the practical limit of infrastructure maturity. Reliable clusters provide capable building blocks, while reliable delivery also requires a shared agreement about how teams use those blocks, what evidence a release produces, where exceptions go, and who owns the outcome. How Cloud Complexity Starts to Compound Follow the service through one release and the pattern becomes clearer: The same components use inconsistent service and environment identifiers across logs and traces.Ownership labels exist in one cluster but not the other.The team copies a pipeline because the shared template cannot sequence migrations.Production access and policy exceptions move through separate ticket queues. The operational cost shows up in the manual coordination required before each deployment. An engineer has to reconstruct which rules apply every time. During an incident, responders can’t move cleanly from an alert to the owning team, deployment record, runbook, and cost center. Finance sees shared-cluster spend that cannot be attributed reliably, while the security team receives evidence in different formats. OpenTelemetry semantic conventions and FinOps allocation practices rely on consistent service, environment, and allocation metadata, so local naming schemes undercut the value of the underlying tools. As the same pattern spreads across clusters and cloud accounts, small differences become a persistent operating burden. Operating Models Set the Terms of Scale The operating model decides who turns those building blocks into a usable delivery system. For our example, the product team owns the order domain, data model, migration safety, scaling behavior, SLOs, and on-call response. The platform team owns the interface through which the service receives a namespace, workload identity, baseline policy, deployment workflow, and telemetry defaults. Security, SRE, and FinOps teams contribute requirements and review the evidence that the workflow produces. This split keeps service-specific decisions close to the people who understand them while centralizing cross-cutting capabilities that every team would otherwise rebuild. CNCF’s platform guidance makes an important distinction here: A platform team is responsible for the interfaces and experience around shared capabilities, even when another team or provider operates the backing service. In practice, a mature platform offers versioned workflows, clear support boundaries, self-service for common requests, and feedback loops based on real usage. In this way, the platform team is an enabler of consistency rather than the operator of every component. Standardization, Autonomy, and Shared Operating Logic The defensible baseline is narrower than a universal application architecture. For this service, shared standards should cover: Service and environment identityWorkload identity and minimum network policyResource requests, quota expectations, and cost-allocation metadataRelease evidence, rollback behavior, and minimum telemetry These rules belong in the shared workflow because inconsistency affects other teams and complicates incident response, security, and cost allocation. The product team still chooses its schema, partitioning strategy, cache design, scaling thresholds, and release timing, and defines SLOs around the behavior users experience. The stateful workload then tests that boundary. A default pipeline built for stateless HTTP services may need a supported hook for migrations and worker rollout. An overly broad standard becomes an approval layer or bottleneck, while an overly narrow one leaves every team maintaining its own release and recovery logic. A bounded extension with an owner, tests, constraints, and review date preserves autonomy without creating an unsupported parallel system. Why Team Friction Turns Into a Scaling Tax Weaknesses in the operating model become most visible in the friction between teams. If a developer must request a namespace, ask another team for credentials, copy a pipeline, and find a production approver, the architecture may be automated while delivery remains ticket-driven. Each handoff adds queue time and loses context. Adding a portal without changing that path gives the developer one more place to check. Effective self-service completes the request, applies policy, records the change, and returns a clear support path. To see whether self-service is reducing friction, track metrics like request-to-environment time, time to first production deployment, exception rate, support demand, and failed-deployment recovery time. CNCF recommends tracking fulfillment and new-service delivery latency; DORA advises applying delivery metrics in the context of a specific service. Together, these measures show whether the workflow reduced coordination overhead or moved it to another queue. When Control Models Backfire The order-processing service example exposes two ways the control model can fail: Overly rigid standardization. A workflow designed only for stateless services forces the team to create a separate migration path, fragmenting release evidence.Unbounded local variation. Unrestricted cluster access allows identity, policy, and resource controls to drift between teams. The scalable approach pairs a narrow baseline, enforced through mechanisms like admission policies, with a documented extension path for legitimate workload-specific behavior, keeping the standard credible without turning each exception into a permanent fork. Operating Assumptions That Fail at Scale Old Assumption Why It Breaks Operating Model Replacement Healthy clusters make a workload portable Storage, identity, policy, and quota profiles differ by environment Versioned environment contract with a shared baseline One shared pipeline can serve every workload Stateful rollout and migration steps don’t fit the default sequence Core workflow with bounded, tested hooks A portal provides self-service Tickets and manual approvals remain behind the interface Workflow that provisions, enforces policy, and records evidence Local conventions remain harmless when teams own their services Metadata and controls drift across services Small enforced baseline with governed exceptions Making Cloud Complexity More Manageable To begin, you don’t need to redesign your entire platform. You can trace one representative delivery workflow and find where coordination breaks. For the order-processing service example, map the path from repository creation to production, including owners, queues, controls, evidence, and exceptions. Improvements should then be tested through adoption and outcomes such as lead time, failed-deployment recovery time, support demand, exception volume, and cost-attribution coverage. This sequence shows whether the platform is reducing operational variation for real workloads before the model expands to more teams and environments. References: Platforms for Cloud-Native Computing, CNCFResource Quotas, KubernetesStorage Classes, KubernetesAdmission Control in Kubernetes, KubernetesResource Semantic Conventions, OpenTelemetryAllocation FinOps Framework Capability, FinOps FoundationService Level Objectives, Google SRESoftware Delivery Performance Metrics, DORA This is an excerpt from DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale.Read the Free Report

By Igboanugo David Ugochukwu CORE

Tell me about that one document that no one has read. Your company's governance policy for AI systems. Forty-something pages, buried in a wiki or Confluence somewhere. The document that legal and compliance teams spent months writing, reviewing it and referencing the National Institute of Standards and Technology Artificial Intelligence Risk Management Framework and the European Union Artificial Intelligence Act, and no doubt a handful of other standards you can't quite recall. Meanwhile, every single model your team has deployed in the last year hasn't consulted that document before deployment. As I've learned in my own experience building enterprise-grade AI solutions, the space between "we have a policy" and "the policy actually prevents something undesirable from happening" is where the headaches for compliance engineers and auditors begin and where the fines from regulators are born. Thus, your governance policy should not be a document. It should be code. Documents Cannot Block a Deployment I understand why companies document governance policies. You need those policies to exist for auditing and legal clarity. But a PDF cannot stop a CI/CD pipeline from deploying code any more than a procedure printed on paper can stop a nuclear reactor from melting down. ISO 42001, NIST’s AI Risk Management Framework, and Annex IV of the EU AI Act all define what actions are necessary, but not where in your stack those actions need to happen. Frequently, the result is that the compliance team drafts a policy, the engineering team gets a slide deck during an all-hands meeting, and everyone returns to their desks confident that they just need to remember to do things differently. Let me give you an example. Say a recommendation model was deployed into production after being trained on a dataset containing personally identifiable information (PII) that was supposed to be restricted to internal analytics. The dataset had been reclassified two months earlier, but no one had updated the pipeline configuration. It took eleven weeks after deployment for a data subject access request to uncover the issue. By then, six million users had received recommendations based on data that no one was permitted to access. It took three people-weeks of work to retrain and redeploy the model, audit all downstream consumers, and report the violation to the relevant regulatory agencies. Yes, this is a theoretical scenario, but it is hardly an unusual one. Readers working on frontier AI models can probably relate, because situations like this happen at large technology companies more often than you might think. That makes it essential to adopt the following techniques to avoid problems like this in the first place. Every Governance Rule Becomes a Gate If you’ve worked in infrastructure engineering, this is not a novel problem. Policy-as-code tools such as Open Policy Agent have existed for years, and today we have Kubernetes clusters in production that simply cannot admit pods that fail to meet specific security requirements. No one manually checks every time a pod is deployed to see whether it has the right security context, because the system itself rejects insecure pods. That is how policy-as-code should work in AI systems: as a zero-trust model. Every assertion in your governance policy should translate into a machine-enforced check applied to some part of the system, returning a non-zero exit code when the check fails. Your policy becomes a set of gates that the system must pass to continue operating and must fail fast when a requirement is not met. There are three ways this becomes useful, so let’s look at each in turn. Before Training Starts: Verify What Data the Model Can See One of the most common requirements in AI governance frameworks is to classify datasets by sensitivity and restrict model training to data with a classification level equal to or below what has been approved for the model. Here’s what that might look like: Python # policy/data_classification.py from governance_engine import PolicyCheck, DataSensitivityTier class DataClassificationGate(PolicyCheck): """ Blocks model training if the dataset's sensitivity level exceeds what this model is approved to access. Maps to: NIST AI RMF GOVERN 1.2, ISO 42001 Section 6.1 """ def evaluate(self, pipeline_context): dataset = pipeline_context.dataset model_approval = pipeline_context.model.data_access_tier # Get the actual classification, not the cached one current_sensitivity = dataset.get_live_classification() if current_sensitivity.tier > model_approval.max_tier: return self.fail( reason=( f"Dataset '{dataset.name}' is classified " f"{current_sensitivity.tier.name} but model " f"'{pipeline_context.model.name}' is only approved " f"for {model_approval.max_tier.name} data. " f"To request reclassification: internal-wiki/data-tier-request" ), remediation_link="internal-wiki/data-tier-request", escalation="data-governance-oncall" ) # Everything checks out, log it for the audit trail self.record_evidence( check="data_classification_pre_training", result="PASS", model_id=pipeline_context.model.id, dataset_id=dataset.id, classification=current_sensitivity.tier.name, timestamp=self.now() ) return self.passed() A few things are worth noting about this example. First, the classification is dynamic. We are not relying on whatever value was assigned to the dataset when it was ingested, because that could easily be out of date. Datasets change, columns are added, and sensitivity is periodically reevaluated. If a dataset’s classification is raised, any model trained on it should have its data access tier raised as well. Second, notice that the failed assertion includes a remediation link. We are giving the person who triggered the failure a direct pointer to what they should do next. This is critically important. One reason companies struggle so much with governance policy is that engineers understandably resist controls that slow down their work. Productivity will suffer if they are blocked without a clear path forward. If your policy is a wall they have to climb over, they will find ways to avoid climbing it. It is better to turn that wall into a door that opens only when the proper permission is presented. At Serving Time: The Gateway That Cannot Be Skipped If you're at all familiar with the announcement of Meta's Llama Guard model as part of the Purple Llama research series in late 2023, you might recognize this next section. To quote the announcement, "Unlike traditional approaches, Llama Guard does not utilize rule lists or keyword matching; instead, it leverages a separate model for enhanced security, trained to discern whether the input or output of Llama models violates any given safety policy." In other words, the Llama Guard model is policy-as-code: all safety policies are encoded directly in the system prompt of the model, allowing it to classify any given prompt or response as either safe or unsafe. This is a tremendously powerful technique, and while you might not be in a position to train dozens of guard models for every system you want to protect, you can most likely implement a similar pattern at the serving gateway layer for your models. YAML # policy/serving_gateway.yaml # This config lives WITH the model definition. # When the model deploys, its governance deploys with it. model_policies: - applies_to: "credit-risk-scoring/v2" enforcement_mode: BLOCK # vs AUDIT_ONLY for rollout pre_inference: - check: caller_authentication require_tier: 3 # Only Tier 3+ services can call this deny_message: "Credit models require Tier 3 service auth. See internal-wiki/model-auth" - check: input_data_restricted_fields blocked_fields: ["social_security_number", "date_of_birth", "ethnicity", "address"] action: STRIP_AND_LOG # Remove the field, log the attempt post_inference: - check: output_contains_pii scan_for: ["ssn_pattern", "phone_pattern", "email_pattern"] action: REDACT - check: fairness_monitor protected_attributes: ["gender", "race", "age_group"] max_disparity_ratio: 1.25 action: ALERT # Don't block, but fire an alert to the fairness oncall always: - check: audit_log_all fields: ["caller_id", "input_hash", "output_hash", "latency_ms", "timestamp"] retention_days: 2555 # 7 years, regulatory requirement This pattern encodes several important governance rules. First, notice that the policies are attached directly to the model itself. When the model is deployed, its policies are deployed with it. You cannot deploy the model without also deploying the policies that govern how it may be used. That means the policies are not stored separately in a governance database or knowledge management system. Instead, they live directly in the model’s configuration. This has several advantages, including making the policies much easier to discover. There is no way to deploy the model without reviewing its policies, which helps ensure those policies are considered and followed. After Deployment: Compliance Drifts Without Continuous Checks So you've deployed your model, and it passed all the policy checks during deployment. Great. Now, assume several months later the model starts serving requests that violate policy. The world changes over time; new regulations come into law; models' concepts drift; users' behaviors evolve. This is entirely realistic, and not hypothetical - the EU AI Act's implementing acts are currently still being drafted. You must perform both continuous checks and deployment-time checks in order to account for changes in the model after deployment: Python # policy/continuous_compliance.py from governance_engine import ContinuousMonitor, AlertSeverity monitor = ContinuousMonitor( schedule="*/5 * * * *", # Every five minutes alert_channel="ai-compliance-oncall" ) @monitor.regulation("eu_ai_act.article_14") def check_human_oversight(system_context): """ EU AI Act Article 14 requires that high-risk AI systems are designed to allow effective human oversight. We interpret this as: there must be a working override endpoint, it must respond fast enough to be meaningful, and a human must have actually reviewed outputs recently. """ if system_context.risk_classification != "HIGH": return # Only applies to high-risk systems # Can a human actually intervene? override = system_context.get_override_endpoint() if not override.is_healthy(): monitor.alert( AlertSeverity.P1, f"Human override endpoint DOWN for {system_context.model_id}. " f"High-risk system operating without oversight capability.", runbook="confluence/internal/ai-override-recovery" ) # Is the override fast enough to matter? if override.p99_latency_ms > 500: monitor.alert( AlertSeverity.P2, f"Override latency {override.p99_latency_ms}ms, too slow " f"for meaningful human intervention.", runbook="confluence/internal/ai-latency-optimization" ) # Has anyone actually looked at this thing recently? days_since_review = system_context.days_since_last_human_review() if days_since_review > 30: monitor.alert( AlertSeverity.P3, f"No human review of {system_context.model_id} in " f"{days_since_review} days. Scheduling mandatory review for the compliance team.", auto_action="schedule_review" ) The value of this approach is that it ties directly back to a specific regulation. If an auditor asks, “How do you know you’re complying with the EU AI Act?” you can show exactly which alerts the check has triggered and when it last ran successfully. That is critical for demonstrating that your systems are actually doing what they are supposed to do. It is much easier to say, “Here is a dashboard showing the results of this check over the last 180 days,” than to explain in natural language all the steps you take to maintain ongoing compliance with a regulation. Mistakes That Cost Us Months (and Probably Millions) Don’t Ship Fifty Policies on Day One Leadership will ask why it is taking so long. Even so, don’t try to deploy data classification, access control, drift monitoring, and fairness audits all at once. Engineers will start filing bypass requests faster than your governance team can process them. Instead, pick one policy that would have prevented your last major incident if it had been in place. Deploy that policy. Let engineers see it, push against it, and learn how it works and how it can help catch real production issues before they become public relations nightmares. Once they understand it, iterate from there. Make Failures Helpful, Not Hostile If a policy fails, make sure it tells the user what they did wrong, why it is a problem, and how to fix it. A policy that simply says “DENIED” will have much lower adoption than one that says, “This dataset contains Tier 3 data, but your model is only approved for Tier 2 datasets. See the link below if you want to formally request a change to your model’s approval tier.” Version Your Policies in Source Control Use proper version control for your policies—not a governance database or a wiki, but real source code version control that enforces pull requests and code review. That way, when a regulation changes, you can update the policy, review the changes, test it, and deploy it to production just like any other code. It also makes it much easier to roll back a change if something goes wrong. The Audit Trail Is the Product Every time a policy runs, capture the outcome along with information about the model, dataset, and any other contextual details that could be useful to an auditor. If someone asks, “How are you preventing the use of sensitive data in model training?” you should be able to show them a dashboard containing the results of every relevant data classification check run over the last 90 days. The Regulatory Reality The regulatory landscape around AI is expanding quickly. The EU AI Act is now in force, the NIST AI Risk Management Framework is becoming an important reference point, state legislatures are passing new laws, and standards such as ISO/IEC 42001 are increasingly becoming de facto requirements. None of these developments are going away, and companies cannot expect a PDF policy reviewed once a year to be enough for compliance. The companies that handle this environment well will treat compliance as a force multiplier rather than overhead. Policy-as-code at deployment time gives you gates, while continuous runtime monitoring gives you guardrails. Together, they give engineers enough context to reason about their systems while adding friction only where it matters most. The result is safer products, more productive engineers, and a compliance posture you can actually demonstrate to an auditor.

In high-volume data platforms, hardcoding validation logic into individual processing pipelines creates significant operational drag. As an enterprise data asset footprint grows, maintaining manual checks for hundreds of tables inevitably leads to mounting technical debt, silent schema drift, and a fragmented audit trail. To achieve data governance at scale, data architects must decouple validation rules from the execution engine. By utilizing a centralized metadata repository to dynamically generate validation suites, organizations can transform data quality from a reactive, script-based bottleneck into a configuration-driven infrastructure asset. The Metadata-Driven Architecture Instead of embedding validation constraints directly inside an ETL/ELT pipeline, this pattern isolates validation rules inside a centralized relational database schema. The orchestration engine programmatically queries this metadata at runtime, constructs the validation suites on the fly, executes them against target tables, and routes the evaluation metrics to an observability layer. This architecture provides three primary engineering advantages: Decoupled Governance: Data stewards can alter business rules or add expectations via simple DML updates without modifying or redeploying production application code.Schema Drift Resilience: The engine dynamically adapts to structural variations by programmatically evaluating target datasets against rules defined at the column level.Centralized Observability: Every rule execution generates a standardized, traceable metric payload, laying a consistent foundation for real-time data auditing and data lineage maps. 1. Defining the Metadata Schema (DDL) To implement this framework in an enterprise Lakehouse ecosystem, the metadata table must act as an immutable source of truth for constraints. Below is the production DDL required to initialize the control directory in Snowflake or Databricks: SQL CREATE TABLE data_quality_rules ( rule_id INT IDENTITY(1,1), table_name VARCHAR(255) NOT NULL, column_name VARCHAR(255) NOT NULL, expectation_type VARCHAR(255) NOT NULL, expectation_kwargs VARIANT NOT NULL, -- Stored as JSON object is_active BOOLEAN DEFAULT TRUE, updated_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(), CONSTRAINT pk_rule_id PRIMARY KEY (rule_id) ); -- Seed metadata rules for execution tracking INSERT INTO data_quality_rules (table_name, column_name, expectation_type, expectation_kwargs) VALUES ('CUSTOMERS', 'CUST_ID', 'expect_column_values_to_not_be_null', '{}'), ('CUSTOMERS', 'AGE', 'expect_column_values_to_be_between', '{"min_value": 18, "max_value": 60}'), ('ORDERS', 'ORDER_ID', 'expect_column_values_to_not_be_null', '{}'); 2. Implementation: The Programmatic Execution Engine The core execution wrapper leverages Python and Great Expectations (gx) to programmatically turn rows of metadata into active validation suites. This script establishes a secure database connection via SQLAlchemy, harvests active constraints, generates runtime batch requests, and triggers structured checkpoints. Python import os import json import logging from datetime import datetime import pandas as pd from sqlalchemy import create_engine import great_expectations as gx from great_expectations.core.batch import RuntimeBatchRequest # Configure structured logging for production auditing logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') logger = logging.getLogger(__name__) class MetadataDataQualityEngine: def __init__(self, connection_string: str): self.engine = create_engine(connection_string) # Initialize Great Expectations ephemeral context for programmatic runtime control self.context = gx.get_context(context_root_dir=None) def fetch_active_metadata(self) -> pd.DataFrame: """Harvests active validation configurations from the centralized store.""" query = """ SELECT table_name, column_name, expectation_type, expectation_kwargs FROM data_quality_rules WHERE is_active = TRUE """ try: df = pd.read_sql(query, self.engine) logger.info(f"Successfully harvested {len(df)} active validation rules.") return df except Exception as e: logger.error(f"Failed to query metadata repository: {str(e)}") raise def compile_expectation_suite(self, table_name: str, rules_df: pd.DataFrame): """Assembles validation rules into a Great Expectations suite on the fly.""" suite_name = f"{table_name}_suite" suite = self.context.add_or_update_expectation_suite(expectation_suite_name=suite_name) # Filter metadata constraints for the specific target asset table_rules = rules_df[rules_df['table_name'] == table_name] for _, row in table_rules.iterrows(): # Parse JSON kwargs configuration gracefully kwargs = row['expectation_kwargs'] if isinstance(kwargs, str): kwargs = json.loads(kwargs) kwargs['column'] = row['column_name'] # Programmatically map string values to structured GX expectation objects expectation_config = gx.core.ExpectationConfiguration( expectation_type=row['expectation_type'], kwargs=kwargs, meta={"notes": f"Automated constraint enforcement for column: {row['column_name']}"} ) suite.add_expectation(expectation_config) self.context.add_or_update_expectation_suite(suite=suite) return suite def execute_quality_checkpoint(self, table_name: str, target_df: pd.DataFrame): """Builds a runtime batch request and evaluates data against the generated suite.""" timestamp = datetime.utcnow().strftime("%Y%m%dT%H%M%SZ") suite_name = f"{table_name}_suite" checkpoint_name = f"{table_name}_checkpoint" # Unique runtime composite signature prevents processing trace collisions batch_request = RuntimeBatchRequest( datasource_name="lakehouse_runtime_datasource", data_connector_name="runtime_data_connector", data_asset_name=f"{table_name}_{timestamp}", runtime_parameters={"batch_data": target_df}, batch_identifiers={"table_name": table_name, "execution_timestamp": timestamp} ) # Register and fire a dynamic checkpoint execution self.context.add_or_update_checkpoint( name=checkpoint_name, config_version=1, class_name="SimpleCheckpoint", validations=[{ "batch_request": batch_request, "expectation_suite_name": suite_name }] ) logger.info(f"Launching data quality checkpoint for table: {table_name}") return self.context.run_checkpoint(checkpoint_name=checkpoint_name) # Production Loop Execution Pattern if __name__ == "__main__": SF_CONN = "snowflake://<user>:<pass>@<account>/<db>/<schema>?warehouse=COMPUTE_WH&role=SYSADMIN" dq_engine = MetadataDataQualityEngine(connection_string=SF_CONN) metadata_rules = dq_engine.fetch_active_metadata() distinct_target_tables = metadata_rules['table_name'].unique() for current_table in distinct_target_tables: try: # Stage current batch dataset from target engine raw_data_df = pd.read_sql(f"SELECT * FROM {current_table}", dq_engine.engine) # Step 1: Build suite dynamically from relational rules dq_engine.compile_expectation_suite(table_name=current_table, rules_df=metadata_rules) # Step 2: Validate batch data and extract metrics payload eval_result = dq_engine.execute_quality_checkpoint(table_name=current_table, target_df=raw_data_df) if not eval_result["success"]: logger.warning(f"Data Quality anomalies detected on asset: {current_table}") else: logger.info(f"Asset {current_table} successfully cleared all metadata expectations.") except Exception as err: # Fault isolation ensures an asset failure never crashes cascading pipeline steps logger.error(f"Processing loop interrupted on asset {current_table}: {str(err)}") continue 3. Production-Grade Engineering Guardrails Building a dynamic system requires putting structural guardrails around the execution engine to prevent it from failing under enterprise pressures. Fault Isolation and Pipeline Resilience: Never let a validation failure on an upstream or non-critical business table halt your entire orchestration loop. Wrapping individual target assets in localized try-except blocks ensures that a failure on a secondary table (like CUSTOMERS) does not block downstream transactional tables (like ORDERS) from completing their validation lifecycles.Idempotency and Batching Identifiers: Every unique quality run must be traceable back to a specific moment in time to avoid overwriting or colliding results in your metadata tracking layer. Pair the table_name with an immutable execution_timestamp (such as a UTC ISO string) as a composite batch identifier. This guarantees an explicit audit trail across parallel streaming windows or backfilled data runs.Metadata-as-Code Frameworks: Treat the validation matrix table with the same operational rigor as production application code. Changes, additions, or deprecations of quality thresholds must follow a strict GitOps progression. Use schema migration version control tools (like Flyway or Liquibase) to manage, track, and deploy DML changes safely across staging and production clusters.Proactive Alerting Integration: Local HTML docs are insufficient for zero-downtime platforms. The metadata evaluation output dictionary must be integrated directly into cloud native alerting systems. Configure Webhook integrations or cloud alerting channels to route failed validation metrics directly to Slack channels or PagerDuty schedules. This ensures on-call engineers are proactively notified the moment a metric payload trends outside acceptable operational thresholds. Summary: The Architectural Impact Transitioning to a metadata-driven approach shifts data quality from a reactive "clean-up" task to an integrated, proactive engineering asset. By treating validation criteria as configurable metadata parameters rather than hardcoded script directives, architects eliminate technical debt and bridge the gap between business semantics and computing layers. This decoupled architecture provides the strict governance framework required to support high-stakes analytics and downstream machine learning layers, ensuring that every data element hitting your warehouse is automatically and transparently vetted before reaching production consumers.

In the early days of the web, monitoring availability was simple: a server either responded to a ping, or it didn't. HTTP checks tightened that up a little — a 200 OK meant the dashboard turned green, and everyone assumed things were fine. That assumption doesn't really hold anymore, though. A modern app can return a picture-perfect 200 OK and still be completely unusable to an actual customer. Take an e-commerce site where the web server is healthy and responding in milliseconds. Somewhere behind it, a third-party inventory service has quietly died, or a CSS change buried the checkout button under a promo banner nobody tested for. Nobody can buy anything. Server's up. Business is down. Legacy monitoring can't see any of this — it was built to check the plumbing, not whether the person standing at the sink can actually get water out of the tap. Uptime Isn't an Infrastructure Metric Anymore In a monolithic architecture, the app and the database lived on one server, and uptime was basically a binary infrastructure question. That's not how most applications get built anymore. A typical SaaS product today is a single-page application backed by dozens of independent microservices spread across regions, plus a stack of external dependencies — an identity provider, a payment processor, a CDN, whatever else. If any one of those goes down, your own servers can be perfectly healthy while your users still can't get through a core workflow. Uptime, in that world, has to mean the continuous availability of the actual business workflow, not a response code. What Synthetic Monitoring Actually Does Synthetic monitoring uses automated clients to simulate real user traffic on a schedule, from multiple locations, around the clock — instead of waiting for a human to hit a broken flow and file a ticket. These aren't simple URL pingers, either. A synthetic monitor opens a real browser, renders the DOM, executes JavaScript, fills out forms, clicks through multi-step flows, and checks that the right data shows up on screen, all while watching the underlying API calls to make sure the backend agrees with what the UI is claiming. If a login flow that normally takes two seconds suddenly takes ten, or a button just stops responding, the monitor flags it right away, typically with a video of the failed session and enough diagnostic detail attached that someone can actually act on it, routed straight into whatever incident tool the team already uses. That's a fundamentally faster loop than "someone tweeted that checkout is broken." Where This Overlaps With QA: Shifting Right QA and production monitoring have traditionally been separate worlds — different teams, different tools, a handoff at the deployment line. That divide doesn't have much justification anymore. If a team's already built solid automated functional tests for CI/CD, there's no real reason to throw those away once code ships. The same script that validates a checkout flow pre-deploy can get repurposed to run every few minutes in production as a synthetic monitor — generally called "shifting right." Done well, it cuts duplicated engineering effort and gets QA and SRE working off the same definition of "healthy" instead of two different ones. Testing Beyond the Front Door: Complex User Journeys Basic uptime monitoring tells you the front door is open. Synthetic monitoring actually walks through the door, picks something up, applies a promo code, checks shipping, completes a transaction — the whole path, not just the entrance. That requires handling state, not just static checks. A monitor testing a healthcare portal needs to log in with synthetic credentials, get through MFA, pull a specific record, and confirm it belongs to the test account and nothing else. One testing a fintech transfer needs to confirm the UI shows success and then separately query the backend to make sure the balances actually moved, because a UI that says "success" while the ledger disagrees is arguably worse than an honest failure. Validating both the interface and the underlying state is what makes this useful for anything regulatory or revenue-critical. The Self-Healing Problem Running scripts against a live production environment is harder than running them in staging, because production changes constantly — new banners, UI experiments, shifting layouts. A rigid script breaks on cosmetic changes it shouldn't even care about, and that's how you end up with false alarms nobody trusts. This is where AI-assisted self-healing has become genuinely useful, rather than just a buzzword bolted onto a monitoring dashboard. If a button's ID changes from submit-order to confirm-purchase, a brittle script just fails. A self-healing monitor uses visual and semantic signals to relocate the element, finishes the check, and logs a low-priority note for someone to review later, instead of paging an engineer at 3 a.m. over what amounts to a rename. Alert Fatigue Is a Design Problem, Not a Tooling Problem Poorly tuned monitoring trains engineers to ignore it, and static thresholds are a big part of why. If an alert fires whenever a page takes longer than three seconds, a one-off network blip pages someone for a problem that resolves itself before anyone even looks at it. A better approach builds a dynamic baseline from historical performance data — per time of day, per day of week — and only escalates when something deviates meaningfully from that baseline. Often it's worth requiring confirmation from more than one geographic location before paging anyone at all, so a regional network hiccup doesn't wake someone up for nothing. Where This Matters Most E-commerce is the obvious one — downtime there is measured in dollars per second, and synthetic checks on cart logic, discount calculation, and payment gateway responses catch the silent revenue leaks a green uptime dashboard would never surface. Multi-tenant SaaS is a quieter version of the same problem: a single shared microservice failing can degrade the experience for every tenant at once, sometimes without anyone noticing for a while. Synthetic scripts that log in under different tenant configurations help confirm data isolation is actually holding and that SLAs are being met in practice, not just assumed on paper because nothing's screamed yet. Healthcare and fintech carry real regulatory weight on top of the operational risk. Synthetic checks that confirm patient records render correctly, or that a banking handshake with a clearing house completes securely, end up functioning as both an operational safeguard and a rough form of continuous compliance evidence — useful when an auditor eventually asks how you know. The Takeaway A green uptime dashboard doesn't mean much anymore if all it's checking is whether a server responds. The failures that actually cost money and trust — a hidden checkout button, a silently failing third-party integration, a broken multi-step flow — live above the infrastructure layer. Only something that behaves like a real user is going to catch them.

There’s always more to our contributors than what you see in their author profiles. For our latest Member Spotlight, I sat down with Shamsher Khan to learn more about his newest project. What started as a frustrating Kubernetes troubleshooting problem has since grown into published research, a new way of thinking about operational evidence, and ongoing open-source work. What first got you interested in digging into complex infrastructure and systems problems? "I’ve always been interested in problems where the visible symptom is not necessarily the real cause. In infrastructure, especially distributed systems, a service can look healthy from one angle while something important is already failing underneath. Troubleshooting becomes less about finding one bad log line and more about understanding how the application, container, node, network, scheduler, and platform interacted over time. That is what made Kubernetes particularly interesting to me. It automates a lot of recovery, which is great operationally, but that also means the system can change very quickly while you are still trying to understand what happened. Over time, I found myself increasingly interested not just in fixing incidents, but in understanding what information engineers actually have available during and after those incidents, what disappears, and where existing tooling helps or still leaves gaps. That curiosity has shaped a lot of my writing and open-source work." Your DZone article, “When Kubernetes Forgets: The 90-Second Evidence Gap,” ended up becoming the starting point for Operational Memory Architecture (OMA). What were you seeing in Kubernetes that made you think, “There’s a bigger problem here”? It came from a very specific frustration during incidents. A pod would crash, Kubernetes would restart it, and by the time I got there to investigate, some of the information I wanted was already gone or had changed. One example is LastTerminationState. Kubernetes keeps information about a container’s most recent termination, but when that container fails again, the previous termination context is replaced. In a fast crash loop, that can happen repeatedly in a short period of time. You can arrive at a pod that has restarted thousands of times and still have only a very small window into how that sequence began. What made me think the problem was bigger was realizing that this was not really a Kubernetes bug. Kubernetes is primarily designed to maintain desired state and restore workloads. Preserving a complete forensic history is a different concern. Once I started looking more systematically, I saw similar boundaries elsewhere. Kubernetes Events have limited retention, short-lived workloads can exist entirely between monitoring samples, and some node- or runtime-level evidence can become difficult or impossible to reconstruct after the underlying state changes. There are already strong observability tools that help with logs, metrics, traces, and events, so the question was not, “Why doesn’t Kubernetes keep everything forever?” That would not be realistic or necessarily desirable. The question became more specific: are there predictable points after which certain diagnostic evidence can no longer be recovered, and can we reason about those points explicitly? I started calling those points evidence horizons. OMA grew from trying to characterize those horizons and explore what evidence may need to be captured before they are crossed." Now that the research is being published in IEEE Access, what do you hope people working with these systems take away from it? And where would you like to see OMA go from here? "The main thing I hope people take away is that recovery and diagnosis are related, but they are not the same problem. A platform can successfully restore an application while still losing some of the context that would have helped explain why it failed. I think many engineers have experienced this without necessarily having a name for it. If you have ever finished an incident review with, “We’re not completely sure what actually triggered this,” disappearing or short-lived evidence may be one reason. I also want to be careful not to suggest that OMA replaces existing observability platforms. Tools for logs, metrics, traces, events, and distributed tracing are already essential. OMA is better thought of as a way of reasoning about when different kinds of evidence remain available and when they may cross a point where recovery becomes difficult or impossible. There is also a practical side to this. Teams doing post-incident reviews, reliability analysis, or audit and compliance work may need to reconstruct what happened after the system has already recovered. Thinking explicitly about evidence retention and recovery boundaries can help teams decide what information is worth preserving. As for where OMA goes next, the research is still early. The work evolved in stages: I first published the foundational OMA idea on arXiv, then extended it with a broader evidence-horizon taxonomy and additional validation before developing it into the peer-reviewed IEEE Access paper. The implementation and experiments are public, and the most useful next step is independent validation in environments different from the ones I tested. There are also limitations in the current work. For example, some node-level evidence across kubelet or node restart boundaries requires deeper integration than the current architecture provides. I documented that rather than trying to claim the problem was solved. Some of these ideas have also influenced practical work I’m doing in OpsCart, an open-source Kubernetes operational triage project. OpsCart is not a replacement for OMA or for established observability tools. I use it more as an engineering testbed for exploring how incident context, workload history, and diagnostic evidence can be surfaced in a way that is useful during everyday Kubernetes troubleshooting. I would like to see other engineers test both the research assumptions and the practical tooling, challenge the model, and point out where it does not hold up. That kind of feedback is more valuable at this stage than claiming the architecture is complete." Research: https://ieeexplore.ieee.org/document/11656328OMA implementation: https://github.com/opscart/k8s-causal-memoryOpsCart: https://github.com/opscart/opscart-k8s-watcher After spending so much time thinking about Kubernetes, what’s your ideal way to completely unplug for a weekend? The first requirement is definitely no Kubernetes dashboards. I spend a lot of time during the week thinking about systems, debugging, writing, and experimenting, so on weekends I like doing almost the opposite: spending time with family, getting outside, going somewhere for the day, or just having time where I’m not trying to solve a technical problem. Infrastructure problems have a way of staying in your head even after you close the laptop, so sometimes the best reset is doing something that has absolutely nothing to do with technology. To see more of Shamsher's content, here's the link to his DZone profile.

For the past several years, technical debt was something that was easy to identify. It came in the form of outdated frameworks, missing documentation, messy databases, etc. It was something companies racked up by moving too fast, skipping the best course of action, and patching up old systems instead of improving them. But now, a new kind of technical debt is fast emerging. At first glance, it may not look broken. However, it may even work perfectly fine initially. The app loads just fine, the feature responds, the workflow runs, and the AI-generated module seems to pass the first round of testing. On the surface, everything looks good. But the problem starts later on, when someone asks just one simple but very crucial question: how did this actually work? That is where many modern software teams begin to really feel the weight that comes with this new reality. They get easy and prompt access to a lot of code that functions, but cannot be easily explained or trusted. This is the new technical debt. Working Code Is No Longer Enough In the past, code that worked was a major milestone for any team. If the product worked, that meant that the team could finally move ahead. If the feature could be deployed, that meant the sprint was successful. If the app did not crash, that was a cause for celebration. That mindset was not the best one to have. Modern software does not live or work in isolation. There are several programs that it’s connected to, ranging from CRMs to cloud platforms, third-party APIs, AI models, and more. Just a single feature on the software may involve five or ten systems behind the scenes. So, if the team making the product does not understand how the code works, they cannot confidently tell what went wrong if the product runs into a wall or something changes, A feature that is running perfectly today may become a liability tomorrow if no one understands how it really works. That is why explainability is becoming just as important as functionality. AI Has Made This Problem Bigger With the assistance of AI in software development, the speed at which software is being created has increased monumentally. Developers can now do their tasks, including generating functions, writing test cases, building interfaces, and troubleshooting issues, at a fraction of the time that it took before. But speed alone does not automatically improve software delivery. According to Google’s 2025 DORA report, AI can be an amplifier of an organization's existing strengths and weaknesses, which means it can improve disciplined teams while exposing weaker engineering processes. This is a major advantage when used responsibly. But it also creates a new risk. AI can help come up with code that looks clean, runs correctly, and solves the immediate problems at hand. But is your development team able to follow along with the logic behind what the AI is doing? Are they accepting the AI-generated solution just because it passes a test, or do they understand every trade-off that it makes? AI tools can help a non-technical founder build a working prototype without knowing how secure or scalable it is. Just like that, your team, though moving faster in the short term, is quietly creating systems that they cannot confidently own in the long term. This does not mean AI-generated code is bad. The issue is not the tool. It is blind adoption. This concern is already showing up in the industry. GitLab’s 2026 research found that 73% of respondents are concerned about the maintainability of AI-generated code, while 82% believe it risks creating a new form of technical debt that their organization is not prepared to manage. When you use AI to accelerate development without reviewing it or passing human judgment on it, you are not just building software faster. You are building uncertainty faster. Explainable Software Requires Better Discipline The solution that I see to this is not to slow down innovation, but to build with more clarity. The basis of explainable software comes with strong requirements. Software development teams need to know what a certain feature is supposed to do, what it’s supposed to not do, what systems it connects with, and what happens when something goes wrong. Without this context, despite how well written the code is, it can become dangerous. The next thing to keep in mind is that the architecture behind the software can and should be understood by more than one person. A strong software system should not depend on one developer’s memory. It needs to have a clear structure, documentation, test coverage, and decision record that help future teams easily understand the system. Code reviews should also evolve. You shouldn’t just be reviewing your code for syntax or performance. Teams need to start asking deeper questions like why was this specific approach chosen? What assumptions does it make? What happens if the API fails? Can another developer understand this system six months from now? These questions may seem simple, but they are what separate working software from dependable software. The New Standard: Code That Works and Can Be Explained The old standard was simple: does the code work? The new standard needs to be stronger. Can it be explained? Can it be maintained? Can it be tested? Can it be scaled? Can a new engineer understand it without reverse engineering the entire system? If the answer is no, then your company may unfortunately be a victim of technical debt. It has only managed to hide it behind a working interface. That hidden debt will eventually come due. With AI tools and automated development workflows becoming common, more and more businesses are capitalizing on the opportunities to build software quickly. I get that it’s exciting, but speed without understanding is not progress. The next generation of technical debt will not always look like bad code. It will look like code that works, until the day someone needs to scale it or explain it. And by then, the real cost will become clear.

When teams first integrate large language models (LLMs) into their software platforms, the initial experience often feels surprisingly simple. A developer writes a few lines of code, sends a prompt to a model API, and receives a response that looks intelligent, contextual, and almost magical. A prototype can be built in days, sometimes hours, and the business quickly starts imagining how AI will transform customer support, automation, analytics, and decision-making. This early success creates a dangerous assumption: that moving from a working AI prototype to a production-grade AI system is simply a matter of increasing traffic and adding more users. In reality, the difficult engineering problems appear after adoption. The moment thousands of users start interacting with an AI-powered application, the hidden costs begin to surface. The model that worked perfectly during testing suddenly becomes expensive. Response times increase. Infrastructure bills grow unpredictably. A model chosen because it produced impressive answers becomes inefficient when handling millions of requests. Teams discover that AI applications are not just software applications with an intelligent component added on top. They are a completely different class of systems where cost, performance, and reliability must be designed from the beginning. I experienced this transition while working on an enterprise AI assistant project designed to help internal teams search knowledge bases, generate reports, and automate operational workflows. During the prototype phase, everything looked straightforward. I connected an application to an LLM provider, built a retrieval pipeline, and added some prompts, and the results were impressive. The first few demonstrations created excitement because the system could answer questions that previously required employees to manually search through thousands of documents. However, when adoption increased, the engineering reality changed. The system was no longer just answering questions. It was processing thousands of conversations, generating large responses, retrieving documents, calling multiple services, and consuming significant compute resources. The biggest lesson from that project was that building an AI capability is easy. Operating it efficiently at scale is where the real engineering begins. The First Hidden Cost: Tokens Become Your New Infrastructure Bill Traditional software systems usually think about infrastructure in terms of servers, databases, memory, and network usage. With generative AI applications, there is another resource that becomes equally important: tokens. Every interaction with a language model is measured through tokens. Input prompts, retrieved documents, conversation history, and generated responses all contribute to token usage. During my early development stage, I focused mainly on improving response quality. I added more context, included more documents, and expanded conversation memory because the model produced better answers when it had more information. The problem was that better answers also meant larger prompts. A simple user question that originally consumed a few hundred tokens could grow into thousands of tokens after adding document retrieval, user history, system instructions, and additional context. The system worked. The answers were good. But the cost model was becoming unsustainable. One of the first changes I made was measuring token consumption at every stage of the pipeline. Instead of treating the model call as a single operation, I started monitoring: Prompt tokensRetrieved context tokensGenerated response tokensTotal tokens per user sessionCost per request A simple monitoring wrapper helped me understand where the money was going. Python response = client.chat.completions.create( model="gpt-model", messages=messages ) usage = response.usage print( f"Input: {usage.prompt_tokens}, " f"Output: {usage.completion_tokens}" ) After implementing token tracking, I also created a cost-monitoring utility to identify expensive requests. TOKEN_PRICE = 0.00001 total_tokens = ( usage.prompt_tokens + usage.completion_tokens ) request_cost = total_tokens * TOKEN_PRICE logger.info( f"Cost per request: ${request_cost:.4f}" ) This small change completely surprised me and changed my approach. I discovered that many expensive requests were not caused by the model itself but by inefficient context management. For example, sending an entire document collection to the model was unnecessary. The model did not need every possible piece of information. It needed the most relevant information. To solve this, I reduced the number of retrieved documents before constructing the prompt. Python retrieved_docs = vector_store.similarity_search( query, k=5 ) context = "\n".join( doc.page_content for doc in retrieved_docs ) This pushed me towards better retrieval strategies, smaller prompts, and smarter context selection. The lesson was simple: In generative AI systems, information is not free. Every additional word sent to the model has a cost. Latency: The User Experience Problem Nobody Notices Early Cost was only one side of the problem. The second challenge was latency. During development, a response time of five or six seconds felt acceptable. I understood that AI models required processing time, and internal users were also patient because they were testing a new capability. Production users are different. A customer waiting for a chatbot response does not think about neural networks, GPUs, or inference pipelines. They simply think the application is slow. As usage increased, I started breaking down latency into individual components. A typical AI request looked like this: User request → Authentication → Retrieval → Database search → Prompt construction → Model inference → Response processing The model was only one part of the delay. In some cases, the retrieval process was adding unnecessary seconds because the system was searching too many documents. In other cases, the application was waiting for large model responses that users did not actually need. I introduced several improvements. First, I reduced unnecessary model calls. A common mistake in AI applications is using an LLM for every decision. Not every task requires intelligence. For example, if a user asks: "Show me my previous reports," there is no reason to call a large language model. A normal database query is faster and cheaper. I implemented a lightweight routing layer. Python def handle_request(query): if "previous reports" in query.lower(): return fetch_reports() return generate_llm_response(query) The model should be used where reasoning is required, not as a replacement for every application function. Second, I streamed responses. Instead of waiting for the entire answer to be generated, users started receiving partial output immediately. Python for chunk in client.responses.stream( model="gpt-model", input=prompt ): print(chunk.delta, end="") Streaming does not reduce the actual processing time, but it improves perceived performance because users see progress immediately. I also introduced latency monitoring. Python import time start = time.time() response = generate_answer(prompt) latency = time.time() - start logger.info( f"Latency: {latency:.2f}s" ) This was an important lesson from my project: AI engineering is not only about making systems faster. It is also about designing experiences where users feel the system is responsive. Model Selection: Bigger Does Not Always Mean Better One of the most expensive mistakes teams make is choosing the largest available model for every task. During my initial implementation, I used a powerful general-purpose model because it produced excellent responses. It was accurate, creative, and handled complex questions well. The problem was that most user requests were not complex. A significant percentage of requests involved simple classification, summary, formatting, or extracting information. Using a premium model for these tasks was like using a heavy database cluster to store a small configuration file. I introduced model routing. The idea was simple: Use smaller, cheaper models for simple tasks. Use larger models only when advanced reasoning is required. My architecture started looking like this: Simplified routing architecture A simplified routing example: Python if request_type == "summary": model = "small-model" else: model = "large-model" response = call_model( model, prompt ) I later automated this process. def select_model(query): if len(query.split()) < 20: return "small-model" return "large-model" This approach reduced cost significantly without affecting user experience. The important mindset shift was understanding that AI systems are not powered by one model. They are powered by a collection of models working together. The future of enterprise AI will not be about finding the single best model. It will be about building intelligent systems that know which model to use and when. Caching: The Forgotten Performance Strategy in AI Systems Caching has existed in software engineering for decades. Databases cache queries. Websites cache pages. Applications cache frequently used data. However, many teams forget that caching is equally important in AI applications. Multi-layer cache architecture During my project, I discovered that many users were asking similar questions repeatedly. Some of the questions were: "What is the company leave policy?" "What are the security requirements?" "How do I request access?" These questions produced almost identical responses every time, and calling an expensive model repeatedly for the same answer made no sense. I introduced multiple caching layers. The first was response caching. If the same question appeared with similar context, I reused the previous response. Python cache_key = hash(user_prompt) if cache_key in cache: return cache[cache_key] response = generate_answer( user_prompt ) cache[cache_key] = response The second was embedding caching. Instead of recalculating document embeddings repeatedly, I stored them and reused them. Python if doc_id not in embedding_cache: embedding_cache[doc_id] = ( embedding_model.embed( document_text ) ) embedding = embedding_cache[doc_id] Caching requires careful design because AI responses are not always identical. User context, permissions, and updated information must be considered. A cached response that ignores security rules can create serious problems. The important lesson is that caching in AI is not just about speed. It is about designing intelligent reuse while maintaining correctness. Infrastructure Optimization: Treat AI Like a Production System As usage increased, I realized that AI systems require the same operational discipline as any production platform. I introduced monitoring across the entire stack. Python metrics = { "latency": latency, "tokens": total_tokens, "model": model_name } send_to_monitoring(metrics) I also implemented rate limiting to prevent traffic spikes from overwhelming the system. Python from flask_limiter import Limiter limiter = Limiter( key_func=get_remote_address ) @limiter.limit("20/minute") def ask_ai(): pass For longer-running workloads such as report generation, I moved requests into asynchronous queues. task_queue.enqueue( generate_monthly_report, report_id ) This prevented expensive background jobs from affecting real-time user requests. The Bigger Lesson: AI Infrastructure Is Becoming a New Engineering Discipline The biggest mistake organizations make is thinking of generative AI as just another API integration, which it is not. Traditional applications are predictable. A database query usually behaves the same way every time. A function returns the same result for the same input. AI systems are different. They introduce uncertainty, variable workloads, expensive computation, and continuously changing behavior. Conclusion Managing generative AI infrastructure at scale requires far more than simply integrating a language model into an application. As usage grows, organizations must carefully balance cost, performance, reliability, and user experience while maintaining operational efficiency. Token consumption, latency optimization, intelligent model routing, caching, monitoring, and infrastructure governance become critical components of a successful AI platform. The organizations that achieve long-term success with generative AI will be those that treat it as a production-grade engineering discipline, designing systems that are scalable, observable, cost-effective, and resilient from the outset rather than attempting to solve these challenges after deployment.

An autoscaling policy can be wrong for months without a single error firing. It isn't built to fail loudly; it's built to keep response times steady, and it'll keep doing exactly that even while making the worst possible call for a GPU-bound job. The mismatch hides in plain sight because nothing looks broken. It stops doing its job without ever raising an alarm, and the first sign usually isn't an alert but a cost report or a training job stuck in a queue. Here's a fairly standard Kubernetes Horizontal Pod Autoscaler config:  YAML apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler spec:   minReplicas: 2   maxReplicas: 10   metrics:     - type: Resource       resource:         name: cpu         target:           averageUtilization: 70 For a stateless web service, this is close to perfect. A pod gets added, utilization dips, another request comes in, utilization climbs again. The whole loop runs slowly enough for the cooldown window to work exactly as intended: plenty of time to observe and react. A training job doesn't move like that. It sits at zero for two days, then needs ten GPUs immediately, then drops back to zero the second the job finishes. CPU utilization barely registers the change, because CPU was never the constraint to begin with. So the autoscaler, watching the wrong metric entirely, does nothing useful. Triggerworks well forbreak down for CPU utilization  Steady, request-driven traffic  GPU-bound training jobs  Queue depth / GPU utilization  Bursty, batch-oriented AI workloads  Legacy web services  Autoscaling wasn't wrong here, exactly. It kept solving the problem it was built for, one that had already stopped being the problem sitting in front of it.   The GPUs Were Right. The Data Never Arrived. There's a second version of this same trap that's easier to miss. Even with the right trigger metric, GPUs can sit idle waiting on data they can't ingest fast enough. Storage throughput and network bandwidth that worked for traditional applications can become bottlenecks when training jobs move terabytes at scale. An idle GPU waiting on data still costs money, but it rarely appears as an autoscaling problem. When the Infrastructure Looks Fine, and the Model Doesn't  Once a model is live and behaving, the infrastructure looks fine. CPU healthy, memory healthy, no alerts firing. Somewhere down the line, though, a flagging rate or an approval rate starts drifting, and nothing in the infrastructure layer notices. Prometheus, Grafana, and OpenTelemetry confirm the service is healthy. None of them tell you whether the model's decisions are still good. That's the split most teams don't plan for going in: infrastructure health and model health are two completely different signals, and only one of them shows up in the tools most cloud teams already trust. Data Quality Still Determines AI Performance  Trace either failure back far enough and it rarely ends at the model. McKinsey's research, AI Data Readiness: The Key to Scaling Impact, found more than two-thirds of high-performing organizations name data, not model selection, not compute, as the real constraint on scaling AI. It shows up constantly in practice: a CRM system, a billing platform, and a support desk defining the same customer three different ways. MLOps tooling can track model versions and deployments, but it cannot fix unreliable data underneath the model. Versioning is not the same as fixing. Models rarely fail because they cannot process data. They fail because they process unreliable data with the same confidence as accurate data. The Regulator's Question Has No Engineering Answer  Eventually, someone always asks the harder question, and it usually isn't an engineer who asks it. A lending platform turns an application down, and the applicant pushes back. A regulator wants to know exactly how that decision got made. Without an audit trail connecting that specific outcome back to the specific inputs the model saw, there's no real answer to give, regardless of how accurate the model has been on average. That almost never blocks a proof of concept. It blocks production, on a timeline nobody controls.  Cloud Placement Becomes a Production Decision for AI Workloads  There's a fourth complication sitting underneath all of this, one that surfaces even later. Where a workload actually runs stops being a footnote once AI enters the picture. AI workloads introduce new constraints around hardware availability, latency, cost, and regulatory requirements. Some workloads have to stay within a specific country's borders for regulatory reasons. Others only perform well on hardware a specific provider happens to offer. A team standardized on one cloud for everything else discovers, usually the hard way, that AI doesn't respect that standardization.  The challenge is no longer choosing one cloud provider. It is deciding where each workload can run effectively while balancing performance, cost, and compliance. What Gets Built Before the Next Incident, Not After None of these four problems — autoscaling, observability, data, governance, and placement — show up in a pilot. That's exactly why they're expensive.  The autoscaling policy either scales for GPU load or it doesn't. The observability stack either catches a model quietly getting worse, or it only notices when a server goes down. The data feeding the model is either governed enough to trust or it isn't. An audit trail either exists before the first real customer sees an output, or it gets built after a regulator asks for one. Someone has either mapped out where each workload needs to run, or that decision is still riding on wherever the last project happened to land.  Right now, real value is going to the teams that got the boring infrastructure work right, not the teams with the fanciest model. 

Over the past year, I've been rebuilding parts of an incident response stack for a client, and the biggest surprise wasn't the AI features themselves. It was how much of the underlying workflow had to change to make those features useful. You can't just bolt an LLM onto a 2015-era ticketing tool and call it AIOps. The queue structure, the alert taxonomy, even the way runbooks are written all need to change. I've written before about the agent side of this shift, in AI Agent Architectures: Patterns, Applications, and Implementation Guide and Observability and DevTool Platforms for AI Agents. This two-part series is the other side of that coin: what happens when you point those same agent patterns at your own production systems instead of at somebody else's AI application. Same reasoning loop, different target. This first part covers incident management specifically, including a category I skipped in my earlier tool roundups: dedicated "AI SRE" agents like Traversal, Resolve.ai, and Cleric, which behave differently from the AIOps platforms most of us grew up with. Part 2 goes past incidents into ITOps, chaos engineering, SLO management, on-call toil, and the rest of what fills an SRE's week. A note on the numbers below: vendor-reported accuracy and MTTR figures in this space move fast and come from the vendors themselves. I've flagged those clearly rather than presenting them as independently verified benchmarks. Where SRE Pain Actually Lives Before getting into tools, it helps to remember what SREs spend their time on. Most postmortems I read over the years have the same three complaints: Too many alerts, not enough signalCorrelating five different dashboards to find one root causeWriting the same postmortem summary for the fourth time this quarter None of these are new problems. What's new is that large language models are actually decent at the second and third ones, if you feed them clean data, and a newer crop of agents is starting to chip away at the first one too. The Traditional Incident Pipeline Here's roughly what an incident used to look like before AI got involved, at most mid-size shops I've worked with: Traditional incident pipeline Every arrow in that diagram is a human doing manual correlation work. That's fine when you have ten services. It falls apart at three hundred, and it's part of why I keep coming back to the point I made in Infrastructure as Code: How Automation Evolved to Power AI Workloads: scale problems in ops rarely get solved by hiring more people to stare at more dashboards. Where AI Fits Into the Pipeline Today The shift isn't "AI replaces the engineer." It's AI collapsing steps B through F into something closer to a single triage step, with the engineer reviewing a proposed root cause instead of hunting for one from scratch. AI-aided pipeline Notice the engineer never disappears from this diagram. They just move from being the one who does the correlation to the one who checks the correlation. That distinction matters, because it changes what you hire and train for. I made a version of this same argument about production-grade agents generally in the Shipping Production-Grade AI Agents refcard: an agent needs a human review layer, or you're just moving risk around instead of removing it. If you want a deeper look at how these review loops are actually structured under the hood, I broke that down recently in Loop Engineering: The Layer After Prompt, Context, and Harness Engineering. Incident Management: What Changed A few concrete things have improved in incident management tools over the last two years: Alert correlation got better. Tools like BigPanda, Moogsoft, and PagerDuty's AIOps features now cluster related alerts using pattern recognition instead of static rules. A database timeout, three downstream service errors, and a spike in 500s used to show up as four separate pages. Good correlation engines now group them as one incident with a suggested cause. Similar-incident retrieval works reasonably well. If your org has a decent history of past incidents with clean postmortems, tools can now surface "this looks like INC-4471 from March" with real accuracy. This only works if your postmortem data isn't garbage, which is a bigger blocker than people admit. Draft postmortems save real time. Not because the AI writes a good postmortem on the first try, but because staring at a blank page is the slowest part of writing one. A rough draft built from the incident timeline, Slack thread, and metrics gives engineers something to edit rather than create. What's newer, and worth its own section, is a class of tools that don't just correlate what you already collected. They go get new evidence during the incident, the way a senior engineer would. The New Category: Dedicated AI SRE Agents This is the part of the landscape that's moved fastest since I last wrote about agent tooling. A handful of startups have built agents whose entire job is investigating production incidents autonomously, not just clustering alerts that already exist. Traversal leans on causal machine learning rather than a general-purpose LLM wrapper. Instead of pattern-matching against similar past incidents, it builds a model of causal dependencies across your services and traces the actual chain of cause and effect, down to the specific deploy or config change that started the failure. It reports strong root-cause accuracy in production at large enterprises and is used for both alert triage and live incident investigation. The pitch is narrower than "full AIOps platform," and that narrowness is the point. Resolve.ai takes a broader angle. It was built by the team that created OpenTelemetry, and it positions itself as an agentic teammate across the whole production lifecycle: investigating incidents, but also touching capacity questions, config drift, and guided code changes. Where Traversal is a specialist in root cause, Resolve.ai is closer to a generalist you'd loop in on almost anything production-related, with the incident work as the anchor use case. Cleric sits in a similar space to both, with a specific focus on autonomous alert triage. It runs a multi-source investigation the moment an alert fires, pulling metrics, logs, traces, and deploy history in parallel, and returns an evidence-backed hypothesis before an on-call engineer has finished opening their second dashboard tab. It runs with read-only access by default, which matters a lot for teams still building trust in the category, and it was named a Gartner Cool Vendor in AI for SRE and Observability in 2025. That "read-only by default" design decision is exactly the kind of guardrail I argued for in Trust No Agent: How to Secure Autonomous Tools on Your Machine: an agent's blast radius should be a deliberate design choice, not an afterthought. Causely and NeuBird round out the space with slightly different angles: Causely focuses on causal reasoning to find the single root cause behind a storm of cascading alerts, and NeuBird targets enterprise IT environments with LLM-driven telemetry analysis at large scale. Here's roughly where an AI SRE agent sits in the pipeline compared to the AIOps correlation tools from the last section: AI SRE agent pipeline The key difference from the earlier diagram: this agent isn't just correlating signals you already collected in a dashboard. It's actively going out and querying your systems the way a human on-call engineer would, forming a hypothesis, testing it, and either confirming or discarding it before it ever pages a person. That's a meaningfully different capability than clustering alerts by similarity, and it's why this category gets its own row in any serious comparison. If you're weighing whether to build this kind of investigation loop yourself versus buying one of these platforms, it's worth reading MCP vs Skills vs Agents With Scripts: Which One Should You Pick? first, since the architecture decision behind "agent that calls tools live" versus "agent with a fixed skill set" applies just as much to SRE tooling as it does anywhere else. What This Actually Brings to the SRE Persona It's worth being specific about what changes for the person on-call, not just what the vendor deck claims: Fewer 2 a.m. investigations that start from zero. The agent has usually already ruled out the obvious suspects by the time a human looks at the page, so the engineer starts from a hypothesis instead of a blank terminal.Less tool-hopping. A lot of incident time isn't spent thinking; it's spent switching between Datadog, Grafana, the CI pipeline, and Slack. An agent that queries all of them in parallel removes a genuinely tedious chunk of the job.A written trail for free. Because the agent's investigation is itself a structured log of what it checked and why, you get a decent postmortem skeleton as a byproduct, not a separate task.A new failure mode to watch for. Engineers can start trusting the proposed root cause without checking the evidence trail, especially under pager pressure. That's a habit worth actively training against, not assuming away. None of this replaces the on-call engineer's judgment. It changes the shape of their shift from "gather evidence, then decide" to "review evidence, then decide," which is faster but only as trustworthy as the evidence the agent actually gathered. Comparing the Tool Landscape Here's how some of the major players stack up on where they've actually invested in AI, versus where it's mostly a checkbox feature. I've split this into two tables, because lumping AIOps correlation platforms in with dedicated AI SRE agents hides a real difference in what these tools do. Established AIOps and incident platforms: ToolAlert CorrelationRoot Cause SuggestionAuto-Drafted PostmortemsPredictive CapacityOwnership / StatusPagerDuty (AIOps)StrongModerateYesLimitedIndependent, public companyMoogsoftStrongStrongNoNoAcquired by Dell Technologies (2023)BigPandaStrongModerateLimitedNoIndependent, privateDatadog (Bits AI)ModerateStrongYesModerateBuilt in-house by DatadogServiceNow (Now Assist)ModerateModerateYesStrong (ITOps)Built in-house by ServiceNowDynatrace (Davis AI)StrongStrongLimitedStrongBuilt in-house by Dynatraceincident.ioModerateLimitedYesNoIndependent, privateRootlyModerateLimitedYesNoIndependent, private Dedicated AI SRE agents: ToolCore ApproachActs Autonomously?Best FitFounded / BackingTraversalCausal ML across dependency graphInvestigation autonomous, remediation guardedTeams with strong existing observability wanting sharper RCA2023, Sequoia and Kleiner PerkinsResolve.aiBroad agentic reasoning over code, infra, telemetryInvestigation autonomous, remediation opt-inTeams wanting one agent across incidents, capacity, and config2024, Greylock-led seedClericMulti-source parallel investigationRead-only by defaultTeams new to AI SRE agents, wary of write access2024, Zetta Venture PartnersCauselyCausal reasoning on cascading alertsInvestigation onlyEnvironments with alert storms and unclear blast radiusPrivate, early stageNeuBirdLLM-driven telemetry analysis at scaleInvestigation, guided remediationLarge enterprise IT environmentsPrivate, early stage A caveat worth stating plainly: I haven't run rigorous side-by-side benchmarks on all of these, and vendor claims move faster than reality, especially in the AI SRE agent table where most of these companies are one to three years old and evolving month to month. Treat both tables as a directional map, not a scorecard, and validate against your own alert volume before picking one. Deterministic AI vs. Generative AI in These Tools This distinction gets muddled in vendor marketing, so it's worth separating clearly. AspectDeterministic / ML-based (older AIOps)Generative AI (LLM-based, newer)ApproachStatistical pattern matching, clustering, anomaly detectionLanguage model reasoning over logs, tickets, chat history, and live queriesPredictabilityHigh, same input gives same outputLower, outputs can vary between runsStrengthCorrelation, anomaly detection at scaleSummarization, hypothesis generation, natural language explanation, draftingWeaknessPoor at explaining "why" in plain languageCan hallucinate a plausible-sounding but wrong root causeWhere it shows upDynatrace Davis AI, Moogsoft's original correlation engineDatadog Bits AI, ServiceNow Now Assist, Traversal, Resolve.ai, ClericTrust level neededCan often auto-remediateNeeds human review before action Most modern platforms now run both in tandem: the deterministic layer does the anomaly detection and correlation, and the generative layer explains it in plain English, forms hypotheses, and drafts the writeup. That combination is doing more real work than either piece alone, and it's basically the same pattern I described for agent observability generally in the AI agent architectures piece linked earlier: a fast, boring, reliable layer underneath a slower, flexible reasoning layer on top. Where We're Headed in Part 2 Incident response gets the spotlight because it's the loudest part of the job, but if you track where an SRE's actual week goes, a lot of it isn't firefighting at all. It's chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. In Part 2, I'll walk through where AI is showing up in ITOps specifically, and then go further into chaos engineering, SLO and error budget management, on-call toil reduction, and capacity planning, the quieter parts of the job that determine whether the incident tools in this article even have a fighting chance.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Most SRE teams do not need another dashboard. They need a safer way to move from "something is wrong" to "we know what to do next." A model that detects anomalies is useful. A model that can touch production can also make a bad incident worse. That is where most conversations about AI in SRE become too optimistic for my taste. The hard part is not only detection. It is deciding how much autonomy the system should have, under which conditions, and with what blast-radius controls. I learned this while working on large-scale cloud services where one customer-facing symptom could turn into a flood of alerts. A degraded dependency might show up as latency in one service, retries in another, queue growth somewhere else, and CPU pressure downstream. During an on-call shift, that can look like five separate problems. Usually, it is one problem echoing through the stack. That experience changed how I think about self-healing infrastructure. The goal is not to build a system that blindly fixes everything. The goal is to build an operational control loop that can separate routine, low-risk recovery from incidents that still need human judgment. The model that has worked best for me is graduated autonomy: Let the system act automatically only when the action is well understood, reversible, and narrow in blast radius. For everything else, the system should collect evidence, recommend the next step, and keep humans in control. Why Static Alerts Stop Scaling Static alerts are not the enemy. I still want to know when disk usage is dangerous, error rates spike, or latency crosses a service-level threshold. But thresholds do not understand context. A CPU spike during a scheduled batch job may be normal. The same spike during steady-state traffic may be a retry storm. A latency increase in one region may be harmless during a controlled deployment, but suspicious if it appears across multiple availability zones with no recent change event. At small scale, engineers can carry that context in their heads. At enterprise scale, they cannot. Services emit hundreds of metrics across regions, dependencies, deployments, and customer paths. Eventually the team is no longer tuning alerts. It is negotiating with noise. In one rollout I was involved with, the most useful improvement was not adding more alerts. It was grouping alerts around dependency context and suppressing repeated downstream symptoms. The on-call experience became calmer because engineers could focus on the likely failure path instead of chasing every red graph independently. That is the kind of problem AI can help with. Not by replacing SRE judgment, but by organizing noisy signals into a more useful operational story. Detection Is Only the First Layer ML-based anomaly detection helps because it learns a service's normal operating shape instead of relying only on fixed thresholds. For cloud metrics, that usually means learning seasonality, traffic cycles, deployment windows, regional differences, and service-specific behavior. An LSTM autoencoder, isolation forest, or well-tuned statistical baseline can all be useful. I care less about the model family than the quality of the telemetry around it. A simple model trained on clean, consistent data will usually beat a sophisticated model trained on messy metrics. A practical anomaly pipeline usually looks like this: Collect metrics, logs, traces, and change events.Normalize them by service, region, dependency, and time window.Score each signal against its learned baseline.Group anomalies by dependency graph and recent changes.Produce an evidence bundle for automation or human review. Here is a simplified version of the scoring stage: Python from dataclasses import dataclass from typing import List @dataclass class MetricWindow: service: str region: str signal: str values: List[float] recent_deploy: bool = False @dataclass class AnomalyScore: service: str region: str signal: str score: float reason: str class BaselineModel: def expected_range(self, service: str, region: str, signal: str): # In production, this may come from a trained model, # feature store, or rolling baseline per service and region. return (0.0, 1.0) def score_window(window: MetricWindow, baseline: BaselineModel) -> AnomalyScore: low, high = baseline.expected_range( window.service, window.region, window.signal, ) latest = window.values[-1] if latest > high: distance = (latest - high) / max(high, 0.001) reason = f"{window.signal} above learned baseline" elif latest < low: distance = (low - latest) / max(abs(low), 0.001) reason = f"{window.signal} below learned baseline" else: distance = 0.0 reason = "within learned baseline" if window.recent_deploy and distance > 0: reason += " during recent deployment window" return AnomalyScore( service=window.service, region=window.region, signal=window.signal, score=min(distance, 1.0), reason=reason, ) The production value is not just the score. It is the metadata around it: ownership, dependency path, recent deploys, feature flag changes, customer impact, and whether the same pattern has appeared before. A single anomalous metric should rarely trigger remediation. Sustained anomalies across correlated signals are more trustworthy than one spike in one chart. Correlation Turns Noise Into an Incident Story During an incident, the useful question is not "Which graph is red?" It is "What changed first, and what depends on it?" That is where dependency-aware correlation becomes more useful than raw anomaly detection. A database issue may surface as API latency, retries, queue saturation, and CPU pressure. Without a dependency graph, every downstream service looks guilty. With one, the system can rank likely causes instead of handing the engineer a wall of symptoms. A useful correlation engine should look at topology, timing, change context, and customer impact. Which dependency failed first? Was there a deployment or config change? Which service is closest to the customer-facing error? The evidence bundle should be readable by a human. If the model says "root cause confidence: 0.86," that is not enough. It should also explain why. JSON { "candidate_root_cause": "identity-token-cache", "region": "example-region-1", "confidence": 0.86, "customer_impact": "elevated authentication latency for a subset of requests", "supporting_signals": [ "p99 latency above learned baseline for multiple consecutive windows", "cache hit rate dropped below its recent operating range", "downstream services showed retry growth after the initial cache anomaly", "no database saturation was observed", "no deployment was detected in the immediate incident window" ], "recommended_action": "drain_and_restart_one_cache_node", "estimated_blast_radius": "single node in a redundant pool", "rollback_plan": "keep node out of rotation if health checks fail after restart" } This is more useful than another alert. It gives the on-call engineer a starting hypothesis and the reasoning behind it. The Graduated Autonomy Model The most important design decision in self-healing infrastructure is not which ML algorithm to use. It is which actions the system is allowed to take. I divide remediation into three tiers. Tier 1: Fully Automated, Low-Risk Actions Tier 1 actions are safe, reversible, and narrow in blast radius. These are actions the system can execute without waiting for a human when confidence is high. Examples include restarting one unhealthy instance, scaling out a stateless service, draining one bad node, flushing a bounded cache, or shifting a small amount of traffic away from a degraded zone. The key phrase is bounded blast radius. Auto-remediation should not restart half the fleet, fail over a primary database, or disable a feature globally just because a model is confident. Confidence is not a substitute for safety. Before I put an action in Tier 1, I expect it to pass these checks: it is reversible, affected capacity is small, redundancy is healthy, there is no active global incident, the same action has not failed recently, rollback is defined, and health checks can verify success quickly. The first Tier 1 actions should be boring. Restarting one unhealthy node is not exciting, but it is exactly the kind of action that can be automated safely when the system has enough evidence. Tier 2: Automated Recommendation With Human Approval Tier 2 is where many real incidents live. The system may know what should happen, but the action still needs human approval. Examples include rolling back a deployment, disabling a feature flag, failing over a database, increasing capacity beyond a normal band, or changing regional routing. For Tier 2, the system should prepare the action, show the evidence, and ask for approval. The human should decide whether the action makes sense, not build the command during the incident. One pattern I have seen repeatedly: the slowest part of remediation is not always finding a likely cause. It is gathering enough confidence to take a risky action. When the system attaches deploy timing, error movement, affected endpoints, config changes, and rollback commands into one review card, the decision becomes easier. Tier 3: Human-Led With AI Context Tier 3 incidents are novel, high-risk, or ambiguous. The system should not execute remediation. It should help humans reason. This includes possible data corruption, multi-region cascading failures, security-sensitive incidents, conflicting signals across dependencies, low-confidence root-cause analysis, or any action with unclear rollback behavior. In Tier 3, the system's job is to summarize what it knows, what changed recently, which hypotheses are most likely, and which dashboards or runbooks are relevant. That alone can save time, but it keeps production control where it belongs. Architecture: A Control Loop, Not a Magic Button A practical self-healing system looks like a control loop with guardrails. Architecture diagram: Graduated autonomy model for self-healing infrastructure The important part of this diagram is the policy gate. Detection and correlation produce a recommendation, but the policy gate decides autonomy. Without that layer, "self-healing" becomes a risky automation script with an ML label attached. The policy gate should evaluate confidence, risk, blast radius, recent action history, service criticality, and rollback readiness. I would express that as policy-driven code: JSON from dataclasses import dataclass from enum import Enum from typing import List class Decision(str, Enum): AUTO_EXECUTE = "auto_execute" REQUEST_APPROVAL = "request_approval" HUMAN_LED = "human_led" @dataclass class RemediationProposal: action: str confidence: float blast_radius_percent: float reversible: bool rollback_defined: bool service_tier: str evidence: List[str] @dataclass class RuntimeContext: active_global_incident: bool recent_failed_action: bool healthy_redundancy: bool minutes_since_last_same_action: int TIER_1_ACTIONS = { "restart_single_instance", "scale_stateless_service", "drain_single_node", "flush_bounded_cache" } TIER_2_ACTIONS = { "rollback_deployment", "disable_feature_flag", "database_failover", "regional_traffic_shift" } def decide_autonomy( proposal: RemediationProposal, context: RuntimeContext ) -> Decision: if context.active_global_incident: return Decision.HUMAN_LED if context.recent_failed_action: return Decision.HUMAN_LED if not proposal.rollback_defined: return Decision.HUMAN_LED if proposal.action in TIER_1_ACTIONS: safe_enough = all([ proposal.confidence >= 0.90, proposal.blast_radius_percent <= 5.0, proposal.reversible, context.healthy_redundancy, context.minutes_since_last_same_action >= 30, len(proposal.evidence) >= 3, ]) return Decision.AUTO_EXECUTE if safe_enough else Decision.REQUEST_APPROVAL if proposal.action in TIER_2_ACTIONS and proposal.confidence >= 0.75: return Decision.REQUEST_APPROVAL return Decision.HUMAN_LED This is not drop-in production code, but the structure is the point: actions are classified, confidence is not the only input, and safety can override the model. In reliable systems, the model proposes; policy disposes. What I Measure Before Expanding Autonomy I would not start by asking, "Can we automate remediation?" I would start by asking whether the system's recommendations are trustworthy. Before allowing Tier 1 execution, I would track root-cause precision, false positives by service, recommendation acceptance, time to useful diagnosis, remediation success, rollback frequency, and any secondary incidents caused by remediation. The last two matter the most to me. A self-healing system that fixes one issue but creates another is not healing. It is moving the incident. My preference is to run in shadow mode first. Let the system detect, correlate, and recommend, but do not let it execute. Compare its recommendations against what engineers actually did. Once the system repeatedly recommends the same low-risk actions humans already take, graduate those actions into Tier 1. That is how trust gets built: not through a big launch, but through repeated correctness in narrow, well-understood situations. Lessons Learned From Building Toward Self-Healing The most useful lessons are not about model architecture. Clean telemetry beats clever models. If service names are inconsistent, regions are missing, logs are unstructured, and ownership metadata is stale, the model will struggle. Before debating LSTMs versus transformers, fix the telemetry pipeline. Change events are first-class signals. Deployments, config pushes, schema changes, and feature flag flips explain many anomalies. If the model cannot see change events, it will treat every incident like a mystery. Alert suppression is not the same as diagnosis. Reducing noise is useful, but the system must preserve the causal path. Suppressing duplicate downstream alerts only helps if the upstream root cause remains visible. Automation needs a memory. Every remediation should leave an audit trail: what was detected, what action was taken, what happened afterward, whether rollback was needed, and whether humans agreed with the recommendation. Start with boring actions. Restarting one bad instance is not glamorous. Draining one node is not a research breakthrough. But these are exactly the kinds of actions that make sense for early autonomy because they are repeatable, reversible, and easy to verify. Where LLMs Fit Large language models are useful in SRE, but I would not put them directly in the execution path for remediation. Their best role is communication and context assembly. An LLM can draft an incident summary, explain the evidence bundle, turn raw telemetry into a timeline, identify runbooks, and prepare a post-incident report. That saves time without giving the model direct control over production. The safer pattern is separation of responsibilities: ML or statistical models detect anomalies, graph correlation ranks likely causes, policy gates decide autonomy, deterministic automation executes approved actions, and LLMs summarize what happened. That separation keeps the high-risk parts deterministic and auditable while still using AI where it helps most. Final Thought Self-healing infrastructure is not about removing SREs from production. It is about removing the repetitive, low-risk work that slows them down during incidents. The best version of AI in SRE is not a magic system that fixes everything. It is a careful control loop: detect early, correlate intelligently, act only within policy, and learn from every outcome. If you are building toward self-healing, do not start with full autonomy. Start with evidence. Then recommendations. Then approval-based actions. Then, only after the system has earned trust, allow narrow automated remediation. That path is slower than the hype cycle, but it is much closer to how reliable infrastructure actually gets built.