Performance

Recently, I made a comment about the idea of there being a “best” monitoring tool: In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case. The comment got some pushback, both privately and in a few forums, and so I wanted to dig into what I meant and why I feel that way. But before I do that, I want to set the record straight: I stand by what I said about there being no “best” monitoring tool. But I was flat-out lying about the other stuff. The hills I’m willing to die on are: The best programming language is PerlThe best car is the 1967 Ford Mustang 390 GT/AThe best pizza style is deep dish, and I’m partial to getting it from Tel Aviv Kosher Pizza in Chicago With that cleared up, let’s get back to monitoring and observability. Zoom Zoom What really got me thinking about the false concept of a “best” monitoring tool was a video my son shared with me, comparing a Lucid Air Sapphire, a Bugatti Chiron, and a Tesla Plaid to see which was the fastest production car of all time. Disclaimer: I am NOT a car person. My son Kaleb (who is 21, in his 2nd year of university to become a mechanical engineer, and on two different Baja SAE teams) very much is. After watching the video, Kaleb pointed out that if the race (which was on a quarter-mile track) had been a half-mile instead, the Bugatti would have won, hands down. This was because (reasons. I honestly couldn’t follow the things he was saying at this point. I leave it to the reader to imagine Star Trek-like technobabble) My point in all this is that “best” — the fastest car in this case — is highly subject to other variables. The type of track (this was on a track that had been pre-treated with VHT, which makes it super sticky and affects traction), the distance, even things like altitude and weather — these all can impact the ultimate outcome. But I’m talking about more than external factors. What is “best” can be affected by the ultimate use case. Cost is the easiest one that comes to mind. Yes, a Bugatti might be fastest. But possibly the “best” car is a Honda Civic because you value cost and reliability over speed. Or perhaps a Kia Sedona might be “best” because you need more seats. Or a Ford F150. Or a Ryder 16-wheeler. This all relates to monitoring and observability in ways that are both important and, sadly, novel to a lot of IT practitioners. We get so caught up in the “speeds and feeds” aspect of tools and solutions — how many flows per second, how many traces per collector, maximum ingest before backpressure occurs — that we often fail to stop and say “Do I need that? Will I ever need that?” I shared this with my friend Kevin Sparenberg (another car guy, who writes occasionally on ), and he added: Monitoring tools are like any other tool. You have your favorite hammer, but it’s not appropriate in every scenario. You don’t (or shouldn’t) use a sledge for setting a nail, nor would you use a claw hammer for forging. You might have a favorite, but you need to consider the type of work you need accomplished. (Besides being a car guy, he’s a home repair weekend warrior. This is just one of the many reasons why we’re friends.) “Best” Isn’t Always Best Part of the blame can be placed at the feet of vendors. I’ve worked at a few, and it’s rare to find one that has the tools to help customers quantify volume and cost before implementation. Most simply say “let’s just get this installed, and we’ll see where it lands, and we can tune from there,” blithely ignoring the way the level of effort (not to mention political maneuvering in the C-suite) to implement a new tool makes sunk-cost fallacy a near-certainty. But that’s only part of the blame. The other part rests at our feet — the monitoring engineers who need to better shoulder the responsibility of understanding and speaking for the needs of our organization. Because if we don’t, who will? Much of this comes back to things I’ve already ranted about: If we don’t have a plan for the monitoring and observability data being collected, any cost is going to seem to be too much. If you have a plan, you’ll know exactly how much the data is worth, and be able to evaluate the cost of a tool. Learn to speak the language of business, to frame things NOT in the technical terms that you find familiar and comfortable, but in terms that make it clear to the business why a new tool is needed and the value it will provide. Solve the problems your organization is actually having. Once again, it’s easy to get swept up by a vendor’s vision. But if your company isn’t having any of the problems that vendor vision describes, it’s all wasted time and money. Kevin added another nuance. Hyper-focusing on a single metric isn’t just sloppy; it can lead to real problems down the road: I would even go so far as to revisit that zero-to-60 metric. What about zero-to-60-to-zero? Sure any car can get to 60, but which car is able to do that AND return to status quo quickly. If you need a good “bad” example, look at the Gen 1 for Dodge Viper. As my dad said, “plenty of giddy-up, virtually no whoa.” How does that translate to monitoring and observability? Think about alerting. LOTS of tools are able to detect and trigger alerts based on extremely specific (and sensitive) triggers. But far fewer have the controls to detect and stop alert storms. My point in all this is to remember that “best” always has to be weighed against YOUR values: Your real-world, actual business needsYour cost-to-benefit ratioYour team’s skillsetsYour tolerance for toil and effort during the transitionYour willingness to support one more tool in perpetuity…and so on. Good Enough is Usually Good Enough There is a story that is equal parts old, hilarious, and fake. It involves a hapless young man (it’s ALWAYS a dude) who decides to mount a JATO rocket to his car to see just how fast he can go. And in the ensuing chaos, this young man (supposedly) earned himself a posthumous Darwin Award. (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) However, one of the aspects of the story that makes it funny (at least in my opinion) is the sheer ridiculousness of it all. Sure, lots of folks want a fast car. Many of those folks are willing to spend a little extra for a car that’s a little faster than the norm. Fewer (but not zero) people might also be willing to go to great lengths to acquire not only a fast car, but “the fastest” car. But strapping a rocket to the top of an old car — one that was clearly never intended to be used this way? That is some serious janky automotive slapstick. It tickles our funny bone by evoking those grainy sepia-tinted images of early 20th century “flying machines” that were nothing more than 2 umbrellas strapped to a piston. As the final image of the JATO story fades in our mind, we can almost see the end-card saying “You just couldn’t leave well-enough alone, could you?” Likewise, we need to foster a habit of self-restraint and technical reflection in our monitoring discipline. We have to recognize when our excitement about a tool’s ability to process 8 million log messages a second clouds our own ability to step back and say “why do I even HAVE 8 million log messages a second?” And if there’s a good reason for that volume, follow up with the question “Why do I need to send every single one of those messages across the internet to a vendor’s storage?” I’m not saying there’s nobody in the world who might need that. You might have a good reason. I just want to suggest you take a second to consider things before you end up creating your own JATO-powered observability disaster. The Solution Is Both-And, not Either-Or This is the thing most vendors don’t say, at least admit within earshot of investors and members of the board, because it flies in the face of the marketing hype and sales pitches they’ve worked so hard to craft. You need more than one monitoring and observability solution. You need to decide which systems (and more fundamentally, which data on those systems) use each tool. You’re going to have to split your budget (time, effort, skills, money) between those tools. This is an ugly but unavoidable truth. Over 27 years working with monitoring and observability tools, the number of times I’ve seen a company that had exactly one monitoring solution is: zero. Even the ones that insist they do, after a little digging, have at least a few pockets of the environment that use something else, whether that’s a class of system (mainframes, minis, Windows NT servers, 6509’s); an organization or team that just went their own way or was acquired but never fully integrated; or a location that — due to their distance from the main organization, either functionally or geographically — has to maintain their own set of tools. And most orgs don’t have “at least a few pockets”. They have a full suite of overlapping solutions. You are going to have — more likely you already DO have — multiple monitoring solutions in place. This is one of those tech realities which is simple, but not easy — like supporting more than one operating system (whether servers or desktops); or moving from branch-based development to feature flags; or building a multi-language, multi-cloud app. It’s a cost of doing real business in the real world. Observability, like life itself, is messy. It also, to quote Ian Malcom, finds a way. So will you. Here’s how: Plan and prepare to identify data types based on complex criteria: it might be a combination of location, system type, data type, and even time frame. Expect that, based on those parameters, you’ll then filter and transform the data before sending it in the correct direction. Also expect that some data types are so valuable, you’ll end up sending the same data in more than one direction. But also prepare to put boundaries in place so you aren’t doing that all the time, because that gets expensive fast. As I said, you need to be ready to support multiple tools, but you should have a plan in place for how you’ll keep track of those tools, identify their primary use case, and even set boundaries on the things they will NOT be permitted to monitor, so your stable of solutions doesn’t explosively get out of control. One way to do that is to set, for every data type or use case, a definitive choice for a primary tool that handles that data, and a secondary that you use as a gut-check. For larger organizations or more important data sets, you might have a tertiary, but draw the line there. Another way (complementary to the first) is to understand that some tools are cheap and do a lot of things mostly ok, so you can spread them like peanut butter across the enterprise; while others are expensive (in time, effort, or money) and only do certain things well. So you should spread THOSE like caviar — only on the systems where they’ll do the most good. Finally, differentiate between management tools that also have a monitoring component and true monitoring and observability solutions. You shouldn’t get rid of management tools, but you also shouldn’t make them the primary source of truth for enterprise monitoring information because they are usually so vendor- or system-specific, and it will lead (again) to that explosion of tools I cautioned against earlier. A Brief Buyer’s Guide A natural question to ask next is “how do I know WHICH tools to get?” Once again, my buddy Kevin has some wise observations: Is it even worth mentioning bake-offs? Do people even do that anymore? Maybe tool A has these features, but tool B has these other ones. And they both share some other capabilities. We need all of it, but can’t get it in one package. There’s nothing wrong with that. Leon’s point about having multiple tools is on target. Bias your decisions to picking the right tool for the right job (but at the same time, try not to collect too many tools. This ain’t Pokémon). Don’t buy for “this tool has this neat feature we don’t need, but maybe someday we’ll want.”. Buy (or deploy) for what you need now and in the near future. IT (and the business) is always growing and evolving. What you THINK is important today may be irrelevant in six months. Hopefully not, but thinking too far ahead — the infamous “five year plan” is just a waste of your effort and time. Taking a Victory Lap The point of this blog is pretty simple: There’s no such thing as “best”, and that goes for everything from cars to programming languages all the way to observability solutions. But more essential is my point about WHY there isn’t a single, specific “best” — it’s because context matters. Use case matters. To be more nuanced, there probably is a “best,” but what is best is extremely particular to you and your circumstances. So the lesson in all this is to make sure you are clear about those circumstances, and that you’re always weighing them against the “we’re the best” marketing hype you’ll hear from many vendors.

The Problem With "Just Add More Workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew, and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformationDiagnosing and fixing data skew with salting and adaptive query execution (AQE)Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant filesGoverning access to the whole pipeline with Unity Catalog Architecture Overview Pipeline shape – a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage – this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1: Set Up Governed Tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. SQL -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics; CREATE SCHEMA IF NOT EXISTS retail_analytics.events; CREATE TABLE IF NOT EXISTS retail_analytics.events.raw_orders ( order_id STRING, customer_id STRING, product_id STRING, quantity INT, event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders'; CREATE TABLE IF NOT EXISTS retail_analytics.events.dim_products ( product_id STRING, category STRING, unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products'; GRANT SELECT ON TABLE retail_analytics.events.raw_orders TO `analysts`; Step 2: Read and Force a Broadcast Join for the Small Dimension Table dim_products is small, so let Spark broadcast it rather than shuffle both sides of the join. Python # pipeline.py from pyspark.sql import functions as F orders = spark.table("retail_analytics.events.raw_orders") products = spark.table("retail_analytics.events.dim_products") Without the explicit broadcast hint, Spark's cost-based optimizer usually picks a broadcast join automatically for small tables, but being explicit avoids surprises when the dimension table grows past spark.sql.autoBroadcastJoinThreshold (default 10MB) without anyone noticing. Step 3: The Aggregation That Triggers a Shuffle groupBy on customer_id is a wide transformation — Spark must shuffle rows so all records for a given key land on the same reducer. Python agg = ( joined .groupBy("customer_id", "category") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) If one customer_id (say, a test account or a large B2B buyer) accounts for a disproportionate share of rows, this is where skew shows up: one reducer task runs for minutes while the rest of the stage finishes in seconds. You'll see this in the Spark UI as a single long-running task in an otherwise short stage. Step 4: Fixing Skew With Salting Adaptive Query Execution (AQE) handles a lot of skew automatically in modern Databricks Runtime, but for known hot keys, explicit salting is still the most predictable fix. Python from pyspark.sql import functions as F import random SALT_BUCKETS = 20 # Add a salt column to spread the hot key across multiple reducers salted = joined.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) partial_agg = ( salted .groupBy("customer_id", "category", "salt") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) # Second pass: combine the salted partial aggregates into the final result final_agg = ( partial_agg .groupBy("customer_id", "category") .agg( F.sum("total_spend").alias("total_spend"), F.sum("order_count").alias("order_count"), ) ) This two-phase pattern — pre-aggregate on a salted key, then combine — is the same trick used inside combiners in older MapReduce systems. It trades a bit of extra shuffle for eliminating the single-reducer bottleneck. Also worth setting explicitly rather than relying on the default of 200: Python spark.conf.set("spark.sql.shuffle.partitions", "auto") # let AQE size it dynamically spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") Step 5: Write to Delta Lake With Optimized Writes Python ( final_agg.write .format("delta") .mode("overwrite") .option("delta.autoOptimize.optimizeWrite", "true") .saveAsTable("retail_analytics.events.customer_spend_summary") ) Optimized writes shuffle data before writing so you get fewer, better-sized files instead of many small ones — this costs some write-time latency in exchange for faster reads later. Step 6: Z-Order the Table for the Queries That Matter If most downstream queries filter on customer_id, co-locate related rows physically so Spark can skip files that can't match the filter. Python OPTIMIZE retail_analytics.events.customer_spend_summary ZORDER BY (customer_id); Z-ordering aims to produce evenly balanced data files by row count rather than raw size, and its effectiveness depends on the column having reasonably high cardinality — Z-ordering by a low-cardinality column like category alone gives little benefit. On newer Delta Lake / Databricks Runtime versions, Liquid Clustering is generally the preferred choice for new tables since it adapts as query patterns change, while ZORDER remains relevant mainly for existing tables not yet migrated. SQL -- Preferred on new tables (Databricks Runtime supporting Liquid Clustering): CREATE TABLE retail_analytics.events.customer_spend_summary CLUSTER BY (customer_id); Comparing Shuffle-Mitigation Techniques TechniqueFixesCostWhen to useBroadcast joinShuffle on the large side of a joinExtra memory on executorsSmall (<~10MB by default) dimension/lookup tablesAQE skew join handlingAutomatic detection of skewed partitionsMinor planning overheadDefault-on; good general safety netManual saltingKnown, severe hot keysExtra shuffle for the two-phase aggregateHot keys that AQE doesn't fully resolveRepartition by keyUneven task distribution before a shuffle stageOne extra shufflePre-shaping data before multiple downstream joinsZ-OrderingSlow reads due to unnecessary file scansShuffle + rewrite during OPTIMIZEExisting tables, high-cardinality filter columnsLiquid ClusteringSame as Z-Order, plus evolving query patternsOngoing background clusteringNew tables on supporting runtime versions Governance: Tying It Back to Unity Catalog Because every table above was created under retail_analytics, access control, audit logging, and lineage are handled centrally rather than per-cluster: SQL -- Restrict PII-adjacent columns without duplicating the table CREATE VIEW retail_analytics.events.customer_spend_summary_masked AS SELECT customer_id, category, total_spend, order_count FROM retail_analytics.events.customer_spend_summary; GRANT SELECT ON VIEW retail_analytics.events.customer_spend_summary_masked TO `bi_readers`; Production Considerations Diagnose before tuning. Check the Spark UI's stage view for one long-running task among many short ones — that's the signature of skew, not just "the job is slow."Predictive optimization can run OPTIMIZE and ANALYZE automatically on Unity Catalog-managed tables, which reduces the need for scheduled maintenance jobs for many workloads.Don't Z-Order everything. It requires shuffling and rewriting the table (or partition), so reserve it for columns that are actually filtered on frequently downstream. References Best practices: Delta Lake — Azure Databricks / Microsoft LearnData skipping (Z-ordering) — Azure Databricks / Microsoft LearnOptimizations — Delta Lake documentationDelta Lake Under the Hood: What Every Data Engineer Should Know — Databricks CommunityMastering Delta Lake Performance: Z-Ordering vs Liquid Clustering — Medium

By Jubin Soni, FBCS CORE

One of the biggest problems that teams managing large-scale distributed systems face is alert noise. Getting precise signals when something is wrong in production services is critical for maintaining the stability of production systems since it enables teams to reduce the time to mitigate issues that impact customers and helps uphold the SLAs promised to customers. In the era of AI, where anyone can write and ship code, reliability becomes a key differentiator for companies. Effective alerting is one of the important aspects of improving and maintaining reliability. In this article, I describe a set of tools and processes that can be incorporated to improve alerting effectiveness for large-scale distributed systems. Best Practices for Production Alerting Treat Alerting Configuration as Production Code Teams typically apply rigorous engineering practices to production code, including unit testing, integration testing, formal code reviews, version control, and pull requests. Applying the same discipline to alerting configurations can significantly improve alert quality, reduce false positives, and make your monitoring system far more reliable. Use Automation as the First Line of Defense Traditional alerting follows a familiar pattern: when a metric crosses a predefined threshold, an alert is triggered and the on-call engineer is paged. The engineer then follows a runbook to diagnose the issue and apply the appropriate mitigation. A more effective approach is to make automation the first line of defense. Automated remediation can resolve many common issues before a human is ever notified, reducing both time to mitigation and the operational burden on on-call engineers. The on-call engineer should only be paged if the automated actions fail to restore the service or require human intervention. Every Alert Should Be Actionable An alert is as good as its actionability. When an alert fires, the on-call engineer should be able to understand what is broken, why it is broken, what immediate action should be taken, and who is responsible. To provide this information, every alert should have proper context, such as links to relevant dashboards, clear remediation steps, outlined escalation procedures, and the contact information of the responsible team to which it needs to be escalated. The runbooks included in the alerts should be regularly reviewed to make sure they are not outdated. Review and Tune Your Alerts Periodically Every alert in the production environment was created for a specific reason. It could have been for a new feature, a repair item for a production incident, etc. However, production systems evolve over a period of time with deployments, configuration changes, new features, and deprecated features. The assumptions for the alert would have changed. It is critical to review alerts periodically to assess the validity of the alert or the underlying conditions and see if the alert thresholds need to be adjusted, or the scope of the alert needs to be changed, or if the alert has to be deprecated. This review can be a weekly or monthly review. Have Mechanisms to Suppress Alerts You may have the perfect alerting system that is noise-free. However, sometimes you may still get valid alerts, but you may need to ignore them for a period of time. For example, you may have a planned production changes which will trigger the alerts which may trigger alerts for expected conditions. Another example is that you may get alerts due to a bug, which could take a while to fix and deploy to production. During this time, you may need to ignore the alerts for a while. Having a mechanism to suppress these alerts will help reduce the expected noise and let your on-call engineers focus on genuine issues. Combine Multiple Metrics to Make Alerting Effective Individual alerts can become misleading in certain cases. Implementing logic to combine multiple signals into a single alert can be effective instead. For example, a spike in system resources such as CPU, memory, etc might be usual during a surge in traffic. However, if there are signals of increased errors or latency, it can indicate that the CPU or memory spike is problematic for the service's health. Combining these two signals can provide accurate alerts instead of using a single signal. Gain a Deeper Understanding of Your System Understanding the service you manage more deeply can be invaluable in creating effective alerts. SRE’s often focus on non-functional aspects of the service. However, spending time on understanding functional aspects of the service can help create an effective alerting system for the service. For example, if you are managing a WebRTC system where latency is paramount to the user experience. Understanding how various components interact with each other and where the latency bottlenecks can arise helps you devise alerts at the individual subsystem level to catch these bottlenecks when they arise with a proper alerting strategy. Find Gaps in Your Alerting While dealing with false positives is extremely important, it is equally important to address false negatives. The impact of not catching issues in production before your customers experience degradation or unavailability of the service can impact SLA’s. Use Chaos Engineering to Find Gaps Netflix pioneered the concept of Chaos Engineering. Chaos Engineering is the practice of proactively injecting controlled failures such as server crashes, network failures, etc., into production systems to understand and identify systematic weaknesses before they can turn out to be outages. We can use this chaos engineering testing as an opportunity to validate if all the alerts that were supposed to fire during the testing actually fired. If there are any gaps, such as improper thresholds or missing alerts, they can be fixed before any real outages occur. Never Let an Incident Go to Waste It’s a best practice to have blame-free post-incident reviews after an incident occurs in production. These reviews should be utilized to review the alerts to verify if the alerts fired effectively and in a timely manner, whether the alerts fired were actionable, and whether the alerts had enough context with them. Any issues or gaps with the alerts should be promptly fixed so that any future incidents are caught in a timely manner and addressed effectively. Conclusion There is no silver bullet to create effective alerts, but following this set of best practices will help to reduce the noise and get precise signals with alerts, which will help catch issues in production in a timely manner, reduce time to mitigate issues, and uphold SLOs.

It is good to hear that the CNCF OpenTelemetry project has become a graduated project (7 years after Fluentd, which encapsulates Fluent Bit, which is also OTLP-compliant). Admittedly, OpenTelemetry is a far larger project, as it provides tooling for tasks such as code auto-instrumentation for applications and many other considerations, such as: Open Agent Management Protocol (OpAMP)ProfilesSemantic Conventions (descriptions of how to carry data for different contexts from Generative AI to Functions as a Service). While the Semantic Conventions will help organizations implement Observability around AI, given its non-deterministic nature, it makes observability essential for understanding more about what has happened. The Open Agent Management Protocol is less notable today than the others, but it has the potential to impact solutions far beyond the OpenTelemetry space. For the rest of this article, I'm going to explore why I think this is the case, and to do that, we need to also understand the key capabilities that OpAMP enables. What Actually Is OpAMP? Let's start with the approach being adopted with OpAMP. Just as the OpenTelemetry project focused on getting an explicit definition and broad support for the OpenTelemetry Protocol (OTLP) before the OTel Collector development got real momentum. Today, OpAMP is essentially a protocol definition, with configuration managed gRPC definitions that can be generated to code stubs for nearly any language through Protobuf. The spec addresses the issue that OTel deployments typically have — lots of Collectors that are very widely distributed, and there is a need to centrally understand and manage the full picture. For example, handle scenarios such as: Being able to examine the operational state in depth by being able to retrieve information about the current configuration being used.Ability to control and command the agent to do things from a central location. Common actions such as updating the configCustom commands for specific. Use casesManage security considerations such as: alive and healthy down to a granular component levelManage the way agents can be identified and accepted dynamicallyCertificate management, e.g. cert rotationOrchestrate the deployment of plugin features, patches, and related assets. This could easily become monolithically large, but the clever decision is that the protocol includes a handshake in which the client and server can declare what they can and can't do, and allow custom commands to be exchanged. The minimum capabilities expected by OpAMP are small — enough to know what is being watched and that it is alive. This flexibility is where the key value for going beyond OpenTelemetry lies. We can add custom commands and use the same framework to control OTel Collectors just as easily as having a custom command handler that could command a database to change its memory allocation to cache, for example. Flexibility can often lead to complexity in message exchanges. While the payloads provide a lot of freedom to describe custom commands, for example, the exchanges aren't complex. The following illustrates one of the most complex exchanges, where authentication between the agent and server is done through certificates, and we need to refresh the certificates that the agent is using. The OpAMP CSR flow, when supported by both the client and server, allows authentication to be managed through certificates. Even with that complexity, only 4 message types are involved. Configuration deployment is not limited to OTLP, so you can dispatch configuration as you wish. This may require you to introduce a custom message so the client/agent understands what to do with the configuration file. Given the OpAMP name, it would be easy to assume that this does mean we have to deploy a new process. The documentation presents two models in which the agent logic can be embedded into the process: either as the OTel Collector or as a Supervisor, which, visually at least, points to the process that launches the thing being managed. Both of these have varying degrees of invasiveness. There is a variation of the Supervisor, which we've called Observer, where the agent knows how to figure out which process they need to monitor through interacting with the host; as a result, it doesn't impact (even to change how the process is launched) the component being observed. In our ChatOps model, we include a set of flows in Fluent Bit (or Fluentd) that perform housekeeping activities and provide feedback through observability channels. To make OpAMP easier for us, we have server logic that can receive instructions directed to a specific agent. The agent then triggers Fluent Bit, for example, to have an observability flow run that can include housekeeping activities. While we could simply give the agent a custom action to perform tasks directly, this separation of concerns reduces risk by isolating permissions. There Are Many Ways to Do That Already It is true that most of these things can be done within a Kubernetes ecosystem. BUT how many companies have their entire software estate purely running on K8S? How many pods do you really want to blow away just to tweak the log level of one package while you look into a possible app error, particularly if your pods are a first-stage transition to a Microservice ecosystem and are still very monolithic? What is interesting is that the OTel blog offers a way to combine OpAMP with the K8S control plane (although today there is a more direct K8S Operator). Yes, we can do a lot of this with GitOps, but not all of it. Not to mention, just because you've pulled the latest configuration doesn't mean the recipient has successfully adopted it. Again, we can go through the argument about how you could solve the problems with Chef or Puppet, etc. What all these arguments miss is that the control plane protocol is unique to the tool concerned. Here, OpAMP has flipped it on its head and said, "This is how you communicate; how you implement is your choice." Currently, the OTel Git repo contains the basic skeleton code. But I can imagine that over time, more flesh will be added to this, just as we saw with the Collector. Essentially, we're talking about the potential for a Universal Control Plane, but where the agents can be from any provider, and the control plane could be from another. This is perfect, as we can apply the same mechanisms in a K8S environment, on bare-metal HPC, on VM-based platforms that host monolithic apps, and for handling devices that may drop in and out of contact. Coming back to the Observability space again, this means that we can start to think about a single control plane that could manage OTel collectors, Fluentd, Fluent Bit, Elastic Stack and Beats components and manage it from a central plane - so now as our signals pass through possibly multiple points before reaching the Log Analytics, metrics dashboards etc we can see that each stage and each part of an end to end is operating. One Concern The biggest concern is possible adoption. Unlike OTLP, which focuses on standardizing how signal sources generate data and route it to the backend, where the core value comes from commercial vendors able to manage their fleets of collectors and perform clever analysis. OpAMP can be seen as commodifying the mid-tier. If a vendor makes their collector OpAMP-compliant, I could come along and start managing that agent (with some caveats around custom commands). If a vendor is charging per agent, they may well see them 'disappear' as the provided control plane no longer manages them. But for Open Source solutions such as Fluentd, Fluent Bit, Beats, and the Elastic Stack, we can now consider simplifying our enterprise. A Protocol Is Not a Full Implementation This is true, and that is where my project comes in. We have started implementing the OpAMP capabilities (and taking advantage of its custom commands), as it provides an ideal vehicle for our ChatOps idea. We've stayed within the Observability space (largely to make it easy for audiences to connect to the implementation). We're implementing the internals of the client and server in an extensible way, so it should be possible to use OpAMP with our initial targets of Fluent Bit and Fluentd, but we intend to expand to support the Elastic Stack/Beats and the OTel Collector. OpAMP complete implementation — taking the protocol and wrapping it with functionality. Including leveraging MCP for natural language (ChatOps) based interactions. Using the protocol, we've addressed the basic needs: what is the agent type, and what configuration is it operating under? That extends to the idea that on the server, we should be able to view the configuration file, ideally maintain it, and deploy it. The latter suggests the need for a configuration editor. So we have built something to support that, which can be entirely configuration-driven and can work with StreamingSQL and Lua scripts. To learn more about the project, try it out, and understand what ChatOps is, go here. A feature or two to help the Fluent stack To deliver some bonus value, we need to understand and potentially amend the Observability agent's configuration. So we have a configuration-driven editor which can be run standalone or as part of the control plane. Conclusion Some parts of the OpAMP protocol have the payload marked as 'Beta' or 'Development'. This may be off-putting until you realise that the OTLP definition was in a 'Beta' state for a very long time before being finally signed off. So don't let this discourage you. The Development message objects are more at risk of change, but this is driven by practical insights, and I don't anticipate any major changes. As for our project implementing OpAMP, it is still young and would benefit from people trying it out and feeding back. There are a lot of possible features we're considering, and we'd welcome feedback and to hear if people have tried to leverage the points of extension. Resources OpAMP official specFluent-OpAMP in GitHub and a landing page (we may have called the project Fluent (as in Fluentd, Fluent Bit), but we're starting to think beyond those two 'collectors').ChatOpsMy blogging on OpAMPFluent BitFluentd

Most SRE teams do not need another dashboard. They need a safer way to move from "something is wrong" to "we know what to do next." A model that detects anomalies is useful. A model that can touch production can also make a bad incident worse. That is where most conversations about AI in SRE become too optimistic for my taste. The hard part is not only detection. It is deciding how much autonomy the system should have, under which conditions, and with what blast-radius controls. I learned this while working on large-scale cloud services where one customer-facing symptom could turn into a flood of alerts. A degraded dependency might show up as latency in one service, retries in another, queue growth somewhere else, and CPU pressure downstream. During an on-call shift, that can look like five separate problems. Usually, it is one problem echoing through the stack. That experience changed how I think about self-healing infrastructure. The goal is not to build a system that blindly fixes everything. The goal is to build an operational control loop that can separate routine, low-risk recovery from incidents that still need human judgment. The model that has worked best for me is graduated autonomy: Let the system act automatically only when the action is well understood, reversible, and narrow in blast radius. For everything else, the system should collect evidence, recommend the next step, and keep humans in control. Why Static Alerts Stop Scaling Static alerts are not the enemy. I still want to know when disk usage is dangerous, error rates spike, or latency crosses a service-level threshold. But thresholds do not understand context. A CPU spike during a scheduled batch job may be normal. The same spike during steady-state traffic may be a retry storm. A latency increase in one region may be harmless during a controlled deployment, but suspicious if it appears across multiple availability zones with no recent change event. At small scale, engineers can carry that context in their heads. At enterprise scale, they cannot. Services emit hundreds of metrics across regions, dependencies, deployments, and customer paths. Eventually the team is no longer tuning alerts. It is negotiating with noise. In one rollout I was involved with, the most useful improvement was not adding more alerts. It was grouping alerts around dependency context and suppressing repeated downstream symptoms. The on-call experience became calmer because engineers could focus on the likely failure path instead of chasing every red graph independently. That is the kind of problem AI can help with. Not by replacing SRE judgment, but by organizing noisy signals into a more useful operational story. Detection Is Only the First Layer ML-based anomaly detection helps because it learns a service's normal operating shape instead of relying only on fixed thresholds. For cloud metrics, that usually means learning seasonality, traffic cycles, deployment windows, regional differences, and service-specific behavior. An LSTM autoencoder, isolation forest, or well-tuned statistical baseline can all be useful. I care less about the model family than the quality of the telemetry around it. A simple model trained on clean, consistent data will usually beat a sophisticated model trained on messy metrics. A practical anomaly pipeline usually looks like this: Collect metrics, logs, traces, and change events.Normalize them by service, region, dependency, and time window.Score each signal against its learned baseline.Group anomalies by dependency graph and recent changes.Produce an evidence bundle for automation or human review. Here is a simplified version of the scoring stage: Python from dataclasses import dataclass from typing import List @dataclass class MetricWindow: service: str region: str signal: str values: List[float] recent_deploy: bool = False @dataclass class AnomalyScore: service: str region: str signal: str score: float reason: str class BaselineModel: def expected_range(self, service: str, region: str, signal: str): # In production, this may come from a trained model, # feature store, or rolling baseline per service and region. return (0.0, 1.0) def score_window(window: MetricWindow, baseline: BaselineModel) -> AnomalyScore: low, high = baseline.expected_range( window.service, window.region, window.signal, ) latest = window.values[-1] if latest > high: distance = (latest - high) / max(high, 0.001) reason = f"{window.signal} above learned baseline" elif latest < low: distance = (low - latest) / max(abs(low), 0.001) reason = f"{window.signal} below learned baseline" else: distance = 0.0 reason = "within learned baseline" if window.recent_deploy and distance > 0: reason += " during recent deployment window" return AnomalyScore( service=window.service, region=window.region, signal=window.signal, score=min(distance, 1.0), reason=reason, ) The production value is not just the score. It is the metadata around it: ownership, dependency path, recent deploys, feature flag changes, customer impact, and whether the same pattern has appeared before. A single anomalous metric should rarely trigger remediation. Sustained anomalies across correlated signals are more trustworthy than one spike in one chart. Correlation Turns Noise Into an Incident Story During an incident, the useful question is not "Which graph is red?" It is "What changed first, and what depends on it?" That is where dependency-aware correlation becomes more useful than raw anomaly detection. A database issue may surface as API latency, retries, queue saturation, and CPU pressure. Without a dependency graph, every downstream service looks guilty. With one, the system can rank likely causes instead of handing the engineer a wall of symptoms. A useful correlation engine should look at topology, timing, change context, and customer impact. Which dependency failed first? Was there a deployment or config change? Which service is closest to the customer-facing error? The evidence bundle should be readable by a human. If the model says "root cause confidence: 0.86," that is not enough. It should also explain why. JSON { "candidate_root_cause": "identity-token-cache", "region": "example-region-1", "confidence": 0.86, "customer_impact": "elevated authentication latency for a subset of requests", "supporting_signals": [ "p99 latency above learned baseline for multiple consecutive windows", "cache hit rate dropped below its recent operating range", "downstream services showed retry growth after the initial cache anomaly", "no database saturation was observed", "no deployment was detected in the immediate incident window" ], "recommended_action": "drain_and_restart_one_cache_node", "estimated_blast_radius": "single node in a redundant pool", "rollback_plan": "keep node out of rotation if health checks fail after restart" } This is more useful than another alert. It gives the on-call engineer a starting hypothesis and the reasoning behind it. The Graduated Autonomy Model The most important design decision in self-healing infrastructure is not which ML algorithm to use. It is which actions the system is allowed to take. I divide remediation into three tiers. Tier 1: Fully Automated, Low-Risk Actions Tier 1 actions are safe, reversible, and narrow in blast radius. These are actions the system can execute without waiting for a human when confidence is high. Examples include restarting one unhealthy instance, scaling out a stateless service, draining one bad node, flushing a bounded cache, or shifting a small amount of traffic away from a degraded zone. The key phrase is bounded blast radius. Auto-remediation should not restart half the fleet, fail over a primary database, or disable a feature globally just because a model is confident. Confidence is not a substitute for safety. Before I put an action in Tier 1, I expect it to pass these checks: it is reversible, affected capacity is small, redundancy is healthy, there is no active global incident, the same action has not failed recently, rollback is defined, and health checks can verify success quickly. The first Tier 1 actions should be boring. Restarting one unhealthy node is not exciting, but it is exactly the kind of action that can be automated safely when the system has enough evidence. Tier 2: Automated Recommendation With Human Approval Tier 2 is where many real incidents live. The system may know what should happen, but the action still needs human approval. Examples include rolling back a deployment, disabling a feature flag, failing over a database, increasing capacity beyond a normal band, or changing regional routing. For Tier 2, the system should prepare the action, show the evidence, and ask for approval. The human should decide whether the action makes sense, not build the command during the incident. One pattern I have seen repeatedly: the slowest part of remediation is not always finding a likely cause. It is gathering enough confidence to take a risky action. When the system attaches deploy timing, error movement, affected endpoints, config changes, and rollback commands into one review card, the decision becomes easier. Tier 3: Human-Led With AI Context Tier 3 incidents are novel, high-risk, or ambiguous. The system should not execute remediation. It should help humans reason. This includes possible data corruption, multi-region cascading failures, security-sensitive incidents, conflicting signals across dependencies, low-confidence root-cause analysis, or any action with unclear rollback behavior. In Tier 3, the system's job is to summarize what it knows, what changed recently, which hypotheses are most likely, and which dashboards or runbooks are relevant. That alone can save time, but it keeps production control where it belongs. Architecture: A Control Loop, Not a Magic Button A practical self-healing system looks like a control loop with guardrails. Architecture diagram: Graduated autonomy model for self-healing infrastructure The important part of this diagram is the policy gate. Detection and correlation produce a recommendation, but the policy gate decides autonomy. Without that layer, "self-healing" becomes a risky automation script with an ML label attached. The policy gate should evaluate confidence, risk, blast radius, recent action history, service criticality, and rollback readiness. I would express that as policy-driven code: JSON from dataclasses import dataclass from enum import Enum from typing import List class Decision(str, Enum): AUTO_EXECUTE = "auto_execute" REQUEST_APPROVAL = "request_approval" HUMAN_LED = "human_led" @dataclass class RemediationProposal: action: str confidence: float blast_radius_percent: float reversible: bool rollback_defined: bool service_tier: str evidence: List[str] @dataclass class RuntimeContext: active_global_incident: bool recent_failed_action: bool healthy_redundancy: bool minutes_since_last_same_action: int TIER_1_ACTIONS = { "restart_single_instance", "scale_stateless_service", "drain_single_node", "flush_bounded_cache" } TIER_2_ACTIONS = { "rollback_deployment", "disable_feature_flag", "database_failover", "regional_traffic_shift" } def decide_autonomy( proposal: RemediationProposal, context: RuntimeContext ) -> Decision: if context.active_global_incident: return Decision.HUMAN_LED if context.recent_failed_action: return Decision.HUMAN_LED if not proposal.rollback_defined: return Decision.HUMAN_LED if proposal.action in TIER_1_ACTIONS: safe_enough = all([ proposal.confidence >= 0.90, proposal.blast_radius_percent <= 5.0, proposal.reversible, context.healthy_redundancy, context.minutes_since_last_same_action >= 30, len(proposal.evidence) >= 3, ]) return Decision.AUTO_EXECUTE if safe_enough else Decision.REQUEST_APPROVAL if proposal.action in TIER_2_ACTIONS and proposal.confidence >= 0.75: return Decision.REQUEST_APPROVAL return Decision.HUMAN_LED This is not drop-in production code, but the structure is the point: actions are classified, confidence is not the only input, and safety can override the model. In reliable systems, the model proposes; policy disposes. What I Measure Before Expanding Autonomy I would not start by asking, "Can we automate remediation?" I would start by asking whether the system's recommendations are trustworthy. Before allowing Tier 1 execution, I would track root-cause precision, false positives by service, recommendation acceptance, time to useful diagnosis, remediation success, rollback frequency, and any secondary incidents caused by remediation. The last two matter the most to me. A self-healing system that fixes one issue but creates another is not healing. It is moving the incident. My preference is to run in shadow mode first. Let the system detect, correlate, and recommend, but do not let it execute. Compare its recommendations against what engineers actually did. Once the system repeatedly recommends the same low-risk actions humans already take, graduate those actions into Tier 1. That is how trust gets built: not through a big launch, but through repeated correctness in narrow, well-understood situations. Lessons Learned From Building Toward Self-Healing The most useful lessons are not about model architecture. Clean telemetry beats clever models. If service names are inconsistent, regions are missing, logs are unstructured, and ownership metadata is stale, the model will struggle. Before debating LSTMs versus transformers, fix the telemetry pipeline. Change events are first-class signals. Deployments, config pushes, schema changes, and feature flag flips explain many anomalies. If the model cannot see change events, it will treat every incident like a mystery. Alert suppression is not the same as diagnosis. Reducing noise is useful, but the system must preserve the causal path. Suppressing duplicate downstream alerts only helps if the upstream root cause remains visible. Automation needs a memory. Every remediation should leave an audit trail: what was detected, what action was taken, what happened afterward, whether rollback was needed, and whether humans agreed with the recommendation. Start with boring actions. Restarting one bad instance is not glamorous. Draining one node is not a research breakthrough. But these are exactly the kinds of actions that make sense for early autonomy because they are repeatable, reversible, and easy to verify. Where LLMs Fit Large language models are useful in SRE, but I would not put them directly in the execution path for remediation. Their best role is communication and context assembly. An LLM can draft an incident summary, explain the evidence bundle, turn raw telemetry into a timeline, identify runbooks, and prepare a post-incident report. That saves time without giving the model direct control over production. The safer pattern is separation of responsibilities: ML or statistical models detect anomalies, graph correlation ranks likely causes, policy gates decide autonomy, deterministic automation executes approved actions, and LLMs summarize what happened. That separation keeps the high-risk parts deterministic and auditable while still using AI where it helps most. Final Thought Self-healing infrastructure is not about removing SREs from production. It is about removing the repetitive, low-risk work that slows them down during incidents. The best version of AI in SRE is not a magic system that fixes everything. It is a careful control loop: detect early, correlate intelligently, act only within policy, and learn from every outcome. If you are building toward self-healing, do not start with full autonomy. Start with evidence. Then recommendations. Then approval-based actions. Then, only after the system has earned trust, allow narrow automated remediation. That path is slower than the hype cycle, but it is much closer to how reliable infrastructure actually gets built.

Site reliability engineering has always been about reducing toil, improving resilience and helping teams respond to incidents with speed and confidence. Agentic SRE takes this idea further, allowing AI systems to observe, reason, and act within operational workflows inside of bounded constraints. The outcome is not a replacement for SREs, but a new operating model in which humans supervise intelligent agents that can help triage, diagnose, and remediate faster than manual processes alone. What Agentic SRE Means Agentic SRE is the use of AI agents to carry out reliability tasks with some autonomy. The agents are able to capture telemetry, correlate signals across systems, propose likely causes, take safe actions, and hand over to humans when the problem exceeds their authority. In practice, this means an AI assistant that can summarise an incident, pull up relevant dashboards, check recent deploys, compare symptoms against runbooks and even trigger low-risk remediation steps. What changes are not the nature of the assistance but the limits of its application. Traditional automation is usually rule-based: if X happens, do Y. Agentic systems are different in that they can adapt to context, select between several paths, and orchestrate steps across tools. This makes them especially useful in complex environments where the same symptom may come from many different root causes. Why SRE Needs Agents The systems today are too big and too interconnected to be run totally by hand. Teams are contending with noisy alerts, fragmented observability data, constant deployments, and ever more dynamic infrastructure. During incidents, engineers often burn precious minutes just to gather context before they can start a real diagnosis. Agentic SRE is attractive because it shortens that time. In a handful of high-friction places, agents can cut toil. They can filter alert storms, enrich alerts with deployment history, draw out meaningful patterns from logs, and surface relevant runbooks. They can also automate repetitive incident response tasks such as opening tickets, notifying owners, checking service health, or validating if a rollback is safe. That doesn't eliminate the need for engineers, but it does take away some of the low-value work that distracts them from judgment-intensive choices. The Human Role Remains Central One common fear is that SREs will be replaced with autonomous systems. Indeed, the human role becomes more, not less, important. Agents are good at pattern recognition, summarisation, and bounded execution. Humans are still better at trade-offs, risk assessment, organisational context, and deciding when not to act. Reliability is not merely a technical problem. It is a business and coordination problem. Humans should set policies, guardrails, and escalation thresholds for agent behaviour. They need to decide which actions can be safely automated, which require approval, and which should never be delegated. So the SRE is evolving from operator to system designer, to policy author, to reliability supervisor. That shift is profound because the skill set you need for the job changes. Where Agents Fit Today The best place to start is with low-risk, high-frequency jobs. These are the areas where automation can provide immediate value without unacceptable risk. Think incident summarisation, alert enrichment, log correlation, runbook retrieval, change impact analysis, and post-incident report drafting. Incident copilots are another strong use case. An agent can also act as a second brain during an outage: it can aggregate timelines, verify recent code changes, search knowledge bases, and suggest next steps. It can help responders avoid duplication of effort and make the first 10 minutes of an incident much more productive. An effective agent can also lessen the cognitive load on on-call engineers by turning the scattered telemetry into a coherent story. A third useful area is remediation assistance. Agents can recommend actions such as scaling a service, restarting a failing job, disabling a faulty feature flag, or rolling back a deployment. In mature setups, these actions can be executed automatically for pre-approved scenarios, while more risky actions still require human confirmation. That combination of automation and oversight is where agentic SRE becomes genuinely powerful. A Practical Architecture An effective agentic SRE system typically has five layers. First, it needs a telemetry layer that includes metrics, logs, traces, events, and deployment data. Without strong observability, the agent is blind and will make incorrect guesses. Second, it requires a reasoning layer, often powered by an LLM, to interpret context and decide what to do next. Third, there should be a tool layer that gives the agent access to safe operational functions, such as querying dashboards, reading configs, opening tickets, or triggering runbooks. Fourth, it needs policies and guardrails that define permissions, approval workflows, rate limits, and failure boundaries. Finally, it should have an audit layer so every decision, action, and recommendation can be traced later. That architecture matters because the danger is not the model itself; the danger is uncontrolled action. A reliable agent is not one that knows everything. It is one that acts only within well-defined limits and remains observable, reversible, and accountable. Guardrails That Matter Trust is the currency of autonomous operations. If teams do not trust the system, they will ignore it. If they trust it too much, they may hand over dangerous actions without oversight. The right answer is neither blind trust nor permanent skepticism. It is a layered trust model built through guardrails. Start with permission scoping. An agent should not have broad access by default. Its permissions should be narrow, explicit, and tied to specific tasks. Next, use action tiers. Low-risk actions can be automatic, medium-risk actions can require confirmation, and high-risk actions should remain human-only. You also need strong rollback paths so any automated action can be quickly reversed. Another essential safeguard is the observability of the agent itself. Just as production systems need monitoring, agents need monitoring too. Teams should track what the agent saw, what it inferred, what action it proposed, and whether the result improved the situation. That makes the system auditable and helps teams refine its behavior over time. The Operating Model Changes Agentic SRE changes incident response from a purely human workflow into a human-agent collaboration loop. In the old model, an engineer gets paged, reads alerts, searches dashboards, checks logs, consults teammates, and then acts. In the new model, the agent can do much of the initial gathering and triage before the human even joins. That shortens the path from detection to understanding. This also changes how teams design runbooks. Instead of static documents that people read under pressure, runbooks become machine-readable operational playbooks. Some of the best runbooks will be written with automation in mind, including clear preconditions, decision points, and action boundaries. That makes them useful both for humans and for agents. Post-incident work also improves. Agents can draft a timeline, collect evidence, identify suspicious changes, and summarize repeated patterns across incidents. That leaves engineers with more time to focus on systemic fixes rather than manual documentation. Over time, the organization develops a stronger feedback loop between incidents, learning, and platform improvements. Risks and Failure Modes Agentic SRE is not free of risk. One failure mode is confident hallucination, where an agent sounds plausible but is wrong. In operations, a wrong answer is not just inaccurate; it can cause downtime. Another risk is over-automation, where teams let agents act in situations that are not actually safe to delegate. There is also the risk of hidden complexity. If an agent stitches together many systems, it can become difficult to understand why it chose a specific action. That opacity can undermine trust and create governance problems. Security is another major concern because an agent with tool access can become an attractive target if permissions are poorly controlled. These risks do not mean agents should be avoided. They mean they must be introduced carefully. The best strategy is to start with narrow, well-understood workflows, measure outcomes, and expand only when confidence is earned. Reliability teams already understand progressive delivery, canary releases, and blast-radius reduction; the same principles should apply to agentic operations. How to Start The easiest entry point is to pick one painful workflow and automate only the first mile. A good candidate is alert triage. An agent can ingest alerts, group duplicates, summarize likely causes, and point responders toward relevant dashboards and runbooks. That alone can save significant time without requiring the agent to make risky changes. Another strong starting point is incident summarization. This is low risk, highly useful, and easy for teams to evaluate. A third option is change impact analysis, where an agent compares recent deploys, feature flag changes, and error spikes to highlight likely correlations. These use cases are valuable because they build trust through usefulness rather than hype. Measure success with clear operational metrics. Look at time to acknowledge, time to diagnose, time to mitigate, alert volume reduction, and after-hours toil reduction. Also measure negative outcomes, such as false suggestions, unsafe recommendations, or overreliance on the agent. Good SRE practice is about evidence, not enthusiasm. A New Reliability Mindset The biggest change Agentic SRE brings is a shift in mindset. It encourages teams to stop viewing automation as just a collection of scripts and to see it instead as a supervised operational partner. This partner can observe faster than a person, summarise quickly, and carry out repetitive tasks more reliably. However, it still requires humans to define the purpose, set limits, and determine acceptable risk. This is why agentic SRE is not merely “AI in operations". It represents a larger redesign of how reliability work is accomplished. The focus shifts from manual responses to intelligent coordination. It changes from isolated dashboards to context-aware agents. It evolves from static runbooks to flexible playbooks. It transforms reactive tasks into guided independence. Organizations that excel with this model will not be the ones that automate everything. They will be the ones that automate thoughtfully, govern effectively, and keep humans involved where decision-making matters most. In this way, Agentic SRE is more about enhancing the reliability system around the engineer than about replacing the engineer themselves. Closing Thoughts Agentic SRE marks a real change in how we can manage modern systems. It provides a way to respond faster, reduce repetitive work, and handle incidents more consistently, but only with strong observability, clear permissions, and human oversight. The future of reliability is not completely automatic or fully manual; it is collaborative, constrained, and constantly improving. For SRE teams, there's a chance to become designers of this new model. This involves creating agent workflows, writing safer runbooks, setting policy limits, and measuring impact with the same attention given to any production system. Teams that excel in this will not only respond more quickly. They will create systems that are more resilient, more adaptable, and much simpler to operate at scale.

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed. Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet. That script eventually became four different experiments, then a collection of small scanners, and eventually the dashboard shown in this article. Over the next eight months, that script evolved into OpsCart Watcher — an open-source operational triage dashboard for Kubernetes. This article is about what the journey taught me, and what I think is still missing from most Kubernetes environments. OpsCart Watcher — operational triage for Kubernetes (6 minutes) The Problem the Script Revealed The script did one thing well: it looked at an entire cluster and listed what was broken. Engineers who ran it kept telling me the same thing — "I had no idea this was there." That response was the important signal. These engineers had Grafana, Prometheus, and kubectl. Visibility was not their problem. The problem was that nothing told them to look at this specific namespace, this specific pod, this specific storage volume — before it became an incident. Consider a pod in CrashLoopBackOff for 19 days with 5,000+ restarts. To a metrics dashboard, that deployment looks healthy: replica count satisfied, a pod exists in Running state between crashes, CPU and memory flat because the container barely lives long enough to consume anything. The dashboard is answering the question it was built to answer — is the cluster meeting its SLOs? — and the answer is yes. The question nobody built tooling for: what deserves attention right now? LayerWhat It AnswersToolsMetricsIs the cluster meeting its SLOs?Prometheus, Grafana, DatadogPer-resource stateWhat is this specific pod doing?kubectl, k9s, LensOperational triageWhat deserves attention right now?Prioritizing operational work across cluster state What Triage Looks Like in Practice Overview page — Incident Score 41/100, KPI bar, Top 5, War Room panel The first time I ran the rebuilt dashboard against a cluster with real failures, the top of the screen didn't show me a CrashLoopBackOff pod. It showed me four CrashLoopBackOff pods spread across three namespaces, collapsed into a single operational problem: Plain Text 1. 4 pods crash-looping CRITICAL payments/fraud-detection (1810 restarts) → kubectl logs fraud-detection-... -n payments --previous That collapsing is the entire idea. Instead of inspecting every deployment individually, I was looking at a ranked list of operational problems — each with a severity, a location, and the exact kubectl command to start investigating. The full output for this environment: Plain Text Incident Score: 41/100 (Degraded) Top 5 Things to Fix: 1. 4 pods crash-looping CRITICAL 4 pods 2. 3 image_pull_backoff issues CRITICAL 3 items 3. 1 privileged_container issue CRITICAL 1 item 4. 1 namespace missing NetworkPolicy HIGH 1 ns 5. 3 orphaned PVCs wasting money MEDIUM 80 GB None of these had triggered an alert. All were present and accumulating before the scan. The Incident Score — a composite 0–100 across reliability, security, and waste — exists for one reason. Engineers fix incidents. Managers remember numbers. "We moved the Incident Score from 41 to 67" is a sentence that sticks. The crash loops and NetworkPolicies are the work behind it. The Step After Detection Finding problems was never the hard part. Knowing where to begin was. The most common feedback on the original bash script was some version of: "I found the problem, but I still didn't know what to do next." In March, I wrote about finding a container with 24,069 restarts that had been accumulating undetected. Finding it took sixty seconds. The next hour was the actual work: what do I run first? Is this configuration or code? Is it customer-facing? The investigation page is my answer to that hour. Investigation page — OpsCart Assessment, Evidence, Recommended Investigation One click from any triage finding opens a dedicated investigation view: Plain Text OpsCart Assessment This workload has restarted 1810 times over 6 days. The restart rate appears stable, suggesting a deterministic configuration or application failure rather than an intermittent infrastructure issue. No referenced ConfigMaps or Secrets were detected in the pod spec — missing configuration is unlikely to be the root cause. Investigation should begin with previous container logs. Estimated time: 5–10 minutes. Evidence [1810 Restarts] [CrashLoopBackOff] [6d] [Deployment/fraud-detection] Recommended Investigation HIGH CONFIDENCE Check previous container logs MEDIUM Verify ConfigMaps and Secrets exist LOW Check for OOMKill in events The assessment is rules-based — no AI. It reads restart count, failure pattern (stable vs accelerating), and referenced configuration objects, then produces a deterministic, auditable summary. The confidence levels reflect how a senior engineer actually reasons: previous logs are almost always the right first move for a crash loop; OOMKill is worth checking but less likely. This is the part kubectl doesn't give you. Neither does Lens, k9s, or Headlamp. From "What Is Broken?" to "What Changed?" The biggest architectural change came when the dashboard gained memory. The first version of the tool answered: "what is broken?" The current version — backed by a small embedded database recording every scan — answers "what changed?" That sounds like a minor distinction. Operationally, it changes everything. An incident that has existed for three days deserves different attention than one that appeared five minutes ago. A cluster whose Incident Score dropped eight points overnight is telling you something that no single scan can. War Room — critical issues with visual differentiation per type Every KPI now carries a trend arrow — critical issues up three since the last scan, waste down one — and the Incident Score shows a seven-point sparkline. Each incident is tracked with first-seen and last-seen timestamps and an active/resolved status, so "CrashLoopBackOff — first detected 6 days ago, still active" replaces "CrashLoopBackOff." Operational memory changed the tool from a scanner into something that remembers the history of a cluster. What This Is Not The triage pattern does not answer when an issue started at the metrics level, why an application is slow, or whether last Tuesday's deployment caused a regression. Prometheus, APM tooling, and deployment audit logs remain the right tools for those questions. The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability stack. The Biggest Lesson When I started, I thought Kubernetes debugging was about collecting more information. It wasn't. Kubernetes already exposes almost everything an operator needs through its API. The difficult part is deciding what deserves attention first. Over eight months, I found myself spending less time searching for failures and more time ranking them. That is ultimately what OpsCart became — not another dashboard, but a prioritization engine for cluster operations. Why Open Source I considered keeping the dashboard private. Instead, I open-sourced it because operational patterns only become useful when they're tested across different clusters. Every environment fails differently, and I wanted the prioritization model to evolve from real-world feedback rather than a single infrastructure. The Remaining Gap The conclusion from my March article is still true: the question worth asking of your environment is not whether these conditions exist — they almost certainly do — but whether your current observability layer would surface them before they become incident preconditions. Eight months of building has only made that conclusion more specific. The gap is not data. The gap is attention: knowing which five things, out of hundreds of resources, deserve a human's time right now. Eight months ago I thought I was building a better debugging script. I wasn't. I was building something that helps operators decide where to spend the next ten minutes. About the environment: The scenarios shown in this article — CrashLoopBackOff pods, orphaned PVCs, missing NetworkPolicies, privileged containers — are representative of what OpsCart finds on real production clusters. The environment shown is a dedicated demonstration cluster configured with realistic failure scenarios. No production data was used. About the tool: OpsCart Watcher is open-source at github.com/opscart/opscart-k8s-watcher. It deploys as a single read-only container: Shell kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80

By Shamsher Khan CORE

Picture an MCP server that keeps “forgetting” what an agent just asked it to do. The agent starts a long-running tool call, such as a database migration. When it checks back for status, the server has no record of the request. There is no crash and no obvious error. The follow-up request simply landed on a different server instance behind the load balancer. That instance never saw the original session. This is a common failure mode once MCP servers move beyond local studio usage and into Streamable HTTP deployments with multiple backend instances. Standard HTTP load balancing treats requests as independent. Stateful MCP sessions expect continuity. When those assumptions collide, the result can be silent state loss. This article walks through why it happens and three practical ways to handle it: sticky sessions, externalized state, and stateless sessions with resumable streams. Why This Happens MCP's client-server model was originally built around a fairly simple mental picture: a host spins up a client, the client talks to a server, and that server sticks around for the life of the session. That works fine when the client launches the server as a local subprocess over stdio, which is still how many local dev setups, IDE integrations, and CLI agents talk to MCP servers. The picture changes the moment a deployment moves to Streamable HTTP for remote, multi-tenant use. Now there are: Multiple server instances behind a load balancerNo shared memory between themA session expected to persist across multiple requestsTool calls that can run long enough for a request to get rerouted mid-session Standard HTTP load balancing assumes requests are independent. MCP sessions assume continuity. Put those two assumptions in the same system, and the result is exactly the bug described above — silent state loss, not a hard failure, which arguably makes it worse. A crash gets noticed quickly. A hallucinated "sure, that's done" doesn't, until someone downstream asks why the migration never ran. What a Naive Setup Looks Like Here's a stripped-down MCP server that will fall over reliably in a multi-instance deployment. It uses in-memory session storage — fine for a demo, a landmine in production: JavaScript // naive-mcp-server.js const sessions = new Map(); // lives only in this process's memory app.post('/mcp', async (req, res) => { const sessionId = req.headers['mcp-session-id']; if (!sessions.has(sessionId)) { sessions.set(sessionId, { createdAt: Date.now(), context: {} }); } const session = sessions.get(sessionId); const result = await handleToolCall(req.body, session); res.json(result); }); Run this on one instance, and it works fine. Put two instances behind a round-robin load balancer, and every second request from the same session can hit an instance that has never seen it before. Because the code creates a new session whenever the ID is missing locally, the server quietly starts over. Any state from the earlier part of the interaction is gone. Tier 1: Sticky Sessions (Session Affinity) The simplest fix, and the one most teams should reach for first, is routing every request from a given session to the same backend instance. The MCP session ID becomes the routing key, instead of relying on the load balancer's default algorithm (round robin, least connections, or whatever else). Here's what that looks like with nginx using a session-aware upstream: Nginx upstream mcp_backend { hash $http_mcp_session_id consistent; server 10.0.1.10:3000; server 10.0.1.11:3000; server 10.0.1.12:3000; } server { listen 443 ssl; location /mcp { proxy_pass http://mcp_backend; proxy_set_header Mcp-Session-Id $http_mcp_session_id; proxy_read_timeout 300s; # long-running tool calls need headroom } } The hash ... consistent directive means the same session ID reliably maps to the same upstream server, and — this part matters — if a server is added or removed, only a fraction of sessions get remapped instead of the whole pool shuffling. Without consistent, scaling the fleet up or down would silently relocate a chunk of active sessions, landing right back at square one. On the application side, it's worth validating that the session actually exists locally, and failing loudly if it doesn't, not silently creating a new one: JavaScript app.post('/mcp', async (req, res) => { const sessionId = req.headers['mcp-session-id']; const session = sessions.get(sessionId); if (!session) { // Routed somewhere it shouldn't have been. Don't fake it. return res.status(409).json({ error: 'session_not_found_on_instance', message: 'Session affinity may have broken. Client should retry or reinitialize.' }); } const result = await handleToolCall(req.body, session); res.json(result); }); That explicit failure does real work here. Whether you return a diagnostic 409 for affinity failure or a spec-aligned 404 for an unknown session, the important part is not pretending the session exists. It turns a silent data-loss bug into a visible, debuggable failure — far better than letting an agent confidently report on work it never actually did. Where sticky sessions fall short: If an instance dies, every session pinned to it dies too. There's no failover. For a lot of teams, that trade-off is completely acceptable — instance restarts are rare, sessions are usually short-lived, and the operational simplicity is worth it. But teams that need real resilience need to look at tier two. Tier 2: Externalized State Instead of keeping session data in process memory, it gets pushed out to something every instance can reach — Redis is the obvious choice, though any shared, low-latency store works. JavaScript const redis = require('redis').createClient(); app.post('/mcp', async (req, res) => { const sessionId = req.headers['mcp-session-id']; const raw = await redis.get(`mcp:session:${sessionId}`); const session = raw ? JSON.parse(raw) : { createdAt: Date.now(), context: {} }; const result = await handleToolCall(req.body, session); await redis.set( `mcp:session:${sessionId}`, JSON.stringify(session), { EX: 1800 } // 30-minute TTL, adjust to session lifecycle ); res.json(result); }); Now any instance can pick up any session — routing no longer matters, and an instance dying doesn't take sessions down with it. The cost is exactly what would be expected: a network round trip on every request, serialization overhead, and one more piece of infrastructure to keep healthy and monitored. For tool calls with large context payloads, that serialization cost isn't trivial — worth benchmarking before committing to it at scale. Tier 3: Stateless Sessions With Resumable Streams The most ambitious approach avoids server-owned session state where possible. Instead, enough context gets encoded in the session token itself (signed, so it can't be tampered with) that any instance can reconstruct what it needs from the request. Combined with MCP's resumability features — replaying missed events from a last-event-ID — this produces a server that doesn't care which instance handles which request, because there's no server-side state to lose. This is the kind of pattern large, multi-region deployments often need, but it's genuinely more work: it requires careful token design, replay handling, and a clear answer for what happens when a tool call's side effects (like that database migration) need to survive a dropped connection mid-execution. Comparing the Three Approaches StrategyFailover on Instance LossImplementation EffortLatency OverheadBest ForSticky SessionsNone — session dies with the instance.Low (load balancer configuration)MinimalSmall fleets, short sessions, early-stage deploymentsExternalized StateYes — any instance can resume the session.Medium (shared store + TTL management)Moderate (network round trip)Most production deployments at moderate scaleStateless + Resumable StreamsYes, if tool/job state is durable elsewhere.High (token design, replay logic)Low per request, high design costLarge-scale, multi-region, high-resilience systems Things Worth Checking Before Deployment A few details that commonly cause trouble, in no particular order: Set a proxy_read_timeout (or the load balancer's equivalent) high enough for the longest expected tool call - the default is almost always too short for anything doing real work.Don't let sessions live forever in Redis or wherever they're stored. A TTL that's too generous quietly turns into a memory leak.Test what happens when the instance pool scales up or down mid-traffic, not just at steady state. This is exactly where sticky-session setups without consistent hashing fall apart.Log session-affinity failures (like that 409 above) separately from normal errors. A spike usually means the load balancer config drifted, not that application logic broke.With Streamable HTTP, double-check that origin validation and auth are still enforced per-request, not just at session creation. Session continuity shouldn't come at the cost of skipping checks on the requests that follow. Make sure clients send MCP-Protocol-Version on subsequent HTTP requests, and validate unsupported versions server-side. The Bigger Picture None of this is really MCP-specific, at a systems level — it's the same stateful-session-behind-a-load-balancer problem that's shown up in every generation of web infrastructure, from sticky sessions in classic app servers to sharded WebSocket connections. What's new is that MCP's tool calls can be long-running and consequential (migrations, deployments, write actions against real systems), which makes silent state loss a lot more expensive than it would be for, say, a shopping cart. Teams still running a single MCP server instance don't need to worry about any of this yet. But the moment horizontal scaling enters the picture — and for any MCP server doing production-critical work, it eventually will — deciding up front which tier to build for beats discovering the gap the hard way, mid-incident.

I maintain a React admin dashboard codebase that had — at last count before upgrading to React 19 — 34 instances of useMemo, 28 instances of useCallback, and 19 components wrapped in memo(). I spent a nontrivial amount of time over two years adding those optimizations, debugging cases where I'd gotten the dependency arrays wrong, and explaining to junior developers why the table re-rendered on every keystroke. React 19 with the compiler deleted most of that work. Here's what actually changed and what still matters. The Compiler Handles What You Used to Do Manually The React Compiler analyzes component render behavior and adds memoization where it's beneficial automatically. You don't specify it. You don't maintain dependency arrays. You don't wrap components in memo(). Before: JSX const Dashboard = memo(function Dashboard({ userId, filters }) { const processedData = useMemo( () => processData(rawData, filters), [rawData, filters] ); const handleFilterChange = useCallback( (newFilter) => updateFilters(newFilter), [updateFilters] ); return <DataTable data={processedData} onFilter={handleFilterChange} />; }); After React 19 with compiler: JSX function Dashboard({ userId, filters }) { const processedData = processData(rawData, filters); const handleFilterChange = (newFilter) => updateFilters(newFilter); return <DataTable data={processedData} onFilter={handleFilterChange} />; } Same performance. Half the code. Zero dependency array bugs. I removed 31 of my 34 useMemo calls after upgrading. The three I kept are genuinely complex computations where I want explicit control. Everything else the compiler handles better than I was doing manually. The One Thing That Still Kills Dashboard Performance The compiler doesn't solve virtualization. If your data table renders 5,000 DOM nodes because your dataset has 5,000 rows — React 19 won't fix that. The browser is still creating and painting 5,000 nodes. Any table with more than 100 rows needs virtualization: JSX import { useVirtualizer } from '@tanstack/react-virtual'; export default function DataTable({ rows }) { const parentRef = useRef(null); const virtualizer = useVirtualizer({ count: rows.length, getScrollElement: () => parentRef.current, estimateSize: () => 52, overscan: 5 }); return ( <div ref={parentRef} style={{ height: '600px', overflow: 'auto' }> <div style={{ height: virtualizer.getTotalSize() }> {virtualizer.getVirtualItems().map(row => ( <div key={row.index} style={{ position: 'absolute', top: 0, transform: `translateY(${row.start}px)`, height: row.size } > <TableRow data={rows[row.index]} /> </div> ))} </div> </div> ); } 5,000 rows. 20 DOM nodes in the viewport at any time. Scroll is smooth. The React Compiler cannot help you here — this is a DOM problem, not a React problem. Optimistic Updates Changed How My Dashboard Feels The biggest perceived performance improvement in React 19 for admin dashboards isn't the compiler. It's useOptimistic. Admin dashboards involve constant small mutations — toggling user status, updating values, changing settings. Before React 19, every mutation waited for the server response before updating the UI. Fast servers meant 200ms delays. Slow servers meant users clicking buttons twice because nothing happened visually. JSX 'use client'; import { useOptimistic, useTransition } from 'react'; function UserStatusBadge({ user, onUpdateStatus }) { const [optimisticStatus, setOptimisticStatus] = useOptimistic( user.status, (_, newStatus) => newStatus ); const [isPending, startTransition] = useTransition(); const toggle = () => { const newStatus = optimisticStatus === 'active' ? 'inactive' : 'active'; startTransition(async () => { setOptimisticStatus(newStatus); // instant UI update await onUpdateStatus(user.id, newStatus); // server in background }); }; return ( <button onClick={toggle} disabled={isPending} className={`badge ${optimisticStatus === 'active' ? 'bg-success' : 'bg-secondary'}`} > {optimisticStatus} </button> ); } Click. Status changes instantly. The server call happens in the background. If the server fails — React reverts the optimistic update automatically. The dashboard feels instant because it is instant from the user's perspective. I added this to every status toggle, every inline edit, every bulk action in my dashboard. The difference in perceived performance is more noticeable to users than any memoization optimization I'd done previously. What React 19 Didn't Fix Route-level code splitting still matters and still requires explicit configuration. If your dashboard loads all route components upfront, the initial bundle is large regardless of React version: JavaScript // Still need this in React 19 const UsersPage = lazy(() => import('./pages/Users')); const AnalyticsPage = lazy(() => import('./pages/Analytics')); const SettingsPage = lazy(() => import('./pages/Settings')); Image optimization still requires next/image or equivalent. Unoptimized images are still the most common performance problem I see in dashboard codebases, and React 19 does nothing about them. Database query performance still determines how fast your data loads. The fastest React rendering in the world doesn't compensate for a 3-second API response. The Summary React 19 removes the memoization overhead that made large React codebases tedious. Use the compiler, stop writing useMemo for everything, and trust that it handles the cases you were handling manually. What still requires your attention: virtualize large tables, add optimistic updates for frequent mutations, split routes with lazy loading, and optimize your images. The performance work that remains is more interesting than the work React 19 eliminated. That's a good trade.

Datadog published the State of AI Engineering 2026 report— real telemetry from over a thousand production environments. Read it. It is the most comprehensive look at AI in production available right now. I want to respond from the reliability engineering perspective, because the data reveals a problem the report names but doesn't fully resolve: agent sprawl is now a production reliability crisis, and the SRE discipline does not yet have governance frameworks for it. What the Data Shows Three findings stand out from an SRE perspective: Framework adoption doubled year over year. LangChain, LangGraph, Pydantic AI, Vercel AI SDK — up from 9% of organizations in early 2025 to nearly 18% by 2026. Services using agentic frameworks: more than doubled. 70%+ of organizations run three or more models. The share running more than six models nearly doubled. Teams are building model portfolios rather than committing to a single provider. Teams add models faster than they retire them. Datadog calls this "LLM tech debt." Each overlapping model introduces its own quality, latency, and cost profile. The report is explicit: this becomes a governance problem. These three findings combine to describe an environment growing faster than it can be governed. I call this Agent Sprawl. Defining Agent Sprawl Agent Sprawl — the condition where AI agent infrastructure complexity (frameworks, models, tool layers, orchestration patterns) grows faster than your ability to measure and govern its reliability. It is structurally identical to the microservices sprawl problem SRE teams faced between 2015 and 2020. Teams added services faster than they added SLOs. The result: production incidents nobody could attribute because the dependency graph was too complex to observe. Agent Sprawl has three specific manifestations: 1. Framework-Invisible Call Complexity When you add LangChain, LangGraph, or any orchestration framework, it adds steps and paths you did not write — retry logic, fallback handlers, context window management, tool routing. All of this happens between your application code and your observability layer. Your SLIs measure at the application boundary. Framework-added calls are invisible. This means your Tool Invocation Efficiency (TIE) baseline — tool calls per task completion — is measuring a mix of your agent's behavior and your framework's behavior. When you upgrade the framework, both change simultaneously. You cannot separate them. In practice, across regulated production environments I've studied, TIE baselines can drift 30 – 40% after a framework major version upgrade with no corresponding change in the agent's task logic. The baseline shift looks like agent degradation. It's actually framework overhead. Teams spend hours on a false RCA. The fix: Instrument at the framework output layer, not the application layer. Capture tool invocations after framework processing. Then freeze your TIE baseline before any upgrade and compare shadow traffic before promoting. 2. Multi-Model SLO Orphaning 70% of organizations running 3+ models means 70% have at least two additional SLO ownership gaps they haven't acknowledged. SLOs are set once — typically when the first model is deployed. As models 2, 3, 4, 5, 6 are added for specific task classes, latency profiles, or cost tiers, nobody revisits the SLO ownership model. Models run in production with no named owner, no baseline, no error budget. When model 3 degrades, there is no owner to page, no baseline to compare against, no runbook to execute. The degradation surfaces as a customer complaint, not an alert. The fix: Treat every model in your fleet like a microservice. Each model gets: a named owner (not a team — a person), a task-class-specific SLO, and a 30-day observation baseline before the SLO is enforced. 3. LLM Tech Debt as a Reliability Liability Deprecated models running in agent chains create silent compatibility risks. When a provider announces deprecation, teams with models buried inside multi-step chains often miss the migration window. The model ages. Safety training falls behind. Decision Quality Rate declines slowly — too slowly to trigger a threshold alert — until accumulated drift surfaces as a production incident. The fix: Treat model deprecation notices the same way you treat dependency CVEs. Automate alerts at 60, 30, and 7 days before end-of-life. Build the migration ticket at announcement time, not at expiry. The Governance Framework Agent Sprawl Needs The Agent Fleet Inventory Before you can govern sprawl, you need to know what you're governing. Maintain a living inventory with, for each component: framework and version, model(s) used, task classes handled, named SLO owner, current TIE/DQR baselines, and deprecation dates. Python from agentsre.sprawl import AgentFleetInventory, FleetComponent, ComponentType inventory = AgentFleetInventory() inventory.register(FleetComponent( component_id="anthropic.claude-sonnet-4-6", component_type=ComponentType.MODEL, agent_id="payment-processor", task_classes=["payment-routing", "fraud-detection"], slo_owner="[email protected]", # named human — not a team baseline_established_at="2026-04-01", deprecation_date="2027-06-01", last_slo_review="2026-04-01", current_tie_baseline=2.4, current_dqr_baseline=91.2, )) report = inventory.quarterly_review_report() print(f"Fleet governance score: {report['fleet_governance_score']}/100") Framework Version Governance — Canary Before Promotion Python from agentsre.sprawl import FrameworkVersionGovernance gov = FrameworkVersionGovernance( tie_drift_threshold=1.15, # block if TIE drifts >15% dqr_drift_threshold=0.85, # block if DQR drops >15% min_shadow_samples=50, ) # Before upgrade: snapshot production baseline gov.snapshot_baseline( agent_id="payment-processor", task_class="payment-routing", framework_version="langchain-0.2.x", tie_values=production_tie_samples, dqr_values=production_dqr_samples, ) # After 48hrs shadow traffic: result = gov.evaluate_upgrade( agent_id="payment-processor", task_class="payment-routing", production_version="langchain-0.2.x", shadow_version="langchain-0.3.x", ) if result.decision == UpgradeDecision.BLOCK: rollback() # framework added hidden overhead — don't promote The Quarterly Multi-Model SLO Review The review should take 30–60 minutes per quarter. For every model in fleet: Verify named owner existsVerify baseline is current (< 90 days old)Check deprecation schedule against provider announcementsReview TIE per-model — models with rising TIE relative to task class baseline are drifting Models scoring below 70 on the governance health score are flagged as governance debt requiring a 30-day remediation window. The Datadog Report's Implicit Challenge The State of AI Engineering 2026 describes an industry in rapid expansion. What it does not fully resolve is the SRE question: who governs all of this, and what does that look like in practice? The SRE community has solved exactly this class of problem before — in distributed systems, in microservices, in cloud infrastructure. The discipline already exists. It needs to be applied to the AI agent layer now, before agent sprawl becomes agent chaos. The Datadog data tells us the window is closing. Framework adoption doubles in a year. Multi-model fleets become the norm. Model debt accumulates. Build the governance layer before the production incidents start. Resources Open-source implementation: [https://github.com/Ajay150313/agentsre]LinkedIn discussion: [https://www.linkedin.com/posts/ajay-devineni_agenticai-sre-reliability-ugcPost-7455786901673902080-BCRM?utm_source=share&utm_medium=member_desktop&rcm=ACoAACIp55QBRGVmAcEbf0D-1PaR5vEbm2yMcJU] What's your biggest agent sprawl challenge right now?