Performance Resources

DZone's Featured Performance Resources

How to Monitor AI Models Without Drowning in Alerts

By Aditya Shrivastava

When putting their model into production, every team or organization encounters the same issue. Failures go unnoticed for days at first because there is no monitoring. As teams begin to fix the issues, they identify areas where production results deviate from the training data, create dashboards for every metric, and set alerts for every threshold. This results in engineers being paged at two in the morning for a bug that fixes itself within an hour, and when an important alert arises, it goes unanswered due to alert fatigue, creating a pipeline that silently feeds garbage into the model. When a team learns to disregard 95% of the issues, they are very likely to disregard the remaining 5% that are actually important, and the solution to this isn’t less monitoring. The good solution to this problem is monitoring, which is tiered, routed, and pruned differently from the infrastructure monitoring that most teams already know. The Problem With Applying Old Monitoring Rules To AI Traditionally, application monitoring used to be binary, which is whether the application or service is up or down, latency is high or low, etc. But AI models don’t fail with these signs; they usually degrade over time. For instance, a recommendation model does not show exceptions when the user behavior shifts; it just silently gets worse at what it was supposed to do. A classifier model does not throw an error when its input distribution changes; it just returns answers confidently with increasingly wrong predictions. An AI application does not crash when it hallucinates; instead, it returns a normal HTTP 200 response with incorrect content. This creates two problems: When AI models fail, the reason for failure is invisible to classical infrastructure monitoring, which causes teams to bolt on multiple checks like data quality checks, drift detectors, and output scorers, each introducing a new source of noise. AI models are statistical in behavior and not deterministic, so setting threshold alerts on them leads to them firing constantly, and training teams have to tune the model. As a result, thorough AI monitoring does not make the application safer; beyond a certain point, it only makes things worse. What to Actually Monitor Monitoring issues that no one will ever take action on is often the first step towards alert fatigue. It is useful to consider it in four layers, each with its own owner and mode of failure. Infrastructure and service: Metrics like inference latency, throughput, Graphics Processing Unit (GPU)/Central Processing Unit (CPU) utilization, error rates, and cost per request and token consumption for anything calling a hosted large language model (LLM) API are classic operational metrics and can usually be monitored with the existing Application Performance Monitoring (APM) tools. Data quality: This is another important thing to keep an eye on because it can cause broken feature pipelines, upstream schema changes, input formats being changed without getting noticed, and null-rate spikes. These are usually the worst failures because you can't see them unless you're looking for them, and the model keeps making predictions based on bad data. Model quality: This can be tracked by looking at changes in the Confidence Score or how much the prediction distribution has changed from what was seen during training. This can be used instead of measuring accuracy because it's hard to tell right away how measures like accuracy are calibrating, because to measure accuracy, you would have to compare the predicted result to the actual correct answer, which doesn't always exist at the time of prediction. Generative artificial intelligence/large language model quality: Metrics like hallucination rate, coherence, factual grounding, toxicity, and susceptibility to prompt injection need different types of tooling to identify them because they are not like traditional metrics and would require human-in-the-loop sampling or an LLM as a judge for identifying them. The mistake many teams make is that they apply the same alerting techniques to all four layers, which is the infrastructure one, as that is the traditional way of setting up monitoring for applications, but issues related to data quality and model quality require a trend-based review. How to Alert Without the Noise Replace static thresholds with adaptive baselines. When systems learn a baseline from historical behavior and trigger alerts on deviations from it, like “alert if latency exceeds 200ms,” this ignores the daily and weekly traffic patterns, and the same is valid for data volume and null rates, which leads to a large number of false alarms being raised. So, teams that have made this switch from static thresholds to adaptive baselines have reportedly reduced noisy alerts by 60–90%. Introduce real severity tiers. When an alert is critical and poses an instant business risk, it is sent to an on-call engineer so that the problem can be fixed right away. Warnings about poor performance that are not critical are sent to a Teams chat channel during business hours, and signals about long-term trends land on the dashboard to be looked at from time to time. This helps to make sure that the notification's urgency matches its real urgency. Correlate and deduplicate before notifying. One change to the schema upstream can cause a dozen problems downstream. Sending a dozen alerts for one root cause either makes the team too busy or forces them to mentally group alerts together, which your tools should be doing for you. Route alerts to whoever can act on them. Misrouting is a common cause of tiredness. If the central platform team doesn't know about the business, they might ignore a spike they can't understand, and the domain team that would be able to understand it would never see the alert. Both problems are solved by linking alerts to the right person by domain, based on where the problem starts. Prioritize by business impact. A system that looks for unusual events handles all alerts the same way because it doesn't know which parts of your system are important to the business. When you think about how important each problem is before choosing how loud to alert, you get a lot fewer alerts overall, and a lot more of them are ones that you should actually act on. Conclusion It's important to understand that all of the ideas we've talked about work together; none of them can be used on their own. For example, adaptive thresholds only give out fewer alerts that aren't differentiated by severity. Without proper routing, severity tiers send the wrong messages about how important something is to the incorrect individuals. To avoid alert fatigue, teams need to take comprehensive actions, which include proper alert designs and organizational practices. They should also ensure that every alert can be acted on, which is better than monitoring everything, because AI monitoring only scales, and not having anyone see a model fail could have serious consequences. Good monitoring means building a system that sends alerts only when it matters, so when it does, people actually act on it. More

Pragmatic Premature Optimization

By Alexander Radzin

“...premature optimization is the root of all evil…” Donald Ervin Knuth Introduction "Premature optimization is the root of all evil." Most software engineers know this, attributed to Donald Knuth, author of The Art of Computer Programming and one of the most influential figures in computer science. Many have also picked up the practical conclusion that followed: "let's make it work first, fix performance later." After all, it's easier to add another EC2 instance than to find the root cause. But here is what Knuth actually wrote: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." A little different, isn't it? The second sentence is almost never quoted — and that is convenient, because it turns a careful statement into a simple excuse. Sometimes for laziness. Sometimes because people assume that optimization means sacrificing readability: cryptic bit manipulation, obscure tricks, code that only the author understands at 2 am. I believe Knuth was indeed warning against that kind of optimization. But that assumption is wrong more often than people think. Good, clean code is frequently efficient code too — not by accident, but because choosing the right tool for the job tends to be both clearer and faster. The examples in this article are proof of that. Scope This article focuses on simple, cheap, and foolproof tips that can be applied universally — regardless of your architecture, framework, or domain. In my experience, they carry virtually no risk of making things worse. Architecture, design, networking, database connectivity, threading — these are deliberately out of scope. Not because they are unimportant, but because they are context-dependent. The right answer depends on your specific system, and each of these topics deserves its own article. Examples String Operations We are all familiar with built-in JDK string utilities like: equals(), startsWith(), endsWith(), contains(): Java s1.equals(s2); s1.startsWith(s2); s1.endsWith(s2); s1.contains(s2); Unfortunately, JDK provides only one function for case-insensitive comparison: Java s1.equalsIgnoreCase(s2) There are no functions for case-insensitive startsWith(), endsWith(), contains(). So, often we combine toLowerCase() or toUppserCase() with startsWith(), endsWith(), contains(): Java s1.toLowerCase().startsWith(s2.toLowerCase()); s1.toLowerCase().endsWith(s2.toLowerCase()); s1.toLowerCase().contains(s2.toLowerCase()); A little verbose and null-prone, but just fine if not on the critical path. However, this technique might cause some performance problems. Do not forget that String is an immutable class, so instead of just a char-to-char comparison between two strings, we create two additional strings that then must be garbage-collected. Considering that String is a wrapper over a char array, the memory allocation may become expensive. The solution is to use case-insensitive utilities provided by different libraries, e.g., Apache Lang3: Java startsWithIgnoreCase(s1, s2); endsWithIgnoreCase(s1, s2); containsIgnoreCase(s1, s2); Or, starting from version 3.18.0: Java Strings.CI.startsWith(s1, s2); Strings.CS.startsWith(s1, s2); Where CI exposes case-insensitive and CS — case-sensitive utilities. Many people like regular expressions and use java.util.Pattern class sometimes, not where it is really necessary. For example: Java Pattern.compile("^prefix.+suffix$").matcher(s).find() Instead of: Java s.startsWith("prefix") && s.endsWith("suffix") Or even: Java Pattern.compile("^prefix").matcher(s).find() instead of s.startsWith("prefix") Pattern.compile("suffix$").matcher(s).find() instead of s.endsWith("suffix") Pattern matching is significantly slower than trivial substring matching. The following table shows evaluation time for 1 million operations: Operation * 1 million times Time, ms s.equals("hello") 7 s.startsWith("hello") 6 s.endsWith("hello") 11 s.contains("hello") 24 s.toUpperCase().startsWith("HELLO") 65 s.equalsIgnoreCase("hello") 5 Pattern.compile("hello").matcher(s).find() 238 pattern.matcher(s).find() 31 What can we see from this table? Performance of equals() and startsWith() is similarendsWith() is 2 times more expensivecontains() is 4 times more expensive than equalsChanging case followed by startsWith() is 10 times (!) more expensiveCase-insensitive comparison functions do not have any performance penaltiesSearching for a substring using a precompiled pattern is about 20% more expensive than using a plain contains() method. Compiling the pattern and using it is almost 10 times more expensive than the plain contains() method. So next time you reach for Pattern.compile(), it is worth pausing for a second: is regex actually needed here, or is a plain string method both simpler and faster? If you really need a pattern, at least compile it in advance — better yet, declare it as a private static final class member. Collections Let’s assume that we want to know whether a given list contains the specific element: Java list.contains("red"); In fact, this call invokes code like this: Java int n = list.size(); for (int i = 0; i < n; i++) { if ("red".equals(list.get(i))) { return true; } } Starting from Java 8, we have a streaming API that just hides from us the same gory details: Java list.stream().anyMatch("red"::equals); This is perfectly fine when the list is short, changes frequently, or is searched only occasionally. But if the list is large, stable, and searched repeatedly, a HashSet is the right tool — offering average O(1) lookup instead of O(n). If you cannot change the original data structure, converting it once at initialization time and searching the Set from that point forward is almost always worth it. If both the guaranteed element order and the fast lookup are needed, we can either hold duplicated data structures — a list for ordering and a set for search or just use LinkedHashSet, which solves both problems. Another common case is case-insensitive search. We already saw above that the combination of toLowerCase() or toUpperCase() with comparison significantly reduces the performance. This can be solved by using TreeSet with custom comparator, e.g. String.CASE_INSENSITIVE_ORDER: Java Set<String> set = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); This gives you a sorted, case-insensitive set with no extra allocations - and the same approach works for TreeMap when your data is key-value pairs. Enum Lookups Everyone knows that an enum entry can be found by its name using a built-in method valueOf(s). However, what to do if the given string is lowercase while enum entries following the naming convention are called using capital letters? Some people use a combination of toUpperCase() and valueOf() that work just fine but have the penalty we discussed above. However, very often people prefer to create a special field representing a “custom” name, so the simple enum like: Java enum Color { RED, GREEN, BLUE } Turns into: Java enum Color { RED("red"), GREEN("green"), BLUE("blue"), … } Let’s mention that this design has at least two disadvantages: Duplicate data: The custom name is the same as a built-in but in a different case, which can be solved much more easily. This allows using really custom names that, according to my experience, in most cases are not needed and just create so-called “edge cases” that, in turn, in most cases are just a signal of bad design and might cause a lot of “stupid” bugs. However, let’s continue. How do people often use this custom name? Java public static Color ofColor(String color) { return Arrays.stream(values()) .filter(c -> c.color.equals(color)) .findFirst() .orElseThrow(() -> new IllegalArgumentException("No enum constant %s.%s".formatted(Color.class.getName(), color))); } The implementation looks pretty nice, but this approach means that each call of ofColor() iterates over the list. Yes, in most cases enums are not huge, so the list is short, but anyway, why do this if we can just create a map from the custom name to the enum entry once during initialization and then use it with O(1) complexity? The following example solves both problems at once: it uses a case-insensitive map where the key is the standard name() of the enum entry during initialization: Java private static final Map<String, Color> colors = Arrays.stream(values()).collect(toMap(Enum::name, e -> e, (existing, replacement) -> replacement, () -> new TreeMap<>(CASE_INSENSITIVE_ORDER))); So, now the method ofColor() becomes trivial: Java public static Color ofColor(String color) { return Optional.ofNullable(colors.get(color)) .orElseThrow(() -> new IllegalArgumentException("No enum constant for " + color)); } One can argue that a map-based implementation is not always possible because sometimes the lookup criteria are too complex to be reduced to a simple key. Although I agree in general, I can say in turn that in many (if not in most) cases this is still possible. So far, the lookup key was a simple string. But what if the search criteria is a range rather than an exact value? Consider a more physically accurate model of colors as ranges of electromagnetic waves. Java public enum Color { BLUE(450, 495), GREEN(495, 570), RED(620, 750); …} How to implement the method ofWaveLength(int waveLength)? The straight-forward way is to iterate over the values of the enum and compare the given wave length with the range for each entry, i.e. implement O(n) search. But we can do better using NavigableMap, which is designed exactly for this kind of range query: Java private static final NavigableMap<Integer, Color> wavelengthMap = Arrays.stream(values()) .collect(Collectors.toMap( color -> color.minNm, color -> color, (existing, replacement) -> existing, TreeMap::new )); Unfortunately, the search method is not as trivial as in the previous example, but still very simple and fast: Java public static Color ofWaveLength(int nm) { return Optional.ofNullable(wavelengthMap.floorEntry(nm)) .map(Entry::getValue) .filter(value -> nm <= value.maxNm) .orElseThrow(() -> new IllegalArgumentException("No enum constant for wavelength: " + nm + " nm")); } Now, let’s compare the performance. Operation * 1 million times Time, ms valueOf(s) 34 valueOf(toUpperCase(s)) 78 Iteration with equals() 40 Color.ofColor() iteration 166 Color.ofColor() map 20 Color.ofWaveLength() map 32 The table shows that: As expected, toUpperCase() reduces performance twiceIteration with call of equals is a little bit more expensive than valueOf() although the enum has only three members and will grow linearly as the enum grows. The more members enum has, the more time iteration takes. Map-based implementation is even faster than one based on the built-in valueOf(). Stream-based iteration (ofColor() iteration) is surprisingly slow. Stream setup overhead (boxing, lambda dispatch, spliterator initialization) is non-trivial for tiny collections Pre-Intitialization The principle here is: do not do something several times if you can do it once. The most trivial example is string or numeric constants: Java private static final String FILE_NAME = "config.json"; private static final int MAX_VALUE = 10_000; However, the same principle applies to heavier objects — and that is where it really matters. Let’s take a look at logging. Most people are used to writing the following “magic” line at the beginning of each class (unless we use Lombok’s @Slf4j annotation): Java private static final Logger logger = LoggerFactory.getLogger(MyClass.class); Are all these modifiers (private static final) really needed? Some people try to save typing time: Java private final Logger logger = LoggerFactory.getLogger(MyClass.class); Moreover, if the logger is not static, we can do even more: Java private final Logger logger = LoggerFactory.getLogger(getClass()); This line looks better because it is error-proof: the class here is not hard-coded, so this line can be copied as-is from one class to another or inherited from the base class. So, what’s the problem? The problem is that retrieving the correct logger is potentially expensive due to synchronized registry lookups. Doing this on every instantiation adds up. A friend of mine told me that once in the company where he worked, this change in some critical path improved performance so much that they managed to reduce the AWS cluster by about one hundred large EC2 machines. The same rule applies to pattern compilation. As the benchmark table showed, compiling a pattern on every method call is nearly ten times slower than reusing a precompiled one. The result of Pattern.compile() should always be stored in a static final field. The only exception is the case when the regular expression is generated dynamically, but we should do our best to avoid such a design. Very often we have to format or parse dates. Traditionally I used SimpleDateFormat. What can be more obvious than this: Java private static final String FORMAT = "yyyy-MM-dd HH:mm:ss"; private static final DateFormat format = new SimpleDateFormat(FORMAT); Frankly speaking, I did this many times following the principle I stated above: there is no reason to create the instance every time we need it if we can create it only once. The problem is that SimpleDateFormat is not thread-safe, so sharing the same instance among different threads can cause the problem. Even worse: we can live with this bug for years without knowing about it, since it only happens under high load and in some cases can just produce slightly wrong results that can be lost in an ocean of valid data. So, should we create instances of SimpleDateFormat every time we need it and cause CPU and GC to work hard? Fortunately, starting from Java 8, we can use DateTimeFormatter instead: Java private static final DateTimeFormatter formatter = DateTimeFormatter.ofPattern(DATE_FORMAT); This class is thread-safe, so we can share its instance among different threads and get consistent results. Conclusion We started with a quote that is almost always cited incomplete. Knuth never said ignore performance — he said don't sacrifice clarity for speculative gains, while reminding us not to pass up opportunities in that critical 3%. The examples in this article live in that 3%. None of the performance issues described here should ever appear in production code. They are not hard to avoid — they require no profiler, no benchmarking framework, no architectural discussion. Just the habit of reaching for the right tool. And that habit pays off. Choosing equalsIgnoreCase() over toLowerCase().equals() is cleaner and faster. A static final logger is simpler and cheaper. A pre-built enum map is more readable and O(1). Good code and efficient code are not in conflict here — they are the same code. The only thing required is the habit of pausing for a second and asking: am I doing this n times when once would do? All code examples from this article are available on Gist. More

How to Diagnose and Recover Stuck Temporal Workflows

By Akhil Madineni

CORE

The 2026 Observability Audit: Separating Single Vendor Silos From Community Innovation

By Chris Ward

CORE

Ampere System Profiler: A Guide to System-Level Profiling

By Tito Reinhart

Alert Fatigue as a System Design Problem: Engineering On-Call Reliability in Modern SRE Teams

Once upon a time, site reliability engineering rested on a linear assumption: monitor more, detect early, and you’ll recover faster. The rise of alert fatigue makes modern SRE teams realize otherwise: Ramadass's (2025) paper, Building an AI-Powered Observability Pipeline for Modern System Reliability, cited research that discovered that: More than two-thirds (82%, actually) of institutions experience alert spikes constantly.Most traditional monitoring tools generate approximately 2,100 alerts daily, with about 70% of them unnecessary and safe to ignore.66% of SRE professionals stated that increased false alerts lead to fatigue, potentially causing them to miss serious issues. How Should We Describe This Situation? Vigilance or Noise? Collaborative systems such as SaaS, third-party APIs, and microservices enhance the degree of observability and notification within systems. Everything is monitored, and occasionally these dependencies may duplicate alerts. When systems request superhuman attention, on-call engineers become fatigued rather than lazy or sloppy. Instead of swift action, alerts are responded to with mistrust. Reliability vs. Experience vs. Metrics Traditional alerting metrics follow traditional reliability practices, that is, error rates, uptime percentages, latency, etc. Although these are essential, they are not actual mirrors of how operators or users experience reliability. Operators may expect reliable alerting to inform decisions, while users may simply define reliability as how well a system enables them to fulfill their intentions. If alerts do not clearly connect to the user experience, there is a gap between detection and action. Over time, the gaps lead to fatigue. On-call engineers begin to “reasonably” ignore these alerts. Why worry over alerts that are not logically related to user outcomes? They may assume. Over time, organizations may end up paying dearly for real issues because alerts were missed or delayed. An On-Call Engineer Experience Here is a typical example of a system design problem an on-call engineer or SRE team may face: 01:15 AM Alert: Latency spikes on a third-party API.01:16 AM Alert: Retry queues are filled.01:16 AM Alert: Timeout alert storms on three dependencies.01:17 AM Alert: Error-rate notification on unrelated endpoints.01:18 AM Alert: Memory and update alerts. And this sequence of alert storms continues, with the on-call engineer receiving more than 20 alerts in just four minutes. The system seems to pass standard observability SRE practice. But what about the long-run reliability suspicions that the bugging signals may create? In this case, the teams are not just grappling with response speed but also with the amplification of confusion when critical alerts are mixed with non-actionable ones. When Detection Outpaces Interpretation We can’t rule out the fact that monitoring in the past decades has taken an advanced leap. And we might be at its cloying stage, where system detection software is outpacing on-call engineers’ interpretation. Systems are “wonder-full” when it comes to identifying when something seems “off.” However, they rarely give explicit descriptions to aid SRE teams’ understanding. An alert can indicate that a queue has exceeded its depth, but may not categorically state whether the issue is temporary or actionable, or whether users are affected. This occurrence spans dozens of dependencies, each with its own signal. The on-call engineer is kept puzzled about the best action to take at the right time. Hence, a reliable response could be excessive caution or delay as the engineer seeks to clarify the situation. The users are negatively impacted. Although the system met technical observability SRE standards, it failed operationally due to its opacity. The Hidden Cost of Alert Overload We rarely see the outcome of alert fatigue overnight. Its effects build up. Delayed response time accumulates. The aftermath incident review loses credibility. Engineers are skeptical of alerts and hesitate to decide first whether they are real or false. The cultural cost of alert fatigue is that on-call roles become a burden SRE teams endure rather than enjoy with a sense of responsibility. In the long run, engineers may feel they have no control over issues due to the confusion that multiple alerts create. Ironically, the same reliability problems that alerts were designed to solve are what they quietly create. Are Alerts Creating a False Sense of Safety? Lots of alerts may seem like a good thing or a sign of strong monitoring at first glance. But here is the truth: alerts could be hiding actual risk. As every deviation is notified, critical and minor alerts blend in. Teams begin to feel alert fatigue and delay response. Then, real problems begin to breed behind the scenes. Remember how SLAs could paint an illusory picture of safety? Similarly, alert volume could do so. Therefore, your SRE team should bind these caveats as the core of their modus operandi. Alerts shouldn’t replace action.Alerts shouldn’t be unsorted (by machines or humans).Alerts shouldn't be discarded. Alerts are signs that our systems need attention, and we should never be tired of listening. SRE Teams Designing Systems that Alert Smartly High-quality systems respond efficiently when dependencies fail. Instead of creating panic, they automatically degrade. SRE teams could design circuit breakers that could inhibit alert storms before they explode. They could also install bulkheads to prevent a single failure from spreading. There could be alert limits and a summary of conditions that resolve the problem of spamming. Instead of relying on metrics, system engineers could set up composite alerts that describe system states. For instance, it’s clearer if a system alert indicates, “Checkout degraded because of latency in payment dependency.” This composite alert is better than 7 alerts that say “Checkout Timeout.” The former shows impact, cause, scope, and urgency. Clarity clears fatigue. Noise does the opposite. Redesigning SRE: Human Reliability That Quells Alert Fatigue We have seen that technical designs may be great, yet other aspects of SRE remain wanting. One such area that could resolve a system design problem is humaneness. To avoid alert fatigue, our design choices must acknowledge human limitations. Therefore, we should accept that some alerts may not require immediate response. Conversely, not every anomaly should trigger an alarm. Understood silence could sometimes be a golden sign that nothing critical is wrong. Advanced SRE teams do not focus on events (or every deviation) but on the states of the system or infrastructure. They are guided by the question: What conditions really impact users, business objectives, or the system's overall health? To achieve this, engineers need to balance product understanding with technical operations. Then they can give a human touch to their designs. Designing systems for human reliability requires a high level of discipline. Site reliability engineers have to continually review, refine, and repair alerts and their trigger commands. Systems are like living organisms that need constant feeding of updates. The evolving nature of alerts could make a helpful one-time alert redundant or harmful in six months. On-Call as a Reliability Interface of SRE No doubt, humans have a role to play in ensuring reliability, but system designs that depend on heroic actions are built not with resilience but with fragility. Reliability is truly achieved when on-call engineers are guided by predefined scripts, models, runbooks, signals, and interfaces. These reduce the tendency to resort to fallible improvisations when issues arise. On-call engineers often take the appellation of “last point of call.” A careful look at their roles shows that they are intermediaries among complex systems, user experience, and consequences. We can thus see that the role of on-call engineers extends beyond problem resolution to stewardship. Conclusion Alert fatigue is a design problem. It often arises when on-call engineers prioritize detection over interpretation, or technical workability over user experience. The dependencies of modern SRE teams make it necessary to align technical alerts with human capability. Alert storms could wear out hardworking engineers who need to take a break. So, system designs need to account for human limitations, recognize that runbooks are better than on-the-spot improvisation, and prioritize clarity over opacity. Designs that account for these factors reduce or eliminate fatigue and preserve the very essence of alerts. In summary, reliability goes beyond resolving many problems to responding to what matters most. When teams can always trust their alerts, they will be more likely to follow up on new cases.

By Oreoluwa Omoike

Reliability Without Control: Operating SRE Practices in Platform–SaaS and API-Dependent Systems

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority. Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control. Unlike in previous systems, modern production is largely assembled rather than built from scratch. For example, a conventional SaaS product is built on interdependencies among payment processors, outsourced data infrastructure such as Amazon Web Services (AWS), messaging services, web hosting, design, AI inference APIs, authentication providers like Google, and more. These useful platforms and products are essentially outside teams' control stations, even though they critically impact users' experience. When they function effectively, you share the glory with the platforms. But when there is a system blackout, your users put you on your toes, even though you have no direct access to resolve the problem on time. Therefore, we shall be exposing SRE practices in platform-SaaS and API-dependent systems and how reliability is getting beyond the control of engineering teams and companies. Why Classical SRE Practices May Fail One major downside of SaaS and dependency on external platforms is that reliability control is often assumed to be in a team's hands, whereas it has been bargained. However, teams must reckon with the fact that the case is reversing. For example, traditional SRE models once alleged that: Service Level Indicators (SLIs) focus on availability or internal uptime and latency.Error budgets arise from changes teams make or deploy.Runbooks still suggest that teams can immediately reconfigure or directly work on faulty components. All these are becoming past cases, especially in platform-SaaS systems. You can have a system indicating 99.99% or even 100% uptime on the back end, while new users are struggling to sign up, probably because an authenticator provider is not fully functional. Dashboards and control panels may indicate green, but in reality, third-party payment APIs have been degraded. A New Definition of Reliability in Operating SRE Practices To resolve the new problem in site reliability engineering (SRE), there needs to be a conceptual shift from component health to an integrated, continuous user experience. Therefore, teams need to undergo a paradigm shift away from questions such as "Is our CPU working maximally?" “Is our API up?” “What are the error rates?” Instead, we should inquire: “Are users checking out seamlessly?” “How fast can they authenticate?” “Can they use the SaaS product to perform its key function?” These types of outcome-based questions span interdependent platforms beyond your full control. The login SLI needs to work with the identity provider; otherwise, its output is meaningless. If the checkout SLO skips payment authorization, then it's both fishy and unreliable. True, there may be some internal errors in a reliable system, but what really matters is an integrated multiplatform experience that the user enjoys. Error Budgets? An SRE Practice to Revisit How many teams would love error budgets to disappear when they give up control? But that’s not so. Instead, they are molecularized. When components of your systems are outsourced, the error budget doesn’t just fade away; it is instead transferred to the interdependent platforms. So, it’s better to plan for the fact that SaaS and API providers will consume some of your reliability budget. Doing so keeps you a few steps ahead and protects your business in the long run. Reliable SRE teams make decisions such as allocating part of their error budget to certain dependencies, setting acceptable parameters for degradation, and defining specific steps to take when a dependency exceeds the stipulated budgets. Here’s an example you can adapt: “We will accept payment authorization failure of 0.0% to 0.2% if it is caused by dependency instability. If it goes above that, we will turn on delayed capture or turn off promotions.” This SRE approach keeps you ready for downtime, as your systems automatically switch to planned or budgeted actions rather than relying solely on integrated platforms. What to Do When Failures Beyond Your Control Arise Actually, some failures may seem beyond your control. The more you attempt to resolve them, the more amplified they become. At this point, your team must adapt to the savvy absorption of such situations. Instead of focusing solely on retrial in an SRE approach, your team needs to design its processes and platforms. This could include failing selectively through circuit breakers, failing fast with timeouts, or failing visibly by keeping users informed. Some core settings should always remain non-negotiable and on standby. These could include the following: Read-only modes/cachesBulkheads that prevent a failure avalanche.Automated circuit breakersDeferred processing These reliable practices ensure there is some form of controlled uptime even when operations seem interrupted. Laser Observability That Proves Reliability In traditional SRE observability, the service boundary is usually the ultimate, but in most modern integrated SaaS platforms, this could be insufficient or worse, dangerous. Operators need to be aware of the actual dependency that is failing, how it is failing (e.g., errors or throttling), and how the failure affects the user experience. Accurate observability for platform-SaaS and API-dependent systems requires these four provisions: Specific dashboard and internal metrics for each vendor.SLI monitoring at the dependency level.Parallel tracing of all outbound calls.Simulation of real-time user experience and workflows. Essentially, whenever there is an emergency, operators should be able to promptly identify whether the source is internal or external. Accuracy and clarity facilitate swift response. Responding to Incidents Without Ownership Another distinct characteristic of modern SRE practice in platform-SaaS is how incidents are responded to. Without ownership, you often cannot debug on your own, roll back a bad deploy, or directly manage other issues. However, you can choose how your system responds by identifying when certain features are disabled, when signals to activate degraded modes are sent, when high traffic is redirected or shed, or when to notify users. To maintain reliability, incident response relies on runbooks to inform decisions. The following questions could help convert the technicality of runbooks to practical solutions: What is the impact on the customer?In what ways can we respond harmlessly?What can we reverse?What should we communicate externally? These questions help resolve incidents, mitigate losses, and intertwine reliability with sound judgment. Is Safety an Illusion in SLAs? SLA providers often readily contract for financial compensation when losses arise, but seldom give absolute reliability guarantees. You may not always expect vendors to consistently meet your availability goals or resolve an avalanche of outages. Safety is a critical consideration when building systems, because when users lose trust in a brand, compensation may not be able to redeem it. Therefore, advanced teams do not consider SLAs as safety nets but as risk pricing. They understand that contractual credits cannot replace trust, brand image, and some almost irredeemable damages. Human Factors in Platform-SaaS and API-Dependent Systems Dependency failures often escalate when cognitive load increases. There could be degraded performance, timeouts without error indicators, partial success, or inconsistent system behavior. Operators may not only focus on machines when dashboards lag or seem to lie. They examine the logs, failure history, or commands. Teams have to design systems with overrides and predictable degradation paths, and observability tools are beyond the failure systems. Reliability goes beyond the correct function of software; it's also about human operations. How Your SaaS and API Platforms Can Imbibe “Good” SRE Practice Effective SRE practices are modern. The following attributes know saas products and API-dependent platforms: Acknowledgment of lack of control very early.Ensuring reliability is embedded in the design.Measuring the outcomes of each SRE criterion or target, instead of just the components.Giving priority to clarity instead of trying to model or control everything because you do not own all the components.Making engineering and operations decisions and products as an integrated whole.Preparing for degradations as inevitable procedures when things fail. Your systems can be reliable if you anticipate failure and accept the reality. Conclusion Modern platform-as-a-service (SaaS) operates in a reliability-without-control manner, leading solid SRE teams to accept that they need to adapt when failures occur. It's simple logic: if you don't absolutely own everything end-to-end, then prepare for the worst: each dependency might fail. It's all about keeping the trust of your users and protecting your brand image.

By Oreoluwa Omoike

When Downtime Means an Unlocked Front Door

Anyone who has carried a pager long enough develops a professional numbness. A queue backs up, a p99 drifts past budget, a deploy does something stupid at 40% rollout. You fix it, you write it up, you go back to sleep. The stakes are real but abstract: revenue per minute, an SLA credit, a churn number on somebody's spreadsheet. That numbness doesn't survive contact with a product people rely on for safety. Picture a regional connectivity degradation after midnight, and an impact estimate on a screen. Normal night, ordinary graph. Then consider what the number is actually counting. For some slice of households in that region, a camera at a front door has gone dark. A parent who checks whether their kid got home. Someone who installed cameras after a break-in and now sleeps better. I want to be careful here, because "our uptime saves lives" is the sort of thing that makes engineers roll their eyes, rightly. Most minutes of most outages harm nobody. But the distribution has a tail, and when a product is somebody's sense of safety, the tail is where the meaning of the work lives. Taking that seriously changes what you measure, what you alert on, and what you're willing to drop under load. Edge Reliability Is a Different Animal Much of my earlier career was conventional infrastructure work with data-center consolidation, phased cloud migrations of several thousand workloads, where most of the energy goes into sequencing risk rather than into any particular technology. Good training in systems thinking. Almost no preparation for how consumer hardware behaves in the wild. Three properties make this class of system unlike a web service. The edge is hostile, and you own none of it. Devices sit on consumer Wi-Fi, behind a cheap router, on an oversubscribed ISP, drawing power that browns out during exactly the storms when people most want their cameras working. You own none of the last mile and all of the customer's expectation of it.Demand is correlated, which breaks naive capacity planning. Web traffic averages out; event traffic from a physical fleet doesn't. A delivery wave or a thunderstorm crossing three states produces millions of events inside the same few minutes. Independent load is easy. Synchronized load is what pages you.Failure costs are wildly uneven. A dropped analytics event is a rounding error. A live view that spins for eight seconds while somebody stands on a porch is arguably worse than an outright failure, because the customer sat there and watched the product not work. If not all errors cost the same, they shouldn't page the same, yet nearly every monitoring setup I've encountered treats them identically. Together, those produce the failure mode that should drive the whole design: you can be green on CPU, memory, and 5xx rate while customers are having a red night. Define "Up" in the Customer's Language The first fight in this kind of environment is about vocabulary. "The service is up" tends to mean "the servers are up," and those are not the same claim. The alternative is SLIs built on journeys a person can perceive. For a home-security-shaped product, that's notification latency from edge event to push landing on the phone, live-view time-to-first-frame, and clip availability and retention. The second one is where I'd push hardest, because there's a tempting shortcut. "Session established" is easy to measure and usually already instrumented. It's also a lie, since a session can establish and then deliver nothing watchable for another three seconds. Move to first rendered frame, and your numbers get noticeably worse — which is how you learn the old metric was flattering you. An objective in that shape reads roughly: 1SLI: proportion of device events whose push notification is2delivered in <= 4s, measured end to end.3SLO: 99.5% of events over a rolling 28-day window. Two choices there matter more than the threshold. Use rolling windows rather than calendar months, because calendar boundaries teach teams to hold their breath until the first and then ship anyway. And give the error budget real authority: burn it, the deploy freeze happens, no case-by-case negotiation with whoever has the loudest roadmap. That second one is where program-management discipline earns its keep more than any architectural decision — governance, entry and exit criteria, named decision rights. It sounds like bureaucracy right until the first freeze holds without anyone having to win a political argument. One test worth applying to every proposed SLO: if breaching it wouldn't change what anybody does next week, it isn't an SLO. It's a dashboard. Measuring Is Harder Than Target-Setting Picking four seconds is easy. Knowing whether you hit it is hard, because the clock starts on a device you don't control and stops on a phone you don't either. The approach that works is correlation across the full path. The device stamps an event ID at capture, and it rides every hop — ingest, media pipeline, notification service, push provider, client ACK. Each hop logs the shared ID with a timestamp, and you reassemble the journey afterward. In Splunk, that looks something like: SQL index=device_events (stage=capture OR stage=push_ack) | stats earliest(_time) as t_capture, latest(_time) as t_ack, values(region) as region by event_id | eval e2e_latency_s = t_ack - t_capture | where isnotnull(t_ack) | eval met_slo = if(e2e_latency_s <= 4, 1, 0) | stats count as total, sum(met_slo) as ok, perc99(e2e_latency_s) as p99_s by region | eval attainment = round(100 * ok / total, 3) | sort - p99_s Unglamorous, and it answers the only question that matters four minutes after waking up: are real people getting notifications on time, and if not, where. Sorting by regional p99 turns "something feels slow" into a location. Now look at the isnotnull(t_ack) filter, because it's the most instructive line in the query. It quietly excludes every journey that never completed, which is the worst outcome for a customer. That exclusion is how an entire class of failure hides in plain sight. The Failure Class That Never Pages You The scenario I'd most want a team to design against is the one that generates no alert at all. A subset of devices with one hardware revision, one firmware version, one specific reconnect path stops delivering notifications while continuing to report healthy. Capture succeeds. The event enters the pipeline. It simply never produces an ACK. Every aggregate metric absorbs it. The affected population is small enough that regional p99 doesn't budge, attainment stays inside budget, and no error rate moves, because nothing errored. Detection ends up coming from a cluster of support tickets, which is the most expensive monitoring you can buy: it means your customers are doing it for you. Two defenses follow directly. Alert on capture events with no matching ACK inside a window, as its own signal rather than folded into latency with slow and failed have different runbooks and different customer meanings. And break tier-1 attainment out by device model and firmware version, accepting the cardinality cost, because a well-behaved aggregate is very good at hiding a badly-behaved cohort. Averages don't just lie about the tail. They lie about who's in it. Running the Room Detection without a fast, calm response is just expensive telemetry. Three things I'd insist on anywhere. Separate the Incident Commander from the person fixing it. Those are genuinely different jobs, and you can watch the debugging degrade in real time when one person does both while fielding stakeholder pings. It's the cheapest reliability improvement available and costs zero engineering hours. Define severity by customer impact, never by component. An internal dashboard degrading is a SEV-3 no matter how loudly its owner complains. Notifications delayed across a region is a SEV-1 immediately, because the product's core promise is broken. Writing that down ends a lot of arguments before they start. Run game days, and expect the first one to be humbling. In my experience, the runbook is wrong in several places, at least one dashboard fails to load under load, and finding the person who can trigger a manual failover takes longer than anyone predicted. By the fifth game day it's boring, which is the entire point. What I'd Tell My Earlier Self Write SLOs in the customer's language and give the budget teeth. Instrument the journey with a correlation key, because component metrics tell you what broke while journey metrics tell you what the customer felt. Decide in advance what you shed under load, and that judgment is too important to make at 3 a.m. Break tier-1 metrics out by cohort, because aggregates hide the people you're failing. And separate the commander from the fixer. The stack will keep moving: more inference at the edge, new codecs, whatever replaces today's push mechanics. The operating philosophy doesn't change. Measure what people actually experience, protect the moments that matter, fail gently, respond as it counts. The best feedback this work gets is silence.

By Naveen Goel

How AI Is Actually Changing SRE Tools, Part 2: ITOps, Chaos Engineering, and the Rest of the Job

In Part 1, I walked through how AI is changing incident response, from correlation engines like BigPanda and PagerDuty's AIOps features to a newer category of dedicated AI SRE agents like Traversal, Resolve.ai, and Cleric that investigate incidents autonomously instead of just clustering alerts you already collected. Incident response gets the spotlight because it's the loudest, most visible part of the job. But if you actually track where an SRE's week goes, a good chunk of it isn't firefighting at all. It's ITOps tickets, chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. This second part covers where AI is showing up in all of that, with the same rule I applied in Part 1: vendor-reported numbers get flagged as vendor-reported, and I say plainly where adoption is still low regardless of how good the tooling has gotten. ITOps: A Slower But Real Shift ITOps has been slower to change than incident management, partly because the data is messier. CMDB entries are stale, ticket categories are inconsistent, and a lot of ITOps work still runs through change advisory boards that move at the speed of a Tuesday meeting. Even so, a few areas have real AI traction: Predictive capacity planning. Rather than static thresholds ("alert at 80% CPU"), some platforms now model usage trends and flag capacity issues days before they'd trip a traditional threshold.Automated ticket triage and routing. Classifying a ticket and routing it to the right queue used to be a rules engine with hundreds of brittle conditions. Language models handle the free-text classification part noticeably better.Change risk scoring. A few platforms now score proposed changes against historical incident data to flag "this type of change caused an outage 3 of the last 20 times." Useful as a second opinion, not a replacement for review. I covered the automation side of this shift, provisioning and managing the infrastructure these ITOps tools sit on top of, in more depth in Infrastructure as Code: How Automation Evolved to Power AI Workloads and Cloud Automation Excellence: Terraform, Ansible, and Nomad for Enterprise Architecture. ITOps AI features are only as good as the infrastructure state they're reasoning about, and that state is usually managed by exactly these kinds of tools. Beyond Incidents and Tickets: The Rest of the SRE Job Incident management The point of this diagram is that the reactive stuff on the left only gets easier if the proactive and human-layer work on the right actually happens. A great incident agent bolted onto a team with no SLOs and a runbook wiki nobody's touched in two years will still struggle. Chaos Engineering Gets a Reasoning Layer Chaos tools like Gremlin, Steadybit, and Harness's chaos engineering module used to require someone to manually design experiments: pick a service, pick a failure mode, guess a reasonable blast radius. That design step is where AI is actually helping now. Harness added generative capabilities that analyze your architecture and operational data to suggest which experiments would teach you the most, instead of you guessing. Steadybit went further and shipped what it calls the first MCP server built for chaos engineering, letting LLM agents query past experiment results directly. That's the same protocol-level pattern I wrote about in MCP vs Skills vs Agents With Scripts: giving an agent a standardized way to query a tool's data instead of scraping a dashboard. In practice, it means an incident investigation agent could eventually ask "have we ever tested this failure mode before" and get a real, structured answer instead of nothing. Worth saying plainly: adoption here is still low industry-wide. Independent research from LogicMonitor's 2026 SRE Report found that resilience engineering is widely valued on paper, but production chaos testing remains uncommon, and many organizations still have low tolerance for deliberate failure injection. AI lowers the design cost of running an experiment, but it doesn't fix the organizational nervousness about deliberately breaking things, and that's a culture problem no agent solves for you. SLOs and Error Budgets Get Easier to Set Up, Not Easier to Enforce Tools like Nobl9 have leaned into AI mostly at the setup stage: pointing at your existing observability data and proposing a reasonable SLO instead of making you guess a number out of thin air, and flagging when a service's error budget burn rate suggests you should stop shipping features and go fix things instead. That second part, "should we stop shipping," is still an organizational decision no tool makes for you. What AI changes here is the friction of getting from zero SLOs to a defensible first draft, which used to take a workshop and a spreadsheet and increasingly takes an afternoon. On-Call Scheduling and Toil Reduction This is the least flashy category and probably the most immediately felt by individual engineers. PagerDuty, Opsgenie, and similar tools have had smart scheduling for years, balancing load and skipping people near PTO. What's newer is toil-specific analysis: some platforms now scan a team's ticket and page history to flag which recurring alerts are pure noise versus which ones represent real, fixable problems, and rank them by engineer-hours wasted. It's a small feature compared to an AI SRE agent doing live root cause analysis, but for a burned-out on-call rotation, "here are your top five noisiest alerts by wasted hours" is sometimes the most useful report in the whole stack. Runbook and Knowledge Management The unglamorous truth about most incidents is that the fix was already documented somewhere, if anyone could find it. This is turning into one of the more genuinely useful applications of retrieval-augmented generation in the SRE space: instead of an engineer grepping a wiki during an active incident, a chat interface pulls the relevant runbook section, the last few times this alert fired, and who fixed it, in one query. Datadog's Bits AI, ServiceNow's Now Assist, and most of the dedicated AI SRE agents from Part 1 all lean on this pattern. The quality ceiling here is entirely set by how good your existing documentation is, which brings us back to a point worth repeating: these tools reward teams that already write things down. If your team is still deciding how to organize that knowledge layer for agents to query safely, that's exactly the ground covered in Trust No Agent: How to Secure Autonomous Tools on Your Machine. Capacity and Cost Optimization I touched on predictive capacity planning under ITOps, but it deserves a wider frame. A lot of what used to be manual FinOps work — right-sizing instances, catching orphaned resources, forecasting when a service will outgrow its current tier — is now a background AI process in platforms like Datadog, Dynatrace, and the major cloud providers' own cost tools. For SREs, the payoff isn't glamorous, but it's real: fewer capacity-related pages, because the system flagged the trend three weeks before it became an incident instead of after. What All of This Adds Up To for the SRE Persona Put the incident agents from Part 1 together with the chaos assistants, SLO copilots, and runbook retrieval from this part, and the actual shift in the job looks less like "AI does SRE work" and more like this: TaskBeforeNowWhat the SRE Still OwnsInvestigating an incidentManual dashboard hoppingAgent proposes root cause with evidenceValidating the evidence, deciding the fixDesigning a chaos experimentManual guesswork on blast radiusAI suggests high-value experimentsDeciding organizational risk toleranceSetting an SLOWorkshop, spreadsheet mathAI proposes a data-backed draftDeciding what the business actually needsFinding the right runbookWiki search during a live pageChat interface surfaces it in secondsJudging if it still appliesScheduling on-callManual rotation and swapsAI balances load, flags toil hotspotsDeciding if the rotation itself is sustainableForecasting capacityManual trend-watchingAI flags the trend earlyApproving the spend The pattern repeating across every row: AI is good at surfacing options and drafts, and still bad at owning the judgment call that has actual consequences. That's not a limitation to apologize for. It's the correct division of labor for now, and probably for a while. What I'd Actually Recommend If you're evaluating tools for your team across either part of this series, a few things I've learned the hard way: Fix your data before buying a tool. Correlation and retrieval are only as good as your alert taxonomy and postmortem history. A brilliant model over garbage data still gives you garbage.Don't let AI auto-remediate anything you haven't tested extensively. Suggestion is fine. Auto-restart-the-production-database is not, unless you've earned that trust over months. This applies doubly to the AI SRE agents from Part 1, most of which default to read-only for exactly this reason.Budget time for the writing habit, not just the tool. Auto-drafted postmortems only help if someone still reviews and improves them. Teams that treat the draft as final start losing institutional knowledge fast.Pilot on one team first. ITOps and chaos engineering rollouts especially tend to get sold org-wide before anyone's tested them against your actual ticket mess or your actual appetite for deliberate failure.Ask what the agent actually queries, not just what it outputs. For the AI SRE agent category specifically, the evidence trail is the product. If a tool can't show you exactly what it checked before proposing a root cause, treat the proposal as a guess with good formatting. Where This Is Headed I keep going back and forth on how much further this goes. The correlation and drafting gains from Part 1 are real, and I use them daily now, and the newer agent-based investigation tools are the first thing in a while that's actually changed how fast I can get from "page fires" to "I know what broke." The proactive side covered here — chaos experiment design, SLO drafting, toil analysis — is quieter progress, but it's the kind that compounds: every noisy alert an AI flags and a team actually fixes is one less 2 a.m. page for good, not just a faster resolution of the next one. But there's a gap between "explains what probably happened" and "understands the system well enough to fix novel failures," and I don't think that gap closes with a bigger model. It closes with better telemetry, better documentation, and engineers who still know how to read a stack trace without an assistant summarizing it for them. If you want a broader look at the open-source side of that telemetry and tooling layer, I put together a rundown in Open-Source LLM Tools Worth Your Time and Developer Tools That Actually Matter in 2026. If your team is evaluating AI features anywhere in your reliability stack this year — incident response, ITOps, chaos testing, or SLO management — start by asking what data problem it's solving, not what model it's built on. The model is rarely the bottleneck. Your alert hygiene, your documentation, and your SLO coverage almost always are.

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Solving Session Persistence for Model Context Protocol Servers at Enterprise Scale

Model Context Protocol (MCP) servers that work perfectly in development can fail intermittently once they are deployed across multiple replicas behind a load balancer. The failure mode is a stream of "session not found" errors that appear at random, and the cause is a mismatch between how certain MCP transports hold session state and how load balancers distribute requests. This article explains why the problem occurs, when it applies, and a concrete pattern for solving it using a shared session store. The problem is easy to miss in early development because it only appears once there is more than one server instance. A single-instance deployment holds every session in local memory, so every request naturally finds its session. Add replicas, and that assumption quietly breaks. The Failure Pattern Consider a deployment with four MCP server replicas behind a round-robin load balancer, serving agents that connect over the Server-Sent Events (SSE) transport. In this configuration, roughly three out of four follow-up requests fail with a "session not found" error. That ratio is not random. With four replicas and round-robin distribution, a follow-up request has only a one-in-four chance of returning to the replica that created the session. The other three times it lands on a replica that has no record of that session. The reason the failures look random at first is that success depends entirely on which replica the load balancer happens to select. The distribution of failures tracks the replica count directly, which is the clearest signal that the load balancer, not application logic, is the source of the problem. Why MCP Sessions and Load Balancers Conflict Not every MCP deployment has this problem, so it helps to be precise about when it applies. A tools-only MCP server can be stateless. Under the streamable HTTP transport, the client caches tool schemas after discovery, and each tool call is a self-contained request that carries everything the server needs to process it. Any replica can handle any request, and load balancing works without special handling. Two situations make a deployment session-bound. The first is the SSE transport. SSE was the only remote transport available for a long time and remains widely deployed. It is stateful by design: the client opens a long-lived connection that the server holds open as a stream, and the server delivers responses back through that open stream rather than through the response to each individual request. The stream physically lives on one replica. When a follow-up request is routed to a different replica, that replica is not holding the stream and cannot associate the request with the session. The result is the "session not found" error. The second is stateful MCP features. Even on a transport that supports stateless operation, an MCP server that must retain per-client state needs sessions. MCP resource subscriptions that push updates when server-side data changes, long-running operations where a client may disconnect and reconnect expecting to resume, and per-client authorization context established at initialization all require the server to hold state across requests. That state must be reachable regardless of which replica receives the next request. The conflict reduces to a single sentence, which is that the session lives on one replica, but the load balancer distributes requests across all of them. How the Connection Is Established The session originates at connection time. The following example uses the Koog framework to connect an agent to an MCP server over SSE, which illustrates where the session comes from: Kotlin import ai.koog.agents.core.agent.AIAgent import ai.koog.agents.mcp.McpToolRegistryProvider import ai.koog.prompt.executor.llms.all.simpleAnthropicAIExecutor import ai.koog.prompt.llm.AnthropicModels import kotlinx.coroutines.runBlocking fun main() = runBlocking { // Open an SSE transport to the MCP server val transport = McpToolRegistryProvider.defaultSseTransport("http://mcp-server:3000/sse") // Build a tool registry from the tools the MCP server val mcpRegistry = McpToolRegistryProvider.fromTransport( transport = transport, name = "records-client", version = "1.0.0" ) val agent = AIAgent( executor = simpleAnthropicAIExecutor(), llmModel = AnthropicModels.Claude.SONNET, toolRegistry = mcpRegistry ) val result = agent.run("Look up the status of record 12345") println(result) } The relevant detail is the transport and the roles it establishes. The client, Koog in this case, opens the SSE connection, and the session lives on the MCP server. Opening an SSE transport creates a stateful connection: the MCP server creates a session bound to that open stream, and from that point the client and server communicate through a channel anchored to one specific server instance. With a single instance, this is invisible. Behind a load balancer, it is the entire problem. The fix belongs on the server side, not in the client. The Fix: A Shared Session Store The solution is to stop storing session state in a replica's local memory and move it to a shared store that every replica can reach. This is an addition to the MCP server implementation. Neither the MCP specification nor the client library provides a distributed session store; the specification defines that sessions exist but does not prescribe how to persist them across instances, so the server-side session handling is the implementer's responsibility. Redis is a natural fit for this role because the access pattern is a simple keyed lookup and the added latency is negligible relative to the rest of an agent request. The mechanism is straightforward. When any replica creates a session, it writes the session record to the shared store rather than to local memory. When any replica receives a request, it reads the session from the shared store before processing. The session no longer belongs to a replica; it belongs to the store, and every replica can reach it. The session record contains what the server would otherwise hold in memory - the session identifier, the negotiated capabilities, any accumulated per-client state, and timestamps for expiry. Assigning each entry a time-to-live allows idle sessions to expire automatically rather than accumulating. The change in the server's request handling can be reduced to the difference between a local map and a shared lookup: Kotlin // Before: the session lives in this replica's memory. // Other replicas have no record of it. val localSessions = mutableMapOf<String, McpSession>() fun handleRequest(sessionId: String, request: McpRequest): McpResponse { val session = localSessions[sessionId] ?: error("session not found") // fails on any other replica return session.process(request) } Kotlin // After: the session lives in a shared store every replica can read. suspend fun handleRequest(sessionId: String, request: McpRequest): McpResponse { val session = sessionStore.get(sessionId) // shared lookup ?: error("session expired or unknown") val response = session.process(request) sessionStore.put(sessionId, session) // persist any state change return response } The SSE transport adds one further requirement. Because the response must travel back through the stream held by a specific replica, the shared store also records which replica holds the stream, and a publish-subscribe channel routes the response to that replica when a request is handled elsewhere: Kotlin // The replica holding the SSE stream subscribes for its sessions sessionBus.subscribe("mcp:response:$sessionId") { payload -> sseStream.send(payload) } // Any replica that processes a request publishes the response sessionBus.publish("mcp:response:$sessionId", response) In this arrangement, the shared store serves two purposes. It is the session store that allows any replica to handle a request, and it is the message bus that routes each response to the replica holding the open stream. A request may arrive at any replica, while the response is delivered to the connection the client is actually listening on. Why Not Sticky Sessions The most immediate alternative is sticky sessions: configuring the load balancer to pin each client to the replica that created its session. This works and is a reasonable temporary measure, but it carries three drawbacks that make it unsuitable as a durable solution. Sticky sessions undermine load distribution, because a high-volume client is concentrated on a single replica while others remain underused. They reintroduce the single point of failure that multiple replicas were intended to eliminate: if the pinned replica fails, every session on it is lost. And they complicate scaling, because newly added replicas receive no existing traffic and take on load only gradually. A shared session store avoids all three. The load balancer can use plain round-robin distribution. Any replica can fail without affecting sessions held by the others. A new replica can serve existing sessions immediately, because it reads them from the same shared store as every other replica. Results With the shared session store in place, the "session not found" errors are eliminated for active sessions, and requests distribute evenly across replicas. Deliberately terminating a replica no longer interrupts active agents, and their requests are absorbed by the remaining replicas. Adding a replica requires no special handling. The shared lookup adds a small step to each request, but the cost is minor in context. A session read is well under a millisecond, while an agent request already spends hundreds of milliseconds or more on model inference and downstream calls. The overhead is not observable in practice. Summary For teams deploying MCP servers at scale, three points are worth carrying forward. Keep the MCP server stateless where possible. A tools-only server on the streamable HTTP transport scales horizontally without any of this complexity. Sessions should be introduced only when genuinely required, for subscriptions, resumable operations, or server-held per-client context. When sessions are required, do not store them on the replica. Move them to a shared store so that any replica can serve any request. This mirrors the lesson web applications settled on years ago for HTTP session state, now recurring in the context of MCP. Account for the SSE response-routing requirement. A shared session store resolves request handling, but the response must still reach the replica holding the open stream, which a publish-subscribe channel provides. Session persistence behind a load balancer is a common example of the operational gaps teams encounter when deploying MCP in production, and the shared-store pattern described here is a direct and durable solution.

By shravya boini

Arm64 Is No Longer the Edge Case

For years, Arm64 was the platform people talked about as a future bet. It was useful in embedded systems, interesting in research, and easy to dismiss as “not the main thing.” That era is over. In a conversation between Dave Neary, Director of Developer Relations at Ampere Computing, and Greg Kroah-Hartman, Linux stable kernel maintainer and long-time kernel developer, the message is clear: Arm64 has become mainstream. It is no longer a special-case architecture. It is a first-class platform in Linux development, deployment, and maintenance. Arm64 Has Become a First-Class Platform in Linux Development Kroah-Hartman’s history with Linux goes back to the late 1990s, when his work in embedded systems led him into kernel development. He started by solving practical device problems, such as getting USB hardware working across many systems. That hands-on work turned into a career built around making Linux more reliable, more portable, and more useful across different hardware. One of the biggest changes he describes is how the Linux community matured. Early on, Linux developers often borrowed ideas from Unix, BSD, and Windows. The goal was to make things function. Over time, Linux moved from catching up to leading. Once that happened, the work became harder. Developers were no longer copying proven models; they were building new infrastructure, new interfaces, and new processes that had to work at scale. That shift also explains why the stable kernel process matters so much. In 2005, Linux moved toward time-based releases and created a stable kernel series focused only on bug fixes. That decision made it possible to keep improving Linux without breaking user space or workloads. For developers, that means a reliable update path. For users, it means confidence that the system will continue to work. Arm64’s growth has made that stability even more important. Today, Arm64 is everywhere: phones, laptops, embedded systems, cloud servers, appliances, and high-performance computing. Linux now runs across all of it. That breadth has changed the ecosystem. When Arm64 breaks, the impact is no longer small. It affects real products and real users across the industry. Upstream Development Improves Arm64 Linux Reliability and Maintainability Kroah-Hartman also highlighted the role of upstream development. The Linux community has long encouraged vendors to work directly on the mainline kernel rather than maintain private patches. That approach saves time, reduces long-term cost, and improves quality. Some vendors learned this the hard way. Others embraced it early and benefited from tighter collaboration with the community. Native Arm64 Testing Gives Kernel Developers Faster Feedback A major practical change for Kroah-Hartman came from using a native Arm64 build server from Ampere. Before that, he mostly tested on x86 and only discovered Arm64 issues later. Now he can build and test Arm64 kernels locally before sending patches out for review. That means fewer mistakes, faster feedback, and less wasted time for everyone involved. The value of that setup is simple: it matches the reality of modern development. Arm64 is no longer a side project. It is part of the core infrastructure of Linux. Native Arm64 tools help developers build better software for the platforms where Linux actually runs. For the Arm64 community, the lesson is direct. Mainstream status brings responsibility. It also brings leverage. The more Arm64 developers work upstream, test locally, and focus on reliability, the stronger the ecosystem becomes. View the full video here: To learn more about Ampere’s developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community. Check out the full Ampere article collection here.

By Craig Hardy

Why Distributed Databases Fail at Coordination Boundaries

Distributed databases are often evaluated through familiar technical dimensions: replication factor, consistency model, partitioning strategy, throughput, latency, and recovery time. These characteristics matter, but they do not fully explain why systems that appear healthy at the component level still experience severe production failures. In many cases, the storage engine is not the weakest part of the architecture. The failure occurs at a coordination boundary. A coordination boundary is any point where independently operating components must agree on timing, ownership, ordering, configuration, or state. These boundaries appear between replicas, partitions, control planes, data planes, load balancers, clients, metadata services, and background maintenance processes. Each component may behave correctly according to its local rules while the overall system produces an incorrect or unstable result. This is why distributed database incidents can be difficult to predict. The database may not fail because a server crashes or a disk becomes unavailable. It may fail because two healthy components temporarily disagree about who owns a partition, whether a node is available, or which version of configuration should be applied. Local Correctness Does Not Guarantee System Correctness Engineers naturally reason about software components individually. A node accepts requests, writes data, replicates changes, responds to health checks, and reports metrics. If each of those behaviors appears correct, the system is assumed to be healthy. Distributed systems challenge that assumption. A replica can be healthy but delayed. A coordinator can be available but operating with stale metadata. A load balancer can route traffic correctly according to its current configuration while that configuration no longer reflects the database topology. A client can retry a failed request according to policy while unintentionally amplifying load during a partial outage. Each component is locally correct. Their interaction is not. Consider a partition ownership transition. One node is being removed, replaced, or scaled down, and another node is taking responsibility for the affected data range. The outgoing node may believe it still owns the partition because it has not received the latest control-plane update. The incoming node may already begin accepting requests because it has received a newer version of the assignment. For a brief period, both nodes may behave correctly according to the information available to them. The system, however, has entered an ambiguous ownership state. That ambiguity can lead to duplicate processing, inconsistent writes, rejected requests, or unexpected latency. The problem does not exist entirely inside either node. It exists at the boundary where ownership information is exchanged and interpreted. Time Is Often the Hidden Coordination Dependency Many distributed database designs avoid relying on perfectly synchronized clocks. Even so, time remains embedded throughout the system. Timeouts determine when a request is considered failed. Leases determine how long a node retains authority. Heartbeats influence failure detection. Retry intervals shape traffic behavior. Expiration policies determine when data should disappear. Background processes decide when to compact, replicate, repair, or rebalance information. These mechanisms create coordination dependencies even when the architecture does not explicitly describe them that way. For example, a client sends a write request and does not receive a response before its timeout. The client cannot immediately know whether the write failed, succeeded, or is still being processed. It retries the request through another route. If the database supports idempotent request handling, the retry may be safe. If it does not, the same logical operation may be applied twice. The first server and the client both followed their expected behavior. The uncertainty appeared between them because completion and acknowledgment were separated by a network boundary. This is a common distributed systems pattern. A timeout provides information about waiting, not about the final outcome of an operation. Cloud architects should therefore treat every timeout as an ambiguity boundary. Timeout behavior must be designed together with idempotency, deduplication, retry limits, load shedding, and observability. Configuring a timeout without defining the system’s response to uncertainty simply moves the failure elsewhere. Metadata Can Become More Critical Than Data Database reliability discussions frequently focus on protecting stored records. Replication, backups, checksums, and repair mechanisms are designed to preserve data durability. However, the metadata that describes how data should be accessed can be just as important. Partition maps, routing tables, node membership, schema versions, configuration states, and feature capabilities determine how requests travel through the system. If this metadata becomes stale or inconsistent, the underlying data may remain fully intact while applications lose the ability to access it reliably. This is particularly important in systems that separate the control plane from the data plane. The control plane decides how infrastructure should be configured. The data plane processes live requests using that configuration. Separating these responsibilities improves scalability and operational isolation, but it introduces another coordination boundary. Configuration changes must move safely from the control plane to every affected data-plane component. During that transition, the system may contain multiple valid configuration versions at once. The engineering question is not merely whether a configuration update can be delivered. It is whether old and new versions can coexist without violating system correctness. Safe configuration rollout often requires versioning, backward compatibility, staged activation, and explicit rollback behavior. Without those protections, a harmless-looking control-plane update can produce a data-plane outage even when no database node has failed. Load Balancing Can Amplify Database Instability Load balancing is sometimes treated as an infrastructure layer outside the database itself. In practice, routing behavior directly influences distributed database reliability. When a node slows down, a load balancer may reduce traffic to it. That appears beneficial, but the remaining traffic must go somewhere. Healthy nodes receive additional load, their latency increases, and health checks may begin failing. The load balancer then removes more nodes, increasing pressure on the smaller remaining pool. This creates a feedback loop. The database causes routing changes, and the routing changes make the database less stable. Neither system is necessarily defective. The failure emerges from their interaction. Aggressive health checks, short timeout thresholds, synchronized retries, and immediate node removal can turn a minor performance issue into a broad outage. A more resilient design considers the rate of change, not only the current health signal. Cloud architects should ask whether routing decisions become less reliable during overload. They should also examine whether the database and load-balancing layers use compatible definitions of health. A node capable of serving read traffic may be temporarily unsuitable for writes. A node completing recovery may be reachable but not ready for production load. Binary healthy-or-unhealthy classifications often hide these operational differences. Background Work Creates Coordination Pressure Distributed databases perform significant work outside the direct request path. Replication, compaction, repair, rebalancing, expiration, backup, and cleanup processes compete for shared resources. These operations are often independently scheduled, which creates additional coordination boundaries. A compaction process may increase disk activity while a rebalance consumes network bandwidth. A repair job may begin during a traffic peak. Expired records may accumulate faster than cleanup processes can remove them. Each mechanism may operate within its configured limits, yet their combined effect can overwhelm the system. Time-to-live functionality provides a useful example. Expiring a record appears to be a simple data operation, but at scale it affects storage layout, indexing, replication, read behavior, and cleanup scheduling. The system must determine when an item is logically expired, when it should stop appearing in reads, and when its physical storage can be reclaimed. Those events may not occur simultaneously. If expiration processing is poorly coordinated, large groups of records can become eligible for deletion at the same time, creating bursts of background work. The feature itself works correctly, but the interaction between expiration timing and resource consumption can destabilize the database. The broader lesson is that operational features should be evaluated as distributed workflows, not isolated functions. Designing for Boundary Failures The most effective way to improve distributed database reliability is to identify coordination boundaries during architecture design. For every boundary, engineers should define what information crosses it, how that information is versioned, how long it remains valid, and what happens when delivery is delayed or duplicated. They should also determine whether the receiving component can safely operate with stale information. Observability should follow the same structure. Monitoring individual nodes is necessary, but it is not sufficient. Teams need visibility into ownership transitions, metadata propagation delays, retry amplification, routing changes, replication lag, and background-work queues. These signals reveal disagreement between components before that disagreement becomes a complete outage. Testing must also include transitional states. Steady-state benchmarks show how a system performs when ownership, routing, and configuration are stable. Production failures frequently occur while those conditions are changing. Architects should test node replacement, delayed configuration propagation, partial network loss, rolling upgrades, uneven clock behavior, repeated retries, overloaded background workers, and conflicting health signals. These scenarios expose the boundaries where local assumptions stop matching global reality. Reliability Lives Between Components Distributed databases rarely fail in the clean, isolated ways described by component diagrams. They fail through timing gaps, stale metadata, ambiguous ownership, retry storms, incompatible health decisions, and overlapping maintenance activity. The database node that appears responsible may only be the place where the problem becomes visible. For cloud architects and engineers, the practical shift is to stop treating coordination as an implementation detail. Coordination is part of the system’s correctness model. Storage engines protect data. Replication protects availability. Load balancing distributes work. Control planes manage change. None of these mechanisms can provide reliability independently. Reliability emerges from how they coordinate, especially when information is delayed, incomplete, duplicated, or temporarily inconsistent. That is where distributed databases are most likely to fail, and where architects should focus first.

By Varsha Ganesh

Zone-Aware Routing in Kubernetes: Reducing Latency, Improving Resilience, and Lowering Cloud Costs

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

By Mayowa Fajobi

Incident Management and the Rise of AI SRE Agents

Over the past year, I've been rebuilding parts of an incident response stack for a client, and the biggest surprise wasn't the AI features themselves. It was how much of the underlying workflow had to change to make those features useful. You can't just bolt an LLM onto a 2015-era ticketing tool and call it AIOps. The queue structure, the alert taxonomy, even the way runbooks are written all need to change. I've written before about the agent side of this shift, in AI Agent Architectures: Patterns, Applications, and Implementation Guide and Observability and DevTool Platforms for AI Agents. This two-part series is the other side of that coin: what happens when you point those same agent patterns at your own production systems instead of at somebody else's AI application. Same reasoning loop, different target. This first part covers incident management specifically, including a category I skipped in my earlier tool roundups: dedicated "AI SRE" agents like Traversal, Resolve.ai, and Cleric, which behave differently from the AIOps platforms most of us grew up with. Part 2 goes past incidents into ITOps, chaos engineering, SLO management, on-call toil, and the rest of what fills an SRE's week. A note on the numbers below: vendor-reported accuracy and MTTR figures in this space move fast and come from the vendors themselves. I've flagged those clearly rather than presenting them as independently verified benchmarks. Where SRE Pain Actually Lives Before getting into tools, it helps to remember what SREs spend their time on. Most postmortems I read over the years have the same three complaints: Too many alerts, not enough signalCorrelating five different dashboards to find one root causeWriting the same postmortem summary for the fourth time this quarter None of these are new problems. What's new is that large language models are actually decent at the second and third ones, if you feed them clean data, and a newer crop of agents is starting to chip away at the first one too. The Traditional Incident Pipeline Here's roughly what an incident used to look like before AI got involved, at most mid-size shops I've worked with: Traditional incident pipeline Every arrow in that diagram is a human doing manual correlation work. That's fine when you have ten services. It falls apart at three hundred, and it's part of why I keep coming back to the point I made in Infrastructure as Code: How Automation Evolved to Power AI Workloads: scale problems in ops rarely get solved by hiring more people to stare at more dashboards. Where AI Fits Into the Pipeline Today The shift isn't "AI replaces the engineer." It's AI collapsing steps B through F into something closer to a single triage step, with the engineer reviewing a proposed root cause instead of hunting for one from scratch. AI-aided pipeline Notice the engineer never disappears from this diagram. They just move from being the one who does the correlation to the one who checks the correlation. That distinction matters, because it changes what you hire and train for. I made a version of this same argument about production-grade agents generally in the Shipping Production-Grade AI Agents refcard: an agent needs a human review layer, or you're just moving risk around instead of removing it. If you want a deeper look at how these review loops are actually structured under the hood, I broke that down recently in Loop Engineering: The Layer After Prompt, Context, and Harness Engineering. Incident Management: What Changed A few concrete things have improved in incident management tools over the last two years: Alert correlation got better. Tools like BigPanda, Moogsoft, and PagerDuty's AIOps features now cluster related alerts using pattern recognition instead of static rules. A database timeout, three downstream service errors, and a spike in 500s used to show up as four separate pages. Good correlation engines now group them as one incident with a suggested cause. Similar-incident retrieval works reasonably well. If your org has a decent history of past incidents with clean postmortems, tools can now surface "this looks like INC-4471 from March" with real accuracy. This only works if your postmortem data isn't garbage, which is a bigger blocker than people admit. Draft postmortems save real time. Not because the AI writes a good postmortem on the first try, but because staring at a blank page is the slowest part of writing one. A rough draft built from the incident timeline, Slack thread, and metrics gives engineers something to edit rather than create. What's newer, and worth its own section, is a class of tools that don't just correlate what you already collected. They go get new evidence during the incident, the way a senior engineer would. The New Category: Dedicated AI SRE Agents This is the part of the landscape that's moved fastest since I last wrote about agent tooling. A handful of startups have built agents whose entire job is investigating production incidents autonomously, not just clustering alerts that already exist. Traversal leans on causal machine learning rather than a general-purpose LLM wrapper. Instead of pattern-matching against similar past incidents, it builds a model of causal dependencies across your services and traces the actual chain of cause and effect, down to the specific deploy or config change that started the failure. It reports strong root-cause accuracy in production at large enterprises and is used for both alert triage and live incident investigation. The pitch is narrower than "full AIOps platform," and that narrowness is the point. Resolve.ai takes a broader angle. It was built by the team that created OpenTelemetry, and it positions itself as an agentic teammate across the whole production lifecycle: investigating incidents, but also touching capacity questions, config drift, and guided code changes. Where Traversal is a specialist in root cause, Resolve.ai is closer to a generalist you'd loop in on almost anything production-related, with the incident work as the anchor use case. Cleric sits in a similar space to both, with a specific focus on autonomous alert triage. It runs a multi-source investigation the moment an alert fires, pulling metrics, logs, traces, and deploy history in parallel, and returns an evidence-backed hypothesis before an on-call engineer has finished opening their second dashboard tab. It runs with read-only access by default, which matters a lot for teams still building trust in the category, and it was named a Gartner Cool Vendor in AI for SRE and Observability in 2025. That "read-only by default" design decision is exactly the kind of guardrail I argued for in Trust No Agent: How to Secure Autonomous Tools on Your Machine: an agent's blast radius should be a deliberate design choice, not an afterthought. Causely and NeuBird round out the space with slightly different angles: Causely focuses on causal reasoning to find the single root cause behind a storm of cascading alerts, and NeuBird targets enterprise IT environments with LLM-driven telemetry analysis at large scale. Here's roughly where an AI SRE agent sits in the pipeline compared to the AIOps correlation tools from the last section: AI SRE agent pipeline The key difference from the earlier diagram: this agent isn't just correlating signals you already collected in a dashboard. It's actively going out and querying your systems the way a human on-call engineer would, forming a hypothesis, testing it, and either confirming or discarding it before it ever pages a person. That's a meaningfully different capability than clustering alerts by similarity, and it's why this category gets its own row in any serious comparison. If you're weighing whether to build this kind of investigation loop yourself versus buying one of these platforms, it's worth reading MCP vs Skills vs Agents With Scripts: Which One Should You Pick? first, since the architecture decision behind "agent that calls tools live" versus "agent with a fixed skill set" applies just as much to SRE tooling as it does anywhere else. What This Actually Brings to the SRE Persona It's worth being specific about what changes for the person on-call, not just what the vendor deck claims: Fewer 2 a.m. investigations that start from zero. The agent has usually already ruled out the obvious suspects by the time a human looks at the page, so the engineer starts from a hypothesis instead of a blank terminal.Less tool-hopping. A lot of incident time isn't spent thinking; it's spent switching between Datadog, Grafana, the CI pipeline, and Slack. An agent that queries all of them in parallel removes a genuinely tedious chunk of the job.A written trail for free. Because the agent's investigation is itself a structured log of what it checked and why, you get a decent postmortem skeleton as a byproduct, not a separate task.A new failure mode to watch for. Engineers can start trusting the proposed root cause without checking the evidence trail, especially under pager pressure. That's a habit worth actively training against, not assuming away. None of this replaces the on-call engineer's judgment. It changes the shape of their shift from "gather evidence, then decide" to "review evidence, then decide," which is faster but only as trustworthy as the evidence the agent actually gathered. Comparing the Tool Landscape Here's how some of the major players stack up on where they've actually invested in AI, versus where it's mostly a checkbox feature. I've split this into two tables, because lumping AIOps correlation platforms in with dedicated AI SRE agents hides a real difference in what these tools do. Established AIOps and incident platforms: ToolAlert CorrelationRoot Cause SuggestionAuto-Drafted PostmortemsPredictive CapacityOwnership / StatusPagerDuty (AIOps)StrongModerateYesLimitedIndependent, public companyMoogsoftStrongStrongNoNoAcquired by Dell Technologies (2023)BigPandaStrongModerateLimitedNoIndependent, privateDatadog (Bits AI)ModerateStrongYesModerateBuilt in-house by DatadogServiceNow (Now Assist)ModerateModerateYesStrong (ITOps)Built in-house by ServiceNowDynatrace (Davis AI)StrongStrongLimitedStrongBuilt in-house by Dynatraceincident.ioModerateLimitedYesNoIndependent, privateRootlyModerateLimitedYesNoIndependent, private Dedicated AI SRE agents: ToolCore ApproachActs Autonomously?Best FitFounded / BackingTraversalCausal ML across dependency graphInvestigation autonomous, remediation guardedTeams with strong existing observability wanting sharper RCA2023, Sequoia and Kleiner PerkinsResolve.aiBroad agentic reasoning over code, infra, telemetryInvestigation autonomous, remediation opt-inTeams wanting one agent across incidents, capacity, and config2024, Greylock-led seedClericMulti-source parallel investigationRead-only by defaultTeams new to AI SRE agents, wary of write access2024, Zetta Venture PartnersCauselyCausal reasoning on cascading alertsInvestigation onlyEnvironments with alert storms and unclear blast radiusPrivate, early stageNeuBirdLLM-driven telemetry analysis at scaleInvestigation, guided remediationLarge enterprise IT environmentsPrivate, early stage A caveat worth stating plainly: I haven't run rigorous side-by-side benchmarks on all of these, and vendor claims move faster than reality, especially in the AI SRE agent table where most of these companies are one to three years old and evolving month to month. Treat both tables as a directional map, not a scorecard, and validate against your own alert volume before picking one. Deterministic AI vs. Generative AI in These Tools This distinction gets muddled in vendor marketing, so it's worth separating clearly. AspectDeterministic / ML-based (older AIOps)Generative AI (LLM-based, newer)ApproachStatistical pattern matching, clustering, anomaly detectionLanguage model reasoning over logs, tickets, chat history, and live queriesPredictabilityHigh, same input gives same outputLower, outputs can vary between runsStrengthCorrelation, anomaly detection at scaleSummarization, hypothesis generation, natural language explanation, draftingWeaknessPoor at explaining "why" in plain languageCan hallucinate a plausible-sounding but wrong root causeWhere it shows upDynatrace Davis AI, Moogsoft's original correlation engineDatadog Bits AI, ServiceNow Now Assist, Traversal, Resolve.ai, ClericTrust level neededCan often auto-remediateNeeds human review before action Most modern platforms now run both in tandem: the deterministic layer does the anomaly detection and correlation, and the generative layer explains it in plain English, forms hypotheses, and drafts the writeup. That combination is doing more real work than either piece alone, and it's basically the same pattern I described for agent observability generally in the AI agent architectures piece linked earlier: a fast, boring, reliable layer underneath a slower, flexible reasoning layer on top. Where We're Headed in Part 2 Incident response gets the spotlight because it's the loudest part of the job, but if you track where an SRE's actual week goes, a lot of it isn't firefighting at all. It's chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. In Part 2, I'll walk through where AI is showing up in ITOps specifically, and then go further into chaos engineering, SLO and error budget management, on-call toil reduction, and capacity planning, the quieter parts of the job that determine whether the incident tools in this article even have a fighting chance.

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

How We Built an LLM Pipeline That Survives Traffic Spikes

We built an LLM pipeline to help a large network operations team stay on top of trouble tickets. It ran quietly in production until the moment it was supposed to earn its keep. In early 2026, a major winter storm swept across a wide region and knocked out power to more than a million people; network equipment failed in bulk, tickets poured in, and the summarizer meant to help engineers triage the chaos went dark. The root cause was not a bug in the usual sense. There was no null pointer and no bad deploy. We hit the Azure OpenAI tokens-per-minute (TPM) limit, our retries made it worse, and we had no fallback. This is the anatomy of that failure, and the architecture we built afterward to treat an LLM like the rate-limited, non-deterministic dependency it actually is. The uncomfortable theme up front: our system was busiest during precisely the event it existed to handle. Demand and failure were correlated. If you put an LLM in front of any incident-driven workload, this will eventually be your story too. What the System Did Tickets in this environment originate from many channels, including network alarms, customer calls, emails, and proactive checks by operations staff. But by the time our pipeline sees them, they are already incidents and cases in ServiceNow. Our scope starts there. ServiceNow streams ticket events out of the box through Stream Connect into Kafka. Our application, running in Azure and orchestrated with LangGraph, consumes those events, retrieves related context from Azure AI Search, and calls the Azure OpenAI API to produce three kinds of summary: Status notifications for the customers affected by an outage,Ticket summaries for the technicians actively working a ticket, andExecutive summaries that roll up what is happening across a region. The value is simple. Ticket logs are long, noisy, and full of machine-generated entries. A technician picking up a ticket, or a manager gauging the blast radius of an outage, does not want to read pages of log. They want five sentences. The LLM gave them five sentences, and on a normal day it sat comfortably within quota. The Failure Timeline Then the storm hit. Equipment failed in bulk. The storm drove power outages past a million customers across a wide region, and our network equipment failed along with the grid. The alarm systems did exactly what they were designed to do: they fired, in volume.Tickets surged. They grew to roughly six times our baseline.The token load surged far faster. This is what caught us. Our load is not measured in requests; it is measured in tokens. Storm tickets did not just arrive more often, each carried a longer log (more alarms, more correlated events). So, a ~6× jump in tickets became closer to a ~15× jump in tokens per minute.We hit the TPM ceiling. Azure OpenAI began returning 429 Too Many Requests with a Retry-After header.Retries deepened the throttle. Every layer that could retry, did. That included the SDK, our wrapper, and LangGraph nodes re-running on failure, all in near-unison, with no jitter. Each retry wave slammed the limit together and pushed our effective token rate higherwhile we were already over budget. And because every retry of a generation is another paid, token-billed call, the retries spent the very budget we had blown..There was no fallback. When retires were exhausted, there was nowhere to go. There was no cheaper model and no degraded path. Summarization simply stopped The cascade. Customer status notifications stalled, technicians lost the ticket summaries they rely on, and executive summaries went stale. So, the team fell back to reading raw logs by hand. Summarization stayed degraded, on and off, for a multi-hour stretch, until we manually provisioned extra capacity and hand-routed traffic to other models to limp through the worst of it. The shape of the overload, with illustrative numbers to make the dynamic concrete: Metric Normal day Storm Summaries per minute ~40 ~240 (≈6×) Tokens per summary (log + context + output) ~3,300 ~8,000 (longer logs) Token demand ~132K TPM ~1.9M TPM Token quota ~250K TPM ~250K TPM Result ~53% utilization ~7.7× over → sustained 429s (The figures are illustrative estimates that preserve the real proportions, not exact production measurements.) The punchline is in the third row: a ~6× rise in tickets became a ~15× rise in tokens. That is the trap of a token-metered dependency, and the rest of this article is what it taught us. How the original outage cascaded — and why naive retries made it worse. Root Cause: An LLM is a Token-Metered Dependency, Not a Request-Metered One Most writing on resilience, including circuit breakers, retries, and bulkheads, is framed around microservices, and most of it applies here. But an LLM API breaks a few assumptions those patterns quietly rely on, and each broken assumption showed up in our incident. 1. The limit is tokens, not requests. Classic rate-limit thinking counts calls; Azure OpenAI quota is measured in tokens per minute. Your load therefore depends on the size of your inputs — and for a summarizer that is the worst possible coupling: it burns the most quota exactly when documents are longest, which during an incident is exactly when logs are longest. A request-rate dashboard would have looked merely elevated while our token rate was off the chart. 2. Retries spend the budget you are already over. On a normal REST API, a retry is cheap. On a token-metered, pay-per-token backend, every retried generation is another full charge against the limit you just exceeded. Naive retries do not just fail to help. They actively deepen the throttle. 3. Synchronized retries are a self-inflicted DDoS. With no jitter, failed calls backed off by the same amount and returned together, re-tripping the limit on a clock. It is the classic retry storm, amplified by point #2 because each retry is token-expensive. 4. No fallback means peak demand is a single point of failure. One model, one deployment, one path is fine until that path is throttled, and it will be throttled at peak. 5. Demand correlates with failure. A summarizer for incident tickets is, by definition, busiest during incidents. The load spike and the operational emergency are the same event. Capacity planned for the average is capacity planned for the calm before the thing you actually built the system for. The Fix: Classify, Route by Severity, and Govern the Token Budget The redesign treats the LLM as a scarce, metered resource and spends it deliberately, turning the frantic, manual capacity-adding and model-rerouting we did by hand during the storm into a permanent, automatic capability. Schedule in Redis, not Kafka. Our Kafka topics are shared by many interfaces and kept generic, so we could not repurpose them for prioritization. Instead, our consumer reads the generic stream and pushes work into Redis priority queues, where all the scheduling logic lives. Kafka stays the durable ingestion layer — a natural backpressure buffer, so a storm surge piles up safely in the log instead of hammering the model, and consumer lag becomes our early-warning storm metric. Classify with a tiny model. A small, local ML classifier scores each ticket by priority, severity (P1–P5), and customer impact, using fields already on the ServiceNow ticket. It is deliberately not an LLM call: during a storm every Azure OpenAI token is contested, so spending premium tokens just to decide how to spend premium tokens is exactly backwards. When the classifier is unsure, it routes up, because under-serving a real P1 is far worse than over-spending on a P4. Route by severity to isolated capacity. Each tier gets the cheapest treatment that still meets its need: Severity Routes to Why P1 / P2 Premium model deployment (related incidents coalesced into one regional rollup) High stakes, exec-facing; worth the tokens P3 / P4 Separate, cheaper model deployment "Good enough" at a fraction of the tokens P5 Non-LLM extractive summary (error counts, key fields, first/last events) Zero tokens; also a universal degraded mode The key trick is that the cheaper tier is a different model, so it draws from a different Azure OpenAI quota pool — a flood of low-severity tickets cannot cannibalize the premium tier's TPM. This needs no provisioned throughput; two standard deployments on different models give you quota isolation for free. Govern the token rate. A shared, Redis-backed token budget gates every LLM call: we estimate a job's tokens before dispatch and only proceed if the rolling per-minute budget allows, per deployment. Retries use bounded exponential backoff with jitter and honor Retry-After; the first worker to see a 429 sets a global cooldown the whole fleet respects, so the retry storm cannot form. Low-priority queues age and get promoted so they are never starved, and at-least-once delivery is made safe with idempotency keyed on ticket plus log version. Put together, the request path becomes: ServiceNow → Stream Connect → Kafka → classifier → Redis priority queue → token governor → the right model (or extractive fallback). The queue absorbs the spike, the governor respects the ceiling, and severity routing decides who gets the scarce premium tokens when there are not enough to go around. The redesigned pipeline: tickets are classified by severity, scheduled through Redis with a token governor, and routed to isolated model tiers. What We Expect (By Design) With this in place, the same storm should behave very differently. The 429 cascade cannot recur by construction. The governor caps dispatch at quota, so overflow becomes bounded queue lag — low-priority summaries delayed by minutes — rather than total failure.Premium capacity is protected. Routing roughly the top 15% of tickets to the premium tier and coalescing related incidents keeps it within quota even under the surge.Cost falls. Moving the bulk of volume to a cheaper model and the long tail to zero-token extraction projects on the order of a 50–65% blended token-cost reduction. Takeaways Plan capacity in tokens, not requests: Your load is driven by input size, which spikes exactly when you can least afford it.Design for the spike, not the average: Assume demand correlates with failure.Make retries jittered, bounded, and 'Retry-After'-aware: Remember each retry costs tokens.Tier your models by importance: Put cheap or non-LLM paths under the long tail, and isolate premium capacity on its own quota pool.Always keep a degraded mode: A rough summary delivered beats a perfect one that never arrives.

By Dileep Mundakkapatta

Performance

DZone's Featured Performance Resources

Monthly Top Performance Experts

The Latest Performance Topics