Culture and Methodologies

I've spent the better part of two years watching teams ship agentic AI systems, and a pattern keeps repeating. Two engineers read the same LangChain docs, attend the same conference talks, and build systems that look identical in a demo. Six months later, one system is handling thousands of requests a day with predictable behavior. The other gets quietly replaced by a simpler rules engine after it embarrassed someone in front of a customer. The gap between those two outcomes has almost nothing to do with model choice or framework familiarity. It comes down to a small set of skills that don't show up on most job postings for AI engineers, and that most online courses skip entirely. What Makes an Agentic AI System Different From a Chatbot Wrapper A chatbot wrapper takes input, sends it to a model, and returns the output. An agentic system makes decisions across multiple steps, calls tools, holds state, and sometimes calls itself. That difference sounds small written down. In practice, it changes everything about how the system fails. A wrapper that gives a bad answer wastes one turn. An agent that makes a bad decision at step two can compound that mistake across steps three through fifteen, calling the wrong API, writing bad data to a database, or looping on a task it can't complete. The failure modes are different in kind, not just in severity, and engineers who haven't built agentic systems before tend to debug them like they would debug a single bad response. They look at the final output instead of the decision trail that produced it. Skill One: Building Evaluation Before Building Features Most teams build the agent first and figure out how to test it later. The engineers who ship reliable systems do the reverse. Before writing the orchestration logic, they write a set of test cases the agent has to pass, with clear pass and fail criteria, and they run those cases against every change to the prompt, the tool definitions, or the model version. This sounds obvious stated plainly. It's rare in practice because agentic systems resist the testing patterns engineers already know. A unit test checks one function against one expected output. An agent's output depends on the conversation history, the tools available at that moment, and the specific phrasing of the user's request, so a single test case doesn't generalize the way a unit test does. Engineers who handle this well build small evaluation harnesses early, often before the agent does anything useful. They run twenty or thirty scenarios that represent the range of things the agent will see in production, including edge cases that look like they shouldn't happen. Then they track pass rate as a number they watch the same way they'd watch latency or error rate. When someone tweaks a system prompt to fix one issue, the harness catches the three other things that broke as a side effect. I've watched a team skip this step on a customer support agent, ship it, and discover three weeks later that a prompt change meant to improve tone had quietly disabled the agent's ability to escalate billing disputes to a human. Nobody caught it because nobody was running scenarios that exercised that path. A harness with even ten well-chosen test cases would have flagged it the same day. Skill Two: Treating Tool Definitions as an API Design Problem The tools an agent calls function as its only way of acting on the world, and most engineers write tool definitions the way they'd write internal function signatures: quick names, minimal descriptions, parameters that make sense to the person who wrote the code. That approach breaks down because the agent reads the tool description the same way it reads everything else, as natural language it has to interpret. A tool called search with the description "searches things" gives the model almost nothing to work with when it's deciding whether to call that tool or a different one, or what to pass as the query. Engineers who get this right write tool descriptions the way a technical writer would write public API documentation. They specify exactly when the tool should be used, what it returns, and what it doesn't do. They name parameters so the intent is obvious without a comment. A tool called search_customer_orders_by_email with a description stating it returns orders from the last 90 days and requires a verified email address gives the model far less room to misuse it than a generic search function does. This matters more as the number of available tools grows. An agent choosing between three tools can often guess right even with weak descriptions. An agent choosing between twenty tools, several of which sound similar, needs descriptions precise enough to disambiguate. Teams that scale past a handful of tools without revisiting this usually see a spike in wrong-tool-selected errors, and the fix is rarely a smarter model. It's better documentation. Skill Three: Designing for Partial Failure Traditional software either works or throws an exception. Agentic systems fail in a third way: the call succeeds, the response looks reasonable, and the content is wrong or incomplete. A tool call to fetch inventory data might return successfully while returning stale numbers. The model might decide a task is complete when it's only handled part of it. Engineers who've shipped production agents build explicit checkpoints into the flow where the system verifies its own progress against the actual goal, not just against whether the last API call returned a 200 status code. This might mean a verification step after a multi-stage task, where a separate prompt checks the agent's claimed output against the original request. It might mean structured outputs at each step that a deterministic function can validate, rather than trusting free text all the way through. The instinct to add more error handling here is correct, but the specific shape matters. Wrapping every tool call in a try-except block catches crashes. It doesn't catch an agent that confidently reports success on a task it didn't finish. That requires building verification logic that understands the task, not just the mechanics of the call. Skill Four: Knowing When Agentic Architecture Is the Wrong Choice The most consistent marker I've found for engineers who build agentic AI systems well is a willingness to argue against using one. 2026 has pushed agentic AI into the default answer for almost any automation problem, and that default is wrong often enough to matter. A task with a fixed sequence of steps and no real decision points doesn't need an agent reasoning through it each time. A deterministic pipeline runs faster, costs less, and fails in predictable ways that are easier to debug at 2 a.m. The engineers I'd trust with a production system are the ones who can look at a proposed agentic workflow and say plainly that a simpler architecture handles 90% of the cases just as well, reserving the agent for the genuine judgment calls. This isn't a popular position to take in planning meetings right now, with enterprise adoption of agentic AI accelerating across every sector and budget approval often tied to whether a project sounds sufficiently advanced. But the systems that hold up under real traffic tend to be the ones where someone pushed back on scope early, kept the agentic part narrow, and let boring code handle everything that didn't need a model making decisions. What This Looks Like Six Months In None of these four skills show up in a typical technical interview. They show up in incident reviews, in the difference between a system that degrades gracefully and one that fails in ways nobody anticipated, and in whether an engineer can explain why their agent made a specific decision three steps into a failed task. The teams shipping agentic systems that survive contact with real users aren't the ones with the most sophisticated prompts or the newest framework. They're the ones who treated evaluation as infrastructure, wrote tool descriptions like public documentation, built verification into the architecture instead of bolting it on after an incident, and stayed honest about when an agent was the wrong tool for the job. That combination is harder to hire for than "experience with LangChain" or "familiarity with RAG pipelines." It's also the actual difference between a demo and a system someone can depend on.

Once upon a time, site reliability engineering rested on a linear assumption: monitor more, detect early, and you’ll recover faster. The rise of alert fatigue makes modern SRE teams realize otherwise: Ramadass's (2025) paper, Building an AI-Powered Observability Pipeline for Modern System Reliability, cited research that discovered that: More than two-thirds (82%, actually) of institutions experience alert spikes constantly.Most traditional monitoring tools generate approximately 2,100 alerts daily, with about 70% of them unnecessary and safe to ignore.66% of SRE professionals stated that increased false alerts lead to fatigue, potentially causing them to miss serious issues. How Should We Describe This Situation? Vigilance or Noise? Collaborative systems such as SaaS, third-party APIs, and microservices enhance the degree of observability and notification within systems. Everything is monitored, and occasionally these dependencies may duplicate alerts. When systems request superhuman attention, on-call engineers become fatigued rather than lazy or sloppy. Instead of swift action, alerts are responded to with mistrust. Reliability vs. Experience vs. Metrics Traditional alerting metrics follow traditional reliability practices, that is, error rates, uptime percentages, latency, etc. Although these are essential, they are not actual mirrors of how operators or users experience reliability. Operators may expect reliable alerting to inform decisions, while users may simply define reliability as how well a system enables them to fulfill their intentions. If alerts do not clearly connect to the user experience, there is a gap between detection and action. Over time, the gaps lead to fatigue. On-call engineers begin to “reasonably” ignore these alerts. Why worry over alerts that are not logically related to user outcomes? They may assume. Over time, organizations may end up paying dearly for real issues because alerts were missed or delayed. An On-Call Engineer Experience Here is a typical example of a system design problem an on-call engineer or SRE team may face: 01:15 AM Alert: Latency spikes on a third-party API.01:16 AM Alert: Retry queues are filled.01:16 AM Alert: Timeout alert storms on three dependencies.01:17 AM Alert: Error-rate notification on unrelated endpoints.01:18 AM Alert: Memory and update alerts. And this sequence of alert storms continues, with the on-call engineer receiving more than 20 alerts in just four minutes. The system seems to pass standard observability SRE practice. But what about the long-run reliability suspicions that the bugging signals may create? In this case, the teams are not just grappling with response speed but also with the amplification of confusion when critical alerts are mixed with non-actionable ones. When Detection Outpaces Interpretation We can’t rule out the fact that monitoring in the past decades has taken an advanced leap. And we might be at its cloying stage, where system detection software is outpacing on-call engineers’ interpretation. Systems are “wonder-full” when it comes to identifying when something seems “off.” However, they rarely give explicit descriptions to aid SRE teams’ understanding. An alert can indicate that a queue has exceeded its depth, but may not categorically state whether the issue is temporary or actionable, or whether users are affected. This occurrence spans dozens of dependencies, each with its own signal. The on-call engineer is kept puzzled about the best action to take at the right time. Hence, a reliable response could be excessive caution or delay as the engineer seeks to clarify the situation. The users are negatively impacted. Although the system met technical observability SRE standards, it failed operationally due to its opacity. The Hidden Cost of Alert Overload We rarely see the outcome of alert fatigue overnight. Its effects build up. Delayed response time accumulates. The aftermath incident review loses credibility. Engineers are skeptical of alerts and hesitate to decide first whether they are real or false. The cultural cost of alert fatigue is that on-call roles become a burden SRE teams endure rather than enjoy with a sense of responsibility. In the long run, engineers may feel they have no control over issues due to the confusion that multiple alerts create. Ironically, the same reliability problems that alerts were designed to solve are what they quietly create. Are Alerts Creating a False Sense of Safety? Lots of alerts may seem like a good thing or a sign of strong monitoring at first glance. But here is the truth: alerts could be hiding actual risk. As every deviation is notified, critical and minor alerts blend in. Teams begin to feel alert fatigue and delay response. Then, real problems begin to breed behind the scenes. Remember how SLAs could paint an illusory picture of safety? Similarly, alert volume could do so. Therefore, your SRE team should bind these caveats as the core of their modus operandi. Alerts shouldn’t replace action.Alerts shouldn’t be unsorted (by machines or humans).Alerts shouldn't be discarded. Alerts are signs that our systems need attention, and we should never be tired of listening. SRE Teams Designing Systems that Alert Smartly High-quality systems respond efficiently when dependencies fail. Instead of creating panic, they automatically degrade. SRE teams could design circuit breakers that could inhibit alert storms before they explode. They could also install bulkheads to prevent a single failure from spreading. There could be alert limits and a summary of conditions that resolve the problem of spamming. Instead of relying on metrics, system engineers could set up composite alerts that describe system states. For instance, it’s clearer if a system alert indicates, “Checkout degraded because of latency in payment dependency.” This composite alert is better than 7 alerts that say “Checkout Timeout.” The former shows impact, cause, scope, and urgency. Clarity clears fatigue. Noise does the opposite. Redesigning SRE: Human Reliability That Quells Alert Fatigue We have seen that technical designs may be great, yet other aspects of SRE remain wanting. One such area that could resolve a system design problem is humaneness. To avoid alert fatigue, our design choices must acknowledge human limitations. Therefore, we should accept that some alerts may not require immediate response. Conversely, not every anomaly should trigger an alarm. Understood silence could sometimes be a golden sign that nothing critical is wrong. Advanced SRE teams do not focus on events (or every deviation) but on the states of the system or infrastructure. They are guided by the question: What conditions really impact users, business objectives, or the system's overall health? To achieve this, engineers need to balance product understanding with technical operations. Then they can give a human touch to their designs. Designing systems for human reliability requires a high level of discipline. Site reliability engineers have to continually review, refine, and repair alerts and their trigger commands. Systems are like living organisms that need constant feeding of updates. The evolving nature of alerts could make a helpful one-time alert redundant or harmful in six months. On-Call as a Reliability Interface of SRE No doubt, humans have a role to play in ensuring reliability, but system designs that depend on heroic actions are built not with resilience but with fragility. Reliability is truly achieved when on-call engineers are guided by predefined scripts, models, runbooks, signals, and interfaces. These reduce the tendency to resort to fallible improvisations when issues arise. On-call engineers often take the appellation of “last point of call.” A careful look at their roles shows that they are intermediaries among complex systems, user experience, and consequences. We can thus see that the role of on-call engineers extends beyond problem resolution to stewardship. Conclusion Alert fatigue is a design problem. It often arises when on-call engineers prioritize detection over interpretation, or technical workability over user experience. The dependencies of modern SRE teams make it necessary to align technical alerts with human capability. Alert storms could wear out hardworking engineers who need to take a break. So, system designs need to account for human limitations, recognize that runbooks are better than on-the-spot improvisation, and prioritize clarity over opacity. Designs that account for these factors reduce or eliminate fatigue and preserve the very essence of alerts. In summary, reliability goes beyond resolving many problems to responding to what matters most. When teams can always trust their alerts, they will be more likely to follow up on new cases.

Many assume that leadership in software engineering starts only when you stop coding and become a manager. I once shared this belief, thinking technology would be simpler than working with people. That was an early misconception. While it is possible to focus your career on code, architecture, databases, and other technical areas, the challenges that shape your impact become less technical over time. Even the best architectural decision has little value if others do not trust, understand, or support it. This does not mean every experienced software engineer should become a manager. Leadership is equally important on the technical track. Senior individual contributors, such as staff engineers, architects, and principal engineers, are expected to influence decisions beyond their own code. As Will Larson discusses in Staff Engineer, advancing beyond senior engineering focuses on technical leadership rather than people management. To increase your technical impact, others must listen to your ideas, trust your judgment, include you in key discussions, and act on your recommendations. You may choose not to manage people, but avoiding leadership will eventually limit your growth as a software engineer. What Do We Mean by Leadership? Leadership predates corporations, job titles, and management frameworks. For example, the Roman military’s success relied not only on superior weapons or armor but also on effective organization. Legions were divided into smaller units, each with defined responsibilities and led by centurions. Leadership was distributed throughout the ranks, enabling coordinated efforts toward larger objectives. A similar concept appears in the term architect, commonly used by software engineers. Derived from the Greek arkhitekton — arkhi meaning chief and tekton meaning builder — an architect was the master builder, responsible for both understanding the craft and directing others. This role closely resembles that of an effective software architect today. Another example comes from the nineteenth-century Prussian military, which developed its General Staff as a professional body focused on planning, coordination, and operational readiness. This approach recognized that complex organizations require skilled individuals to address critical challenges without making each one the commander. The model became influential and was adopted by other militaries. Software engineering faced a similar challenge: how can experienced engineers expand their organizational impact without moving into people management? An early solution appeared in the British Royal Navy, where managing large fleets required separating command authority from technical expertise. Naval operations relied on both captains and skilled officers responsible for navigation, planning, logistics, and coordination. This staff function supported fleet-level decision-making without direct command and influenced how organizations approach distributed expertise and coordination. Modern software organizations independently adopted a similar approach. Titles such as Staff Engineer, Principal Engineer, and Distinguished Engineer now represent technical leadership roles. Will Larson highlights this distinction in his 2021 book, Staff Engineer: Leadership Beyond the Management Track, which explores Staff-plus engineering as leadership outside the traditional management ladder. This distinction is essential: management is a role, while leadership is an activity. Managers have formal responsibilities for people, performance, hiring, priorities, and processes. Technical leaders may lack formal authority, but their influence comes from expertise, judgment, communication, trust, and guiding better technical decisions. In software engineering, leadership does not require direct reports; it often means being the trusted engineer who provides direction in complex situations. Why Should a Software Engineer Care About Leadership? While understanding leadership is valuable, it is even more important to consider why a Software Engineer who does not plan to become a manager should invest time in developing these skills. As your career advances, your impact depends not only on your technical skills but also on your ability to influence decisions, collaborate effectively, and guide the organization toward better technical outcomes. 1. Software Development Is About People Software development is inherently a social activity. Software is built collaboratively with engineers, product managers, designers, architects, clients, managers, and other stakeholders. Even highly technical decisions must be explained, discussed, challenged, negotiated, or approved by others. A database migration may seem purely technical until it impacts another team. An architectural decision becomes a communication challenge when multiple teams must adopt it. Even an elegant solution can fail if it does not address the client’s real needs. While much of your day may involve working with machines, software exists for people, is created by people, and ultimately serves people. Choosing not to pursue management does not eliminate the human aspect of software engineering. 2. Technical Expertise Needs Trust, Influence, and Access Technical correctness alone is not sufficient. You might understand why a particular architecture will not scale, recognize that a technology introduces unnecessary complexity, identify an important security risk, or propose a significantly better design. However, your expertise has limited impact if others do not listen. Technical expertise leads to organizational impact only when you can influence the organization’s actions. Influence rarely stems from expertise alone. People must trust your judgment, see that you understand the context, listen to opposing views, explain trade-offs clearly, and adapt your position when evidence changes. This is also why relationships matter in a technical career. As your responsibilities increase, many key decisions occur outside the codebase, such as during architecture reviews, design discussions, planning sessions, incident reviews, roadmap meetings, and cross-team or stakeholder meetings. If you want to influence those decisions, you need to be part of those conversations. Leadership helps you build the credibility and trust needed to participate in these discussions and ensures your voice is heard. A strong technical leader does more than provide correct answers. They create conditions where good technical decisions can be understood, challenged, accepted, and implemented. 3. If You Do Not Lead, Someone Else Will Make the Decision When experienced engineers avoid leadership, an uncomfortable consequence arises. The decisions do not disappear. Someone else will make them. And that person may have considerably less technical understanding of the consequences. This can lead organizations to measure engineering productivity using questionable proxies such as lines of code, number of commits, tickets closed, or tokens consumed by AI tools. When engineers encounter such decisions, their natural reaction is often: “Who thought this was a good idea?” A better question might occasionally be: “Which experienced engineers were involved when this decision was made?” Leadership ensures that technical knowledge is represented in decisions affecting engineering. You do not need to control every decision, but you should be willing to participate in the important ones. 4. Leadership Multiplies Your Technical Impact There is a natural limit to how much software one person can build individually. Even exceptional engineers have limited time each day. Leadership enables your expertise to extend beyond those limits. By helping others make better design decisions, establishing reusable architectural approaches, mentoring engineers, improving practices, or preventing costly mistakes, your impact exceeds your individual contributions. This marks an important transition in senior technical careers. Early in your career, your value is largely based on your individual contributions. Later, your value increasingly comes from enabling other engineers and teams to succeed. Your code remains important, but it is no longer the sole measure of your contribution. 5. Leadership Becomes Part of Technical Career Progression Leadership becomes increasingly integral to career progression beyond the Senior Software Engineer role. Staff Engineers, Principal Engineers, Distinguished Engineers, and Software Architects may remain Individual Contributors, yet their responsibilities typically extend beyond implementing individual features. They are expected to provide technical direction, navigate ambiguity, resolve difficult trade-offs, connect teams, mentor engineers, challenge assumptions, and influence decisions whose consequences may extend across an organization. None of those responsibilities inherently requires becoming a people manager. But almost all of them require leadership. Treating leadership as exclusive to management can eventually limit the career growth of experienced Software Engineers. You can choose not to manage people. You can choose to remain deeply technical. As your scope and impact grow, leadership increasingly becomes integral to technical work. Conclusion Leadership in software engineering does not mean leaving the technical path or becoming a manager. It means understanding that software is built by people, and that technical expertise must be paired with trust and influence to drive change. Key decisions need experienced engineers involved, and leadership enables your knowledge to reach beyond your own code. As you advance to roles like Staff Engineer, Principal Engineer, or Software Architect, leadership becomes essential for greater impact. You can remain an Individual Contributor and stay deeply technical, but to increase your influence, you must also guide, communicate, build trust, and help others make better decisions.

By Otavio Santana CORE

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority. Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control. Unlike in previous systems, modern production is largely assembled rather than built from scratch. For example, a conventional SaaS product is built on interdependencies among payment processors, outsourced data infrastructure such as Amazon Web Services (AWS), messaging services, web hosting, design, AI inference APIs, authentication providers like Google, and more. These useful platforms and products are essentially outside teams' control stations, even though they critically impact users' experience. When they function effectively, you share the glory with the platforms. But when there is a system blackout, your users put you on your toes, even though you have no direct access to resolve the problem on time. Therefore, we shall be exposing SRE practices in platform-SaaS and API-dependent systems and how reliability is getting beyond the control of engineering teams and companies. Why Classical SRE Practices May Fail One major downside of SaaS and dependency on external platforms is that reliability control is often assumed to be in a team's hands, whereas it has been bargained. However, teams must reckon with the fact that the case is reversing. For example, traditional SRE models once alleged that: Service Level Indicators (SLIs) focus on availability or internal uptime and latency.Error budgets arise from changes teams make or deploy.Runbooks still suggest that teams can immediately reconfigure or directly work on faulty components. All these are becoming past cases, especially in platform-SaaS systems. You can have a system indicating 99.99% or even 100% uptime on the back end, while new users are struggling to sign up, probably because an authenticator provider is not fully functional. Dashboards and control panels may indicate green, but in reality, third-party payment APIs have been degraded. A New Definition of Reliability in Operating SRE Practices To resolve the new problem in site reliability engineering (SRE), there needs to be a conceptual shift from component health to an integrated, continuous user experience. Therefore, teams need to undergo a paradigm shift away from questions such as "Is our CPU working maximally?" “Is our API up?” “What are the error rates?” Instead, we should inquire: “Are users checking out seamlessly?” “How fast can they authenticate?” “Can they use the SaaS product to perform its key function?” These types of outcome-based questions span interdependent platforms beyond your full control. The login SLI needs to work with the identity provider; otherwise, its output is meaningless. If the checkout SLO skips payment authorization, then it's both fishy and unreliable. True, there may be some internal errors in a reliable system, but what really matters is an integrated multiplatform experience that the user enjoys. Error Budgets? An SRE Practice to Revisit How many teams would love error budgets to disappear when they give up control? But that’s not so. Instead, they are molecularized. When components of your systems are outsourced, the error budget doesn’t just fade away; it is instead transferred to the interdependent platforms. So, it’s better to plan for the fact that SaaS and API providers will consume some of your reliability budget. Doing so keeps you a few steps ahead and protects your business in the long run. Reliable SRE teams make decisions such as allocating part of their error budget to certain dependencies, setting acceptable parameters for degradation, and defining specific steps to take when a dependency exceeds the stipulated budgets. Here’s an example you can adapt: “We will accept payment authorization failure of 0.0% to 0.2% if it is caused by dependency instability. If it goes above that, we will turn on delayed capture or turn off promotions.” This SRE approach keeps you ready for downtime, as your systems automatically switch to planned or budgeted actions rather than relying solely on integrated platforms. What to Do When Failures Beyond Your Control Arise Actually, some failures may seem beyond your control. The more you attempt to resolve them, the more amplified they become. At this point, your team must adapt to the savvy absorption of such situations. Instead of focusing solely on retrial in an SRE approach, your team needs to design its processes and platforms. This could include failing selectively through circuit breakers, failing fast with timeouts, or failing visibly by keeping users informed. Some core settings should always remain non-negotiable and on standby. These could include the following: Read-only modes/cachesBulkheads that prevent a failure avalanche.Automated circuit breakersDeferred processing These reliable practices ensure there is some form of controlled uptime even when operations seem interrupted. Laser Observability That Proves Reliability In traditional SRE observability, the service boundary is usually the ultimate, but in most modern integrated SaaS platforms, this could be insufficient or worse, dangerous. Operators need to be aware of the actual dependency that is failing, how it is failing (e.g., errors or throttling), and how the failure affects the user experience. Accurate observability for platform-SaaS and API-dependent systems requires these four provisions: Specific dashboard and internal metrics for each vendor.SLI monitoring at the dependency level.Parallel tracing of all outbound calls.Simulation of real-time user experience and workflows. Essentially, whenever there is an emergency, operators should be able to promptly identify whether the source is internal or external. Accuracy and clarity facilitate swift response. Responding to Incidents Without Ownership Another distinct characteristic of modern SRE practice in platform-SaaS is how incidents are responded to. Without ownership, you often cannot debug on your own, roll back a bad deploy, or directly manage other issues. However, you can choose how your system responds by identifying when certain features are disabled, when signals to activate degraded modes are sent, when high traffic is redirected or shed, or when to notify users. To maintain reliability, incident response relies on runbooks to inform decisions. The following questions could help convert the technicality of runbooks to practical solutions: What is the impact on the customer?In what ways can we respond harmlessly?What can we reverse?What should we communicate externally? These questions help resolve incidents, mitigate losses, and intertwine reliability with sound judgment. Is Safety an Illusion in SLAs? SLA providers often readily contract for financial compensation when losses arise, but seldom give absolute reliability guarantees. You may not always expect vendors to consistently meet your availability goals or resolve an avalanche of outages. Safety is a critical consideration when building systems, because when users lose trust in a brand, compensation may not be able to redeem it. Therefore, advanced teams do not consider SLAs as safety nets but as risk pricing. They understand that contractual credits cannot replace trust, brand image, and some almost irredeemable damages. Human Factors in Platform-SaaS and API-Dependent Systems Dependency failures often escalate when cognitive load increases. There could be degraded performance, timeouts without error indicators, partial success, or inconsistent system behavior. Operators may not only focus on machines when dashboards lag or seem to lie. They examine the logs, failure history, or commands. Teams have to design systems with overrides and predictable degradation paths, and observability tools are beyond the failure systems. Reliability goes beyond the correct function of software; it's also about human operations. How Your SaaS and API Platforms Can Imbibe “Good” SRE Practice Effective SRE practices are modern. The following attributes know saas products and API-dependent platforms: Acknowledgment of lack of control very early.Ensuring reliability is embedded in the design.Measuring the outcomes of each SRE criterion or target, instead of just the components.Giving priority to clarity instead of trying to model or control everything because you do not own all the components.Making engineering and operations decisions and products as an integrated whole.Preparing for degradations as inevitable procedures when things fail. Your systems can be reliable if you anticipate failure and accept the reality. Conclusion Modern platform-as-a-service (SaaS) operates in a reliability-without-control manner, leading solid SRE teams to accept that they need to adapt when failures occur. It's simple logic: if you don't absolutely own everything end-to-end, then prepare for the worst: each dependency might fail. It's all about keeping the trust of your users and protecting your brand image.

Large language models have evolved from simple chat interfaces into autonomous systems capable of planning, reasoning, and interacting with external tools. The next stage of this evolution is multi-agent software engineering, where specialized AI agents collaborate to solve complex business workflows instead of relying on a single monolithic model. A planner may decompose work, researcher agents retrieve enterprise knowledge, coding agents generate implementations, reviewer agents validate outputs, and execution agents perform approved actions. Although this architecture appears attractive, production deployments reveal that coordinating multiple agents resembles building a distributed system far more than writing prompt chains. The primary challenge is not model intelligence but system reliability. Every additional agent introduces another opportunity for hallucinations, context loss, latency, retries, and cascading failures. A workflow containing five agents with individually high accuracy can still produce inconsistent outcomes because each handoff becomes another source of uncertainty. The engineering challenge therefore shifts from prompt engineering toward orchestration, state management, resilience, and observability. Most successful enterprise implementations begin with a planner-worker architecture. Instead of allowing every agent to communicate freely, a planner receives the business objective, decomposes it into smaller tasks, distributes work to specialized agents, and aggregates the responses into a final result. This pattern simplifies coordination, enables centralized policy enforcement, and provides a single location for monitoring execution. Java AgentPlan plan = planner.createPlan(request); List<CompletableFuture<AgentResult>> workers = plan.tasks().stream() .map(task -> CompletableFuture.supplyAsync( () -> worker.execute(task))) .toList(); List<AgentResult> results = workers.stream() .map(CompletableFuture::join) .toList(); return aggregator.combine(results); Bottlenecks Arise As the number of agents increases, direct synchronous communication quickly becomes a bottleneck. Event-driven messaging provides better scalability by allowing each agent to publish completed work while downstream agents subscribe only to events they understand. Kafka is particularly effective because partitions naturally distribute workloads across worker instances while preserving message ordering for individual workflows. The orchestration layer no longer manages worker availability directly and instead publishes work to topics, allowing consumer groups to handle scaling and recovery. A durable workflow engine becomes equally important. Stateless orchestration fails whenever a process crashes, a deployment occurs, or an agent exceeds execution time. Platforms such as Temporal persist workflow history so execution resumes from the last successful checkpoint rather than restarting an expensive reasoning process. This separation between orchestration and agent execution prevents duplicated work while making long-running AI workflows operationally reliable. Addressing Context Management Context management presents another significant engineering problem. Passing the complete conversation between every agent rapidly increases token consumption while reducing response quality. Instead, enterprise systems maintain workflow state separately from prompts. Business context is stored in persistent databases, semantic knowledge resides in vector stores, and external capabilities are exposed through Model Context Protocol (MCP) servers. Each agent retrieves only the information required for its current task instead of inheriting the entire execution history. Java workflowRepository.save( WorkflowState.builder() .workflowId(id) .currentAgent("SecurityReviewer") .status(Status.RUNNING) .context(serializedContext) .build() ); Standardizing communication between agents also improves maintainability. Rather than exchanging natural language, production systems often define structured contracts that include workflow identifiers, task types, priorities, and correlation identifiers. JSON { "workflowId": "WF-2041", "source": "Planner", "target": "CodeReviewer", "task": "Validate generated API", "traceId": "9bdc-421" } Structured messaging enables retries, auditing, replay, and interoperability across heterogeneous agents developed by different teams. It also aligns naturally with emerging protocols designed for agent interoperability. Reliability patterns from distributed systems remain equally valuable in AI applications. Agent failures should never stall an entire workflow. Timeouts, retries, circuit breakers, and dead-letter queues prevent individual components from consuming unlimited resources while protecting downstream services from cascading failures. Java try { AgentResponse response = future.get(20, TimeUnit.SECONDS); } catch (TimeoutException ex) { retryQueue.publish(task); circuitBreaker.recordFailure(); } Additional Issues to Consider Unlike conventional microservices, however, AI systems introduce another category of failure called reasoning loops. An agent may repeatedly invoke different tools while attempting to improve its answer without ever reaching completion. Runtime safeguards therefore extend beyond traditional retry limits to include maximum reasoning depth, token budgets, and execution deadlines. These controls prevent runaway costs while ensuring workflows terminate predictably. Production systems require complete visibility into every agent interaction. Traditional application logs reveal infrastructure failures but rarely explain why an AI workflow produced an incorrect decision. Distributed tracing with OpenTelemetry allows each planner, worker, and tool invocation to emit correlated telemetry containing workflow identifiers, agent names, execution latency, token usage, and tool calls. A single trace can reconstruct the entire reasoning path, making failures reproducible instead of mysterious. Java Span span = tracer.spanBuilder("agent-execution").startSpan(); span.setAttribute("workflow.id", workflowId); span.setAttribute("agent.name", "SecurityReviewer"); span.setAttribute("tokens.input", 1350); span.setAttribute("tokens.output", 512); worker.execute(task); span.end(); Observability should extend beyond infrastructure metrics. Enterprises benefit from tracking reasoning iterations, tool invocation frequency, retrieval latency, hallucination rates, retry counts, and token consumption. These operational metrics quickly reveal inefficient prompts, unreliable tools, or expensive reasoning loops before they impact production workloads. Testing also changes significantly. Traditional unit tests validate deterministic functions, whereas AI agents produce probabilistic outputs. Instead of asserting exact responses, enterprise pipelines evaluate workflows against acceptance criteria such as schema validation, factual correctness, safety policies, latency budgets, and execution cost. Regression suites should replay representative business workflows after every prompt, model, or orchestration change to ensure quality remains stable despite model updates. Security becomes increasingly important as agents gain permission to execute external actions. Every tool invocation should follow least-privilege principles, while generated code executes only inside isolated containers or sandboxes. Human approval remains essential for high-impact operations such as financial transactions, infrastructure changes, or customer-facing decisions. Durable workflow engines make this straightforward by pausing execution until approval arrives rather than blocking application threads. Standardizing Integrations The emergence of Model Context Protocol (MCP) further standardizes enterprise integrations. Instead of creating custom connectors for every application, MCP exposes databases, repositories, APIs, and enterprise tools through a consistent interface that any compliant agent can consume. Combined with Kafka-based messaging and workflow engines such as Temporal, MCP enables independently developed agents to cooperate without tightly coupling business logic to individual AI models. Despite growing enthusiasm, multi-agent architectures should not become the default solution. Many business problems remain better served by a single agent with carefully selected tools. Every additional agent increases latency, infrastructure complexity, operational cost, and potential failure points. Multi-agent systems become valuable only when tasks naturally decompose into specialized responsibilities requiring parallel execution, independent security boundaries, or domain-specific reasoning. Successful production deployments therefore resemble distributed systems more than prompt engineering experiments. Planner-worker orchestration, durable workflow persistence, event-driven communication, standardized protocols, resilient execution, comprehensive observability, and continuous evaluation collectively determine whether an AI system scales beyond demonstrations. A Final Word Multi-agent software engineering represents an important architectural evolution rather than simply a larger collection of language models. Organizations that approach agent collaboration with the same engineering discipline applied to microservices, distributed messaging, and cloud-native platforms will build systems capable of remaining reliable under production workloads. Those that treat agent orchestration as little more than chained prompts will likely encounter escalating costs, inconsistent behavior, and operational instability long before realizing the expected productivity gains.

By Uthej Mopathi CORE

TL; DR: The “Agile to the Product Operating Model” Survey Results Between August 2 and August 10, 2026, 48 practitioners participated in my Agile to Product Operating Model (POM) survey, which tries to shed light on what is actually changing. Let me summarize the answers for you: the reported transformations change decision-making less than the Cagan framework suggests. Where respondents report improvements, they appear in delivery and collaboration rather than in business results. Unfortunately, the human side of the transition is the least encouraging part of the answers. Thesis: Product operating model transformations mostly change vocabulary and organizational structure while leaving the decision system — who decides what gets built, on what evidence, at what speed- largely untouched. AI is changing product decisions independently of POM transformations. Who Answered, and Why I Report Counts, Not Percentages Of the 48 respondents of the Agile to Product Operating Model, 26 work in organizations that have adopted a product operating model or are moving toward it (we refer to them as the “movers” from here on). The other 22 work in organizations that are not moving; these organizations are still discussing the issue, have decided against it, or have never considered it. So you get absolute counts, and every finding below is directional, not definitive. Additional limitations are: 24 of the 48 respondents are Scrum Masters or Agile Coaches, the roles under the most pressure from this shift.38 of 48 work in organizations with more than 250 people; the sample contains 2 respondents from startups and 2 from scale-ups. Apparently, the people who have participated in the survey are those who still care enough about agile practice to read a newsletter about it; people who left “Agile” entirely are structurally missing. Also, as I did not ask respondents to identify employers, the 48 participants report on their organizations, not 48 distinct verified organizations. Putting it all together, this is a report from inside large organizations, where the product operating model makes its boldest promises. One Fundamental Change in 26 Answers Asked which statement best reflects their experience so far, the 26 movers split like this: 9 say mostly relabeling, the same way of working with new vocabulary9 say real change in some areas, relabeling in others7 say it is too early to tell. Exactly one participant reports a fundamental change in how the organization decides what to build, which is the most interesting learning from the responses: changing the vocabulary, changing the organizational structure, and changing how decisions actually get made are three different things. Better Delivery, Inconclusive Business Results, Worse Morale The Agile to Product Operating Model survey asked movers to compare six outcomes against their previous way of working. Aggregating “somewhat” and “clearly” in each direction: Speed of delivery: Better (9), No change (10), Worse (1), Too early / cannot tell (6)Value delivered to customers: Better (8), No change (8), Worse (2), Too early / cannot tell (8)Business results: Better (3), No change (8), Worse (4), Too early / cannot tell (11)Team motivation and morale: Better (7), No change (4), Worse (9), Too early / cannot tell (6)Developers’ satisfaction: Better (4), No change (8), Worse (7), Too early / cannot tell (7)Collaboration with stakeholders: Better (9), No change (6), Worse (4), Too early / cannot tell (7) The table does not say that the transitions are failing: Delivery speed leans positive: 9 better against 1 worse.Customer value leans positive: 8 against 2.Collaboration with stakeholders leans positive: 9 against 4.Business results are inconclusive, with 11 of 26 unable to say and the rest split. The people dimensions are the only ones that lean consistently negative: morale at 9 orgs is worse than at 7, where it improved; developers’ satisfaction is similar: worse at 7 orgs and better at 4 orgs. Two ways to interpret this information, and the survey does not help to choose: (1) The improvements concentrate on the things enterprises already know how to optimize (flow, coordination, delivery), while the transformations may be falling short of the things the model claims to change most fundamentally. (2) Operational effects precede commercial effects, because business outcomes have longer feedback cycles, and 11 of 26 respondents explicitly cannot yet say. Consider that flow, coordination, or delivery are easier to measure and attribute than commercial effects, which is a significantly fuzzier area. Therefore, the interesting question is how this changes over time, not how it looks at one moment. The range of individual experience behind those aggregates is wide. One respondent, 12 months into adoption at a large financial-services organization, described the hardest part as “surviving the political game that the POM shift has been” and reports worse answers on five of the six dimensions. The other two respondents who have been at 12 or more months report the opposite: better value, better morale, and better collaboration. Three long-tenured accounts cannot agree on whether the practical transition mechanics, culture, affected products and services, or the organization itself makes the difference. (That is the curse of a small sample size.) Agile to Product Operating Model: The Empowerment Gap, and What Settles Uncertainty Cagan’s own test separates empowered product teams from feature teams: in a product team, the team is tasked with solving a problem and owns the solution. In a feature team, “the value and business viability are the responsibility of the stakeholder or executive that requested the feature.” The survey asked movers which pattern operates today, in the middle of their transitions: 9 of 26 report the empowered pattern (leadership sets goals and problems to solve, teams decide what to build)8 of 26 report that leadership decides which features get built and teams implement them.4 of 26 report stakeholder requests are filling the product backlog reactively.4 of 26 say it is genuinely unclear or contested right nowA single one reports teams setting their own direction. So even among respondents whose organizations are actively adopting a model whose entire premise is empowered teams, the “feature-list or reactive-backlog” pattern (12) outnumbers the “empowered” pattern (9). A related question asked what settles it when the organization is uncertain whether something is worth building: 6 said debate and prioritization before anything gets built5 said whoever has the most authority decides5 said things just get built and shipped, and an explicit decision rarely follows. 4 said research evidence5 said it varies too much to say. In total, only five respondents fall into the categories I pre-registered as evidence-led: research evidence, disposable prototypes, or ship-and-measure. There is one case among all the answers: a respondent reports that a disposable AI-assisted prototype reduces build uncertainty: build to learn, decide, and throw it away. That same respondent, 12 months into adoption, also reports better responses across all six outcome dimensions listed above. Of course, one case is not evidence of a relationship. It is a question worth asking in a larger sample, not a claim I am entitled to make here. In Little Code, Big Waste, I argued that cheap AI code removes the cost gate that used to force a should-we-build-this decision, and that “when generating plausible code becomes cheap, every hour spent building the wrong thing becomes waste that can now be produced at scale”. This survey says the prototype-to-decide pattern barely exists yet in this sample. Agile to Product Operating Model Survey Shows: The Two Transitions Run Separately If you believe the keynote version of events, AI is forcing organizations to redesign their operating model around it. The 26 movers report something different. Only one respondent says AI is a core reason for the change; 6 of 26 call it one factor among several; 16 say AI adoption runs in parallel, but separately from, the operating model change; and 3 report little or no role. Now the other side. Among the 22 respondents in non-moving organizations, 15 report that AI is changing how product decisions are made anyway: 6 noticeably, 9 in pockets, without any formal model change. Despite the small sample size, the data support a narrow claim: for most movers in this sample, AI and operating-model transformation are separate initiatives, while for most respondents in non-moving organizations, AI is changing product decisions without any model change at all. Formal operating-model change is neither a prerequisite for AI-driven changes to product decision-making nor, so far, organized around them. The survey did not measure how AI enters these organizations or who authorizes it, so I will not claim more than that. The disconnect between the two is already the finding. My Pre-Survey Hypothesis Scoreboard Here are my verdicts on my four hypotheses, H1-H4, that led to the creation of the Agile to Product Operating Model survey: H1, consultant-driven transitions produce more relabeling than leadership-driven ones: Relatively consistent, but no verdict: 3 of the 4 respondents whose transition is driven by consultants or a transformation office report “mostly relabeling,” against 4 of 16 in product-led or executive-led transitions. But those are just four participants. H2, the empowerment gap: The feature-list plus reactive-backlog patterns (12) outnumber the empowered pattern (9). That is observed in this sample, however, not established beyond it. H3, where AI drives the move, product judgment is the scarcest capability: incorrect on my side: Among the 7 respondents whose organizations treat AI as a core reason or a factor in the move, the most frequent scarcity answer is “delivery capacity is still the bottleneck” (3), ahead of product judgment (2). Across all 26 movers, the scarcity question splits three ways: “stakeholder alignment and decision speed” (7), delivery capacity (6), and product judgment (6). One distinction keeps H3’s rejection from closing the question. The survey measures perceived constraints, and perception rarely reflects reality accurately. AI may have changed the economics of implementation faster than organizations have updated their sense of where the workflow bottleneck sits. Whether delivery capacity objectively remains the constraint is a different question, and this survey cannot answer it. What it can say: practitioners today experience decision speed, delivery, and judgment as roughly competing constraints, and the judgment-scarcity era I anticipated is not the world they report living in. H4, organizations that build without explicit decisions report worse customer value than evidence-led ones: rejected as stated: 1 of 5 build-without-deciding respondents reports better customer value, against 2 of 5 evidence-led ones. Again, the number of replies is too small. Changing the Structure Without Changing the Decision System Three of the findings above belong side by side. Only 1 of 26 movers reports a fundamental change in how build decisions are made. Only 9 of 26 report the empowered decision pattern operating today. Only 5 of 26 fall into the evidence-led categories for resolving build uncertainty. Together, they suggest a more precise diagnosis than “the transformation is theater.” Transformations change three different layers, and the layers move at different speeds: Vocabulary changes first: Product operating model, empowered teams, outcomes over outputs; you get the idea.Structure changes second, and often really does change roles, reporting lines, team topologies, or artifacts.The decision system changes last, if at all: Who decides, based on what evidence, under what uncertainty, with what authority, at what speed. This survey reads like a snapshot of organizations renaming the first layer, reorganizing the second, and leaving the third largely untouched. That is why the 1-of-26 count deserves the weight I put on it earlier. One of the product operating model’s defining promises is a different way of deciding what to build. If only one of the 26 respondents experiences a fundamental change in that mechanism, the question is no longer whether the transformation is proceeding fast enough, but what is being transformed. That mental model offers one possible explanation for the outcome table: structural change could improve coordination and flow before it alters the quality of product decisions. Whether that explains the pattern here is impossible to establish from 26 responses. A long-term study of organizations and involved practitioners would need to test whether decision-system change predicts eventual business outcomes. (Consider that a hypothesis this survey generated, not a result it delivered.) Agile to Product Operating Model: Why Product Washing Is Easier Than Empowerment The survey cannot tell us why the decision system resists change. What follows is my hypothesis, argued from two decades of watching transformations, not a survey result. The tempting explanation is that these organizations are implementing the product operating model badly, and that a proper implementation would deliver. I spent those two decades watching the Agile community run exactly that defense. Every failed adoption was “not real Scrum.” The argument is unfalsifiable, and it taught an entire industry to blame practitioners instead of examining incentives. I will not run the same defense for the product operating model. In November 2024, I described Product Washing: the hollow adoption of product practices that “leaves companies stuck in the same old dynamics but with a new vocabulary,” transformation by reprinting business cards. My hypothesis for the mechanism: product washing is not an implementation failure but what enterprise incentives produce when you ask powerful people to redistribute their own power. The model demands that stakeholders with budget authority hand problem-selection to product leadership and solution-selection to teams. Budget authority is power, and in many large organizations, “product leadership” has become a new title for the same stakeholders who control the money. Meanwhile, middle layers face asymmetric career payoffs: a visible failure damages a career far more than a shared success advances it. Under those payoffs, routing decisions through committees and sign-offs is rational self-protection, and it does not vanish because the org chart was redrawn. Two honest caveats bound this hypothesis: First, in regulated industries, some of that scrutiny is very relevant: when a named person must answer to a regulator, a sign-off chain is accountability, and the respondents from banking and the public sector live with constraints no product operating model erases. The skill worth having is telling required governance apart from accountability theater; most organizations run both and label neither. Second, this survey is a cross-section, not a time series, and two competing explanations fit the same data: The transition hypothesis: decision authority changes slowest of all the layers, and these organizations are simply not there yet.The attractor hypothesis: enterprise incentives pull transformations toward renamed feature factories and hold them there. My incentive argument predicts the attractor. With only 3 respondents at 12 or more months, this survey cannot distinguish between them. That is a testable question for a future survey. The Non-Movers’ Catch-22 The 22 respondents in non-moving organizations deserve more attention than transformation literature usually grants them. Their top reasons for staying put: Leadership sees no need or has other priorities (15),Lacking the product-management maturity to build on (13), andThe cost and fatigue of yet another transformation (10). Only 6 claim their current way of working performs well enough. The dominant non-mover position is not a confident endorsement of the status quo. And inside the reasons sits a genuine Catch-22. Organizations supposedly need the product operating model because their product capabilities are weak, while 13 of 22 respondents say weak product capabilities are precisely why their organization cannot adopt it. A transformation that requires the maturity it promises to create is a hard sell to people who have already survived several that made the same offer. Conclusion: Three Questions for Monday Morning Three questions locate your own organization on this map: First: who decided the last thing your team built, and would your CPO name the same person? If the answers differ, you have found the gap between the model on the slides and the model in operation. Second: what settled your organization’s last genuinely contested build decision: evidence, debate, or seniority? “Whoever has the most authority decides” got 5 votes out of 26 in this survey. Be honest about whether your organization would add a sixth. Third: what would have to be true for a disposable prototype to settle the next contested decision instead? That is a political question, not a technical one: who would have to accept evidence as a tiebreaker, and what would it cost them? Forty-eight answers later, the better question is no longer which operating model organizations adopt. It is how they make decisions when the cost of trying something has collapsed while the cost of deciding has not: Who decides on what evidence, and how quickly can the organization act on what it learns? That is where my work is heading next.

By Stefan Wolpers CORE

You have a checkout flow. You have 40 tests. They're green. Now: what happens when a payment webhook arrives after the user cancels? What happens when a retry lands on a session that already expired? What happens on the fourth failed attempt when autoRenew is off and the period boundary has already passed? You don't know. Not because you're careless — because a state machine with 6 states, 7 actions, and 3 payload values has thousands of reachable (state, action, data) combinations, and your 40 tests visit 40 of them. The bugs that page you at 2 am live in the other several thousand. Polygraph is a Claude Code plugin and standalone CLI that walks all of them. Why You'd Bother Polygraph is for stateful code: reducers, workflow engines, protocol handlers, session managers, order state machines, anything with a dispatch(state, action) shape. If your code is a pile of pure functions, go use property-based testing. If it's a state machine, keep reading. Narrow on shape, not on language. The model reads your source in whatever it's written in — you name it once as lang in the contract — and the trace format is just NDJSON, so any runtime that can log a {pre, action, data, post} line per step can feed it. What's always JavaScript is the derived spec and your rules, because those are what the replayer and model checker execute on Node. What you get back is not a lint warning. It's a shortest action sequence that reaches a state violating a rule you wrote. Something like: Shell ✗ never-charged-twice [state] — pred returned false init {"status":"new","attempts":0,"hasDue":false} CREATE({}) -> {"status":"active","attempts":0,"hasDue":false} RENEW_CHARGE({"result":"5xx"}) -> {"status":"grace","attempts":1,"hasDue":true} RENEW_CHARGE({"result":"ok"}) -> {"status":"grace","attempts":2,"hasDue":true} That's a repro. You paste it into a test file, and you have a failing test in about ninety seconds. A real one: On a production SaaS subscription-billing machine, Polygraph flagged a disagreement on exactly one window: a 5xx from the payment processor during renewal moved the row to grace and marked it due, when the dunning path in the same codebase correctly treated 5xx as ambiguous. The next retry rotated the idempotency key. If the 503'd transfer had actually settled, the customer got charged twice. A human reviewer had found the same bug by hand; five independent model-derived readings of the source landed on it blind. And in the controlled seeded-bug eval, the split is worth knowing: replaying real traces against the derived spec found 0 of 5 seeded bugs. Model checking found 5 of 5, with counterexamples. Trace replay tells you whether to trust the model. Model checking is where the bugs actually are. What It Actually Does Three artifacts, all diffable, all in your repo: 1. contract.json — the scope. Which state fields matter, which actions the machine accepts, what data each action can carry, which states are terminal, and lang is the language your source is written in. 2. A spec — a JavaScript model of your code, written by an LLM from your source (whatever language that is). It's a strict SAM v2 module: every action it ignores has to say why via reject(reason), it can't hide bookkeeping state, and it declares its own action/data domains — so the checker knows what to explore with zero config. Several specs are generated independently and vote, so one bad generation doesn't decide anything. JavaScript export const stateInvariants = [ { name: 'locked-only-at-limit', pred: (s) => s.status !== 'locked' || s.attempts >= 3 }, ]; export const transitionInvariants = [ { name: 'expired-never-verifies', pred: (pre, action, data, post) => !(action === 'ATTEMPT' && data?.expired) || post.status !== 'verified' }, ]; 3. invariants.mjs — your rules, as plain JS predicates: This part is yours and can't be automated away. Code with a bug is a perfectly faithful description of the wrong behavior. Invariants are where your intent enters the system. Then two checks run. Replay asks "is the spec faithful?" Real traces ({pre, action, data, post} windows, captured by wrapping your dispatch once) are replayed against each spec, with positive and negative controls proving the harness can tell good from bad. Model check asks "where are the bugs?" It iterates the faithful spec exhaustively from init against your invariants and prints the shortest path to every violation. The Caveats "Exhaustive" means exhaustive over the finite (action, data) domain declared in your contract. A machine whose behavior depends on unbounded counters or arbitrary strings is checked only at the representative values someone chose. That's the standard TLA+ modeling move, and the gap between declared domain and real data is real.It's a consistency check, not a proof. A clean run means your code's observable behavior matches an independent reading of its own source. Nothing more.Every finding is a lead to investigate, not a verdict. There is no triage step that discharges "real invariant break with no observable consequence."It's experimental and not peer-reviewed. Don't make it your only safeguard on safety-critical code. API Key and Cost Only three things call the Anthropic API: spec generation, code authoring (polygen), and polynv's optional headless invariant harvest. You need ANTHROPIC_API_KEY in your environment for those, including inside Claude Code, where the skills shell out to the same scripts and do not use your session credentials. Ballpark, on a typical machine: you runkey?costverify.mjs --source … (generate + replay)yes~$0.50polygen.mjs --intent … (author new code — JS/TS output only)yes~$2replay saved specs, model check, --tla, polyvers, polynv, polyrunno$0 That second row is the load-bearing one. Everything that checks (replay, the exhaustive model check, version gating, the mutation grade, TLC escalation) is keyless, local, and deterministic on Node ≥ 20. Which is precisely what makes CI viable: you commit the spec, and the gate re-runs it on every merge request for free. No key in CI, no per-MR API bill, no nondeterminism in your pipeline. That gate is polygate, and there's a GitLab reference implementation at <POLYGATE_GITLAB_URL> — a .gitlab-ci.yml you can copy that runs corpus validation, replay, and the model check against your committed artifacts and fails the MR on a violation. (Contrast: Specula, the closest comparable agentic TLA+ pipeline, reports a median of $57 and 3.7 hours per system. Excellent tool, structurally can't run on every MR.) Getting Started Prerequisite, and it's a hard one: The stateful code has to be runnable in isolation, because traces are ground truth from the code actually executing. A clean step boundary: a dispatch, reducer, or handler, from experience, Claude will refactor it easily for you. If it only runs against a live DB or device, stand up doubles first (in Claude Code, the agent will build them). Note this is the only place your language matters, and only for convenience: the bundled withTracing / tapReducer helpers are JS, so a Go or Python machine means writing the {pre, action, data, post} NDJSON lines yourself. It's about ten lines. Zero-cost first (no key, five minutes): Shell git clone https://github.com/cognitive-fab/polygraph cd polygraph && npm test # validates the bundled corpus, runs the controls npm run verify:turnstile-v2 # replays bundled specs — see the output shape Then on your own machine, as a plugin: Shell /plugin marketplace add cognitive-fab/polygraph /plugin install polygraph@polygraph …and just ask: "verify this state machine", or /polygraph:polygraph for the guided end-to-end run (Claude drafts the contract, instruments the boundary, captures traces, runs controls, triages with you). Trace capture is historically what made this expensive; it's the step the agent now carries. Or plain CLI, no Claude Code: Wrap your dispatch once, projecting only the contract's observable keys (JS shown; in another language, emit the same NDJSON shape by hand): JavaScript import { withTracing } from '<plugin>/scripts/instrument/trace-emitter.mjs'; const dispatch = withTracing( rawDispatch, () => ({ status: m.status }),'traces/s1_normal.ndjson' ); Note --source takes your real file, in your real language: Shell node scripts/validate_corpus.mjs contract.json traces/ # no key node scripts/verify.mjs --contract contract.json --source src/machine.ts \ --traces traces/ --model opus-5 --n 5 --out out/ # key, ~$0.50 That writes out/findings.md and the generated specs to out/specs/. Commit the winning one, and from then on the loop is free: Shell node scripts/check.mjs --spec out/specs/spec_0.js --contract contract.json \ --invariants invariants.mjs # no key, forever There's no default model: pass --model. Use opus-5 or better; deriving a faithful transition function is a hard reasoning task and lighter models don't clear the bar. If you see empty specs, you lowered --max-tokens below what the reasoning block needs; put it back to 32000. Apache-2.0. The method is written up in arXiv:2607.05076. Your test suite is a sample. This is the census.

A few weeks ago, I participated in a 24-hour AI hackathon where we built a product using AI. Necessary tools were provided, a large number of engineers participated eagerly, and a few business folks also joined to bring their ideas into a real-world product using AI. During brainstorming, people drafted end-to-end process flow diagrams and started working on development, using all the recent available models and platforms to build their product. When the development time window ended, it was time for presentations. As I watched each team present their results, I observed that they couldn’t automate the end-to-end process flow. What they had planned during brainstorming didn’t turn out to be a complete, end-to-end product. Most of the solutions followed the same pattern: they did something in one product and took the output to another product, and the output of that product went somewhere else to finish the loop. Since the transition was handled by a human, it was considered “human in the loop.” No doubt this speeds up the process, but it left a gap — between an AI system that helps finish certain tasks in a process flow and an AI system that finishes the process end to end. Which one is the better model to build? Let’s Try to Define Assist and Complete An assisting system sits inside a workflow. AI takes some kind of input and produces an output. A human then takes the output, reviews it, and decides what to do with it — moving to the next system with or without AI, and continuing the process flow. If we removed the AI from this process flow, it could still run, and the work would still get done. It would just take more time. For example, in a real-world tech process, a business team uses AI to document a requirement, which is converted into a JIRA story using the Confluence AI tool, and a developer reads it and uses Copilot or another AI assistant to develop the code. A complete system owns the workflow’s outcome. Here, the system takes the input, processes it, makes the decision, triggers an action, and writes a log documenting the actions it has performed. If we remove any component from this process flow, it won’t just slow down — it will completely stop. As a real-world example, there is a solution that takes all the reviews for a company, analyzes them, categorizes them, summarizes them, and then raises a ticket with that summary and assigns it to the respective team. As an evolution of this process, teams often look only at their current work and how they can leverage AI in it, rather than looking at the end-to-end process flow integration. This is one of the reasons why AI investment doesn’t show up in the metrics that matter. It was architected to help, for sure, but not to finish. The Four-Part Test After watching this pattern across a number of different workflows, I have formed a set of four questions to address whether a process is structurally ready for a complete system, or whether it’s just going to support an assisting one. Is the trigger unambiguous? Something has to start the cycle — a batch starts on a schedule, a microservice is triggered by a button in the UI, a sensor crosses a threshold. If the start of the workflow is itself a judgment call that varies by who is looking at it, then we don’t have a clear trigger, and the downstream process isn’t ready to be fully automated. Can you define the decision logic? “Define” isn’t simple, but the system’s decision logic can be made explicit, even if the underlying model is complex. In a reliability process flow system, you can define what should be considered an error, what the threshold is, and when to act. Contrast that with deciding whether a customer complaint is about to escalate into a PR problem — the reasoning experienced staff uses to answer that is often genuinely tacit, inconsistent between two employees, and resistant to being written down.Is it executable in a real system? A complete system needs more than just an AI model. It might need an API, a write permission, a button it can press — not a dashboard a human reads and acts on separately based on what they see in it. This sounds obvious, doesn’t it? But teams often build excellent reasoning and never build the hooks to act on it. Can the outcome be verified? This is the most important question of all. After the action completes, can the system check and confirm whether the right action was taken and that it worked? Did the email send, did the case close correctly, did the transaction process without a downstream error? Verifiability is the closing step of a fully automated system, and it’s what makes the system safe. It’s what catches a bad decision before it compounds tenfold. When a workflow answers all four of these questions well, it is a strong candidate for a complete system. When even one question fails, that’s a signal that the workflow should stay an assisting one, regardless of how sophisticated the underlying model is. Why Most Systems Assist But Don’t Fully Complete A simple, straightforward question is why so many production systems stop at assist. In my experience, there are three structural reasons: Narrow decision logic: If a system has narrow decision logic in the process flow — binary answers like yes or no, or a defined set of cases — it can often be handled within the process flow regardless of AI involvement. But people try to introduce AI here anyway, and when it picks an answer that wasn’t pre-defined, that’s treated as a failure.Evidence validation failure: Teams give AI the middle of the workflow because it’s recoverable — a human validates the output at the end and catches problems before real damage occurs. Handing over the full cycle means trusting the outcome, but most organizations haven’t built the measurement discipline to know when that trust is justified. They deploy the AI and measure in time savings, but they never run the harder experiment: tracking what the system’s success rate would actually be if it were allowed to complete the cycle unsupervised. Unresolved accountability: This one is more cultural than technical, but it’s a real blocker. When a human closes the loop, it’s clear who is responsible. But when a system closes it, that question gets harder — especially in any environment where decisions face external scrutiny, like an auditor, a regulator, or a customer dispute. Most organizations keep a human in the closing position by default, regardless of whether that person is actually adding judgment in the majority of cases. The teams that get real value out of AI in the next few years won’t be the ones who bolt AI onto every step they can find. They will be the ones who took the time to evaluate, workflow by workflow, whether they were building something that helps or something that finishes.

Over the past year, I've been rebuilding parts of an incident response stack for a client, and the biggest surprise wasn't the AI features themselves. It was how much of the underlying workflow had to change to make those features useful. You can't just bolt an LLM onto a 2015-era ticketing tool and call it AIOps. The queue structure, the alert taxonomy, even the way runbooks are written all need to change. I've written before about the agent side of this shift, in AI Agent Architectures: Patterns, Applications, and Implementation Guide and Observability and DevTool Platforms for AI Agents. This two-part series is the other side of that coin: what happens when you point those same agent patterns at your own production systems instead of at somebody else's AI application. Same reasoning loop, different target. This first part covers incident management specifically, including a category I skipped in my earlier tool roundups: dedicated "AI SRE" agents like Traversal, Resolve.ai, and Cleric, which behave differently from the AIOps platforms most of us grew up with. Part 2 goes past incidents into ITOps, chaos engineering, SLO management, on-call toil, and the rest of what fills an SRE's week. A note on the numbers below: vendor-reported accuracy and MTTR figures in this space move fast and come from the vendors themselves. I've flagged those clearly rather than presenting them as independently verified benchmarks. Where SRE Pain Actually Lives Before getting into tools, it helps to remember what SREs spend their time on. Most postmortems I read over the years have the same three complaints: Too many alerts, not enough signalCorrelating five different dashboards to find one root causeWriting the same postmortem summary for the fourth time this quarter None of these are new problems. What's new is that large language models are actually decent at the second and third ones, if you feed them clean data, and a newer crop of agents is starting to chip away at the first one too. The Traditional Incident Pipeline Here's roughly what an incident used to look like before AI got involved, at most mid-size shops I've worked with: Traditional incident pipeline Every arrow in that diagram is a human doing manual correlation work. That's fine when you have ten services. It falls apart at three hundred, and it's part of why I keep coming back to the point I made in Infrastructure as Code: How Automation Evolved to Power AI Workloads: scale problems in ops rarely get solved by hiring more people to stare at more dashboards. Where AI Fits Into the Pipeline Today The shift isn't "AI replaces the engineer." It's AI collapsing steps B through F into something closer to a single triage step, with the engineer reviewing a proposed root cause instead of hunting for one from scratch. AI-aided pipeline Notice the engineer never disappears from this diagram. They just move from being the one who does the correlation to the one who checks the correlation. That distinction matters, because it changes what you hire and train for. I made a version of this same argument about production-grade agents generally in the Shipping Production-Grade AI Agents refcard: an agent needs a human review layer, or you're just moving risk around instead of removing it. If you want a deeper look at how these review loops are actually structured under the hood, I broke that down recently in Loop Engineering: The Layer After Prompt, Context, and Harness Engineering. Incident Management: What Changed A few concrete things have improved in incident management tools over the last two years: Alert correlation got better. Tools like BigPanda, Moogsoft, and PagerDuty's AIOps features now cluster related alerts using pattern recognition instead of static rules. A database timeout, three downstream service errors, and a spike in 500s used to show up as four separate pages. Good correlation engines now group them as one incident with a suggested cause. Similar-incident retrieval works reasonably well. If your org has a decent history of past incidents with clean postmortems, tools can now surface "this looks like INC-4471 from March" with real accuracy. This only works if your postmortem data isn't garbage, which is a bigger blocker than people admit. Draft postmortems save real time. Not because the AI writes a good postmortem on the first try, but because staring at a blank page is the slowest part of writing one. A rough draft built from the incident timeline, Slack thread, and metrics gives engineers something to edit rather than create. What's newer, and worth its own section, is a class of tools that don't just correlate what you already collected. They go get new evidence during the incident, the way a senior engineer would. The New Category: Dedicated AI SRE Agents This is the part of the landscape that's moved fastest since I last wrote about agent tooling. A handful of startups have built agents whose entire job is investigating production incidents autonomously, not just clustering alerts that already exist. Traversal leans on causal machine learning rather than a general-purpose LLM wrapper. Instead of pattern-matching against similar past incidents, it builds a model of causal dependencies across your services and traces the actual chain of cause and effect, down to the specific deploy or config change that started the failure. It reports strong root-cause accuracy in production at large enterprises and is used for both alert triage and live incident investigation. The pitch is narrower than "full AIOps platform," and that narrowness is the point. Resolve.ai takes a broader angle. It was built by the team that created OpenTelemetry, and it positions itself as an agentic teammate across the whole production lifecycle: investigating incidents, but also touching capacity questions, config drift, and guided code changes. Where Traversal is a specialist in root cause, Resolve.ai is closer to a generalist you'd loop in on almost anything production-related, with the incident work as the anchor use case. Cleric sits in a similar space to both, with a specific focus on autonomous alert triage. It runs a multi-source investigation the moment an alert fires, pulling metrics, logs, traces, and deploy history in parallel, and returns an evidence-backed hypothesis before an on-call engineer has finished opening their second dashboard tab. It runs with read-only access by default, which matters a lot for teams still building trust in the category, and it was named a Gartner Cool Vendor in AI for SRE and Observability in 2025. That "read-only by default" design decision is exactly the kind of guardrail I argued for in Trust No Agent: How to Secure Autonomous Tools on Your Machine: an agent's blast radius should be a deliberate design choice, not an afterthought. Causely and NeuBird round out the space with slightly different angles: Causely focuses on causal reasoning to find the single root cause behind a storm of cascading alerts, and NeuBird targets enterprise IT environments with LLM-driven telemetry analysis at large scale. Here's roughly where an AI SRE agent sits in the pipeline compared to the AIOps correlation tools from the last section: AI SRE agent pipeline The key difference from the earlier diagram: this agent isn't just correlating signals you already collected in a dashboard. It's actively going out and querying your systems the way a human on-call engineer would, forming a hypothesis, testing it, and either confirming or discarding it before it ever pages a person. That's a meaningfully different capability than clustering alerts by similarity, and it's why this category gets its own row in any serious comparison. If you're weighing whether to build this kind of investigation loop yourself versus buying one of these platforms, it's worth reading MCP vs Skills vs Agents With Scripts: Which One Should You Pick? first, since the architecture decision behind "agent that calls tools live" versus "agent with a fixed skill set" applies just as much to SRE tooling as it does anywhere else. What This Actually Brings to the SRE Persona It's worth being specific about what changes for the person on-call, not just what the vendor deck claims: Fewer 2 a.m. investigations that start from zero. The agent has usually already ruled out the obvious suspects by the time a human looks at the page, so the engineer starts from a hypothesis instead of a blank terminal.Less tool-hopping. A lot of incident time isn't spent thinking; it's spent switching between Datadog, Grafana, the CI pipeline, and Slack. An agent that queries all of them in parallel removes a genuinely tedious chunk of the job.A written trail for free. Because the agent's investigation is itself a structured log of what it checked and why, you get a decent postmortem skeleton as a byproduct, not a separate task.A new failure mode to watch for. Engineers can start trusting the proposed root cause without checking the evidence trail, especially under pager pressure. That's a habit worth actively training against, not assuming away. None of this replaces the on-call engineer's judgment. It changes the shape of their shift from "gather evidence, then decide" to "review evidence, then decide," which is faster but only as trustworthy as the evidence the agent actually gathered. Comparing the Tool Landscape Here's how some of the major players stack up on where they've actually invested in AI, versus where it's mostly a checkbox feature. I've split this into two tables, because lumping AIOps correlation platforms in with dedicated AI SRE agents hides a real difference in what these tools do. Established AIOps and incident platforms: ToolAlert CorrelationRoot Cause SuggestionAuto-Drafted PostmortemsPredictive CapacityOwnership / StatusPagerDuty (AIOps)StrongModerateYesLimitedIndependent, public companyMoogsoftStrongStrongNoNoAcquired by Dell Technologies (2023)BigPandaStrongModerateLimitedNoIndependent, privateDatadog (Bits AI)ModerateStrongYesModerateBuilt in-house by DatadogServiceNow (Now Assist)ModerateModerateYesStrong (ITOps)Built in-house by ServiceNowDynatrace (Davis AI)StrongStrongLimitedStrongBuilt in-house by Dynatraceincident.ioModerateLimitedYesNoIndependent, privateRootlyModerateLimitedYesNoIndependent, private Dedicated AI SRE agents: ToolCore ApproachActs Autonomously?Best FitFounded / BackingTraversalCausal ML across dependency graphInvestigation autonomous, remediation guardedTeams with strong existing observability wanting sharper RCA2023, Sequoia and Kleiner PerkinsResolve.aiBroad agentic reasoning over code, infra, telemetryInvestigation autonomous, remediation opt-inTeams wanting one agent across incidents, capacity, and config2024, Greylock-led seedClericMulti-source parallel investigationRead-only by defaultTeams new to AI SRE agents, wary of write access2024, Zetta Venture PartnersCauselyCausal reasoning on cascading alertsInvestigation onlyEnvironments with alert storms and unclear blast radiusPrivate, early stageNeuBirdLLM-driven telemetry analysis at scaleInvestigation, guided remediationLarge enterprise IT environmentsPrivate, early stage A caveat worth stating plainly: I haven't run rigorous side-by-side benchmarks on all of these, and vendor claims move faster than reality, especially in the AI SRE agent table where most of these companies are one to three years old and evolving month to month. Treat both tables as a directional map, not a scorecard, and validate against your own alert volume before picking one. Deterministic AI vs. Generative AI in These Tools This distinction gets muddled in vendor marketing, so it's worth separating clearly. AspectDeterministic / ML-based (older AIOps)Generative AI (LLM-based, newer)ApproachStatistical pattern matching, clustering, anomaly detectionLanguage model reasoning over logs, tickets, chat history, and live queriesPredictabilityHigh, same input gives same outputLower, outputs can vary between runsStrengthCorrelation, anomaly detection at scaleSummarization, hypothesis generation, natural language explanation, draftingWeaknessPoor at explaining "why" in plain languageCan hallucinate a plausible-sounding but wrong root causeWhere it shows upDynatrace Davis AI, Moogsoft's original correlation engineDatadog Bits AI, ServiceNow Now Assist, Traversal, Resolve.ai, ClericTrust level neededCan often auto-remediateNeeds human review before action Most modern platforms now run both in tandem: the deterministic layer does the anomaly detection and correlation, and the generative layer explains it in plain English, forms hypotheses, and drafts the writeup. That combination is doing more real work than either piece alone, and it's basically the same pattern I described for agent observability generally in the AI agent architectures piece linked earlier: a fast, boring, reliable layer underneath a slower, flexible reasoning layer on top. Where We're Headed in Part 2 Incident response gets the spotlight because it's the loudest part of the job, but if you track where an SRE's actual week goes, a lot of it isn't firefighting at all. It's chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. In Part 2, I'll walk through where AI is showing up in ITOps specifically, and then go further into chaos engineering, SLO and error budget management, on-call toil reduction, and capacity planning, the quieter parts of the job that determine whether the incident tools in this article even have a fighting chance.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

When we started to work on microfrontend migration on one of our projects, the architecture looked great on paper (like always): one host shell, several remote apps, and teams could deploy independently on their own timelines. But in practice it wasn't so clean. One part kept getting on my nerves: actually mounting remote React components inside the host. Each microfrontend came with the same glue code. Load the remote bundle, create a React root, render the component, keep track of the mounted instance, push updated props into it when the host re-renders, and clean up listeners on unmount. And do not forget to handle load failures. It wasn't especially hard code. But it was just the kind of code nobody wants to repeat. Another problem is type safety, which had a habit of disappearing exactly where I wanted it most. Inside the remote, TypeScript understood the component props perfectly. But at the host boundary, that often collapsed into unknown and as any. If a remote added a required prop or renamed an existing one, the host usually did not find out from the compiler. After doing this a few times across different projects, I decided the pattern deserved a real abstraction instead of one more copy-pasted wrapper. What I Wanted It should be part of my toolkit package and shouldn't be really hard. Something much more practical. The goal was simple: Remove repetitive host-side boilerplateKeep prop types across the host/remote boundaryWork with separate bundles and separate React rootsAvoid shared stores, global registries, and code generationFit into an existing Module Federation setup without changing how remotes are versioned or deployed That idea transformed to @mf-toolkit/mf-bridge. The Base The package has two parts: one wrapper on the remote side, and one host component that takes care of the integration. On the remote side, you define the entry once: TypeScript import { createMFEntry } from '@mf-toolkit/mf-bridge/entry' import { CheckoutWidget } from './CheckoutWidget' export const register = createMFEntry(CheckoutWidget) On the host side, you render the bridge where the remote should appear: import { MFBridgeLazy } from '@mf-toolkit/mf-bridge' <MFBridgeLazy register={() => import('checkout/entry').then(m => m.register)} props={{ orderId, userId } fallback={<CheckoutSkeleton/>} /> That’s all. With MFBridgeLazy, the host doesn’t have to deal with all the hassle of loading things on demand, setting up the root, updating stuff, cleaning up, or handling event listeners — the tool does it all. Plus, because the register function has clear types, the host can automatically figure out what props the remote component needs. If the remote component suddenly needs a new prop, you’ll see a TypeScript error right away during development, not after the app is already live and causing problems. How Prop Updates Travel This was the part I wanted to keep as boring and predictable as possible. Once a remote component is mounted, it lives in its own React root. That means the host cannot simply re-render it as if it were a normal local child. The host still needs a way to send updated props into that remote tree every time its own state changes. There are plenty of ways to solve this: shared stores, shared context, global event buses, custom registries. I wanted the smallest possible mechanism that stayed local to each mounted microfrontend. So `mf-bridge` uses the one thing both sides already share: the mount element. When the host re-renders with new props, the bridge dispatches a `CustomEvent` on that specific DOM element. The remote listens to events on that same element and re-renders with the new props. That is it. I like this approach for a few reasons. First, it is naturally isolated. If you have several microfrontend slots on the same page, each one has its own mount element, so updates do not bleed across instances. Second, it does not need a shared module graph or global state container just to move props around. Third, it keeps the contract very explicit: the host owns the mount point, and the props, and the remote owns how it renders them. Internally, the package wraps this in a small typed DOM event bus, but consumers do not really need to think about those details. Why This Helped More Than Just Saving Lines of Code The obvious benefit is less boilerplate. If a page has five remote slots, I no longer end up with five slightly different wrappers all doing the same lifecycle work. But the bigger benefit is moving problems earlier in the process. Before this, the host/remote boundary was often exactly where type information got blurry. That made one of the most important contracts in the system feel surprisingly fragile. A remote could evolve, and the host would not always know it had fallen out of sync. With mf-bridge, prop inference flows from the remote entry to the host usage. That changes the feedback loop. A contract mismatch becomes a compile-time problem instead of an incident report. There is also a reliability benefit in the lifecycle handling. The package takes care of the repetitive, easy-to-forget parts: Lazy loading with a fallback UIClean mount and unmount behaviorProp streaming on re-rendersListener cleanupError handling when the remote fails to loadOptional preloading and retry behaviorOptional hooks for setup and teardown on the remote side when you need DI or per-mount initialization None of these features are individually groundbreaking. The value is that they come together in one small, reusable bridge instead of being re-implemented in every host wrapper. The Cases I Wanted to Be Sure About When the basic version started to work, I spent a bit more time on some of the scenarios that usually make microfrontend wrappers fragile. One of those cases was multiple instances of the same remote on a single page — a widget in the main content area, a compact version in a sidebar, or the same remote mounted in a few different places. I wanted to make sure what updates stayed local to the exact mount point instead of leaking. Using the DOM element itself as the transport turned out to be a very practical way to preserve that isolation. Another important case was failed loading. I didn't want the host to end up with a blank hole in the UI just because a remote bundle failed on the first attempt. That is why the bridge supports fallbacks, preloading, and retry behavior. I think that kind of thing makes an integration feel solid. And sure, we should not forget about what happens when the problem is rendering. If a remote drops during render, I do not want that failure to destabilize the whole host page. So error handling became part of the design too: we keep the failure contained to the mount point, surface the error to the host, and make recovery possible when new props arrive. Then there is setup and unmount — that case is covered, too. Where It Fits Compared to React.lazy or Portals This package is not a replacement for React.lazy, and it is not trying to be cleverer than React. If your component lives in the same bundle and the same React tree, React.lazy is still the natural tool. If you just want to render into a different DOM node inside the same tree, portals are great. mf-bridge is for the awkward case those tools do not cover well: a component living across a Module Federation boundary, loaded from a separate bundle, mounted into its own React root, but still expected to behave like a first-class part of the host page. That is the gap I wanted to close. A Small Package, Not a New Platform I also cared quite a bit about keeping the package lightweight. It has zero production dependencies and uses the browser's native CustomEvent API for prop streaming. In practice, that means less surface area, fewer moving parts, and one less utility layer to debug when something goes wrong. The goal was never to build a microfrontend platform. It was simply to remove a recurring nuisance and make the host/remote boundary feel safer. Sometimes that is enough to justify a package. I published it as @mf-toolkit/mf-bridge. Repository, docs, and examples: github.com/zvitaly7/mf-toolkit. If you are working with Module Federation and you already have a small pile of hand-written wrappers around remote React components, this may save you some time. And if you have solved the same problem in a completely different way, I would genuinely be curious to compare notes.