Culture and Methodologies Resources

DZone's Featured Culture and Methodologies Resources

From Agile to the Product Operating Model

By Stefan Wolpers

CORE

TL; DR: The “Agile to the Product Operating Model” Survey Results Between August 2 and August 10, 2026, 48 practitioners participated in my Agile to Product Operating Model (POM) survey, which tries to shed light on what is actually changing. Let me summarize the answers for you: the reported transformations change decision-making less than the Cagan framework suggests. Where respondents report improvements, they appear in delivery and collaboration rather than in business results. Unfortunately, the human side of the transition is the least encouraging part of the answers. Thesis: Product operating model transformations mostly change vocabulary and organizational structure while leaving the decision system — who decides what gets built, on what evidence, at what speed- largely untouched. AI is changing product decisions independently of POM transformations. Who Answered, and Why I Report Counts, Not Percentages Of the 48 respondents of the Agile to Product Operating Model, 26 work in organizations that have adopted a product operating model or are moving toward it (we refer to them as the “movers” from here on). The other 22 work in organizations that are not moving; these organizations are still discussing the issue, have decided against it, or have never considered it. So you get absolute counts, and every finding below is directional, not definitive. Additional limitations are: 24 of the 48 respondents are Scrum Masters or Agile Coaches, the roles under the most pressure from this shift.38 of 48 work in organizations with more than 250 people; the sample contains 2 respondents from startups and 2 from scale-ups. Apparently, the people who have participated in the survey are those who still care enough about agile practice to read a newsletter about it; people who left “Agile” entirely are structurally missing. Also, as I did not ask respondents to identify employers, the 48 participants report on their organizations, not 48 distinct verified organizations. Putting it all together, this is a report from inside large organizations, where the product operating model makes its boldest promises. One Fundamental Change in 26 Answers Asked which statement best reflects their experience so far, the 26 movers split like this: 9 say mostly relabeling, the same way of working with new vocabulary9 say real change in some areas, relabeling in others7 say it is too early to tell. Exactly one participant reports a fundamental change in how the organization decides what to build, which is the most interesting learning from the responses: changing the vocabulary, changing the organizational structure, and changing how decisions actually get made are three different things. Better Delivery, Inconclusive Business Results, Worse Morale The Agile to Product Operating Model survey asked movers to compare six outcomes against their previous way of working. Aggregating “somewhat” and “clearly” in each direction: Speed of delivery: Better (9), No change (10), Worse (1), Too early / cannot tell (6)Value delivered to customers: Better (8), No change (8), Worse (2), Too early / cannot tell (8)Business results: Better (3), No change (8), Worse (4), Too early / cannot tell (11)Team motivation and morale: Better (7), No change (4), Worse (9), Too early / cannot tell (6)Developers’ satisfaction: Better (4), No change (8), Worse (7), Too early / cannot tell (7)Collaboration with stakeholders: Better (9), No change (6), Worse (4), Too early / cannot tell (7) The table does not say that the transitions are failing: Delivery speed leans positive: 9 better against 1 worse.Customer value leans positive: 8 against 2.Collaboration with stakeholders leans positive: 9 against 4.Business results are inconclusive, with 11 of 26 unable to say and the rest split. The people dimensions are the only ones that lean consistently negative: morale at 9 orgs is worse than at 7, where it improved; developers’ satisfaction is similar: worse at 7 orgs and better at 4 orgs. Two ways to interpret this information, and the survey does not help to choose: (1) The improvements concentrate on the things enterprises already know how to optimize (flow, coordination, delivery), while the transformations may be falling short of the things the model claims to change most fundamentally. (2) Operational effects precede commercial effects, because business outcomes have longer feedback cycles, and 11 of 26 respondents explicitly cannot yet say. Consider that flow, coordination, or delivery are easier to measure and attribute than commercial effects, which is a significantly fuzzier area. Therefore, the interesting question is how this changes over time, not how it looks at one moment. The range of individual experience behind those aggregates is wide. One respondent, 12 months into adoption at a large financial-services organization, described the hardest part as “surviving the political game that the POM shift has been” and reports worse answers on five of the six dimensions. The other two respondents who have been at 12 or more months report the opposite: better value, better morale, and better collaboration. Three long-tenured accounts cannot agree on whether the practical transition mechanics, culture, affected products and services, or the organization itself makes the difference. (That is the curse of a small sample size.) Agile to Product Operating Model: The Empowerment Gap, and What Settles Uncertainty Cagan’s own test separates empowered product teams from feature teams: in a product team, the team is tasked with solving a problem and owns the solution. In a feature team, “the value and business viability are the responsibility of the stakeholder or executive that requested the feature.” The survey asked movers which pattern operates today, in the middle of their transitions: 9 of 26 report the empowered pattern (leadership sets goals and problems to solve, teams decide what to build)8 of 26 report that leadership decides which features get built and teams implement them.4 of 26 report stakeholder requests are filling the product backlog reactively.4 of 26 say it is genuinely unclear or contested right nowA single one reports teams setting their own direction. So even among respondents whose organizations are actively adopting a model whose entire premise is empowered teams, the “feature-list or reactive-backlog” pattern (12) outnumbers the “empowered” pattern (9). A related question asked what settles it when the organization is uncertain whether something is worth building: 6 said debate and prioritization before anything gets built5 said whoever has the most authority decides5 said things just get built and shipped, and an explicit decision rarely follows. 4 said research evidence5 said it varies too much to say. In total, only five respondents fall into the categories I pre-registered as evidence-led: research evidence, disposable prototypes, or ship-and-measure. There is one case among all the answers: a respondent reports that a disposable AI-assisted prototype reduces build uncertainty: build to learn, decide, and throw it away. That same respondent, 12 months into adoption, also reports better responses across all six outcome dimensions listed above. Of course, one case is not evidence of a relationship. It is a question worth asking in a larger sample, not a claim I am entitled to make here. In Little Code, Big Waste, I argued that cheap AI code removes the cost gate that used to force a should-we-build-this decision, and that “when generating plausible code becomes cheap, every hour spent building the wrong thing becomes waste that can now be produced at scale”. This survey says the prototype-to-decide pattern barely exists yet in this sample. Agile to Product Operating Model Survey Shows: The Two Transitions Run Separately If you believe the keynote version of events, AI is forcing organizations to redesign their operating model around it. The 26 movers report something different. Only one respondent says AI is a core reason for the change; 6 of 26 call it one factor among several; 16 say AI adoption runs in parallel, but separately from, the operating model change; and 3 report little or no role. Now the other side. Among the 22 respondents in non-moving organizations, 15 report that AI is changing how product decisions are made anyway: 6 noticeably, 9 in pockets, without any formal model change. Despite the small sample size, the data support a narrow claim: for most movers in this sample, AI and operating-model transformation are separate initiatives, while for most respondents in non-moving organizations, AI is changing product decisions without any model change at all. Formal operating-model change is neither a prerequisite for AI-driven changes to product decision-making nor, so far, organized around them. The survey did not measure how AI enters these organizations or who authorizes it, so I will not claim more than that. The disconnect between the two is already the finding. My Pre-Survey Hypothesis Scoreboard Here are my verdicts on my four hypotheses, H1-H4, that led to the creation of the Agile to Product Operating Model survey: H1, consultant-driven transitions produce more relabeling than leadership-driven ones: Relatively consistent, but no verdict: 3 of the 4 respondents whose transition is driven by consultants or a transformation office report “mostly relabeling,” against 4 of 16 in product-led or executive-led transitions. But those are just four participants. H2, the empowerment gap: The feature-list plus reactive-backlog patterns (12) outnumber the empowered pattern (9). That is observed in this sample, however, not established beyond it. H3, where AI drives the move, product judgment is the scarcest capability: incorrect on my side: Among the 7 respondents whose organizations treat AI as a core reason or a factor in the move, the most frequent scarcity answer is “delivery capacity is still the bottleneck” (3), ahead of product judgment (2). Across all 26 movers, the scarcity question splits three ways: “stakeholder alignment and decision speed” (7), delivery capacity (6), and product judgment (6). One distinction keeps H3’s rejection from closing the question. The survey measures perceived constraints, and perception rarely reflects reality accurately. AI may have changed the economics of implementation faster than organizations have updated their sense of where the workflow bottleneck sits. Whether delivery capacity objectively remains the constraint is a different question, and this survey cannot answer it. What it can say: practitioners today experience decision speed, delivery, and judgment as roughly competing constraints, and the judgment-scarcity era I anticipated is not the world they report living in. H4, organizations that build without explicit decisions report worse customer value than evidence-led ones: rejected as stated: 1 of 5 build-without-deciding respondents reports better customer value, against 2 of 5 evidence-led ones. Again, the number of replies is too small. Changing the Structure Without Changing the Decision System Three of the findings above belong side by side. Only 1 of 26 movers reports a fundamental change in how build decisions are made. Only 9 of 26 report the empowered decision pattern operating today. Only 5 of 26 fall into the evidence-led categories for resolving build uncertainty. Together, they suggest a more precise diagnosis than “the transformation is theater.” Transformations change three different layers, and the layers move at different speeds: Vocabulary changes first: Product operating model, empowered teams, outcomes over outputs; you get the idea.Structure changes second, and often really does change roles, reporting lines, team topologies, or artifacts.The decision system changes last, if at all: Who decides, based on what evidence, under what uncertainty, with what authority, at what speed. This survey reads like a snapshot of organizations renaming the first layer, reorganizing the second, and leaving the third largely untouched. That is why the 1-of-26 count deserves the weight I put on it earlier. One of the product operating model’s defining promises is a different way of deciding what to build. If only one of the 26 respondents experiences a fundamental change in that mechanism, the question is no longer whether the transformation is proceeding fast enough, but what is being transformed. That mental model offers one possible explanation for the outcome table: structural change could improve coordination and flow before it alters the quality of product decisions. Whether that explains the pattern here is impossible to establish from 26 responses. A long-term study of organizations and involved practitioners would need to test whether decision-system change predicts eventual business outcomes. (Consider that a hypothesis this survey generated, not a result it delivered.) Agile to Product Operating Model: Why Product Washing Is Easier Than Empowerment The survey cannot tell us why the decision system resists change. What follows is my hypothesis, argued from two decades of watching transformations, not a survey result. The tempting explanation is that these organizations are implementing the product operating model badly, and that a proper implementation would deliver. I spent those two decades watching the Agile community run exactly that defense. Every failed adoption was “not real Scrum.” The argument is unfalsifiable, and it taught an entire industry to blame practitioners instead of examining incentives. I will not run the same defense for the product operating model. In November 2024, I described Product Washing: the hollow adoption of product practices that “leaves companies stuck in the same old dynamics but with a new vocabulary,” transformation by reprinting business cards. My hypothesis for the mechanism: product washing is not an implementation failure but what enterprise incentives produce when you ask powerful people to redistribute their own power. The model demands that stakeholders with budget authority hand problem-selection to product leadership and solution-selection to teams. Budget authority is power, and in many large organizations, “product leadership” has become a new title for the same stakeholders who control the money. Meanwhile, middle layers face asymmetric career payoffs: a visible failure damages a career far more than a shared success advances it. Under those payoffs, routing decisions through committees and sign-offs is rational self-protection, and it does not vanish because the org chart was redrawn. Two honest caveats bound this hypothesis: First, in regulated industries, some of that scrutiny is very relevant: when a named person must answer to a regulator, a sign-off chain is accountability, and the respondents from banking and the public sector live with constraints no product operating model erases. The skill worth having is telling required governance apart from accountability theater; most organizations run both and label neither. Second, this survey is a cross-section, not a time series, and two competing explanations fit the same data: The transition hypothesis: decision authority changes slowest of all the layers, and these organizations are simply not there yet.The attractor hypothesis: enterprise incentives pull transformations toward renamed feature factories and hold them there. My incentive argument predicts the attractor. With only 3 respondents at 12 or more months, this survey cannot distinguish between them. That is a testable question for a future survey. The Non-Movers’ Catch-22 The 22 respondents in non-moving organizations deserve more attention than transformation literature usually grants them. Their top reasons for staying put: Leadership sees no need or has other priorities (15),Lacking the product-management maturity to build on (13), andThe cost and fatigue of yet another transformation (10). Only 6 claim their current way of working performs well enough. The dominant non-mover position is not a confident endorsement of the status quo. And inside the reasons sits a genuine Catch-22. Organizations supposedly need the product operating model because their product capabilities are weak, while 13 of 22 respondents say weak product capabilities are precisely why their organization cannot adopt it. A transformation that requires the maturity it promises to create is a hard sell to people who have already survived several that made the same offer. Conclusion: Three Questions for Monday Morning Three questions locate your own organization on this map: First: who decided the last thing your team built, and would your CPO name the same person? If the answers differ, you have found the gap between the model on the slides and the model in operation. Second: what settled your organization’s last genuinely contested build decision: evidence, debate, or seniority? “Whoever has the most authority decides” got 5 votes out of 26 in this survey. Be honest about whether your organization would add a sixth. Third: what would have to be true for a disposable prototype to settle the next contested decision instead? That is a political question, not a technical one: who would have to accept evidence as a tiebreaker, and what would it cost them? Forty-eight answers later, the better question is no longer which operating model organizations adopt. It is how they make decisions when the cost of trying something has collapsed while the cost of deciding has not: Who decides on what evidence, and how quickly can the organization act on what it learns? That is where my work is heading next. More

AI Assist vs AI Complete: The Real Gap in Most AI Workflows Today

By Muralidharan Lakshmanan

A few weeks ago, I participated in a 24-hour AI hackathon where we built a product using AI. Necessary tools were provided, a large number of engineers participated eagerly, and a few business folks also joined to bring their ideas into a real-world product using AI. During brainstorming, people drafted end-to-end process flow diagrams and started working on development, using all the recent available models and platforms to build their product. When the development time window ended, it was time for presentations. As I watched each team present their results, I observed that they couldn’t automate the end-to-end process flow. What they had planned during brainstorming didn’t turn out to be a complete, end-to-end product. Most of the solutions followed the same pattern: they did something in one product and took the output to another product, and the output of that product went somewhere else to finish the loop. Since the transition was handled by a human, it was considered “human in the loop.” No doubt this speeds up the process, but it left a gap — between an AI system that helps finish certain tasks in a process flow and an AI system that finishes the process end to end. Which one is the better model to build? Let’s Try to Define Assist and Complete An assisting system sits inside a workflow. AI takes some kind of input and produces an output. A human then takes the output, reviews it, and decides what to do with it — moving to the next system with or without AI, and continuing the process flow. If we removed the AI from this process flow, it could still run, and the work would still get done. It would just take more time. For example, in a real-world tech process, a business team uses AI to document a requirement, which is converted into a JIRA story using the Confluence AI tool, and a developer reads it and uses Copilot or another AI assistant to develop the code. A complete system owns the workflow’s outcome. Here, the system takes the input, processes it, makes the decision, triggers an action, and writes a log documenting the actions it has performed. If we remove any component from this process flow, it won’t just slow down — it will completely stop. As a real-world example, there is a solution that takes all the reviews for a company, analyzes them, categorizes them, summarizes them, and then raises a ticket with that summary and assigns it to the respective team. As an evolution of this process, teams often look only at their current work and how they can leverage AI in it, rather than looking at the end-to-end process flow integration. This is one of the reasons why AI investment doesn’t show up in the metrics that matter. It was architected to help, for sure, but not to finish. The Four-Part Test After watching this pattern across a number of different workflows, I have formed a set of four questions to address whether a process is structurally ready for a complete system, or whether it’s just going to support an assisting one. Is the trigger unambiguous? Something has to start the cycle — a batch starts on a schedule, a microservice is triggered by a button in the UI, a sensor crosses a threshold. If the start of the workflow is itself a judgment call that varies by who is looking at it, then we don’t have a clear trigger, and the downstream process isn’t ready to be fully automated. Can you define the decision logic? “Define” isn’t simple, but the system’s decision logic can be made explicit, even if the underlying model is complex. In a reliability process flow system, you can define what should be considered an error, what the threshold is, and when to act. Contrast that with deciding whether a customer complaint is about to escalate into a PR problem — the reasoning experienced staff uses to answer that is often genuinely tacit, inconsistent between two employees, and resistant to being written down.Is it executable in a real system? A complete system needs more than just an AI model. It might need an API, a write permission, a button it can press — not a dashboard a human reads and acts on separately based on what they see in it. This sounds obvious, doesn’t it? But teams often build excellent reasoning and never build the hooks to act on it. Can the outcome be verified? This is the most important question of all. After the action completes, can the system check and confirm whether the right action was taken and that it worked? Did the email send, did the case close correctly, did the transaction process without a downstream error? Verifiability is the closing step of a fully automated system, and it’s what makes the system safe. It’s what catches a bad decision before it compounds tenfold. When a workflow answers all four of these questions well, it is a strong candidate for a complete system. When even one question fails, that’s a signal that the workflow should stay an assisting one, regardless of how sophisticated the underlying model is. Why Most Systems Assist But Don’t Fully Complete A simple, straightforward question is why so many production systems stop at assist. In my experience, there are three structural reasons: Narrow decision logic: If a system has narrow decision logic in the process flow — binary answers like yes or no, or a defined set of cases — it can often be handled within the process flow regardless of AI involvement. But people try to introduce AI here anyway, and when it picks an answer that wasn’t pre-defined, that’s treated as a failure.Evidence validation failure: Teams give AI the middle of the workflow because it’s recoverable — a human validates the output at the end and catches problems before real damage occurs. Handing over the full cycle means trusting the outcome, but most organizations haven’t built the measurement discipline to know when that trust is justified. They deploy the AI and measure in time savings, but they never run the harder experiment: tracking what the system’s success rate would actually be if it were allowed to complete the cycle unsupervised. Unresolved accountability: This one is more cultural than technical, but it’s a real blocker. When a human closes the loop, it’s clear who is responsible. But when a system closes it, that question gets harder — especially in any environment where decisions face external scrutiny, like an auditor, a regulator, or a customer dispute. Most organizations keep a human in the closing position by default, regardless of whether that person is actually adding judgment in the majority of cases. The teams that get real value out of AI in the next few years won’t be the ones who bolt AI onto every step they can find. They will be the ones who took the time to evaluate, workflow by workflow, whether they were building something that helps or something that finishes. More

Code Generation Is Solved; Trust Is the Bottleneck

By Jean-Jacques Dubray

From raw manifests to self-service Kubernetes apps: creating enterprise-ready open platforms

By DZone Staff

Incident Management and the Rise of AI SRE Agents

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Building Internal Developer Platforms as Products: A Practical Guide for IDP Architects

Why Most Platforms Fail to Become Products Many companies are heavily investing in internal developer platforms (IDPs) with the expectation that they will speed up delivery and governance, and increase developer productivity. Despite significant investment in Kubernetes, CI/CD, observability, security tooling, and cloud infrastructure, many platforms struggle to gain adoption. The reason is simple: they are built and operated like infrastructure projects, not products. Infrastructure teams are often very focused on technical excellence: automation, scalability, reliability, and compliance. Developers, on the other hand, are interested in a different goal — getting their applications into production quickly and safely without having to go through so much complexity. IDP is successful when developers choose it voluntarily because it makes their lives easier. That shift requires platform architects to think less like infrastructure engineers and more like product managers. Building an IDP is like operating an airport. Nobody travels because they love airports. They travel because they want to reach a destination efficiently. Similarly, developers do not care about Kubernetes clusters, pipelines, secrets management, or observability stacks. They care about shipping features to customers. The platform's job is to make the journey smooth, fast, and safe. This article explores the core practices that differentiate successful product-centric platforms from infrastructure-centric ones. Practice 1: Start With Developer Journeys, Not Technology Choices Imagine constructing a shopping mall by selecting elevators, security systems, and air-conditioning units before understanding customer traffic patterns. The result is often technically impressive but operationally frustrating. The same happens with developer platforms. Architects should first map the customer journey (developer journey) before designing platform capabilities. Many platform initiatives begin with questions like: Which Kubernetes distribution should we use?Which GitOps framework is best?Which CI/CD tool should be standardized? These are important questions, but they should not be the starting point. Successful platform architects begin by understanding developer workflows: How does a new service get created?How long does environment provisioning take?Where do deployment delays occur?What causes support tickets?Which activities are repetitive and manual? The goal is to identify friction and eliminate it. Organizations using platforms based on technologies like Red Hat OpenShift, IBM Cloud Kubernetes Service, or other cloud-native platforms have found that developers adopt only when the platform team focuses on reducing the friction in workflow rather than adding more infrastructure features to the platform. Practice 2: Create Golden Paths, Not Golden Handcuffs A highway encourages drivers to use the fastest route while still allowing exits when necessary. Successful IDPs behave like highways. Developers naturally choose the Golden Path because it is easier and safer than building everything from scratch. One of the most powerful concepts in modern platform engineering is the Golden Path. A Golden Path provides: Recommended architecturesStandard deployment patternsPre-approved security controlsBuilt-in observabilityAutomated CI/CD workflows Developers should be able to move fast along a paved road while retaining flexibility for unique requirements. Platform teams that leverage services from cloud provider environments often realize that standardized self-service templates drive significantly higher adoption than restrictive governance models. Practice 3: Make Self-Service the Primary Interface Every banking transaction once required a visit to a physical branch. Today, customers expect to do everything from a mobile app. Developers hope for the same experience from inside their own software. Nothing kills developer productivity faster than dependency queues. Consider a common case of dependency queues. Open a ticket for infrastructure.Wait for approval.Wait for provisioning.Request secrets.Request monitoring.Request deployment access. Weeks can pass before development even begins. Modern platforms must provide self-service experiences where developers can do the following without opening tickets. Create environmentsProvision databasesConfigure pipelinesAccess observability dashboardsRequest infrastructure resources An IDP should function like a digital banking application—secure, streamlined, and available on demand. Below is the Product-Centric IDP reference architecture. Developers consume platform capabilities through self-service experiences, while the platform embeds security, observability, governance, and delivery capabilities and exposes them through Golden Paths. Practice 4: Treat Platform APIs as Products A power drill might have sophisticated engineering in it. Users judge it by a very simple standard: “Can I drill a hole fast and reliably?" Many platform teams are focused on infrastructure automation and not developer experience. Each API, template, workflow, and portal interaction is a product interface. Questions worth asking include: Is the API predictable?Is documentation clear?Are error messages actionable?Is onboarding intuitive?Can developers discover capabilities easily? Developers evaluate IDPs the same way. They are not interested in the complexity underneath. They care about usability. This principle is especially important when integrating observability services, cloud provisioning layers, or deployment automation platforms. For example, IBM Cloud's managed services can significantly simplify operational complexity, but value is realized only when developers experience that simplicity through intuitive platform workflows. Practice 5: Build Observability into the Platform, Not Around It Imagine when you are driving a car without any speedometer, fuel gauge or warning indicators. You may still reach your destination but the risk increases dramatically. Observability is the dashboard for software systems. Observability is often treated as an afterthought. A team deploys an application and later attempts to add: MetricsLogsTracesDashboardsAlerting This approach creates inconsistency and operational blind spots. Platform teams should embed observability from day one. Every service created through the platform should automatically include: Logging standardsDistributed tracingMetrics collectionHealth monitoringService dashboards Whether organizations use IBM Cloud Observability, Instana, OpenTelemetry, Prometheus, Grafana, or other solutions, the platform should make observability automatic rather than optional. Practice 6: Make Security Invisible but Ubiquitous When entering a modern office building, people rarely think about security. Access badges, surveillance, and emergency controls are built into the environment — the building is secure without requiring employees to become security experts. The same principle applies to IDPs. In immature environments, security is seen as a series of checkpoints, review meetings, manual compliance approvals, vulnerability assessments, and audit evidence collection. Developers find it as friction because it arrives late in the delivery lifecycle. Traditional security models operate as gates. Platform-centric security operates as guardrails. The objective is not fewer security controls — it is fewer manual interactions. Build Secure-by-Default Golden Paths Every new service created through the platform should automatically inherit: Secure CI/CD pipelines with dependency and container image scanningSecret detection and policy enforcementAccess control standards and audit loggingEncryption best practices Automate Policy Enforcement Manual compliance verification is one of the biggest sources of deployment delays. Platform teams should adopt policy-as-code (PaC) approaches that automatically validate deployment configurations, infrastructure standards, and regulatory controls. Instead of asking, "Did someone review this configuration?" the platform asks, "Does this configuration satisfy our policies?" Reduce Security Cognitive Load Developers should not need deep expertise in every security domain. The platform should abstract identity management, secrets management, certificate management, and vulnerability remediation workflows—particularly in hybrid and multi-cloud environments where security complexity grows rapidly. A useful measure of progress: the percentage of security controls inherited from the platform versus manually implemented by application teams. The higher the inheritance rate, the lower the cognitive load. Practice 7: Measure Platform Success Like a Product A gym owner does not measure success by counting treadmills—they measure it by member outcomes. Platform teams should apply the same logic. Traditional infrastructure metrics like cluster utilization, pipeline counts, and resource consumption tell you whether the platform is running. They do not tell you whether it is working for developers. Product-oriented platform teams focus on: Developer satisfactionPlatform adoptionTime to first deploymentDeployment frequencyLead time for changes If developers still circumvent the platform, no amount of technical sophistication matters. The Developer Experience Scorecard Measuring developer experience requires balancing sentiment, effort, and adoption. High-performing platform teams track four key measures: Metric What It Measures How to Collect Developer Satisfaction Score (DSS) Overall platform sentiment Quarterly survey, 1–10 scale Platform NPS Willingness to recommend the platform "How likely are you to recommend this platform?" scored 0–10 Ease-of-Use Score How intuitive common workflows feel Per-task rating, 1–5 scale Developer Effort Score How much work is required to achieve an outcome Survey question on effort per task Together, these reveal not just whether developers are using the platform but whether they genuinely value it. Satisfaction Is a Leading Indicator Most delivery metrics lag behind—deployment frequency (e.g., lead time, incident count) and other metrics. Developer satisfaction is a leading indicator. Developers discover friction long before it is observable from the data. A declining DSS today will result in a decline in productivity and adoption tomorrow. Listening early allows platform teams to respond before problems grow into organizational challenges. The real measure of success is not how many developers use the platform—it is how they feel while using it. The IDP Health Dashboard High-performing platform teams monitor a balanced set of metrics across four categories: Category Metrics Sentiment DSS, Platform NPS, Ease-of-Use ratings Adoption Golden Path adoption, self-service usage, onboarding rates Friction Support ticket volume, documentation search failures, manual approval requests Productivity Time to First Deployment (TTFD), environment provisioning time, lead time for changes A platform succeeds not when developers are forced to use it, but when they prefer to use it. Practice 8: Reduce Cognitive Load Relentlessly The automotive industry spent decades simplifying the driving experience so drivers could focus on reaching their destination rather than understanding the mechanics of their vehicles. IDPs should do the same. As organizations evolve into cloud-native architectures, developers are expected to navigate containers, Kubernetes, CI/CD, IaC, security policies, service meshes, observability tools, and compliance requirements all at once. Each one solves a very important problem individually. As a whole, they overwhelm developers and take focus away from developing business capabilities. A successful platform is not one that exposes every infrastructure capability. It is one that hides unnecessary complexity while providing simple, intuitive paths to outcomes. The goal of platform engineering is not to eliminate complexity. It is to absorb complexity so developers don't have to. Common indicators of excessive cognitive load: Developers struggling to find documentationFrequent support requests for routine tasksLong onboarding times for new servicesMultiple handoffs between teamsTool sprawl across the engineering ecosystem Reduce Tool Sprawl Every tool a developer must learn introduces new interfaces, terminology, documentation, and configuration models. Platform teams should create a unified experience through a developer portal, service catalog, or platform API, that minimizes the number of decisions and interfaces developers encounter. Minimize Context Switching Every transition between tools, teams, or approval processes introduces cognitive overhead. Platform teams should ask: Can this be automated? Can these steps be consolidated? Can approvals be replaced with automated guardrails? The goal is fewer interruptions between code creation and deployment. Platform Teams Are Complexity Brokers Complexity never disappears — it moves. Organizations can either push complexity onto every development team, or centralize and manage it within the platform. High-performing platform teams choose the latter, absorbing operational, security, infrastructure, and compliance complexity so application teams can focus on features. Practice 9: Obsess Over Time to First Deployment The first experience developers have with a platform often determines whether they embrace it or avoid it. Imagine a shopping mall where opening a new store requires twelve forms, multiple approval queues, and manual setup of every utility. Store owners would go elsewhere. The best malls provide ready-made spaces where businesses can start operating almost immediately. Developer platforms should do the same. High-performing platform teams focus relentlessly on Time to First Deployment (TTFD) — the time between creating a service and successfully deploying it. The Biggest Contributors to Poor TTFD Bottleneck Root Cause Fix Manual infrastructure provisioning Ticket-driven approval chains Self-service IaC, service catalogs, platform portals CI/CD pipelines built from scratch No standard templates Pre-built, reusable pipeline templates Security reviews at the end Late-stage compliance gates Shift left — embed scans and policy checks in Golden Paths Observability setup delays Manual metrics/dashboard configuration Auto-provision logging, tracing, and health checks by default Too many decisions Choice overload at onboarding Provide Golden Paths with sensible defaults Measure Every Stage Stage Target Service creation < 5 mins Repository creation Automated Pipeline creation Automated Infrastructure provisioning < 10 mins First build < 5 mins First deployment < 15 mins Observability enablement Automatic TTFD = Provisioning Time + Setup Time + Approval Time + Deployment Time Many organizations discover that approval time is larger than all technical activities combined. The fastest platforms replace approvals with automated guardrails. Practice 10: Build a Platform Community, Not Just a Platform Team Cities flourish when residents contribute feedback and shape growth. Cities planned entirely from a central authority often struggle to meet citizen needs. IDPs are no different. The best platforms evolve through continuous collaboration. Platform teams should create feedback loops through office hours, community forums, developer councils, internal documentation reviews, and experience surveys. Developers become co-creators rather than consumers. Community Health Metrics Running community mechanisms is not enough — each one needs a way to know whether it is working. Track these six indicators to measure community health: Metric What It Measures Healthy Signal Monthly Active Community Members Developers engaging in forums, channels, or office hours Steady growth quarter over quarter Developer-to-Developer Answer Rate % of forum questions answered by non-platform-team members Above 40% indicates a self-sustaining community External Contributions per Quarter Pull requests or documentation edits from application teams Increasing trend Roadmap Items from Community Input % of platform backlog items originating from developer feedback Above 50% signals product-centric culture Office Hours Repeat Attendance Rate % of attendees who return across multiple sessions Above 60% indicates ongoing value Support Ticket Deflection Rate % of issues resolved via community before a ticket is opened Rising deflection reduces platform team toil The ultimate sign of a mature platform community is a change in how developers talk about the platform—from something that happens to them to something they help shape. Practice 11: Think in Products, Roadmaps, and Customer Value Smartphones succeeded because manufacturers continuously improved user experience. Customers did not buy phones because of processor specifications. They bought outcomes—better communication, productivity, and convenience. Developers adopt platforms for the same reason. The strongest indicator that a platform is becoming a product is a change in language. Instead of asking: What infrastructure should we standardize? Platform teams begin asking: What developer problems should we solve next? Which user journeys create the most friction?Which capabilities deliver the highest value?What does our product roadmap look like? Features matter only when they improve the developer experience. Practice 12: Design for Platform Reliability, not Just Application Reliability Imagine a city that invests heavily in building roads, bridges, and public transport for its citizens, but has no maintenance crew, no traffic monitoring, and no plan for when a bridge closes. The infrastructure exists, but without reliability commitments, citizens cannot depend on it. Internal developer platforms face exactly the same risk. Most platform engineering conversations focus on the reliability of applications running on the platform — uptime, error rates, latency SLOs for customer-facing services. What is rarely discussed is the reliability of the platform itself. Yet the platform is load-bearing infrastructure for every engineering team in the organisation. When the CI/CD pipeline degrades, every team's delivery stops. When the service catalog is unavailable, no new services can be provisioned. The platform's reliability is a multiplier — a single failure can simultaneously impact dozens of teams. Define Platform SLOs Before Developers Define Them for You Platform teams that do not define their own Service Level Objectives will find that developers define them informally — through frustration, workarounds, and loss of trust. Effective platform SLOs cover the experiences developers depend on most: Pipeline availability — what percentage of CI/CD pipeline executions succeed without infrastructure-related failures?Provisioning latency — how long does environment or resource provisioning take at the 95th percentile?Portal availability — is the developer portal and service catalog accessible during working hours?Golden Path build time — how long does a standard pipeline template take to complete? These are the experience metrics developers encounter every day. A platform team that publishes and tracks these SLOs operates as a reliable internal service provider. A team that does not is invisible until something breaks. IDP Maturity Model Stage Characteristics Infrastructure Platform Standardized infrastructure, clusters, CI/CD tooling Self-Service Platform Service catalogs, automation, infrastructure on demand Developer Platform Golden Paths, integrated observability and security, DevEx focus Platform Product Platform roadmaps, adoption metrics, developer satisfaction measurement Adaptive Platform Continuous feedback loops, AI-assisted operations, continuous platform evolution Most organizations do not start with a Platform Product. They evolve toward it. The goal of the maturity model is not to reach the highest stage overnight, but to identify the next set of capabilities that will improve developer experience and platform adoption. High-performing platform teams treat platform maturity as a journey rather than a destination. Assessing Your Current Stage To identify where your platform currently sits, ask three diagnostic questions: How do developers access platform capabilities today? If the answer is "by opening a ticket," the platform is at the infrastructure stage. If developers provision resources on demand without human approval, they are at the self-service stage or beyond.Do developers choose the platform voluntarily or use it because they must? Voluntary adoption driven by speed and simplicity signals a developer platform or platform product. Mandatory usage with frequent workarounds signals an earlier stage.Does the platform team maintain a product roadmap prioritized by developer feedback? A yes here is the clearest indicator of a platform product. The absence of a roadmap almost always reflects an infrastructure or self-service mindset. Moving to the Next Stage Each stage has a single dominant unlock that drives progression: Infrastructure → Self-Service: Replace ticket-driven provisioning with self-service automation and a service catalog.Self-Service → Developer Platform: Introduce Golden Paths that embed security, observability, and CI/CD by default.Developer Platform → Platform Product: Establish a formal platform roadmap, measure developer satisfaction (DSS, NPS), and treat developer feedback as a product backlog.Platform Product → Adaptive Platform: Build continuous feedback loops, introduce AI-assisted operations, and invest in platform telemetry that proactively surfaces friction before developers report it. The most common mistake is attempting to skip stages. Teams that build Golden Paths before self-service exists create well-designed paths nobody can access independently. Teams that adopt satisfaction metrics before Golden Paths exist measure friction without the tools to address it. Progress through the stages in order. The IDP Architect's Checklist Before launching any new platform capability, ask: ✅ Does this feature remove friction from a developer workflow? ✅ Can developers access it through self-service? ✅ Is it aligned with a Golden Path? ✅ Is observability included by default? ✅ Is security built into the platform? ✅ Is governance automated rather than manual? ✅ Can success be measured through developer outcomes? ✅ Does it reduce cognitive load? ✅ Does it improve Time to First Deployment? ✅ Would developers choose this platform if they had alternatives? If the answer to several of these questions is "no," the capability is probably infrastructure-focused rather than product-focused. Final Thoughts The future of platform engineering is not about building more infrastructure. It is about delivering better developer experiences. The most successful IDPs combine the discipline of site reliability engineering (SRE), the automation of cloud-native technologies, and the mindset of product management. Whether your foundation runs on IBM Cloud, OpenShift, hyperscaler cloud services, or a hybrid environment, the winning formula remains the same: Treat developers as customers. Treat the platform as a product. Treat developer productivity as the ultimate business metric. When platform architects embrace this mindset, platforms stop being collections of tools and start becoming accelerators of innovation—and that's when platforms truly become products.

By Josephine Eskaline Joyce

CORE

How to Design a Distributed Job Scheduler

Almost every backend eventually needs to run code on a schedule. Send the invoice at midnight. Retry the failed payment in five minutes. Generate the weekly report every Monday at 7 AM. Clean up expired sessions every hour. On one server, this is easy. You write a cron line and move on. The trouble starts when one server becomes ten. Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single “scheduler” box, and that box becomes a single point of failure. Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it. Any job due during that downtime window silently never fires. A distributed job scheduler solves this. It runs jobs reliably across a fleet of machines, fires each job once even when nodes crash, and keeps working when parts of the system fail. This post walks through how to design one, the trade-offs at each step, and the mistakes that bite teams in production. What the Scheduler Has to Do Before drawing boxes, it helps to pin down the requirements. They split into two groups. Functional requirements: Run a job once at a specific time (a one-time job).Run a job on a repeating schedule, usually defined with cron (a recurring job).Support job dependencies, where job B runs only after job A succeeds.Retry a job automatically when it fails.Respect priority, so urgent jobs run before bulk jobs.Cancel or pause a job that is scheduled or already running. Non-functional requirements: Durability. Once the system accepts a job, it must not lose it, even if a node dies one second later.At-least-once execution. Every due job runs at least one time.Scale. The design should handle millions of jobs per day across many workers.Fault tolerance. A crashed worker must not block other jobs, and its work should be picked up by someone else. One requirement is worth calling out early. People often ask for “exactly-once” execution. In a distributed system, you cannot truly get it. What you can build is at-least-once delivery plus idempotent jobs, which together behave like exactly-once from the outside. More on that later. The Core Architecture The single most important idea in this design is to separate deciding when a job runs from actually running it. These are two different problems with different scaling needs, so they become two different components. A clean design has four parts: A scheduler that watches the clock and decides which jobs are due.A queue that holds ready-to-run jobs and hands them out.A pool of stateless workers that pull jobs and execute them.A datastore that holds job definitions and execution history, and acts as the source of truth. Why decouple the queue from the workers at all? Because load is bursty. At midnight, a thousand daily jobs may become due at the same second. If the scheduler called workers directly, that spike would hit them all at once. The queue absorbs the spike and lets workers drain it at a steady rate. It also lets you scale workers up and down without touching the scheduler. This is the same reason queues show up across system design, which I covered in detail in Role of Queues in System Design. Modeling Jobs in the Database The datastore is the source of truth, so the schema matters. A common approach uses two tables. One holds the recurring definition, the other holds individual runs. SQL CREATE TABLE jobs ( id BIGINT PRIMARY KEY, name TEXT NOT NULL, cron TEXT, -- null for one-time jobs payload JSONB, next_run_at TIMESTAMPTZ, -- when this job is next due enabled BOOLEAN DEFAULT TRUE ); CREATE TABLE job_runs ( id BIGINT PRIMARY KEY, -- unique id per run job_id BIGINT REFERENCES jobs(id), status TEXT NOT NULL, -- PENDING, RUNNING, SUCCEEDED, FAILED, DEAD attempt INT NOT NULL DEFAULT 1, scheduled_at TIMESTAMPTZ, started_at TIMESTAMPTZ, lease_until TIMESTAMPTZ ); CREATE INDEX idx_jobs_due ON jobs (next_run_at) WHERE enabled = TRUE; The partial index on next_run_at is the workhorse. The scheduler asks “which jobs are due now” many times per second, and this index keeps that query fast even with millions of rows. Each run moves through a small set of states. Drawing the state machine makes the retry and failure logic obvious. Defining Schedules With Cron Recurring jobs need a way to express “every day at 2:30 AM” or “every 15 minutes.” Cron is still the standard. A classic cron expression has five fields: Plain Text minute hour day-of-month month day-of-week 30 2 * * * -> 2:30 AM every day The Java world often uses Quartz cron, which adds a seconds field at the front and a year field at the end, giving six or seven fields. The two formats look similar but are not interchangeable, and mixing them up is a frequent source of jobs that never fire. The scheduler stores the cron string and computes a concrete next_run_at timestamp from it. After a run is enqueued, it computes the next one. This raises a real question: what happens if the scheduler was down for an hour and three runs were missed? This is the misfire problem. You generally pick one of two policies: Catch up. Run every missed occurrence in order. Correct for billing, expensive for everything else.Skip. Run only the next future occurrence and forget the missed ones. Right for jobs like cache refreshes where stale runs add no value. Make this an explicit setting per job. Teams that leave it implicit get surprised after the first outage. Picking Which Jobs to Run The scheduler needs to find due jobs and hand them off. There are three common ways to find them. Polling. Every second, query the database for jobs where next_run_at <= now(). Simple and reliable. The partial index keeps it cheap. The cost is a small delay, up to your poll interval.Timer wheel. Keep upcoming jobs in an in-memory structure sorted by time. Very precise and great for short delays, but you have to rebuild it from the database after a restart.Push. An external timing service fires an event when a job is due. Real-time, but now you depend on another moving part. For most systems, polling with a one-second interval is the right default. It is boring, and boring is good for a component you are trusting with billing runs. The harder problem is concurrency. If you run several scheduler instances for availability, they will all poll the same table at the same time. Without care, two of them pick the same job, and it runs twice. The clean fix in PostgreSQL is row locking with SKIP LOCKED: SQL SELECT id FROM jobs WHERE enabled = TRUE AND next_run_at <= now() ORDER BY next_run_at LIMIT 100 FOR UPDATE SKIP LOCKED; FOR UPDATE locks the rows this instance selects. SKIP LOCKED tells other instances to ignore locked rows and grab the next free ones instead. Many schedulers can now poll in parallel, each claiming a different batch, with no coordination service and no duplicate pickups. Airflow uses exactly this approach instead of a heavier consensus protocol, which is a good reminder that the simplest mechanism that meets the requirement usually wins. Why Exactly-Once Is a Myth Here is the scenario that breaks naive designs. A worker pulls a job, runs it successfully, and then crashes before it can tell the system “done.” The system still thinks the job is running. The lease expires, another worker picks it up, and the job runs a second time. You charged the card twice. You cannot delete this scenario. Networks drop messages and processes die at the worst moment. So you stop chasing exactly-once delivery and instead make the work safe to repeat. That means two things working together: At-least-once delivery. The system guarantees a due job runs at least one time, accepting that it may occasionally run more than once.Idempotent jobs. Running the same job twice has the same effect as running it once. The standard trick is an idempotency key built from stable identifiers, for example {job_id, run_id, attempt}, or a key tied to the business action like invoice_2026_06_charge. The worker records that key before committing side effects. If the same key shows up again, the worker sees the work is already done and acknowledges without repeating it. This is why each run gets its own unique id. A time-ordered id such as a Snowflake id or a ULID works well, because it is unique across the whole fleet without coordination and it sorts by creation time, which keeps the job_runs table naturally ordered. I explained the structure of these ids in How Snowflake IDs Work, and the deduplication pattern itself in Idempotent Receiver Pattern. There is one more subtle gap. The worker has to update the database and publish to the queue, and those are two systems. If it writes to the database and then dies before publishing, the job is lost. The transactional outbox pattern closes this gap by writing the job and an outbox row in one local transaction, then publishing from the outbox separately. I covered that in The Transactional Outbox Pattern. Coordinating at Scale A single scheduler instance has a throughput ceiling. Past a certain number of jobs per second, one process polling one database cannot keep up. There are two ways to grow. The first is leader election. You run several scheduler instances, but only one is active at a time. The others stand by and take over if the leader dies. A coordination service like etcd or ZooKeeper holds the leadership lock. This is simple to reason about, but the single active leader is still a throughput bottleneck. The second is sharding. You split the job space across many active schedulers. A simple scheme hashes the job id into one of N partitions, and each scheduler owns a set of partitions. Every job has exactly one owner, so there are no duplicate pickups, and throughput grows by adding schedulers. Consistent hashing makes it cheaper to add or remove schedulers without reshuffling everything. Sharding has one sharp edge. During a handover, while leases for a partition are changing hands, two schedulers can briefly believe they own the same partition. This is split brain. You do not try to make it impossible, because that is expensive. Instead, you let the worker-side idempotency check be the final safety net. If both schedulers enqueue the same run, the idempotency key means it still executes once. Google’s cron service takes a stricter route for its most sensitive launches. It writes the launch record to a quorum using Paxos before the job actually starts, so a failover cannot lose or double-fire it. For most teams, leases plus idempotency are enough, and full consensus is overkill. Detecting Failures and Recovering Workers crash. The scheduler has to notice and reassign their work, without stealing jobs from workers that are simply slow. The mechanism is a lease with a heartbeat. When a worker claims a run, it sets lease_until to a short time in the future, say 30 seconds. While the job runs, the worker periodically extends the lease. If the worker dies, it stops extending, the lease expires, and a recovery sweep moves the run back to PENDING so another worker can take it. SQL -- recovery sweep: reclaim runs whose lease has expired UPDATE job_runs SET status = 'PENDING' WHERE status = 'RUNNING' AND lease_until < now(); Two details make this robust. First, the lease timeout must be comfortably longer than a normal heartbeat interval, or a brief pause will cause a healthy job to be wrongly reclaimed. Second, you need protection against a zombie worker, one that froze on a long garbage collection pause, lost its lease, and then woke up and tried to finish writing results. A fencing token solves this. The reclaimed run gets a higher token, and the datastore rejects any write carrying an older token. I went deeper on time-bound ownership and fencing in The Lease Pattern in Distributed Systems. Retries Done Right A failed job should usually be retried, but retrying badly makes outages worse. If a downstream service is struggling and every failed job retries immediately, you pile on more load at the exact moment it can least handle it. The fix is exponential backoff with jitter. Each retry waits longer than the last, and a random jitter spreads the retries out so they do not all fire at the same instant. Plain Text attempt 1 fails -> wait ~1s attempt 2 fails -> wait ~2s attempt 3 fails -> wait ~4s attempt 4 fails -> wait ~8s (each wait randomized by +/- a few hundred ms) After a fixed number of attempts, stop. A job that keeps failing should not retry forever. Move it to a dead letter queue, a separate place for runs that exhausted their retries, and alert a human. The dead letter queue keeps a poisoned job from clogging the pipeline while preserving it for investigation. Operating the Thing A scheduler is infrastructure other teams depend on, so it has to be observable and controllable. For observability, track the metrics that tell you the system is healthy: Queue depth. A queue that keeps growing means workers cannot keep up.Scheduling lag, the gap between when a job was due and when it actually started.Run outcomes per minute, split by succeeded, failed, and dead.Lease reclaims, which spike when workers are crashing. For control, give operators real knobs. They should be able to pause a queue, drain a worker before a deploy so it finishes current jobs and takes no new ones, and replay a dead-lettered job after fixing the cause. Building these in from the start saves a lot of pain during the first incident. How Real Systems Approach This None of this is theoretical. The same building blocks show up across well-known tools, each making a different trade-off. Quartz. A mature Java scheduler. Multiple instances coordinate through a shared database using row locks, the same idea as the SKIP LOCKED approach above.Airflow. Orchestrates dependency graphs of tasks. Its scheduler uses database locks rather than a consensus protocol, favoring operational simplicity.Temporal. Models workflows as code and replays an append-only event history to recover state after a crash, which sidesteps a whole class of mid-task failure bugs.Celery. A popular task queue in Python, with a beat component that handles periodic scheduling.Kubernetes CronJobs. Run containerized jobs on a cron schedule inside a cluster, with configurable policies for missed runs and concurrency. See the Kubernetes CronJob docs.Google distributed cron. Writes launch state to a Paxos quorum before launching, so a leader failover never loses or doubles a run. The pattern across all of them is consistent. Decouple scheduling from execution, lean on the database or a quorum for coordination, accept at-least-once and make jobs idempotent, and design for failure as the normal case. Takeaways If you remember five things from this, make it these. Separate the decision of when a job runs from the work of running it. They scale differently.Do not chase exactly-once. Build at-least-once delivery and make every job idempotent.Use the database as a coordination primitive. SELECT ... FOR UPDATE SKIP LOCKED lets many schedulers poll safely.Use leases with heartbeats and fencing tokens to detect dead workers and reclaim their runs without double execution.Retry with exponential backoff and jitter, cap the attempts, and send the rest to a dead letter queue. A good scheduler is not clever. It is careful. It assumes nodes will die, messages will duplicate, and clocks will drift, and it keeps running anyway.

By Ajit Singh

Build Your Own Local AI QA Engineer With Docker, Ollama, LibreChat, and Playwright MCP

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri

CORE

How I Built a Star Wars Grogu Product Research Agent With Codex, Lark, and SerpApi

Cross-border e-commerce sellers often spend hours comparing the same products across different Amazon marketplaces. Prices, reviews, and seller signals vary by country, but the process is still largely manual. I wanted to see how far I could automate it with a small AI agent built using Codex, SerpApi, and Lark. Suppose I ask in Codex: Compare Grogu products in the US and Japan The workflow is straightforward: Search Amazon marketplacesFetch product detailsSummarize seller signalsSend a report to Lark I kept the stack intentionally simple: Python 3.12uvPydanticSerpApi Python SDKOpenAI structured outputsLark custom bot webhookCodex for both development and as the conversational interface Running the Agent From Codex Once everything was wired together, I could simply ask Codex: Compare Grogu products in the US and Japan Codex understood the request, triggered the CLI workflow, collected marketplace data via SerpApi, and delivered a structured report to Lark. Here’s the entire flow in action: The Workflow The overall flow looks like this: Starting with a natural-language request in Codex, the agent first validates that the question is related to cross-border Amazon research. OpenAI then translates the request into a structured command. SerpApi handles both product discovery and detailed product retrieval, while OpenAI extracts seller-focused insights from the collected data. Finally, the results are packaged into a Lark card and delivered via a Custom Bot webhook. I intentionally did not start with FastAPI, background jobs, or a database. For the MVP, the important question was simpler: Can I go from a natural-language product question to a useful cross-border product card? Translating Natural Language into Commands The natural-language entry point uses OpenAI to translate user requests into structured commands. For example: Compare Grogu products in the US and Japan becomes: YAML query="Grogu" marketplaces=["us","jp"] output_mode="send_lark" Using a strict schema keeps the pipeline predictable and much easier to debug. Instead of letting the model orchestrate everything, I only ask it to generate structured commands. Searching Amazon With SerpApi This project relies on two SerpApi endpoints. The Amazon Search API is used to discover candidate products, while the Amazon Product API enriches them with much richer details. Search results provide ranking context and thumbnails, while product pages contain detailed information such as images, availability, and descriptions. Combining both produced much better product cards than using either endpoint alone. Modeling Product Data Amazon pages are messy. Some products have ratings but no availability. Some have images but no variants. Some fields simply don't exist. Missing data is normal, not an exception. My first instinct was to say, "Why not model most fields as optional?" That is true, but it is not the whole solution.The real issue was that SerpApi returns useful product data from several different places. Some fields are flat. Some fields are nested. Some fields have different names depending on whether they came from Amazon Search API or Amazon Product API. Some products should not be shown at all if they are missing the signals a seller actually needs. To make the data usable, I: Normalized both endpoints into a common Product model;Filtered out products with weak seller signals;Merged Product API details back into search results;Treated missing fields as expected rather than failures. The Product model still uses optional fields, because missing data is normal: But optional fields alone were not enough. I also filtered search results before choosing products for detail lookup: This was important for the Lark card. A cross-border seller does not want a table full of Rating: N/A and Reviews: N/A. Those rows make the card noisy and less actionable. The next issue was nested and inconsistent JSON. For price, SerpApi may return a string, a number-like value, or a nested object: For availability, the Product API does not always use one stable field. I had to check availability, stock, and sometimes delivery: Keeping the Agent Narrow The most interesting part of the project isn't Grogu. It's scope. The entry point only supports cross-border Amazon product research. If somebody asks: What is today's weather? the app simply rejects the request. Requests outside the project scope are rejected locally before calling OpenAI. For supported requests, OpenAI returns a strict command object: This keeps the agent predictable. It translates requests into commands instead of improvising actions. I deliberately avoided turning this into a general chatbot. The agent only knows one workflow: cross-border Amazon product research. That narrow scope makes the behavior easier to explain, test, and trust. It also keeps OpenAI API responsible for translation rather than improvisation. Generating Seller-Friendly Insights and Delivering Them to Lark Once product data has been collected, OpenAI generates seller-focused insights. I intentionally constrain the model to use only information returned by the APIs, avoiding hallucinated prices, ratings, or availability. Instead of producing generic summaries, the analysis focuses on demand signals, social proof, pricing, and obvious risks — information that is much more useful for sellers. The final result is delivered through a Lark Custom Bot webhook. Error Handling External APIs fail. That's normal. I treated each layer independently. If OpenAI fails, no command is generated. If SerpApi fails, the analysis stops with a clear message. If Lark delivery fails, the report can still be viewed locally. This separation keeps failures localized and prevents one component from bringing down the entire workflow. Conclusion The most interesting lesson from this project wasn’t the Grogu theme or the Lark card. It was learning where the boundaries should be. The agent works because it stays narrow. That narrowness makes the system easier to understand, debug, and trust. And if you’re building tools for cross-border e-commerce, SerpApi’s Amazon API provides a surprisingly rich source of product data. They made this entire workflow possible. If you’re working on product research, seller analytics, or marketplace intelligence, I’d recommend giving them a try. Check out the full SerpAPI article collection here.

By Magenta Qin

AI in SRE: A Practical Autonomy Model for Self-Healing Infrastructure

Most SRE teams do not need another dashboard. They need a safer way to move from "something is wrong" to "we know what to do next." A model that detects anomalies is useful. A model that can touch production can also make a bad incident worse. That is where most conversations about AI in SRE become too optimistic for my taste. The hard part is not only detection. It is deciding how much autonomy the system should have, under which conditions, and with what blast-radius controls. I learned this while working on large-scale cloud services where one customer-facing symptom could turn into a flood of alerts. A degraded dependency might show up as latency in one service, retries in another, queue growth somewhere else, and CPU pressure downstream. During an on-call shift, that can look like five separate problems. Usually, it is one problem echoing through the stack. That experience changed how I think about self-healing infrastructure. The goal is not to build a system that blindly fixes everything. The goal is to build an operational control loop that can separate routine, low-risk recovery from incidents that still need human judgment. The model that has worked best for me is graduated autonomy: Let the system act automatically only when the action is well understood, reversible, and narrow in blast radius. For everything else, the system should collect evidence, recommend the next step, and keep humans in control. Why Static Alerts Stop Scaling Static alerts are not the enemy. I still want to know when disk usage is dangerous, error rates spike, or latency crosses a service-level threshold. But thresholds do not understand context. A CPU spike during a scheduled batch job may be normal. The same spike during steady-state traffic may be a retry storm. A latency increase in one region may be harmless during a controlled deployment, but suspicious if it appears across multiple availability zones with no recent change event. At small scale, engineers can carry that context in their heads. At enterprise scale, they cannot. Services emit hundreds of metrics across regions, dependencies, deployments, and customer paths. Eventually the team is no longer tuning alerts. It is negotiating with noise. In one rollout I was involved with, the most useful improvement was not adding more alerts. It was grouping alerts around dependency context and suppressing repeated downstream symptoms. The on-call experience became calmer because engineers could focus on the likely failure path instead of chasing every red graph independently. That is the kind of problem AI can help with. Not by replacing SRE judgment, but by organizing noisy signals into a more useful operational story. Detection Is Only the First Layer ML-based anomaly detection helps because it learns a service's normal operating shape instead of relying only on fixed thresholds. For cloud metrics, that usually means learning seasonality, traffic cycles, deployment windows, regional differences, and service-specific behavior. An LSTM autoencoder, isolation forest, or well-tuned statistical baseline can all be useful. I care less about the model family than the quality of the telemetry around it. A simple model trained on clean, consistent data will usually beat a sophisticated model trained on messy metrics. A practical anomaly pipeline usually looks like this: Collect metrics, logs, traces, and change events.Normalize them by service, region, dependency, and time window.Score each signal against its learned baseline.Group anomalies by dependency graph and recent changes.Produce an evidence bundle for automation or human review. Here is a simplified version of the scoring stage: Python from dataclasses import dataclass from typing import List @dataclass class MetricWindow: service: str region: str signal: str values: List[float] recent_deploy: bool = False @dataclass class AnomalyScore: service: str region: str signal: str score: float reason: str class BaselineModel: def expected_range(self, service: str, region: str, signal: str): # In production, this may come from a trained model, # feature store, or rolling baseline per service and region. return (0.0, 1.0) def score_window(window: MetricWindow, baseline: BaselineModel) -> AnomalyScore: low, high = baseline.expected_range( window.service, window.region, window.signal, ) latest = window.values[-1] if latest > high: distance = (latest - high) / max(high, 0.001) reason = f"{window.signal} above learned baseline" elif latest < low: distance = (low - latest) / max(abs(low), 0.001) reason = f"{window.signal} below learned baseline" else: distance = 0.0 reason = "within learned baseline" if window.recent_deploy and distance > 0: reason += " during recent deployment window" return AnomalyScore( service=window.service, region=window.region, signal=window.signal, score=min(distance, 1.0), reason=reason, ) The production value is not just the score. It is the metadata around it: ownership, dependency path, recent deploys, feature flag changes, customer impact, and whether the same pattern has appeared before. A single anomalous metric should rarely trigger remediation. Sustained anomalies across correlated signals are more trustworthy than one spike in one chart. Correlation Turns Noise Into an Incident Story During an incident, the useful question is not "Which graph is red?" It is "What changed first, and what depends on it?" That is where dependency-aware correlation becomes more useful than raw anomaly detection. A database issue may surface as API latency, retries, queue saturation, and CPU pressure. Without a dependency graph, every downstream service looks guilty. With one, the system can rank likely causes instead of handing the engineer a wall of symptoms. A useful correlation engine should look at topology, timing, change context, and customer impact. Which dependency failed first? Was there a deployment or config change? Which service is closest to the customer-facing error? The evidence bundle should be readable by a human. If the model says "root cause confidence: 0.86," that is not enough. It should also explain why. JSON { "candidate_root_cause": "identity-token-cache", "region": "example-region-1", "confidence": 0.86, "customer_impact": "elevated authentication latency for a subset of requests", "supporting_signals": [ "p99 latency above learned baseline for multiple consecutive windows", "cache hit rate dropped below its recent operating range", "downstream services showed retry growth after the initial cache anomaly", "no database saturation was observed", "no deployment was detected in the immediate incident window" ], "recommended_action": "drain_and_restart_one_cache_node", "estimated_blast_radius": "single node in a redundant pool", "rollback_plan": "keep node out of rotation if health checks fail after restart" } This is more useful than another alert. It gives the on-call engineer a starting hypothesis and the reasoning behind it. The Graduated Autonomy Model The most important design decision in self-healing infrastructure is not which ML algorithm to use. It is which actions the system is allowed to take. I divide remediation into three tiers. Tier 1: Fully Automated, Low-Risk Actions Tier 1 actions are safe, reversible, and narrow in blast radius. These are actions the system can execute without waiting for a human when confidence is high. Examples include restarting one unhealthy instance, scaling out a stateless service, draining one bad node, flushing a bounded cache, or shifting a small amount of traffic away from a degraded zone. The key phrase is bounded blast radius. Auto-remediation should not restart half the fleet, fail over a primary database, or disable a feature globally just because a model is confident. Confidence is not a substitute for safety. Before I put an action in Tier 1, I expect it to pass these checks: it is reversible, affected capacity is small, redundancy is healthy, there is no active global incident, the same action has not failed recently, rollback is defined, and health checks can verify success quickly. The first Tier 1 actions should be boring. Restarting one unhealthy node is not exciting, but it is exactly the kind of action that can be automated safely when the system has enough evidence. Tier 2: Automated Recommendation With Human Approval Tier 2 is where many real incidents live. The system may know what should happen, but the action still needs human approval. Examples include rolling back a deployment, disabling a feature flag, failing over a database, increasing capacity beyond a normal band, or changing regional routing. For Tier 2, the system should prepare the action, show the evidence, and ask for approval. The human should decide whether the action makes sense, not build the command during the incident. One pattern I have seen repeatedly: the slowest part of remediation is not always finding a likely cause. It is gathering enough confidence to take a risky action. When the system attaches deploy timing, error movement, affected endpoints, config changes, and rollback commands into one review card, the decision becomes easier. Tier 3: Human-Led With AI Context Tier 3 incidents are novel, high-risk, or ambiguous. The system should not execute remediation. It should help humans reason. This includes possible data corruption, multi-region cascading failures, security-sensitive incidents, conflicting signals across dependencies, low-confidence root-cause analysis, or any action with unclear rollback behavior. In Tier 3, the system's job is to summarize what it knows, what changed recently, which hypotheses are most likely, and which dashboards or runbooks are relevant. That alone can save time, but it keeps production control where it belongs. Architecture: A Control Loop, Not a Magic Button A practical self-healing system looks like a control loop with guardrails. Architecture diagram: Graduated autonomy model for self-healing infrastructure The important part of this diagram is the policy gate. Detection and correlation produce a recommendation, but the policy gate decides autonomy. Without that layer, "self-healing" becomes a risky automation script with an ML label attached. The policy gate should evaluate confidence, risk, blast radius, recent action history, service criticality, and rollback readiness. I would express that as policy-driven code: JSON from dataclasses import dataclass from enum import Enum from typing import List class Decision(str, Enum): AUTO_EXECUTE = "auto_execute" REQUEST_APPROVAL = "request_approval" HUMAN_LED = "human_led" @dataclass class RemediationProposal: action: str confidence: float blast_radius_percent: float reversible: bool rollback_defined: bool service_tier: str evidence: List[str] @dataclass class RuntimeContext: active_global_incident: bool recent_failed_action: bool healthy_redundancy: bool minutes_since_last_same_action: int TIER_1_ACTIONS = { "restart_single_instance", "scale_stateless_service", "drain_single_node", "flush_bounded_cache" } TIER_2_ACTIONS = { "rollback_deployment", "disable_feature_flag", "database_failover", "regional_traffic_shift" } def decide_autonomy( proposal: RemediationProposal, context: RuntimeContext ) -> Decision: if context.active_global_incident: return Decision.HUMAN_LED if context.recent_failed_action: return Decision.HUMAN_LED if not proposal.rollback_defined: return Decision.HUMAN_LED if proposal.action in TIER_1_ACTIONS: safe_enough = all([ proposal.confidence >= 0.90, proposal.blast_radius_percent <= 5.0, proposal.reversible, context.healthy_redundancy, context.minutes_since_last_same_action >= 30, len(proposal.evidence) >= 3, ]) return Decision.AUTO_EXECUTE if safe_enough else Decision.REQUEST_APPROVAL if proposal.action in TIER_2_ACTIONS and proposal.confidence >= 0.75: return Decision.REQUEST_APPROVAL return Decision.HUMAN_LED This is not drop-in production code, but the structure is the point: actions are classified, confidence is not the only input, and safety can override the model. In reliable systems, the model proposes; policy disposes. What I Measure Before Expanding Autonomy I would not start by asking, "Can we automate remediation?" I would start by asking whether the system's recommendations are trustworthy. Before allowing Tier 1 execution, I would track root-cause precision, false positives by service, recommendation acceptance, time to useful diagnosis, remediation success, rollback frequency, and any secondary incidents caused by remediation. The last two matter the most to me. A self-healing system that fixes one issue but creates another is not healing. It is moving the incident. My preference is to run in shadow mode first. Let the system detect, correlate, and recommend, but do not let it execute. Compare its recommendations against what engineers actually did. Once the system repeatedly recommends the same low-risk actions humans already take, graduate those actions into Tier 1. That is how trust gets built: not through a big launch, but through repeated correctness in narrow, well-understood situations. Lessons Learned From Building Toward Self-Healing The most useful lessons are not about model architecture. Clean telemetry beats clever models. If service names are inconsistent, regions are missing, logs are unstructured, and ownership metadata is stale, the model will struggle. Before debating LSTMs versus transformers, fix the telemetry pipeline. Change events are first-class signals. Deployments, config pushes, schema changes, and feature flag flips explain many anomalies. If the model cannot see change events, it will treat every incident like a mystery. Alert suppression is not the same as diagnosis. Reducing noise is useful, but the system must preserve the causal path. Suppressing duplicate downstream alerts only helps if the upstream root cause remains visible. Automation needs a memory. Every remediation should leave an audit trail: what was detected, what action was taken, what happened afterward, whether rollback was needed, and whether humans agreed with the recommendation. Start with boring actions. Restarting one bad instance is not glamorous. Draining one node is not a research breakthrough. But these are exactly the kinds of actions that make sense for early autonomy because they are repeatable, reversible, and easy to verify. Where LLMs Fit Large language models are useful in SRE, but I would not put them directly in the execution path for remediation. Their best role is communication and context assembly. An LLM can draft an incident summary, explain the evidence bundle, turn raw telemetry into a timeline, identify runbooks, and prepare a post-incident report. That saves time without giving the model direct control over production. The safer pattern is separation of responsibilities: ML or statistical models detect anomalies, graph correlation ranks likely causes, policy gates decide autonomy, deterministic automation executes approved actions, and LLMs summarize what happened. That separation keeps the high-risk parts deterministic and auditable while still using AI where it helps most. Final Thought Self-healing infrastructure is not about removing SREs from production. It is about removing the repetitive, low-risk work that slows them down during incidents. The best version of AI in SRE is not a magic system that fixes everything. It is a careful control loop: detect early, correlate intelligently, act only within policy, and learn from every outcome. If you are building toward self-healing, do not start with full autonomy. Start with evidence. Then recommendations. Then approval-based actions. Then, only after the system has earned trust, allow narrow automated remediation. That path is slower than the hype cycle, but it is much closer to how reliable infrastructure actually gets built.

By Shraddhaben Gajjar

Why Developers Must Be Part of the Customer Validation Process

Agile has made it faster to design, build, and ship features. Teams track sprints, display burndown charts to show progress, and conduct sprint retros. With the inclusion of AI in almost every phase of software development, teams are equipped to accelerate feature delivery even more. But all these improvements in delivery time haven’t really increased customer satisfaction or led to a greater rate of adoption among customers. Perhaps it’s because, at its core, being agile alone does not help software development teams develop features iteratively. We’re still in waterfall when it comes to requirements — they are created by product managers and handed over to the software developers, who create and ship these features without any real feedback from customers. Customer validation and feedback must be intentionally included in agile sprints, and software developers must be a part of this process. Understanding what problems a feature can solve and watching how customers may use a feature can empower developers to think outside the box and deliver features that are easy for customers to learn, adopt, and use. But how can teams realistically achieve this? Talking to and observing how customers behave is time-consuming and can take away time from actual development, leading to reduced velocity and longer delivery times — none of which are acceptable outcomes. These activities are also very much in the product management wheelhouse. Hence, developers must work with the product management team to ensure they are getting the maximum learning in the most efficient way. This is where the concept of build-measure-learn cycles can be folded into the software development process. What Is a Build-Measure-Learn (BML) Cycle? A BML cycle is a continuous feedback loop that provides a disciplined way to reduce waste, validate assumptions quickly, and ensure we’re delivering features customers will use. Build a set of features that allow you to test full functionality or a meaningful portion of itMeasure how customers behave and use the featureLearn whether to continue, pivot, or in some cases abandon the feature in question Set Up of a BML Cycle Duration A build, measure, and learn cycle is generally a fixed number of sprints. If you’re unsure of how many sprints/weeks should make up a build, measure, and learn cycle, start with 2 sprints or 4 weeks. 4 weeks is generally sufficient time to ensure there is meaningful progress to demo to customers. Goals Every BML cycle must have a defined goal. For example, if you’re building a new application, a goal can be to demonstrate the menu bar and placement of all elements of the menu bar. If a feature is very big, instead of waiting till the end to demo the entire feature, BML cycles can be set up to demo parts of the feature to ensure feedback is incorporated early. Participants Choose a set of customers who will be your BML partners; this step is owned and executed by the product manager. Choosing too many customers can lead to feedback fatigue mid-sprint, and choosing too few can cause feedback to be skewed towards a certain segment of customers. Choosing 5–6 customers who represent 80% of your user segments is a great way to ensure that BML cycles produce meaningful feedback. BML customers are generally a subset of your larger alpha and beta customer cohort. These are customers who are engaged and will give you feedback on a regular basis. Role of Engineers in a BML Cycle The role of engineers in a build-measure-learn (BML) cycle is not just to build the feature; they are active partners in customer demos during the measure phase and important stakeholders making decisions during the learn phase. Build — Engineers Shape the Demo, Not Just the Code Engineers are the stars of the show during the build stage. They understand the requirements, write the code, peer review code, run research spikes for future sprints, and locally test their code before it is handed over to QA. Engineers decide and direct how much of the feature/product is being built and what can be shown in a demo to customers. They can also quickly wire a feature just for the demo to get early insights from customers without having to build the entire feature if the team is undecided on certain elements of the feature. Measure — Engineers Observe, Instrument, and Interpret Signals In a BML demo, engineers are not silent observers — they are data collectors and pattern recognizers. They learn and start to think about making the feature better by: Watching how users interact with the prototype and understanding how to make the feature more usableIdentifying edge cases that Product Managers may missInspecting system behavior and identifying latency, misfires, and false positivesAsking clarifying questions on workflow to deduce technical constraints that need to be resolved Engineers often notice user behavior and draw conclusions such as: The user paused before clicking — the UI needs to be more intuitiveThe model misclassified because the lighting in the user’s environment is different These observations are essential to ensure that the final product delivered meets customer expectations. Learn — Engineers Help Decide Whether to Persevere or Pivot on Key Architecture Constraints After the demo, engineers help translate observations into technical insights and challenges. They contribute by: Explaining to the larger team why certain behaviors such as model limitations, workflow mismatches, or timeouts occurredIdentifying what’s easy vs. hard to change and thereby influencing what is feasible for MVP timelinesHelping refine the goals and objectives for the next BML cycleHelp prioritize fixes or feature changes based on impact and effort Having engineers as part of the BML cycle builds trust with customers by showing credibility and assuring customers that the product is being built by experts. Engineers encourage an atmosphere of transparency on what is achievable quickly versus what needs time to be built. They also create a collaborative atmosphere where customers are motivated to be more forthcoming because they feel heard by the people who build the system.

By Susan Isaac

Lift-and-Shift vs. Modernize: A Decision Framework for Enterprise Workloads

One of the most consequential decisions in any enterprise cloud migration is deceptively simple to state and surprisingly hard to answer: do we move the workload as-is, or do we modernize it first? Having worked through cloud migrations across dozens of enterprise customers spanning both AWS and Azure. I can tell you this question rarely has a universal answer. The right path depends on the workload, the business context, and the maturity of the team inheriting it in the cloud. What follows is the decision framework I use when guiding customers through this choice. Understanding the Two Paths Lift-and-shift (also called rehost) means moving a workload to the cloud with minimal or no code changes. You are essentially taking an on-premises virtual machine (VM), an application server, or a database and running it on cloud infrastructure instead. Tools like Azure Migrate and AWS Migration Hub (Application Migration Service, or MGN) are purpose-built for this. Modernization is a broader term that can mean refactoring an application to use cloud-native services (databases-as-a-service, managed Kubernetes, serverless functions), re-platforming to a container-based architecture, or rebuilding from scratch as a microservices application. The spectrum between these two poles includes re-platforming, for example, moving a SQL Server workload to Azure SQL Managed Instance, which preserves the database engine behavior while offloading infrastructure management. This middle path is often underrated. The Core Tension Lift-and-shift is fast and low-risk. You can move a workload in weeks, not months. Your teams do not need to rearchitect anything. Applications continue to behave exactly as they did on-premises. The downside is that you carry your technical debt into the cloud. A poorly designed, resource-hungry application that cost you money on-premises will likely cost you more in the cloud, where idle compute is billed by the hour. You also miss out on cloud-native capabilities: autoscaling, managed resilience, and pay-per-use economics. Modernization promises better long-term economics and agility. But it is expensive up front, requires skill sets your team may not yet have, and introduces real delivery risk. Projects that start as modernization efforts frequently run over time and budget. The goal of a decision framework is to apply the right approach to the right workload, not to pick a single philosophy and apply it everywhere. Five Questions That Drive the Decision 1. What Is the Business Criticality of This Workload? Tier 1: Applications that directly generate revenue or are customer-facing warrant investment in modernization, especially if they have growth potential. The engineering effort pays back through scalability, resilience, and feature velocity. Tier 3: Internal tools, reporting systems, or legacy applications used by a handful of employees are strong lift-and-shift candidates. The cost of modernizing rarely justifies the benefit. A fast triage: Ask the application owner what happens if the application is down for four hours during business hours. The answer tells you a lot about where to invest. 2. Is the Application End-of-Life or Actively Developed? If an application is on a deprecation path, to be replaced in 18 to 36 months, lift-and-shift is almost always the correct call. You want the application in the cloud for consolidation, cost, or data center exit reasons, but you do not want to invest engineering resources in something you are going to retire. Conversely, if an application is actively developed and your engineering team ships features to it regularly, modernization has a compounding return. Every sprint benefits from cloud-native capabilities. 3. What Are the Licensing and Dependency Constraints? Some applications are locked to specific operating system versions, middleware versions, or third-party components that are not certified on modern platforms. A manufacturing execution system or a financial ledger application from 2008 may have an ISV (Independent Software Vendor) support contract that explicitly requires Windows Server 2012 R2. In those cases, your choice is not lift-and-shift versus modernization. It is lift-and-shift or do nothing. Azure and AWS both offer extended security update programs for legacy OS versions, making rehost viable even for older stacks. 4. What Are the Team's Skills and Capacity? Modernization is an engineering-intensive activity. If your team is composed primarily of infrastructure engineers skilled at VM management but with limited experience in Kubernetes, Terraform, or cloud-native PaaS (Platform as a Service) services, a forced modernization will stall. Honest capacity and skills assessment matters. I have seen organizations attempt to modernize a monolithic Java application to microservices while simultaneously running a datacenter migration. Both programs suffered. A phased approach often works better: lift-and-shift first to get out of the datacenter, then modernize workloads incrementally once the team is stable on the cloud platform. 5. What Are the Unit Economics Over a Three-Year Horizon? Run the numbers. This is non-negotiable. Tools like Azure's Total Cost of Ownership (TCO) calculator or AWS Pricing Calculator can model lift-and-shift costs quickly. For modernization, you will need to factor in engineering labor costs, which are often 3x to 5x the infrastructure savings in the first year. The business case shifts in favor of modernization when: The workload has high and variable traffic (autoscaling delivers real savings)The team plans significant feature development (cloud-native accelerates delivery)The current architecture requires expensive licensed middleware that PaaS services can replace The business case favors lift-and-shift when: The workload has predictable, flat traffic (reserved instances close the cost gap)Engineering capacity is constrainedThe migration is driven by a hard datacenter exit deadline A Decision Matrix Factor Favor Lift-and-Shift Favor Modernize Business criticality Low to medium High, customer-facing Development activity Stable / end-of-life Active development Technical debt Manageable High and growing Team skill set Infrastructure-focused App dev / cloud-native capable Timeline Hard deadline Flexible Licensing constraints ISV-locked Open or replaceable Traffic pattern Flat, predictable Variable, spiky The Re-Platform Middle Path Before forcing a binary choice, evaluate re-platforming for database workloads. Moving from SQL Server on a VM to Azure SQL Managed Instance, or from Oracle to Amazon RDS, is a lift-and-shift at the application layer and a modernization at the data layer. You eliminate OS patching, get automated backups, built-in high availability, and elastic scaling without refactoring a single line of application code in most cases. This is often the highest-return migration move available to enterprise customers and is underutilized because teams think in binary terms. What I See Go Wrong The most common failure mode is scope creep driven by modernization enthusiasm. A team scopes a lift-and-shift, then someone says “while we’re at it, let’s containerize it.” Twelve months later, the application is still not in production. The second most common failure mode is lift-and-shift without right-sizing. Teams migrate on-premises VMs 1:1 to cloud VMs without analyzing actual CPU and memory utilization. Azure Migrate’s performance-based assessments and AWS Compute Optimizer exist for exactly this reason. A VM provisioned at 16 cores on-premises is often running at 8% CPU utilization. Moving it as-is is leaving money on the table. Both mistakes are avoidable with a disciplined assessment phase before migration execution begins. Putting the Framework Into Practice In a typical enterprise migration engagement, I recommend the following sequencing: Discover and classify: Run an agentless discovery (Azure Migrate or AWS MGN) to inventory all workloads. Classify each by tier, development activity, and licensing constraints.Apply the decision matrix: Score each workload and assign a migration strategy: rehost, re-platform, or modernize.Sequence by risk: Start migrations with lower-criticality, lower-complexity workloads to build team confidence on the target platform.Right-size before you migrate: Use performance data to set cloud VM sizes. Do not replicate on-premises provisioning patterns.Modernize in waves: Once lift-and-shift workloads are stable in the cloud, identify the top candidates for modernization based on business value and team readiness Closing Thoughts There is no universally correct answer between lift-and-shift and modernize. The decision is contextual, and applying the wrong strategy to a workload modernizing something that should have been retired, or lifting-and-shifting something that needed to be rebuilt creates costs that compound over time. The framework above does not eliminate judgment. It structures the judgment so it is applied consistently, documented, and defensible to stakeholders who will inevitably ask why you chose the path you did.

By Srinivasarao Thumala

One of Waterfall's Most Resilient Artifacts

The waterfall model of software development was formally described in 1970. It was critiqued decisively by the early 1980s. It was officially succeeded by iterative and incremental methods through the 1990s and rendered obsolete, in many professional contexts, by the Agile Manifesto of 2001. However, it is still alive and embedded in the structure of the work in many organizations, not as a named process but as an unexamined assumption. Not "we do waterfall" — nobody says that. But: "the feature goes to QA when development is done." "The sprint doesn't end until QA signs off." "We're blocked on QA." "QA found twenty bugs; we can't ship until they're resolved." These phrases are not descriptions of Agile. They are descriptions of waterfall with a two-week cycle time. The phase is still there. The hand-off is still there. Testing still follows development, a verification step rather than a continuous property. This article examines why the phase assumption is so persistent and what it costs in organizational terms. It also addresses the difficulties of the transition from quality-as-phase to continuous quality. The Waterfall Assumption in Agile Clothing Waterfall: Requirements → Design → Development → Testing → Release. Agile-in-name: Plan → Sprint Development → Sprint Testing → Sprint Review → Release. The cycle shortened. The phase sequence did not. Testing still follows development. QA is still the last gate. Quality is still the property of a function, not of the process. The structural assumption that makes waterfall expensive is present in both. The shorter cycle merely reduces the interval between its consequences. Why the Phase Persists: A Structural Analysis The persistence of the QA-as-phase model is not irrational. It has structural causes that are worth understanding, because understanding them is the prerequisite for addressing them. Organizational Gravity Testing-as-phase is the default because it maps onto the most natural division of labor in software development: some people build things, other people check them. This division is intuitive, easy to staff, manage, and account for. You know who the developers are, you know who the QA engineers are. You also know when one group's work ends, and the other's begins. The hand-off is the organizational seam. Quality as a continuous property has no equivalent seam. It is distributed across the team. It is harder to manage, harder to staff, and harder to account for. Organizational gravity pulls toward the legible structure — even when the legible structure is less effective. Measurement Gravity The phase model produces measurable outputs: test cases executed, bugs found, bugs resolved, pass rate. These can be counted, tracked, reported, and displayed on dashboards. A VP of Engineering can look at a sprint report and understand, at a glance, the state of QA. Quality as a continuous property produces outputs that are distributed, contextual, and harder to aggregate: a requirements ambiguity caught in review, a design flaw surfaced before implementation, a unit test that prevented a defect from propagating, an exploratory session that identified a risk the team had not considered. These are real contributions to quality. They are almost impossible to count in a way that maps onto an existing reporting structure. Organizations have traditionally measured the output of a phase. So the phase persists. Skills Gravity Testing-as-phase concentrates quality skills in a specialist function. QA engineers know testing. Developers know development. The specialization is clean, and the skills are developed within it. Quality as a continuous property requires that testing thinking is distributed — that developers write meaningful tests, that product owners specify with testability in mind, that architects consider failure modes, that DevOps engineers understand quality signals in production. This requires a different skills profile across the entire engineering organization, not just in a QA function. Building that profile takes years of deliberate investment. Most organizations have not made that investment, because the phase model did not require it. The result is a workforce that is structurally dependent on the phase: remove the QA checkpoint and quality does not become everyone's responsibility; it becomes no one's. The Cost of the Phase: Where the Damage Accumulates The QA-as-phase model does not produce a single, dramatic failure. It produces a chronic, compound cost that accumulates across the SDLC in ways that are individually explainable and collectively damaging. The following four cost categories are where most of that damage concentrates. The Rework Cost When quality is a downstream phase, defects discovered during that phase require rework. This is often more expensive than prevention. This is the oldest argument in software quality economics, and it remains true: a requirement misunderstood at the requirements stage costs almost nothing to correct before any code is written. However, it takes significant time to correct after the code, tests, and potentially dependent features have been built around the misunderstanding. The phase model makes late discovery structurally inevitable. Testing does not begin until development is complete. By the time testing begins, the cost of correction is already elevated. By the time testing finds a defect in a dependency, the cost is elevated further. By the time a defect reaches production — as some always do — the cost is at its maximum. The insidious quality of this cost is that it feels normal. Teams that have always operated in the phase model have calibrated their expectations around a level of rework that they could reduce substantially by shifting quality earlier. To them, the excess cost is not an excess. It is a natural difficulty of software development. The Context-Switch Cost In the phase model, there are cases when a defect discovered during QA requires a developer to return to code they wrote days or weeks ago. The developer must reconstruct the context of that code, understand the failure, and fix it — without introducing new defects in the process. This context reconstruction is expensive since context switching is one of the most significant drains on engineering throughput. The cost is not just the time to fix. It is the time to remember, to understand, and to re-enter a cognitive state that the developer was in when the code was written. For complex code, in a complex system, this can take hours before a line of correction is written. In a continuous quality model, where defects are caught by a failing unit test, by a CI failure, or by a developer's own review, the context is immediately available. In this case, the cost of correction is a fraction of the phase-model equivalent. The Blocking Cost When QA is a terminal phase, it is also a bottleneck. The rate at which software can be released is limited by the rate at which QA can process it. In organizations where development outpaces QA capacity — which is most organizations, because QA headcount is typically smaller than development headcount — a queue forms. Work in progress accumulates. Developers complete features that cannot be tested, so they pick up new features, creating more work in progress and deepening the queue. This queuing dynamic has consequences beyond the immediate delay. Work sitting in a queue is work that simultaneously creates risk (it may depend on other work, accumulate merge conflicts, or become outdated relative to the codebase) and consumes organizational attention (it must be tracked, managed, and prioritized when it eventually moves). The phase model does not just slow release — it creates inventory of partially-complete work that costs money to maintain. # The queuing model, simplified # # Development team: 8 engineers, completing ~2 stories/sprint each = 16 stories/sprint # QA team: 2 engineers, testing ~6 stories/sprint each = 12 stories/sprint # # Net accumulation per sprint: 16 - 12 = 4 stories # After 3 sprints: 12 stories in queue # After 6 sprints: 24 stories in queue # # Organizational responses (all suboptimal): # 1. Hire more QA -> increases throughput, preserves the phase structure # 2. Reduce dev velocity -> reduces throughput, demoralizes developers # 3. Reduce QA thoroughness -> increases throughput, reduces evidence quality # 4. Accept the queue -> ships slower, builds technical debt in WIP # # The phase model has no good solution to this problem. # Continuous quality dissolves it: quality activities happen in parallel # with development, so the testing bottleneck does not exist. The arithmetic of the phase bottleneck. None of the responses address the structural cause. The Signal Latency Cost The most consequential cost of the phase model is one that rarely appears in project management reports: the latency of quality signals. In a phase model, information about defects travels slowly — from the moment of injection during development, through the queue, through the testing phase, back to the developer through the defect report, and eventually to the fix. This latency can span days, weeks, or, in organizations with long release cycles, months. During that interval, the defect is not idle. If it is a requirement misunderstanding, other features may have been built on the same misunderstanding. If it is a design flaw, other components may have been built to interface with the flawed design. If it is a logic error, other code may have been written that depends on the erroneous behavior. Late signals are not just expensive to act on — they are expensive because of everything that has been built on top of the undetected problem before the signal arrives. Why The Transition Is Hard Understanding why the transition from quality-as-phase to continuous quality is genuinely difficult is not an excuse for not making it. It is a prerequisite for making it successfully. The Skills Gap Continuous quality requires that developers write tests that are meaningful, not merely present. Most developers were not trained to do this, and many have spent careers in environments where testing was delegated downstream. Writing tests that cover critical behavior, that are maintainable, that fail for the right reasons, and that are specific enough to be useful diagnostically — these are skills that must be developed deliberately, and they take time. Attempting to move to a continuous model without investing in developer testing capability produces a predictable outcome: developers write tests, the test suite grows, CI runs the tests, and the organization believes it has achieved continuous quality. What it has actually achieved is a large, expensive, poorly designed test suite that provides weak evidence and incurs high maintenance costs. This is coverage illusion at the process level — the phase is gone in name, but the quality of the testing has not improved. The Short-Term Productivity Hit When a team transitions from phase-based to continuous quality, the short-term impact on delivery velocity is negative. Developers are spending time on tests they previously delegated. Requirements reviews take longer because testability is now considered. CI pipelines are slower because they run more tests. The work that was previously concentrated in a downstream phase is now distributed across the cycle, and the team feels the additional load before it feels the reduction in rework. This short-term hit is real, predictable, and temporary. It typically resolves as the team develops the practices and the test infrastructure matures. But it requires leadership to hold the course during the transition — to resist the pressure to revert to the phase model because "it felt faster" — and to communicate clearly that the short-term cost is the price of the long-term gain. The Accountability Gap In the phase model, quality accountability is clear: QA is responsible for it. When a defect escapes to production, there is a function whose remit includes finding that defect. In the continuous model, quality accountability is distributed. It belongs to the team. And distributed accountability, without careful organizational design, can become no accountability — everyone responsible could mean no one responsible. Avoiding this requires that the continuous model includes explicit quality accountability at the team level — that retrospectives include quality evidence review, that release decisions require documented risk assessment, and that escape rates are tracked and discussed as team metrics, not as QA metrics. Quality becomes no one's job title and everyone's job responsibility. That transition requires deliberate structural design, not just an announcement that quality is now everyone's concern. The Transition Checklist Skills investment: have developers received training and practice in behavior-driven test design? Process change: are requirements reviewed for testability before development begins? Infrastructure investment: does the CI pipeline provide feedback within ten minutes? Definition of Done: does it include quality evidence as a non-negotiable element? Metrics change: have defect counts been replaced with escape rate and behavior coverage? Leadership commitment: has the short-term productivity hit been acknowledged and planned for? Accountability design: is quality ownership at the team level explicit, documented, and reviewed? QA role redesign: have QA engineers been given the charter and the investment to become quality system architects rather than phase executors? Wrapping Up The QA-is-a-phase assumption is one of waterfall's most resilient artifacts. It survived the transition to Agile because it maps onto natural organizational divisions, produces measurable outputs, and concentrates quality skills in a specialist function. These are real advantages, but they are outweighed by the structural costs they impose. The phase model makes late discovery structurally inevitable. It creates bottlenecks that cannot be solved by hiring. It produces signal latency that allows defects to compound before they are detected. It concentrates quality accountability in a function that cannot own it alone. And it measures quality through metrics that are legible but poorly correlated with actual system reliability. Continuous quality is not a methodology or a tool. It is a structural property of the development process: quality evidence is generated at the stage where it is cheapest to act on, by the people who are in the best position to generate it, as a non-negotiable part of the definition of done. Achieving it requires skills investment, process change, infrastructure investment, and leadership commitment to absorb a short-term cost for a long-term gain. The QA phase is not a safety net. It is a delay mechanism that makes defects more expensive without making them less frequent.

By Stelios Manioudakis

CORE

The Rise of Agentic SRE: Humans, Agents, and Reliability

Site reliability engineering has always been about reducing toil, improving resilience and helping teams respond to incidents with speed and confidence. Agentic SRE takes this idea further, allowing AI systems to observe, reason, and act within operational workflows inside of bounded constraints. The outcome is not a replacement for SREs, but a new operating model in which humans supervise intelligent agents that can help triage, diagnose, and remediate faster than manual processes alone. What Agentic SRE Means Agentic SRE is the use of AI agents to carry out reliability tasks with some autonomy. The agents are able to capture telemetry, correlate signals across systems, propose likely causes, take safe actions, and hand over to humans when the problem exceeds their authority. In practice, this means an AI assistant that can summarise an incident, pull up relevant dashboards, check recent deploys, compare symptoms against runbooks and even trigger low-risk remediation steps. What changes are not the nature of the assistance but the limits of its application. Traditional automation is usually rule-based: if X happens, do Y. Agentic systems are different in that they can adapt to context, select between several paths, and orchestrate steps across tools. This makes them especially useful in complex environments where the same symptom may come from many different root causes. Why SRE Needs Agents The systems today are too big and too interconnected to be run totally by hand. Teams are contending with noisy alerts, fragmented observability data, constant deployments, and ever more dynamic infrastructure. During incidents, engineers often burn precious minutes just to gather context before they can start a real diagnosis. Agentic SRE is attractive because it shortens that time. In a handful of high-friction places, agents can cut toil. They can filter alert storms, enrich alerts with deployment history, draw out meaningful patterns from logs, and surface relevant runbooks. They can also automate repetitive incident response tasks such as opening tickets, notifying owners, checking service health, or validating if a rollback is safe. That doesn't eliminate the need for engineers, but it does take away some of the low-value work that distracts them from judgment-intensive choices. The Human Role Remains Central One common fear is that SREs will be replaced with autonomous systems. Indeed, the human role becomes more, not less, important. Agents are good at pattern recognition, summarisation, and bounded execution. Humans are still better at trade-offs, risk assessment, organisational context, and deciding when not to act. Reliability is not merely a technical problem. It is a business and coordination problem. Humans should set policies, guardrails, and escalation thresholds for agent behaviour. They need to decide which actions can be safely automated, which require approval, and which should never be delegated. So the SRE is evolving from operator to system designer, to policy author, to reliability supervisor. That shift is profound because the skill set you need for the job changes. Where Agents Fit Today The best place to start is with low-risk, high-frequency jobs. These are the areas where automation can provide immediate value without unacceptable risk. Think incident summarisation, alert enrichment, log correlation, runbook retrieval, change impact analysis, and post-incident report drafting. Incident copilots are another strong use case. An agent can also act as a second brain during an outage: it can aggregate timelines, verify recent code changes, search knowledge bases, and suggest next steps. It can help responders avoid duplication of effort and make the first 10 minutes of an incident much more productive. An effective agent can also lessen the cognitive load on on-call engineers by turning the scattered telemetry into a coherent story. A third useful area is remediation assistance. Agents can recommend actions such as scaling a service, restarting a failing job, disabling a faulty feature flag, or rolling back a deployment. In mature setups, these actions can be executed automatically for pre-approved scenarios, while more risky actions still require human confirmation. That combination of automation and oversight is where agentic SRE becomes genuinely powerful. A Practical Architecture An effective agentic SRE system typically has five layers. First, it needs a telemetry layer that includes metrics, logs, traces, events, and deployment data. Without strong observability, the agent is blind and will make incorrect guesses. Second, it requires a reasoning layer, often powered by an LLM, to interpret context and decide what to do next. Third, there should be a tool layer that gives the agent access to safe operational functions, such as querying dashboards, reading configs, opening tickets, or triggering runbooks. Fourth, it needs policies and guardrails that define permissions, approval workflows, rate limits, and failure boundaries. Finally, it should have an audit layer so every decision, action, and recommendation can be traced later. That architecture matters because the danger is not the model itself; the danger is uncontrolled action. A reliable agent is not one that knows everything. It is one that acts only within well-defined limits and remains observable, reversible, and accountable. Guardrails That Matter Trust is the currency of autonomous operations. If teams do not trust the system, they will ignore it. If they trust it too much, they may hand over dangerous actions without oversight. The right answer is neither blind trust nor permanent skepticism. It is a layered trust model built through guardrails. Start with permission scoping. An agent should not have broad access by default. Its permissions should be narrow, explicit, and tied to specific tasks. Next, use action tiers. Low-risk actions can be automatic, medium-risk actions can require confirmation, and high-risk actions should remain human-only. You also need strong rollback paths so any automated action can be quickly reversed. Another essential safeguard is the observability of the agent itself. Just as production systems need monitoring, agents need monitoring too. Teams should track what the agent saw, what it inferred, what action it proposed, and whether the result improved the situation. That makes the system auditable and helps teams refine its behavior over time. The Operating Model Changes Agentic SRE changes incident response from a purely human workflow into a human-agent collaboration loop. In the old model, an engineer gets paged, reads alerts, searches dashboards, checks logs, consults teammates, and then acts. In the new model, the agent can do much of the initial gathering and triage before the human even joins. That shortens the path from detection to understanding. This also changes how teams design runbooks. Instead of static documents that people read under pressure, runbooks become machine-readable operational playbooks. Some of the best runbooks will be written with automation in mind, including clear preconditions, decision points, and action boundaries. That makes them useful both for humans and for agents. Post-incident work also improves. Agents can draft a timeline, collect evidence, identify suspicious changes, and summarize repeated patterns across incidents. That leaves engineers with more time to focus on systemic fixes rather than manual documentation. Over time, the organization develops a stronger feedback loop between incidents, learning, and platform improvements. Risks and Failure Modes Agentic SRE is not free of risk. One failure mode is confident hallucination, where an agent sounds plausible but is wrong. In operations, a wrong answer is not just inaccurate; it can cause downtime. Another risk is over-automation, where teams let agents act in situations that are not actually safe to delegate. There is also the risk of hidden complexity. If an agent stitches together many systems, it can become difficult to understand why it chose a specific action. That opacity can undermine trust and create governance problems. Security is another major concern because an agent with tool access can become an attractive target if permissions are poorly controlled. These risks do not mean agents should be avoided. They mean they must be introduced carefully. The best strategy is to start with narrow, well-understood workflows, measure outcomes, and expand only when confidence is earned. Reliability teams already understand progressive delivery, canary releases, and blast-radius reduction; the same principles should apply to agentic operations. How to Start The easiest entry point is to pick one painful workflow and automate only the first mile. A good candidate is alert triage. An agent can ingest alerts, group duplicates, summarize likely causes, and point responders toward relevant dashboards and runbooks. That alone can save significant time without requiring the agent to make risky changes. Another strong starting point is incident summarization. This is low risk, highly useful, and easy for teams to evaluate. A third option is change impact analysis, where an agent compares recent deploys, feature flag changes, and error spikes to highlight likely correlations. These use cases are valuable because they build trust through usefulness rather than hype. Measure success with clear operational metrics. Look at time to acknowledge, time to diagnose, time to mitigate, alert volume reduction, and after-hours toil reduction. Also measure negative outcomes, such as false suggestions, unsafe recommendations, or overreliance on the agent. Good SRE practice is about evidence, not enthusiasm. A New Reliability Mindset The biggest change Agentic SRE brings is a shift in mindset. It encourages teams to stop viewing automation as just a collection of scripts and to see it instead as a supervised operational partner. This partner can observe faster than a person, summarise quickly, and carry out repetitive tasks more reliably. However, it still requires humans to define the purpose, set limits, and determine acceptable risk. This is why agentic SRE is not merely “AI in operations". It represents a larger redesign of how reliability work is accomplished. The focus shifts from manual responses to intelligent coordination. It changes from isolated dashboards to context-aware agents. It evolves from static runbooks to flexible playbooks. It transforms reactive tasks into guided independence. Organizations that excel with this model will not be the ones that automate everything. They will be the ones that automate thoughtfully, govern effectively, and keep humans involved where decision-making matters most. In this way, Agentic SRE is more about enhancing the reliability system around the engineer than about replacing the engineer themselves. Closing Thoughts Agentic SRE marks a real change in how we can manage modern systems. It provides a way to respond faster, reduce repetitive work, and handle incidents more consistently, but only with strong observability, clear permissions, and human oversight. The future of reliability is not completely automatic or fully manual; it is collaborative, constrained, and constantly improving. For SRE teams, there's a chance to become designers of this new model. This involves creating agent workflows, writing safer runbooks, setting policy limits, and measuring impact with the same attention given to any production system. Teams that excel in this will not only respond more quickly. They will create systems that are more resilient, more adaptable, and much simpler to operate at scale.

By Neel Shah

Spec-Driven Development Renamed an Old Problem; It Didn't Solve It

Spec Kit, OpenSpec, BMAD, Kiro — all of it is built on the assumption that a spec can stay the source of truth. It can't, for the same reason design docs and wikis never stayed current either. I think the interesting unsolved problem in this space isn't "more rigorous specs," it's "specs that don't require a human to remember to update them." Curious if people running these in production agree. The Pitch Everyone's Making Right Now Spec-driven development has become the default answer to "AI agents write plausible-looking code that's subtly wrong." The idea: stop prompting, write a structured spec, let the agent execute against it, review the diff. GitHub Spec Kit has 90K+ stars and /speckit.specify → /speckit.plan → /speckit.tasks → /speckit.implement is basically a known workflow at this point. OpenSpec does the same thing lighter, with delta specs and openspec validate --strict. BMAD goes the other direction and simulates a full 12-agent team. AWS built an entire IDE (Kiro) around the idea. Tessl raised $125M on it. It's not hype for no reason — it works better than raw prompting in many real-world cases. But I've been going deep on this for the last few weeks, reading every practitioner write-up I can find instead of just the vendor pages, and one problem keeps showing up that none of these tools actually solve. Keeping Specs True Is the Same Problem We've Had Since READMEs We've tried to keep documentation in sync with code for 20+ years — design docs, architecture wikis, READMEs. It's never worked, for a boring reason: editing the code is fast and gets you something shipped; editing the doc is slow and gets you nothing visible. When a team is moving, the doc loses every time. SDD just gives the same artifact a new name and calls it "the source of truth." But the asymmetry is identical. You're three tasks into a spec, the agent hits a constraint nobody documented, fixes it inline, ships it — and the spec still describes the old plan. Multiple people running this in production describe specs drifting within days, not months. None of the major frameworks have a real answer for this beyond "discipline" — which is exactly the thing that's failed every previous time we asked for it. Even the Premise Has a Real Critique Kent Beck's objection (picked up by Martin Fowler in January) is worth sitting with: most SDD write-ups assume you write the whole spec before implementation starts, which quietly assumes you won't learn anything during implementation that should change the spec. That's a strange assumption to build a methodology on — it's the opposite of the feedback-loop principle XP was built around. Fowler's point was that AI should speed up the feedback loop, not become an excuse to front-load more upfront documentation. ThoughtWorks' Radar flagged basically the same risk: heavy upfront specs trending toward big-bang releases, which is the exact failure mode agile exists to prevent. A Spec Can Pass Every Check and Still Be Wrong One account that stuck with me: a dev ran a real, multi-month, fully spec-driven project. Thorough specs, the agent built exactly what was specified, every acceptance criterion passed. The system was still wrong, because the spec couldn't capture the tacit, runtime, cross-system knowledge that only shows up once you're actually building. He changed one infra decision mid-project and the entire downstream spec graph broke, because everything after that point had silently inherited an assumption that no longer held. And independently — Scott Logic ran a real SDD workflow against a real project and got: ~10x slower, more ceremony, same number of bugs as their normal process. I don't think that means SDD is useless. I think it means the rigor itself isn't the bottleneck people assume it is. What I Think Is Actually Underexplored Not another framework asking teams to write more documents more carefully. Something closer to: infer and update the spec from what the code and the running system actually do, the same way you'd treat generated docs or a lockfile — maintained automatically, not maintained by someone remembering to do it. I don't have a working version of this, and I'm not sure it's even tractable at the granularity you'd need for it to be useful (anyone who's tried to auto-generate accurate architecture diagrams from a real codebase knows how that usually goes). I'm putting this out mostly because I'd rather find out it's a dead end from people who've actually hit this wall than after building something. Genuinely Curious, If You're Running Any of These in Prod Does your spec survive past week 2–3, or does it quietly become aspirational?When it drifts, what's the actual trigger — a missed edge case, a changed dependency, a scope call nobody backported into the spec?Has anyone tried generating/updating specs from code or telemetry instead of writing them forward? Did it work, or did it just produce noise?If you've used openspec validate --strict or Spec Kit's constitution model in anger — where did it actually hold up, and where did it just add ceremony? Not pitching anything here, genuinely trying to figure out if this is a real gap or if I'm missing a tool/pattern that already handles it.

By Sam K