Culture and Methodologies

If you are responsible for keeping a production system alive, you have probably been pitched an "AI SRE" in the last few months. Maybe more than one. Maybe more than one this week. When we sat down at Bronto to map that space, we ended up with 64 tools across seven camps: pure-play investigation agents, observability platforms with AI built in, incident management tools with AI on top, AIOps and alert correlation, hyperscaler agents like AWS DevOps Agent and Azure SRE Agent, Kubernetes copilots and open source projects, and a long tail of adjacent tools. All of them use the same two words on their landing page, and they solve very different problems. So the first question most teams ask is the classic one: do we buy, or do we build? In our landscape article, we argue that the choice isn't binary. You can go full DIY, build on open source foundations like HolmesGPT or K8sGPT, buy a product and extend it, or buy it as-is. Most teams land somewhere in the middle. In this article, I want to make a different argument for building one, at least once: build to understand what you buy. Even if you never run your own DIY agent in production, putting one together by hand shows you where its answers come from, what it can't see, and why it goes wrong when it does. After building one, you know which questions to ask a vendor, and you know how to get better results from whatever you end up running. That's the profit you should take away from this post. The fun is the building and seeing your agent try to investigate an incident: it might get it wrong, it might get it right, but you'll see how it interacts with your tools and your data. This post is based on a hands-on lab we ran at an AWS and Bronto evening, and everything is available through bronto-community/ai-sre-lab. By the end, you will have a small agent with the following capabilities: reads logs, traces, and metrics through an MCP server,reads the application's source code and commit history through GitHub's MCP server,writes its hypothesis, evidence, and confidence into a GitHub issue. Note that in the lab we use Bronto as the observability backend. I work there, so I am biased to think it is your best option, but you can of course use any other solution. What an AI SRE Is AI SRE is mostly a marketing term. In short, it's an agentic loop pointed at the production system that can answer questions about it and help you troubleshoot. Like any agentic loop, it has inputs on one side (an alert, a question from a human), a model in the middle, tools underneath (mostly MCP servers these days, for telemetry, code, cloud APIs, runbooks), and outputs on the other side (a Slack message, a ticket, a pull request). So, it's an agent like any other. It's what you connect to it that makes it an agent for SRE. The Patient: A Small Online Shop Our AI SRE needs something to investigate. The lab comes with a tiny storefront: four Python services instrumented with OpenTelemetry, plus a payment processor mock standing in for a third party. You can, of course, point the agent directly at your own data. It's read-only, so it is not able to harm your system. What You Need To run this lab, you need the following: A container engine, like Docker DesktopAn AWS account with access to a model on Amazon Bedrock.A Bronto API key (there is a free trial), or another observability backend with an MCP server.A GitHub account and a fine-grained personal access token. It needs to read the contents of your copy of the storefront repo and write issues to it.Optional, for nicer output: jq and glow. You can watch this video to follow along as we walk through each step: Start the Shop Clone the lab and generate the storefront repository. The script builds a small Git history with a handful of commits and tags two releases, v3.1.0 (healthy) and v3.2.0: Shell git clone https://github.com/bronto-community/ai-sre-lab cd ai-sre-lab ./build-storefront.sh Your agent will later read this code through GitHub, so push it to a repository of your own, tags included: Shell cd storefront gh repo create storefront --public --source . --push git push --tags cd .. Then point the Collector at your telemetry backend and bring everything up: Shell cd demo echo "BRONTO_API_KEY=<your key>" > .env make up Note, that demo/collector.yaml ships to Bronto's EU ingestion endpoint. If your account lives in another region, or you want a different backend, that file is the only thing you need to change. Next, run make up, which deploys v3.1.0 and starts the traffic. It prints a message that you should leave it about ten minutes before you do anything else. An investigator with no baseline has nothing to compare against. So, take your time going through the first few next steps so it can build up! Step 1: The Loop All agents start with a basic file that receives prompts, sends them to the model, and returns the results. It's a simple first agent, so here is more or less the code in full: Python import os from bedrock_agentcore import BedrockAgentCoreApp from strands import Agent from strands.models import BedrockModel model = BedrockModel( model_id=os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-haiku-4-5-20251001-v1:0"), region_name=os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION", "us-west-2"), ) app = BedrockAgentCoreApp() @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model) return {"result": str(agent(payload.get("prompt", "Hello!")))} if __name__ == "__main__": app.run() # POST /invocations, GET /ping on :8080 Two libraries do the work. Strands Agents runs the agent loop: send the prompt to the model, run any tool the model asks for, feed the result back, repeat until the model has an answer. The Bedrock AgentCore SDK wraps that in an HTTP server with the /invocations and /ping contract AgentCore Runtime expects. Today that runs on your laptop. Later, the same container can run on AgentCore Runtime without changes. Every step of the lab is a prebuilt image, so you don't need to build anything. Create a .env next to where you run the commands: Shell AWS_DEFAULT_REGION=us-west-2 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... BEDROCK_MODEL_ID=us.anthropic.claude-haiku-4-5-20251001-v1:0 With this setup, you can start the agent for the first time: Shell docker run --rm -p 8080:8080 --env-file .env ghcr.io/bronto-community/ai-sre-agent:step-1 Open a second terminal and ask it something: Shell curl -s localhost:8080/invocations -d '{"prompt": "Who are you?"}' | jq -r .result It will say, "Claude, an AI assistant made by Anthropic." Correct, and completely useless for incident response. Right now this is a model behind an endpoint, with no identity and no tools. Step 2: Identity The second step changes only one line of code. Every Markdown file in prompts/ becomes part of the system prompt: Python SYSTEM_PROMPT = "\n\n".join(p.read_text() for p in sorted(Path("prompts").glob("*.md"))) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT) ... The image ships with two files. role.md: Markdown You are the on-call SRE for Storefront, an online shop. You investigate production incidents: what broke, where, and why. And rules.md: Markdown - Only state what you can back with evidence from your tools. - If you have no tool that can answer, say so and name what you would need. - Be brief. You also get to add your own. Create a my.md: Markdown - Answer in at most three bullet points. - End every answer with your confidence: low, medium or high. - Sign off as "your friendly on-call dino". Start step 2 as follows. The command contains your custom instructions as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-2 Shell curl -s localhost:8080/invocations -d '{"prompt": "Checkout is slow. What is wrong?"}' | jq -r .result You should get a short answer that says it has no access to telemetry, rates its own confidence as low, and signs off as a dino. That's a good result. The rule "say so and name what you would need" did its job: an agent that admits it can't see anything is far more useful than one that guesses. Step 3: Eyes Now the agent gets something to look at. Step 3 adds one tool source, Bronto's MCP server: Python bronto = MCPClient(lambda: streamable_http_client( os.environ.get("BRONTO_MCP_URL", "https://mcp.eu.bronto.io/mcp"), http_client=httpx2.AsyncClient( headers={"X-BRONTO-API-KEY": os.environ["BRONTO_API_KEY"]}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) ... That's the whole change. The MCP server's tools (search logs, list datasets, and so on) become tools the agent can call, and Strands handles the plumbing. This is also the point where MCP pays off: if your telemetry lives somewhere else, you swap this client for that vendor's MCP server and keep everything else. Add BRONTO_API_KEY to your .env (and BRONTO_MCP_URL if your account isn't in the EU region), then: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-3 Start with a question that tells you what it can see: Shell curl -s localhost:8080/invocations -d '{"prompt": "What services do you know about?"}' | jq -r .result It should list the storefront services. Go ahead and ask it some more questions about the state of those services: what HTTP endpoints are available? What is the average error rate right now? With this, you have a first working version. Next we want to investigate an incident. To do so, ship the next release of the storefront: Shell make release According to the Makefile, symptoms take about 60 seconds to show up in p95. So wait a minute or two and then ask it: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result It will come back with a hypothesis about what is wrong and what you might be able to do about it. Take a look at the telemetry yourself and check: is the analysis correct, or did the agent pick up some false information? Step 4: Mouth An answer in your terminal is nice for a demo, but in production you would want to have it sent to a Slack channel, a pager app, or an issue tracker. In our example, we send the answer as a GitHub issue. This would let you tag your coding agent to review the issue and open a pull request that fixes it. You now have two options for how this gets filed. Either you give your agent another tool, and it can decide for itself where to send the hypothesis, or you make the code file the issue, so every analysis gets pushed into a dedicated issue: Shell def file_issue(question: str, hypothesis: str) -> str: r = httpx2.post( f"https://api.github.com/repos/{os.environ.get('GITHUB_REPO', 'ai-sre-lab/storefront')}/issues", headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, json={"title": f"[investigation] {question}"[:120], "body": hypothesis}, timeout=30, ) r.raise_for_status() return r.json()["html_url"] @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) hypothesis = agent(question) return {"result": file_issue(question, hypothesis)} To make sure that what the agent provides fits the output, an additional new prompt file tells the agent how to write it: Markdown Your answer is filed as a GitHub issue for the on-call team. Write it as one: a one-line hypothesis, the evidence for it, your confidence, and what you could not check. Before running the next version of your agent, add your GitHub token and your repository to .env: Shell GITHUB_TOKEN=github_pat_... GITHUB_REPO=<your-user>/storefront Run step 4: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-4 Ask it again to investigate the issue: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result The response is now a link. Open it and read the hypothesis. Is it right? How confident is it? There's a good chance it's confidently wrong. While testing the lab, one run concluded that checkout was down because a query returned nothing: an empty result is not proof of an outage, but nothing told the agent that. Step 5: Better Instructions What you do next doesn't involve code. You do what you'd do with a new human on the rotation: you tell them how you investigate. Add these lines to your my.md: Markdown - Slow is broken too: compare latency (p50, p95) before and after, not only errors. - Compare against a quiet window earlier the same day. - Look for `deploy` spans near the onset: they name the commit that went live. Each line encodes a habit experienced SREs have, and models don't bring by default. Checkout can be very broken without throwing a single error. A number means little without a baseline. And the first question in most incidents is "what changed?" The prompt files are read when the container starts, so restart step 4 with the same command and ask a more realistic question: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Compare the new issue with the old one. Is it better? Worse? Did it fail outright? Step 6: Better Model As you might now, a different model can create a different output, so if you are not satisfied with the results yet, change one line in .env: Shell BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0 Restart your agent, ask the question again, and compare the outputs again: better or worse? Step 7: Code So far the agent can see that something changed and maybe even when. If your instructions worked, it found the deploy. What it can't do is look at what was deployed. Step 7 adds the second MCP server, GitHub's, restricted to read-only repository tools: Python github = MCPClient(lambda: streamable_http_client( os.environ.get("GITHUB_MCP_URL", "https://api.githubcopilot.com/mcp/x/repos/readonly"), http_client=httpx2.AsyncClient( headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto, github]) ... To make it useful to the agent, add one more prompt file, code.md. It connects the telemetry to the code: Markdown The shop's source code is in the GitHub repository `ai-sre-lab/storefront`. Every service reports the commit it runs as `deployment.commit`. Replace the repository path with your own, then mount it as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" \ -v "$PWD/code.md:/app/prompts/code.md" \ ghcr.io/bronto-community/ai-sre-agent:step-7 Ask the same question again: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Does it find the deploy? Does it open the release and pick the right commit? In our testing, it found the deploy marker and the commit, and then stopped. The release contains several commits, and the last hop, reading every one of them and figuring out which change explains the symptom, didn't happen on its own. If you experience the same, try to fix it yourself by refining your system prompts or by trying yet another model. When you're done, make rollback puts the healthy version back, and make down cleans everything up. If you want to give your agent a harder case, make psp-slow degrades the third-party payment processor. So, Buy or Build? Now you know what an AI SRE is and how it works, from the inside. That's a much better position from which to answer the question this post started with. If you buy, a lot of what hurt along the way becomes someone else's problem: confidently wrong answers that need endless prompt tuning, context windows overflowing with search results, wiring up every integration yourself. AWS DevOps Agent, for example, starts investigating when an alert fires and connects to the tools you likely already run, and you can still extend it with custom skills and your own tools. Take the four questions from the landscape article with you into every demo: Where does it get its data? Does it act, or only advise? Can you see its reasoning? What happens when it's wrong? After this lab, you'll know what good and bad answers to those look like. If you build, what you have now is a skeleton, and you can build more features towards a complete picture on top of it: Let alerts trigger it instead of curl.Run it on AgentCore Runtime instead of your laptop (agentcore deploy).Give it more tools: your cluster, your cloud APIs, your past incidents and postmortems.Observe the agent itself. Its own traces tell you which tools it called, how many tokens it burned, and where it went wrong, and that's the input you need to improve it. Either way, the data underneath matters more than the agent on top. Every improvement in this lab depended on the telemetry being there: a baseline from ten quiet minutes, latency and not just errors, a deployment.commit attribute that connects a span to a line of code. An agent can only reason about what your system tells it. That was true for humans long before agents showed up. So: what would your agent get wrong? The full lab, including the slides, the agent images for every step, and the storefront harness, is at github.com/bronto-community/ai-sre-lab. Content is CC BY 4.0; code is MIT. If you build something on top of it, I'd love to hear about it. I've shown how easy it is to build a simple AI SRE agent for fun. If you want to see what a production-ready one looks like, read our post on the Bronto blog, Introducing Bronto's Open-Source Alert Investigation Agent. It's an open-source agent that replicates the Alert Investigation feature in Bronto, and it's essentially our production version of this lab: a Bronto monitor triggers it instead of curl, it sticks to a constrained set of tools and reasoning paths instead of improvising, and it hands you a report with likely root causes and suggested next steps. The code is at bronto-community/bronto-alert-investigation-agent.

TL; DR: Reusing Agile Artifacts to Avoid Accumulating AI Debt In this video from the 76th Hands-on Agile Meetup, I walk you through the A3 Delegation System and show how it helps avoid AI debt by borrowing artifacts and practices from Agile, such as the Definition of Done and Retrospectives. What the Video on the A3 Delegation System Addresses AI Debt and the Delegation Lifecycle: The delegation, documentation, and ownership failures you already know from Agile reappear the moment an AI workflow runs unattended and undocumented, and the six-stage Delegation Lifecycle gives you a place to catch each one before it reaches a customer.The A3 Framework, Assist, Automate, Avoid: Every task lands in one of three boxes, Assist (AI drafts, you own the outcome), Automate (AI executes on explicit rules with an audit cadence), or Avoid (work that stays human because failure damages trust), and the real risk is Assist quietly drifting into Automate because the output has looked fine for four weeks.Model Routing by Cost Tier: Good enough is a deliberate decision with an owner, not a default, so you match each task class to local, mid-tier hosted, or frontier models by cost and stakes, and you keep proprietary process data out of the big labs because that data is your actual moat.The AI Definition of Done per Task Class: One Definition of Done for everything works in Scrum, but breaks under AI, so you write a separate one for each task class across four levels: verification, provenance disclosure, data hygiene, and sufficiency tier.The Delegation Audit and Governance Trail: Drift happens slowly, and approving a draft in four seconds is rubber-stamping, not reviewing, so you run a monthly audit for output drift, sufficiency drift, reversibility, and review creep, and the artifacts you create along the way answer any CFO, CTO, or procurement due diligence question without extra reporting work.

By Stefan Wolpers CORE

For the past decade, the gold standard of Site Reliability Engineering (SRE) was "all your telemetry in one place, beautifully rendered." Centralized platforms allowed teams to build massive grids of charts tracking CPU usage, memory allocation, request latencies, and error rates. Yet this approach masks a critical flaw: traditional dashboards show only symptoms, forcing humans to act as data archaeologists to find the root cause. With microservices handling highly distributed workloads, a simple user request might cross a dozen internal boundaries. When latency spikes, a static dashboard tells you what is broken, but it cannot tell you why. The modern engineering directive has fundamentally shifted from reactive visibility to deep system understanding. The conventional approach of isolating metrics, logs, and traces into discrete dashboard tabs or separate tools has broken down under cloud-native complexity: The Contextual Chasm: When an alert fires, engineers are forced to pivot between systems, manually trying to correlate a metric spike in one graph with a raw text error in a log file. This manual synchronization creates a high mean time to resolution (MTTR).Pre-Aggregation Destruction: Traditional monitoring relies on pre-aggregated data averages to control cloud storage costs. This destroys fine-grained details, making it impractical to debug highly specific "unknown-unknown" incidents affecting rare user cohorts.Anticipation Bias: Dashboards assume that a software engineering team can accurately predict every failure mode ahead of time to build a chart for it. Complex systems fail in entirely unpredictable ways that static, predefined views cannot capture. The Observability Revolution: Core Paradigms Replacing the Dashboard Rather than forcing engineers to stare at screens waiting for a line to turn red, modern observability treats system telemetry as a unified, exploratory analytics problem: High-Cardinality Unified Data Layers: Modern architectures store unaggregated, raw event streams. By preserving trace IDs, customer IDs, and container headers, engines can instantly pinpoint the precise thread or database index responsible for a system failure.Contextual Topology and Causality: Advanced systems continuously map software dependencies and traffic patterns. Instead of sending a dozen isolated alert notifications, the system assembles an incident narrative, proving how a configuration drift upstream actively broke a payment processing gate downstream.Agentic AI Remediation: The industry is moving from monitoring toward autonomous operations. Specialized AI agents consume telemetry streams directly to execute preventative actions like scaling production clusters or rolling back buggy code deployments. The traditional dashboard view: application health, load, response times, and error rates show that something is wrong, but not why. The engineer still has to hunt for the cause. The high-cardinality view: instead of scanning charts, the failing dimension is isolated directly — here, errors tied to a specific service version (v350.10) and HTTP 401 status — pointing straight at the root cause. Adapting to the Age of Non-Deterministic Infrastructure The massive expansion of large language model (LLM) agents and generative software layers has further accelerated the demise of legacy dashboards. Traditional systems check whether an API endpoint returns a 200 OK HTTP status code. An AI-driven microservice can operate perfectly within technical boundaries, response times, and system limits, while still delivering entirely hallucinated or factually incorrect output to an enterprise client. Observability has adapted by monitoring semantic tokens, prompt guardrails, and vector pipeline drift, complex factors that cannot be displayed on a standard bar chart. The metric of success for a modern technology infrastructure is no longer the number of operational dashboards under management. Success is defined by autonomous, rapid understanding and instantaneous path-to-resolution, which is why the static visual graph has been displaced as the primary debugging surface. The Strategic Dilemma Transitioning from static, pre-aggregated dashboards to high-cardinality observability (tracking raw, un-aggregated event data across highly granular dimensions like user_id, container_id, and ip_address) is essential for modern debugging and reducing MTTR. However, storing raw, multi-dimensional event streams exponentially inflates cloud infrastructure and data ingest costs. Without strict architectural guardrails, observability spend can grow to consume an unsustainable percentage of total cloud budgets. The True Drivers of High-Cardinality Cost Exploding Volume of Unique Dimensions: Cloud-native elements (ephemeral containers, serverless functions) spin up and down constantly, generating billions of unique metric combinations that saturate traditional relational structures.Data Ingestion Overhead: Commercial observability vendors charge based on data volume ingested and index size, and self-hosted open-source clusters (e.g., OpenSearch, or Prometheus with Cortex/Thanos) carry the same cost pressure through the infrastructure required to store and index that volume.Hot Storage Reliance: Keeping raw, unaggregated logs and traces instantly queryable requires expensive, high-throughput SSD cloud storage, driving up baseline cloud infrastructure costs. Architectural Framework for Cost Mitigation To balance the technical necessity of high-cardinality data with strict fiscal control, tech leaders should mandate a tiered storage approach: Implement Intelligent Data Sampling Not all telemetry is created equal. SRE teams should configure tail-based sampling rules. For instance, store 100% of traces that result in an HTTP 5xx error or high latency, but sample only 1% to 5% of healthy, predictable HTTP 200 OK traces. Adopt Tiered Storage Horizons Hot Tier (0–7 Days): Keep full, high-cardinality raw telemetry in expensive, high-performance storage for immediate active incident response.Warm Tier (8–30 Days): Move data to aggregated metrics blocks or lower-cost search-optimized tiers for historical trend analysis.Cold Tier (31+ Days): Compress and offload raw data into highly economical object storage (such as Amazon S3, Google Cloud Storage, or Azure Blob) strictly for long-term compliance and audit requirements. Tiered storage keeps high-cardinality observability affordable: recent data stays hot for fast incident response, then ages into progressively cheaper, slower storage for trend analysis and compliance. Enforce Strict Lifecycle Policies. Automate the purging or down-sampling of high-cardinality dimensions as data ages. A specific container ID from three weeks ago holds little operational value for daily SRE workflows. Verdict Do not scale back observability goals to save on infrastructure bills. Instead, optimize how and where that telemetry lives. By implementing data sampling at ingestion and automated lifecycle workflows, organizations can unlock the full power of high-cardinality root-cause analysis while keeping data storage costs linear and predictable.

TL;DR: A 2×2 Matrix on AI for Agile Practitioners You probably think making sense of AI means keeping up with every new tool, agent, and pricing tier. Steve Jobs faced a similar mess at Apple in 1997, with a dozen versions of the Macintosh, and he fixed it with a two-by-two grid on a whiteboard. Almost 30 years later, the “Jobs Matrix for AI” sorts the AI tool market and, more usefully, the AI use cases of Scrum Masters, Product Owners, Agile Coaches, and anyone else in agile product development. Replace Consumer and Pro with Team and Organization, and Desktop and Portable with Chatbot and Agent, and you get four boxes that show where a use case belongs, what it takes to move it, and which box should stay empty. Thesis: This article applies Steve Jobs’ 1997 Consumer/Pro, Desktop/Portable matrix first to the 2026 AI tool market, and then to agile practices, producing a 2×2 matrix that sorts AI use cases by required decision rights and by delegated authority. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, and summarization. It is a production tool, not a substitute for thinking. Kudos to Claude Design for creating the graphics for this post. Four Boxes Saved Apple From Its Own Product Catalog In 1997, after weeks of product reviews, Jobs had seen enough. According to Walter Isaacson’s account in the Harvard Business Review, he stopped the meeting, walked barefoot to a whiteboard, and drew a two-by-two grid with “Consumer” and “Pro” above the columns and “Desktop” and “Portable” beside the rows: four great products, one per box, and everything else canceled. “Deciding what not to do is as important as deciding what to do,” he later told Isaacson. (Remember Simplicity–the art of maximizing the amount of work not done–is essential?) Jobs presented the matrix publicly in his iMac introduction on May 6, 1998, and at Seybold in September 1998, he added the reason that matters to anyone running a portfolio: with only four platforms, Apple could put its A team on every one of them instead of a B or C team on any. The same keynote showed the matrix’s most useful feature, the empty box: Apple had no consumer portable yet, and Jobs could only promise one for the following year, so everyone, the competition included, could see what was missing until the iBook filled that box in July 1999. The AI Tool Market Fits the Same Matrix, for Now Keep Consumer and Pro, and replace Desktop and Portable with Chatbot and Agent. A chatbot returns material that a person then uses, whereas an agent receives the authority to carry out a bounded task through several steps or systems. ChatGPT defines the Consumer Chatbot; enterprise chat, such as Copilot Chat in Microsoft 365, holds the Pro Chatbot box; and the Pro Agent box holds coding agents like Claude Code and Codex, plus Microsoft’s Copilot Cowork, which, according to Microsoft, runs defined work end to end, returns a finished result, and stays off until an administrator enables it. The Consumer Agent box was the thin one until Meta launched Muse on September 8, 2026; by September 24, it had passed 3.4 million downloads, according to Sensor Tower estimates reported by TechCrunch. Downloads measure interest rather than sustained use, which makes Muse the first candidate for the “iBook position,” with its staying power still unproven. The boxes leak, too: Muse lives in a chat interface, and Microsoft 365 Copilot offers a toggle from chat into Cowork, so I would treat the tool matrix as a snapshot (my guess: it will not survive the next 6 months unchanged). Your Jobs Matrix for AI: Team or Organization, Chatbot or Agent In a professional environment, nobody is a consumer, and the organization buys the tools anyway, so the columns describe the context and decision rights that the work requires. Two questions place any use case in a box: What authority does the AI have after the request? If it returns material for a person to use, it is a chatbot use case; if it carries out a bounded task through several steps or systems, with defined review and approval points, it is an agent use case. What starts it does not matter: a Developer can ask an agent to implement a change, and a schedule can trigger a plain summary.Who must contribute, authorize, or respond? If one product team, its Product Person included, can understand and act on the work, it is a Team use case; if several teams or organizational functions must get involved, it is an Organization use case. Typical use cases that come to mind are: Team Chatbot Summarizing the notes from Sprint Planning, the Sprint Review, or the Retrospective, including a scheduled summary that a person reads before anyone acts on it.Challenging a Product Backlog item for ambiguities and missing acceptance criteria before refinement.Drafting test cases from acceptance criteria.Designing a Retrospective around a specific team problem.Explaining legacy code or domain terms to a new team member. Requires: A practitioner with judgment who makes AI use visible to the team. Where it fails: Private, invisible use, for example, a Product Owner who generates Product Backlog items instead of talking to stakeholders. Team Agent After every event, extracting the agreed actions, checking each one with its owner, and posting only the approved items where the team works.Implementing a well-specified Product Backlog item and opening a pull request that the Developers review against the Definition of Done and development standards.Checking the Product Backlog weekly for stale, duplicate, or unrefined items and proposing changes to the Product Owner.Scanning the team’s codebase nightly for vulnerable dependencies and opening draft fixes.Publishing release notes from the shipped changes once the Product Owner has approved the draft. Requires: A named review standard for every output and enough review capacity to apply it. (Capacity refers to humans; an LLM-as-a-judge-only approach is a tricky proposition.) Where it fails: Output grows faster than the team’s capacity to review it, and approval shrinks to a click. Organization Chatbot Asking “Who decided X, when, and why?” across the organization’s documents.Asking which teams work on anything related to a given company objective.Checking governance and compliance questions, for example, whether customer data may be used in an experiment.Synthesizing past quarterly reviews before a strategy session.Querying the organization’s own user and market research. Requires: Current, well-kept documentation and clear permissions. Where it fails: It retrieves what the organization wrote down, including every stale and contradictory page. Organization Agent Producing a daily digest of the past 24 hours of customer feedback and support issues, compared with the product roadmap, listing gaps and emerging patterns for a product leadership report.Scanning all teams’ plans for dependencies before they turn into blockers.Compiling a weekly portfolio report from the actual work instead of from status meetings.Routing product usage anomalies to the owning team.Tracking competitor releases and comparing them with the roadmap. Requires: Current data from several sources, clear decision rights, and governance over what the agent may do unattended. Where it fails: Automated reporting replaces the conversation, and the agents’ lists of gaps quietly become the strategy. Deciding Which Boxes of the Jobs Matrix for AI Stay Empty Jobs used his matrix to cancel products, and an agile practitioner’s matrix should do the same. First, ask whether AI should touch the task at all, since a script, a checklist, or a ten-minute conversation often does the job better (the first question of the A3 Delegation System; see below). Second, before a use case moves into an agent box, name four things: Standard: What counts as good output.Owner: Who answers for the result.Permissions: What the agent may read, change, and send.Review burden: Who checks the output, how often, and at what cost. If you cannot name all four, do not delegate the task to an agent; keep a person in charge of the work, or leave AI out of it. That test also exposes a costly shortcut: leadership that starts with the Organization Agent box because it wants visibility first, dashboards are in high demand, and it seems likely to repeat the Agile transformations that began with tooling and reporting. The standard differs by output, which is why the Definition of Done and development practices cover code but not every piece of agent work: Code implementing a Product Backlog item: Review standard: The agreed-upon development practices and, likely, the Definition of Done.Who approves: The Developers.Action items from an event: Review standard: Each owner confirms their item.Who approves: The item’s owner.Release notes: Review standard: Accuracy against the shipped changes and a tone that fits the audience.Who approves: The Product Owner.Feedback digest and gap list: Review standard: Named sources and flagged unsupported claims.Who approves: The owner of the digest.Portfolio report: Review standard: Figures traceable to the work.Who approves: Whoever sends it upward. One Use Case, Two Boxes For example, take the customer feedback digest and keep it in the Organization column. As a chatbot use case, someone in product leadership asks Claude to examine last week’s support tickets and survey comments against the roadmap and decides which findings deserve a conversation with the teams. As an agent use case, an agent gathers feedback from the same sources every morning, compares it with the roadmap, and delivers a recurring list of gaps. Sources and columns stay the same, yet the move requires new authority, ownership, and review: someone must own the sources, define what counts as a gap, check claims the data does not support, and decide when to pause the digest. The agent may gather and compare, whereas deciding whether a gap deserves attention remains product strategy, a decision whose accountability stays with people. That is the rule across all four boxes: AI may prepare or carry out bounded work, and people keep the decisions and accountability defined for that work. If nobody can take on those jobs, leave the agent box empty. Jobs Matrix for AI – Conclusion: Find Your Empty Box Take last week’s AI use in your team, and put every instance into one of the four boxes. If you end up with an empty box, it reflects a decision about capacity and authority, so decide whether it is empty because nobody has got to it yet or because nobody could own it if you filled it. Key Questions This Article Answers What Was Steve Jobs’ 2×2 Product Matrix at Apple? Steve Jobs’ product matrix was a two-by-two grid he drew in 1997 to cut Apple’s lineup to four computers, with Consumer and Pro across the top and Desktop and Portable down the side. According to Walter Isaacson, Apple canceled everything outside the four boxes. By 1998, the iMac, Power Macintosh G3, and PowerBook G3 filled three boxes, and the iBook filled the consumer portable box in July 1999. What Is the Difference Between an AI Chatbot and an AI Agent? The difference is the authority the AI has after the request that starts it. A chatbot returns material for a person to read and decide how to use. An agent carries out a bounded task through several steps or systems, with defined review and approval points. A Developer can ask an agent to implement a change, and a schedule can trigger a plain chatbot summary. How Can Agile Teams Categorize Their AI Use Cases? Agile teams can sort AI use cases into a 2×2 matrix modeled on Steve Jobs’ 1997 product grid. The columns ask who must contribute, authorize, or respond: one Scrum Team, the Product Owner or Product Manager, or several teams and functions. The rows ask what authority the AI has: returning material (chatbot) or carrying out a bounded task (agent). A Retrospective summary is Team Chatbot; a daily customer-feedback digest compared with the roadmap is Organization Agent. When Should a Team Delegate a Task to an AI Agent? A product team should delegate a task to an AI agent only when it can name four things: the standard for good output, the owner who answers for the result, the permissions defining what the agent may read, change, and send, and the review burden. If any of the four is missing, a person stays in charge of the work, or AI stays out of it. Should Every Box of an AI Use Case Matrix Be Filled? Not necessarily: an empty box can reflect a sound decision about capacity and authority. Apple’s consumer portable box stayed empty until the iBook arrived in July 1999, and everyone could see what was missing. In an Agile organization, the Organization Agent box should stay empty when nobody can own the sources, define what counts as a gap, check unsupported claims, and decide when to pause the agent.

By Stefan Wolpers CORE

Most machine learning (ML) platforms still hand out accelerators through static quotas, spreadsheet-driven approvals, and hallway negotiation. That model was tolerable when capacity was cheap, and demand was steady. Neither holds for modern AI workloads. This article walks through why static allocation breaks down, the five components of a capacity orchestration layer (intent capture, policy, hardware brokering, continuous reallocation, and goodput accounting), and a 90-day rollout plan that won't destroy your users' trust in the scheduler. The Contradiction You Have Probably Already Measured If you run infrastructure for a large ML organization, you have likely seen a dashboard that says something uncomfortable. The fleet is 90-plus percent allocated. Teams are filing tickets about blocked capacity. And a sampled profile of the same fleet shows a meaningful fraction of devices doing no useful work. Buying more accelerators does not resolve this. It scales the inefficiency. The contradiction comes from a specific architectural choice: treating an accelerator reservation as ownership rather than as a lease with conditions. The industry has spent years optimizing acquisition: supply agreements, multi-year commitments, reserved cloud capacity. Comparatively little engineering has gone into what happens after the hardware lands in the fleet. Why Static Quotas Fail for ML Specifically Conventional service capacity planning works because service demand is autocorrelated. Today's traffic predicts tomorrow's. ML workload demand does not behave this way, for four reasons. Demand is event-driven. An eval result at 2 p.m. can turn a 16-device research project into a 512-device training run by the next morning. No quarterly quota cycle accommodates that. Jobs differ in shape, not just size. A gang-scheduled training job with all-reduce dependencies across 64 devices and a stateless inference replica are both "GPU requests" to a naive scheduler, but they have incompatible placement, topology, and preemption semantics. A fleet fragmented without topology awareness ends up with allocatable devices that no large job can actually use. Reservation and consumption diverge. A job holding 32 devices while blocked on an input pipeline is occupying capacity and producing nothing. Device-level utilization metrics will not reliably catch this, because a device spinning on a synchronization barrier can still report activity. Returning capacity is asymmetrically risky. This one dominates. If reacquiring capacity takes two weeks of negotiation, the rational move for every team is to hold everything they might need at peak, indefinitely. Static allocation rewards hoarding. Each over-request is a locally correct decision inside a badly designed system, which is why exhortations to "be good citizens" never fix it. The Architecture A capacity orchestration layer sits between your workload submission API and the underlying scheduler, whether that is Kubernetes with Kueue or Volcano, Slurm, Ray, or an internal system. It does not replace the scheduler. It feeds the scheduler better information and revisits its decisions after admission. 1. Intent Capture: A Device Count Is an Answer, Not a Requirement When a user requests "32 A100s," they have already collapsed their actual requirement into one number. The scheduler needs the requirement itself: what the workload can substitute, what it can tolerate, and what it cannot give up. In practice that looks like a spec along these lines: YAML apiVersion: capacity.internal/v1 kind: WorkloadIntent metadata: name: ranking-hparam-sweep spec: class: research-batch # drives policy, not placement preferred: accelerator: a100-80gb count: 32 substitutable: - accelerator: h100-80gb count: 20 # based on measured step-time on this model - accelerator: tpu-v5e count: 64 requiresRecompile: true # XLA path validated in CI topology: gangScheduled: true minInterconnect: nvlink-domain elasticity: minReplicas: 8 maxReplicas: 32 resizeCost: 90s # checkpoint + reshard preemption: tolerated: true gracePeriod: 120s checkpointInterval: 600s deadline: "2026-08-14T00:00:00Z" Two fields carry most of the value. substitutable is what makes cross-architecture brokering possible. preemption.tolerated, paired with a real checkpointInterval, is what makes a job safely reclaimable. Interruption tolerance should be verified, not self-declared. Run a preemption drill in CI: kill the job, confirm it resumes from checkpoint within the stated grace period. A self-reported tolerance that has never been exercised will eventually be exercised in production, on a bad day. Be honest about the engineering cost. Portable workload definitions, container images built for multiple backends, and checkpoint formats that survive a device-count change are all real work. Elastic resharding in particular is hard for models with complex parallelism strategies. Treat the full intent spec as something teams grow into. A job that declares only preferred still schedules; it just gets fewer options. 2. Policy Engine: Encode the Argument Once The point of a policy engine is not to remove human judgment. It moves judgment out of per-request negotiation and into rules that can be reviewed, versioned, and revised. On Kubernetes, Kueue's cohort model already expresses the core mechanics through nominal quota, borrowing, and lending: YAML apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: research-shared spec: cohort: fleet-accelerators preemption: reclaimWithinCohort: Any withinClusterQueue: LowerPriority resourceGroups: - coveredResources: ["nvidia.com/gpu"] flavors: - name: h100-80gb resources: - name: "nvidia.com/gpu" nominalQuota: 256 borrowingLimit: 512 # may burst into idle cohort capacity lendingLimit: 192 # keeps a protected floor of 64 devices Read those preemption lines together with the limits, because the pair is what changes team behavior. lendingLimit guarantees a floor that is never lent out, and reclaimWithinCohort: Any guarantees that lent capacity comes back the moment the owner needs it, by preempting borrowers if necessary. Once returning idle capacity carries no risk of losing it, the economic case for hoarding mostly evaporates. Beyond quota mechanics, useful policy inputs include production criticality, launch dates, queue age, the historical efficiency of the requesting workload, and per-organization fairness floors. Dominant Resource Fairness (Ghodsi et al., NSDI 2011) is a sensible default for multi-resource sharing, with queue aging layered on top so low-priority work eventually runs instead of starving. One requirement is non-negotiable: explainability. Every scheduling decision needs a retrievable answer to "why is my job still queued?" naming the binding policy, the preempting workload, and a projected start time. Without that, engineers conclude the system is arbitrary and route around it. An orchestration layer people distrust is worse than static quotas, because now you have a scheduler plus a shadow economy of private allocations. 3. Hardware Broker: One Portfolio, Not Several Fiefdoms Many organizations run accelerator families as separate silos, each with its own team, toolchain, and queue. The fragmentation hides substitution opportunities and strands capacity on whichever side happens to be idle. A broker matches substitutable declarations against live supply and makes the tradeoff explicit: 32 A100s in four hours, versus 20 H100s now, versus 64 TPU v5e chips now plus a 12-minute recompile. Comparing those options requires measured throughput per accelerator class per model family, which in turn means maintaining a benchmark corpus and refreshing it as software evolves. Compiler-level changes alone can shift the relative ranking of hardware classes over time; our work on compiler autotuning for TPU fleets (CATWILD, MLSys 2026) found that tuning decisions which were optimal at one point degrade as workloads and toolchains drift, and the same staleness problem applies to any performance model a broker relies on. A stale model will confidently route work to the wrong hardware. Scope this realistically. Workloads with hand-tuned CUDA kernels, custom collectives, or architecture-specific memory layouts are not portable, and forcing portability on them wastes effort. The goal is narrower: preserve flexibility where it is technically sound, and make sure the jobs that genuinely are portable are visible to the scheduler as such. 4. Continuous Reallocation: Placement Does Not End at Admission Traditional schedulers decide placement once, at admission, and then stop thinking. AI infrastructure needs a control loop that keeps asking whether each device is still attached to the highest-value feasible work. Conditions the loop should detect and act on: Idle reservations. Allocated devices with sustained near-zero productive compute past a grace threshold get reclaimed.Over-provisioned elastic jobs. When scaling efficiency degrades past a threshold, the marginal device is buying almost nothing. Resize down and return the difference.Blocked jobs. A job stalled on data loading, or an external dependency for longer than its checkpoint interval, gets checkpointed and requeued rather than left squatting.Priority inversions. A launch-critical workload arriving after lower-priority jobs have claimed capacity triggers graceful preemption, per declared policy.Substitution openings. A portable queued job starts now on a different accelerator class instead of waiting for its preferred one. The safety constraints matter as much as the actions. Rate-limit preemptions per team per hour. Never preempt a job whose checkpoint drill has not passed. Cap how many times a single job can be preempted before it gets promoted, or some unlucky workload will never finish. And write every reallocation event into the job's own log with a reason attached. Unexplained preemption is the fastest way to lose organizational support for the whole system. 5. Goodput Accounting: Allocated Is Not Productive You cannot manage what your metrics obscure, and device-level utilization obscures a lot. A layered view separates the questions: LayerQuestionRough definitionAllocation rateIs the fleet handed out?Allocated device-seconds / total device-secondsOccupancyAre allocated devices active?Active device-seconds / allocated device-secondsGoodputIs the activity producing committed progress?Device-seconds contributing to non-discarded work / allocated device-secondsMFUHow close is the work to the hardware roofline?Achieved model FLOPs / peak FLOPs Goodput is the layer that changes behavior. Computing it means joining hardware telemetry with job-level progress signals such as steps committed, checkpoints written, and requests served, so that restart replay, discarded partial work, and stalled collectives count as loss rather than as utilization. A training job that crashes and replays six hours from a stale checkpoint looks fine by occupancy and terrible by goodput, and goodput is telling the truth. Model FLOPs utilization, introduced in the PaLM paper (Chowdhery et al., 2022), is worth tracking but answers a different question. MFU is a compiler and kernel efficiency signal, not an allocation signal. A job can be perfectly scheduled and still leave performance on the floor; that is a different team's bug. The Fairness Constraint A system that optimizes purely for near-term commercial value fails predictably. Exploratory research never runs, because its value is uncertain by definition, and small teams lose every contest against large ones. The safeguards are concrete: a protected research pool that revenue workloads cannot preempt, queue aging that raises effective priority over time, per-team caps within each cohort, and reservation expiry so held capacity gets re-justified periodically. All of these cost measured efficiency, and that cost should be budgeted deliberately. The alternative is an infrastructure layer that quietly defunds the products you have not built yet. A 90-Day Rollout Days 0 to 15: instrument before changing anything. Inventory accelerator types, ownership boundaries, queue mechanisms, and approval paths. Separate reservation time from productive execution time in telemetry, and publish the gap. If you cannot produce a defensible goodput number, everything downstream is guesswork. Expect the first honest measurement to be politically uncomfortable. Days 16 to 45: write the allocation contract. Define priority classes, fairness floors, preemption rules, reservation expiry, and substitution policies, then circulate them with engineering, infrastructure, and product leadership before automating anything. Decide which workloads carry hard guarantees and which are explicitly flexible. This phase is mostly negotiation rather than implementation, and skipping it is the most common way these projects die. Days 46 to 90: automate one bounded pool. Choose a cohort with a contained blast radius. Batch research capacity is usually right, since it tolerates interruption. Enable reclamation, resizing, and substitution there only, and track queue time, end-to-end completion time, goodput, failed migrations, and policy exceptions. Then hold. Do not expand scope until teams trust both the decisions and the explanations behind them. Trust, not scheduler throughput, is the binding constraint on adoption. Closing Accelerator demand is not going to flatten. Models grow, inference spreads into more product surfaces, and experiment volume rises with headcount. Newer hardware improves per-device throughput while doing nothing about contention; if anything, it raises the stakes, since a stranded H100 is a more expensive mistake than the stranded V100 it replaced. Organizations that scale well will stop treating accelerators as assets assigned to teams and start managing them as a continuously rebalanced portfolio governed by explicit, explainable policy. Procurement still matters. Orchestration determines how much of what you procured turns into shipped product. The most valuable accelerator in your fleet is not the next one you buy. It is the one you already own, currently attached to the wrong job.

A research agent can spend minutes or hours accumulating search results, model outputs, citations, and intermediate synthesis. Losing that work because a Python worker restarts is not an acceptable failure mode. The useful design target is therefore not an immortal worker process but a disposable worker whose completed research remains durable. Temporal’s official LangGraph integration provides that boundary: LangGraph still defines agent state and control flow, while Temporal supplies durable execution, Activity retries, timeouts, and replay-oriented recovery. As of September 2026, the integration is in Public Preview; its Python API is still marked experimental, and it requires temporalio 1.27.0 or later. Make the Process Disposable, Not the State Temporal treats a Workflow Execution as durable state rather than as the lifetime of one worker process. After a crash or infrastructure interruption, execution can be reconstructed and resumed; Temporal explicitly describes its execution model as surviving crashes, network failures, and infrastructure outages. That property changes how LangGraph persistence should be approached. In a conventional LangGraph deployment, a checkpointer persists thread-scoped graph state, and InMemorySaver explicitly loses checkpoints when the process restarts. Under the Temporal integration, however, Temporal owns execution durability; the integration documentation specifically recommends InMemorySaver only when LangGraph itself requires a checkpointer, such as for interrupts, rather than adding PostgreSQL or Redis merely to duplicate Workflow durability. That separation matters because two persistence systems for the same execution create two authorities for progress. A database checkpoint may represent one view of graph advancement while Temporal history represents another. The cleaner model makes Temporal’s Workflow history the execution ledger and keeps LangGraph state suitable for durable orchestration. External storage still has a role for application data, documents, embeddings, and cross-run memory, but not as a second scheduler checkpoint for the same run. Temporal’s integration makes this ownership explicit by assigning execution durability to Temporal while retaining LangGraph’s graph semantics. Put Uncertainty Behind Activities The integration requires every LangGraph node or Functional API task to declare where it executes. Network calls, LLM requests, database access, file I/O, current-time reads, randomness, and other nondeterministic operations belong in Temporal Activities. Pure state transformations and lightweight routing can run inside the Workflow, where replay requires deterministic behavior. Activity nodes receive Temporal timeout and retry policies; the plugin rejects LangGraph retry policies for these nodes and expects Temporal RetryPolicy configuration instead. A research graph can make that boundary explicit without changing the overall LangGraph programming model: Python research = StateGraph(ResearchState) research.add_node( "search", search_sources, metadata={ "execute_in": "activity", "start_to_close_timeout": timedelta(minutes=2), "retry_policy": RetryPolicy(maximum_attempts=4), }, ) research.add_node( "merge", merge_evidence, metadata={"execute_in": "workflow"}, ) research.add_edge(START, "search") research.add_edge("search", "merge") The search node is an Activity because its result depends on an external service and may fail transiently. The merge node can remain in the Workflow when it only combines already-recorded evidence. This is the central recoverability rule: expensive uncertainty crosses an Activity boundary; deterministic orchestration stays replayable. Conditional-edge functions deserve the same discipline because the integration always runs them in the Workflow and requires them to be deterministic and asynchronous. Let Temporal Become the Execution Ledger The graph becomes available to Workflows through LangGraphPlugin, while execution occurs on a normal Temporal Worker. Shared timeout and retry defaults can live on the plugin, but execute_in still has to be declared on each node so that an accidental nondeterministic Workflow node cannot hide behind a global default. Python plugin = LangGraphPlugin( graphs={"research": research}, default_activity_options={ "start_to_close_timeout": timedelta(minutes=2), "retry_policy": RetryPolicy(maximum_attempts=4), }, ) worker = Worker( client, task_queue="research-agent", workflows=[ResearchWorkflow], plugins=[plugin], ) await worker.run() Inside the Workflow, invoking the registered graph remains small: Python app = graph("research").compile() result = await app.ainvoke({ "query": request.query, "evidence": [], "draft": None, }) If the worker disappears after an Activity has completed and its outcome has become part of durable execution, recovery does not depend on reconstructing arbitrary Python process memory. If an Activity attempt is interrupted by a worker crash or transient failure, Temporal can retry that Activity according to its policy rather than restarting the entire research graph. Temporal’s LangGraph documentation explicitly identifies worker crashes as a condition under which an Activity-wrapped node can rerun, while Temporal’s platform model preserves Workflow execution across process and infrastructure failures. Retries also change the design of external writes. Any Activity whose side effect can survive a failed attempt should carry a stable operation key so a retry resolves to the same logical write rather than creating another one. This follows directly from the fact that Activity attempts can rerun after failures. A persistence Activity can use the run and revision as that identity: Python async def save_draft(state: ResearchState) -> dict: operation_id = f"{state['run_id']}:{state['revision']}" await repository.upsert(operation_id, state["draft"]) return {"saved_revision": state["revision"]} The retry-safe behavior in this example comes from the storage contract around operation_id, not from assuming that an Activity executes only once. Keep Memory and Progress Separate LangGraph’s normal persistence model distinguishes thread checkpoints from longer-lived stores. Temporal narrows that choice inside this integration. The documentation states that LangGraph Store objects are not available inside Activity-wrapped nodes because live store state cannot cross the Activity boundary and an Activity may execute on a different worker. Per-run memory therefore belongs in Workflow or graph state. Shared memory across runs belongs in an external database that every relevant worker can access independently. This distinction also prevents a subtle mistake around InMemorySaver. LangGraph documentation correctly warns that InMemorySaver is process-local and disappears on restart. Temporal’s integration can still recommend it for LangGraph features such as interrupt() because it is not being asked to provide crash durability there; Temporal is. The two statements are compatible once the ownership boundary is explicit. Streaming needs similar treatment. Activity-side LangGraph streaming has at-least-once delivery semantics per Activity attempt. A failed attempt can publish chunks that appear again when the Activity retries because earlier stream writes are not rolled back. Progress streams should therefore be treated as advisory presentation data or deduplicated with sequence identifiers. Temporal’s documentation explicitly recommends deduplication or relying on the final Workflow result as authoritative state. Keep Long Research From Becoming Long History A deep-research agent may loop through dozens of retrieval, critique, and synthesis passes. Temporal records execution events, so sufficiently long graphs can eventually approach Workflow history limits. The LangGraph integration addresses this with Continue-As-New plus a task-result cache. cache() returns a serializable representation of completed LangGraph task results; passing that cache into graph(..., cache=...) in the new run allows previously completed nodes to avoid re-execution. Python app = graph("research", cache=prior_cache).compile() state = await app.ainvoke(state) workflow.continue_as_new( args=[state, cache()] ) The transition should occur at a deliberate research boundary before history becomes excessive. Continue-As-New starts fresh execution history while application state and the LangGraph result cache can be carried into the successor run. The distinction is important: a new Temporal run does not need to imply a new research effort. Previously completed node results remain reusable even as the accumulated history is reset for continued execution. Recovery Is the Feature A recoverable research agent is produced by assigning durability to the correct layer, not by trying to keep a worker alive forever. LangGraph should express the agent’s graph and state transitions; Temporal should own replay-oriented recovery, retries, timeout enforcement, and durable execution; external databases should hold genuinely shared application memory and retry-safe side effects. With nondeterministic work isolated in Activities, deterministic control flow kept inside the Workflow, streams treated as non-authoritative, and Continue-As-New used for long histories, a worker can be terminated at an inconvenient moment without turning completed research into lost work. The process becomes replaceable, while the research remains part of the durable execution.

By Akhil Madineni CORE

Throughout my career, I've watched sprints stall in the same place. Everything was going fine: the features were merged, the requirements were clear. But then we hit a roadblock: waiting on test cases. Because the QA team was hand-converting Jira stories into steps and expected results. Most test automation strategy conversations skip right past this authoring bottleneck. I've done the conversion work myself. It's not hard work, but it's necessary, slow, and repetitive. Worst of all, it's disconnected from the actual skills that made anyone want to work in quality engineering in the first place. In my previous article, I drew a line between two architectures that both get sold as "AI test generation." One is simply a large language model behind a prompt template. The other is an actual agent that reads your requirements, your attachments, and your existing test libraries before it writes anything. Agentic test creation means an AI agent reads your requirements, attachments, and existing test library, then drafts structured test cases traced to each acceptance criterion, with a human review gate before anything enters the suite. That article included a 6-step example of what an agentic pipeline does with a promo code user story. The result: 7 cases reused, 6 generated, 1 of which was rejected at review. This time I want to expand that example into a full walkthrough, drawn from my own experience. The example isn't special. These are the same plain-language requirements your team probably already writes. But the process and outcome are interesting. Starting With a Plain-Language Requirement When I say plain-language requirements, I mean the artifacts an agile team already produces. User stories. Acceptance criteria, whether they're bullet points or structured Gherkin scenarios. The Jira ticket itself, with its comments and revision history. Even the wireframe or screenshot somebody attached during refinement. None of it is specifically written for an AI. But the good news is that none of it needs to be. That's the premise behind AI test case generation from user stories: the story is the spec, and its criteria are the test conditions. It's also what shift-left testing looks like in practice. If the requirement itself is the test input, test design starts the moment the story is written. The Agent Loop vs. an LLM Call I covered the agent loop in detail last time, so I'll just give an overview here. A typical single LLM call is a stateless function. You call, you get an answer. You move on to the next call. But an agent is a loop: it reasons about a goal, calls tools to gather information, observes the results, and revises its plan before producing output. That's the reasoning-and-acting loop the ReAct paper formalized (Anthropic's Building Effective AI Agents is my recommended primer). Applied to test creation, the agent loop parses the story and its criteria, pulls the linked test cases, reads the attachments, and builds a coverage map — all before generating any steps. The Agentic Test Case Walkthrough Here is our requirement: As a returning customer, I want to apply a promo code at checkout so that my discount is reflected in the order total. The acceptance criteria attached to the story (let's call it PROMO-214): A valid promo code reduces the order total and shows a discount line item.An invalid or expired code shows an inline error and leaves the total unchanged.A promo code and a gift card can be applied to the same order; the promo discount is applied first.Percentage discounts round half up to the nearest cent at the order level, not per item.Removing a code restores the original total. Plus there's an attachment: checkout-mockup.png, showing the promo field, the discount line, and the gift card entry point. That's our entire input package. Here's what it looks like: What the Agent Surfaces Before Generating Before writing anything, the agent first checks the test cases linked to the checkout area. And it finds 40! It maps the story's scenarios against them and proposes 7 for reuse: valid code, invalid code, expired code, empty field, case sensitivity, code removal, and re-application after removal. In a generic tool, those 7 would come back as duplicates. Here they arrive as reuse suggestions with their existing case IDs, so their regression history stays attached. The Output Our agent now generates 6 new cases against actual gaps. Here's one: TC-1207 · Apply a promo code and a gift card to the same order (AI-generated · awaiting review) Preconditions Returning customer signed in. Cart contains 2 items ($39.97 and $40.02) totaling $79.99. Valid 20% code SAVE20. Gift card balance of $30.00 on the account. Saved card on file. Step 1 Proceed to checkout. Expected: order summary shows $79.99. Step 2 Enter SAVE20 and select Apply. Expected: discount line shows −$16.00; total updates to $63.99. Step 3 Apply the gift card. Expected: gift card line shows −$30.00; total updates to $33.99. Step 4 Place the order. Expected: confirmation lists both adjustments; $33.99 is charged to the saved card; remaining gift card balance is $0.00. Traceability PROMO-214, acceptance criteria 1, 3, 4 (primary: 3) Let's look at what the agent did with acceptance criterion 3. The criterion states an ordering rule: promo first, then gift card. The generated steps verify the amounts in that order, with the expected results as the actual arithmetic. That arithmetic came from reading the criteria together: 20% off $79.99 is $15.998, which becomes $16.00 because criterion 4 rounds at the order level. Rounded per item, the same discount would be $15.99. The rate, the ordering rule, and the rounding rule all interact. A second generated case, TC-1208, targets the expired-code path in a checkout that already has a saved payment method on file. It renders like this: The remaining 4 cases cover rounding at the half-cent boundary, a gift card that exceeds the discounted total, code removal after a gift card is applied, and discount persistence across a session timeout. None of these are uncommon, but they're typical cases we might skip when we run out of time in a sprint. Just as telling is what the agent didn't generate. Nothing in the batch covers 2 browser tabs applying codes to the same cart, and nothing checks whether the inline error is announced to a screen reader ... because no criterion mentions either. The agent's coverage tracks what's written down—and when it strays past that, the review gate catches it. Deciding what should have been written down is still your job. One objection you might raise: these are structured test cases, not automated end-to-end scripts. That's true. But the structure is the point. A case with discrete steps and explicit expected results is an automatable artifact. Whether a person executes TC-1207, an automation engineer scripts it, or an execution agent picks it up downstream, the hard part is already done: what to verify, in what order, and with what data. Prose test ideas can't make that handoff. But steps with expected results can. Your Output Is Only as Good as Your Input Next, let's take a look at the limits of this approach. Like most solutions, agentic test cases have some best practices that should be followed. First, vague acceptance criteria produce vague steps. If criterion 4 hadn't specified order-level rounding, the agent would have had to infer a rounding rule, and a reviewer would have needed to confirm it. What Does a Testable Criterion Look Like? So what does a testable criterion look like? Compare "discounts should work with gift cards" to criterion 3 above: a promo code and a gift card can be applied to the same order, and the promo discount is applied first. The first version tells the agent a feature exists. The second gives it an ordering rule it can verify with arithmetic. A few habits close this gap: give ordering rules explicitly, specify rounding and limits, name the expected error behavior, and attach your mockups. None of this is new, right? Testers have been pushing for the best practices for years. The agent just makes it more important. Second, a thin or messy test library weakens the reuse step. An agent can only propose reusing cases it can find! And finally, treat the review gate as mandatory. In the original run of this example, our reviewer edited 2 cases and rejected 1. In my previous article, I recommended tracking the reviewer rejection rate; I'll refine that here: track 2 numbers separately, the rejection rate and the edit rate. Rejection rates going up usually means the context feeding the agent is broken. Maybe it's a thin test library, or stories that aren't linked to their cases, or even criteria that are silent on a whole scenario. Edit counts going up usually means your acceptance criteria are vague. Rejected Test Cases Let's pause for a moment to look back at our rejected case from earlier. We didn't dig into that much. Agentic failures sometimes don't look like failures. In this example, our reviewer rejected a session-timeout case. It had clean steps, specific expected results, and was formatted exactly like the other 5. But the problem was that we gave no acceptance criterion on what happens to a discount when a session expires. The agent inferred a behavior and then tested its own inference. That's the pattern you should use to train reviewers: the most dangerous output isn't the sloppy case; it's the really good-looking case... that just happens to verify a requirement that no one wrote. The good news is that the rejection itself was useful. It made its way back to the product owner as a requirements question, which is precisely the kind of gap that previously wouldn't surface until production. Implementing Agentic Test Creation Finally, let's look at options for implementing agentic test creation. As is often the case, you can build, or you can buy. Buy. Vendors have started building true agentic solutions. For example, Agentic Test Creation in Tricentis qTest is one implementation of this pattern. It runs the loop inside the test management platform itself, so the reuse suggestions and the review gate land where the test cases already live. Build. You can also build this yourself. The building blocks are all there: the ReAct loop, an LLM that calls your tools, and community-maintained open-source connectors like the MCP Atlassian server that lets an agent read Jira stories directly. A motivated platform team can assemble this pipeline themselves. The benefit to buying? The plumbing. Existing test libraries, traceability links, and review workflows are already wired together. This can matter more than you might think. Reuse detection is only as good as the agent's view of what exists, and a platform that already holds your test library already has that view. Same story for the review gate: it's a workflow with roles, permissions, and an audit trail, which is exactly the kind of thing that's boring to build and easy to underestimate. The benefit to building, of course, is flexibility. You pick the model, you own the prompts, you can encode your team's house style for test cases, and you can wire in internal systems no vendor will support. But be sure you budget honestly. Writing an agent that drafts steps from a story might be a weekend prototype. But the review workflow, the permissions, and keeping the prompts current as your requirements evolve are real work. That you now own. How to Evaluate Agentic Test Creation Tools Whichever route you take, I'd ask these 3 questions: Does the solution read everything attached to the requirement, including images?Does the solution propose reuse before it generates?Does every machine-written case pass through a review gate before entering the project? A test automation strategy is ultimately a decision about where humans create the most value. When structured test cases can be drafted from the requirements your team already writes, the hours that went into transcription can instead move to the work only humans can do: exploratory testing, risk analysis, and deciding what should be tested in the first place. Revisiting Your Test Automation Strategy My readers may recall my personal mission statement, which I feel can apply to any IT professional: "Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else." — J. Vester Your team already writes user stories and acceptance criteria. Those artifacts are enough to drive end-to-end test creation, provided the system reading them knows what already exists. Transcribing them into test steps by hand never really added much value. But a context-aware agent drafted from them just might. Have a really great day!

By John Vester CORE

Software organizations require rules to ensure predictability, knowledge sharing, and sound technical decisions. However, excessive rules, processes, and standards can obstruct progress if they eclipse desired outcomes. This longstanding tension in software engineering is now more pronounced as AI accelerates code generation, solution exploration, and implementation of change. The main challenge is not the number of rules, but how they fit with context, autonomy, and responsibility. Some organizations have minimal constraints, others tightly control decisions, and a few adapt principles to specific situations. Noticing these models helps engineering leaders assess their organization, understand associated risks, and move toward more effective technical governance. Software Architecture Is Also an Organizational Problem Software architecture is often defined by technical elements such as components, APIs, databases, infrastructure, quality attributes, and architectural styles. While these matter, they exist within a wider context. People design and maintain software, influenced by business constraints, communication structures, incentives, policies, and authority levels. Architecture is not only a technical artifact; it also reflects the environment in which teams make technical decisions. This broader perspective reflects the socio-technical nature of software engineering and is consistent with standards such as ISO/IEC/IEEE 42010, which recognize that architectural concerns encompass organizational, economic, regulatory, and community factors. Conway's Law shows that organizational communication structures often mirror system design. If organizational structure shapes software, then the distribution of authority, management of disagreement, response to failure, and governance of technical decisions likewise influence architecture. Therefore, engineering culture and governance are architectural concerns, not just management issues. Culture Shapes How Engineering Decisions Are Made If architecture is formed by its organizational context, organizational culture also directly influences engineers’ daily decisions. Sociologist Ron Westrum offers a system for understanding how organizations respond to problems and opportunities, particularly in high-risk environments. His model focuses on information flow, including cooperation, safe communication of bad news, responsibility management, failure investigation, and acceptance of new ideas. Westrum identified three main cultural patterns: pathological (power-oriented), bureaucratic (rule-oriented), and generative (performance-oriented) organizations. This distinction matters in software engineering, where architectural decisions depend on information moving across teams, technical boundaries, and leadership levels. Organizations that conceal problems, discourage disagreement, or fragment responsibility make different technical choices than those that discuss risks openly, encourage collaboration, and treat failures as chances to learn. DORA adopted Westrum's model in its research and consistently links generative, high-trust cultures to stronger software delivery and organizational performance. Culture isn't just the environment around engineering work; it shapes how teams interpret technical information, who can challenge decisions, how rules are enforced, and how architecture evolves. PathologicalBureaucraticGenerativePower orientedRule orientedPerformance orientedLow cooperationModest cooperationHigh cooperationMessengers “shot”Messengers neglectedMessengers trainedResponsibilities shirkedNarrow responsibilitiesRisks are sharedBridging discouragedBridging toleratedBridging encouragedFailure leads to scapegoatingFailure leads to justiceFailure leads to inquiryNovelty crushedNovelty leads to problemsNovelty implemented Mode 1: Wild West Engineering Wild West Engineering refers to environments with weak governance, unclear ownership, and few shared engineering principles. Teams may appear highly autonomous, but such autonomy lacks architectural direction, reliable feedback, and consistent accountability. Technical decisions are frequently reactive and localized, with tools and approaches selected for immediate needs or personal preference rather than the wider system context. The problem is not simply a lack of rules. Experienced teams can operate effectively with minimal rules if they share strong principles and understand the impact of their decisions. Wild West Engineering occurs when autonomy exists without sufficient shared context, accountability, or coordination. Over time, architectural knowledge becomes siloed, similar problems are solved inconsistently, and changes become riskier as no one fully understands the system. Westrum’s pathological culture is a relevant comparison: cooperation is limited, responsibilities are avoided, bad news is suppressed, and failures result in blame rather than inquiry. In software organizations, this often creates a hero culture where a few engineers become indispensable for managing disorder. AI can worsen this problem; when teams generate code, add dependencies, or make architectural decisions without shared guardrails, inconsistency and technical debt can spread quickly. Indicators Typical signs involve unclear ownership, multiple solutions for similar problems, undocumented architectural decisions, reliance on tribal knowledge, fear of changing legacy components, recurring blame after incidents, and dependence on a few key engineers. Another important sign is resignation: engineers stop proposing improvements because changing the system seems riskier than leaving existing problems unresolved. How to Move Forward Imposing heavy bureaucracy is not the solution to chaos. The first step is to establish minimum shared guardrails: clarify ownership, document key architectural decisions, define core engineering principles, set baseline expectations for security and operability, and create safer ways to discuss failures and technical debt. The goal is to support autonomy with enough structure to ensure local decisions conform to a coherent system. Mode 2: Bureaucratic Engineering Bureaucratic Engineering represents the opposite extreme, where governance is pervasive. Standards, approval processes, architectural reviews, mandatory technologies, and detailed procedures dictate how teams operate. While this structure creates predictability and consistency, problems arise when adherence to process outweighs knowing its purpose. Teams may focus on compliance rather than evaluating whether rules remain relevant. Westrum’s bureaucratic culture reflects similar traits: limited cooperation, narrow responsibilities, minimal cross-boundary collaboration, and a tendency to view novelty as problematic. In engineering, this occurs when architectural decisions are centralized, outdated standards persist, and exceptions are hard to secure. While these mechanisms may guarantee a baseline of quality, they may also limit performance. The core issue is not the presence of rules, but the erosion of judgment. Governance becomes rigid when organizations treat principles as permanent mandates and compliance as a substitute for engineering quality. AI can increase this tension, as organizations may respond to new risks with broad restrictions, approved-tool lists, or complex authorization processes. Although these measures may reduce risk, they can also hinder teams from pursuing valuable opportunities, making the organization safer but less nimble and flexible. Indicators Typical signs include rules missing explicit justification, standards defended only by “we have always done it this way,” centralized approval for minor technical decisions, difficulty obtaining exceptions, and architectural decisions that ignore local context. Status quo bias helps explain this persistence. Samuelson and Zeckhauser found that people often prefer existing or default options. In engineering organizations, this reinforces architectural inertia: once a technology, methodology, or rule becomes standard, replacing it requires far more justification than maintaining it. As a result, rules persist not because they are optimal, but because they are easier to retain. Organizational silence is another key warning sign. Morrison and Milliken define this as a collective tendency to withhold concerns when employees believe speaking up is unwise or ineffective. In engineering, this occurs when developers privately disagree with decisions but remain silent in meetings, believing that contesting established processes will not lead to change. Persistent silence should not be mistaken for agreement; it may signal that the organization discourages dissent. Another warning sign is when success is measured mainly by process compliance rather than software outcomes. Teams may meet all requirements yet become slower, less innovative, and disconnected from the initial intent of these controls. How to Move Forward To move beyond Bureaucratic Engineering, organizations should redefine governance rather than eliminate it. Leaders must frequently review the purpose of rules, distinguish primary constraints from preferences, and replace unnecessary approvals with clear principles. Teams need defined boundaries, authority to make decisions within them, and a clear process for exceptions when justified by context. A practical test is to ask whether a rule is still justified by its intended outcome. If no one can explain the risk it addresses, the value it protects, or evidence of its need, the organization may be maintaining the process for its own sake. The aim is to shift from prescriptive control to outcomes-based control, with accountability and feedback. Mode 3: Context-Driven Engineering Context-Driven Engineering integrates governance with local judgment. Rules are tools for managing risk and ensuring consistency, not ends in themselves. Teams work within defined principles, constraints, and responsibilities, while retaining the autonomy to adapt decisions to their particular technical and commercial context. The core assumption is that the same rule may yield different outcomes in different environments. This approach corresponds with Westrum’s generative culture, defined by high cooperation, shared risk, cross-boundary collaboration, inquiry following failures, and openness to innovation. In engineering, teams are expected to follow standards, understand their purpose, and recognize when exceptions are warranted. Architectural decisions are treated as explicit trade-offs, not automatic pattern applications. Teams evaluate technologies such as microservices, hexagonal architecture, event-driven systems, or specific cloud platforms based on the problem, constraints, and desired outcomes, rather than assuming they are universally correct. Applying a rule in the wrong context can lead to poor decisions. The same applies to architectural patterns: they address recurring problems in specific contexts, and applying them without considering context can create anti-patterns. For example, a lifeboat is essential in the ocean but irrelevant in a desert. Similarly, providing water is life-saving for someone dehydrated, but ineffective for someone drowning. A tool or rule's effectiveness depends on the context in which it is used. This environment requires greater maturity, not reduced governance. Autonomy must be balanced with accountability, feedback, and a readiness to revisit decisions as new information arises. AI fits well with this model, as its use can be governed based on context and risk: low-risk tasks may permit broad experimentation, while sensitive data, production changes, or high-impact decisions demand stricter controls. The objective is responsible judgment within clear boundaries, not unrestricted freedom. Indicators Key indicators include clear engineering principles, documented decision rationales, explicit ownership, and constructive disagreement. Teams should be able to explain both the rule and its rationale. Exceptions are allowed but must be justified. Failures prompt learning rather than blame, and architecture teams periodically review standards instead of letting them become outdated. Another strong indicator is that teams can contest established practices without equating disagreement with disloyalty or process violations. A context-driven organization distinguishes between non-negotiable constraints and situation-dependent choices. Security, legal requirements, data protection, and regulatory obligations remain strict, while implementation details such as architectural style, framework selection, or deployment strategy are evaluated locally. This distinction ensures autonomy does not devolve into unstructured or uncontrolled practices. How to Sustain This Model The main risk is regression. Insufficient discipline can lead to disorder, whereas excessive concern for consistency can create bureaucracy. Upkeeping this model requires ongoing review of principles, active feedback loops, documented architectural decisions, clear exception processes, and leaders willing to reexamine their assumptions. A practical test is whether the organization can answer four questions: What problem are we solving? Which constraints are truly non-negotiable? What trade-off are we accepting? What evidence would prompt us to revisit the decision? When teams can answer these questions consistently, governance functions as a learning system rather than a control mechanism. Conclusion Three Engineering Governance models inspired by Ron Westrum Engineering organizations improve not by simply adding or removing rules, but by ensuring teams understand their purpose, have the required context, and are trusted to exercise judgment within clear boundaries. Wild West Engineering lacks the structure for long-term growth, while Bureaucratic Engineering frequently leads to rigidity and limits learning. Context-Driven Engineering seeks balance through prioritizing principles over prescriptions, fostering autonomy with accountability, and encouraging decisions based on trade-offs rather than routine. As AI accelerates code generation and change, maintaining this harmony is increasingly important. The most adaptable organizations will foster responsible, context-aware decision-making.

By Otavio Santana CORE

Development in small companies and startups often slows down at the boundaries between people and teams. A developer waits for a product decision. Frontend work stalls because the API response is unclear. QA discovers that two teams interpreted the same requirement differently. Three practical habits can reduce these delays: document the feature, discuss its technical risks, and agree on interface contracts before dependent implementations diverge. Keep each step proportional to the size and uncertainty of the task. Start With the Biggest Uncertainty Before distributing work, identify what the team needs to learn first. Unclear user experience? Start with a UI prototype. Frontend developers can use mocks to explore the flow while the team clarifies requirements.Uncertain business logic, performance, or integration? Start with a focused backend investigation or technical prototype.Unclear user problem or business value? Start with product discovery, involving users, product stakeholders, and technical specialists as needed. Having a UI does not automatically make frontend work the first priority. Choose the starting point that resolves the most important uncertainty, then agree on enough shared detail to let work proceed in parallel. 1. Maintain Feature Documentation and Define Responsibilities A concise Product Requirements Document (PRD) gives the team a shared explanation of what to build, why it matters, and how to evaluate it. For a small feature, a short page may be enough. QuestionWhat to recordWhat are we building?The intended behavior, MVP scope, and explicit exclusions.Who is it for?The users and the problem they face.Why is it needed?The expected benefit to users and the business.How will we evaluate it?Acceptance criteria, success metrics, a baseline or comparison group, and a measurement period. Name the person responsible for product decisions and identify the people needed to implement and validate the feature. These are responsibilities; a small team may combine several of them in one role. ResponsibilityTypical ownerSet goals, decide scope, and resolve product questionsProduct ownerAssess feasibility, design interfaces, and implement the featureDevelopers and technical leadDefine test scenarios and verify behaviorQA and developersDesign the user experienceDesigner, where neededDefine instrumentation and evaluate impactAnalyst or another explicitly assigned team member Example: A Post Recommendation System The following is an illustrative PRD. Its numerical targets are examples to agree on for a particular product, rather than measured results or universal benchmarks. SectionExampleProblem and hypothesisUsers may struggle to find relevant posts in a chronological feed. We expect recommendations based on their interests to improve content discovery.UsersReaders discovering posts; creators whose eligible posts can be recommended.MVP behaviorReturn up to 20 eligible posts, ranked by popularity within topics inferred from likes and subscriptions. Recompute lists every 24 hours. Exclude deleted posts and posts the requesting user cannot access when serving the response. Use a general popularity list when there is insufficient interest history.Experiment supportAssign eligible users to stable control and treatment groups and record recommendation impressions and clicks. Include this in the MVP so the first version can be evaluated.Out of scopeML models, similarity between users, and advertising recommendations. Decide whether to add these after evaluating the MVP.PerformanceIllustrative target: server-side p95 response time below 200 ms for serving precomputed lists at 1,000 requests per second on a representative test dataset. Measure the batch recomputation job separately and require it to finish within the 24-hour refresh window.Acceptance criteriaTests verify ranking on a fixed dataset, the popularity fallback, access filtering, and feedback recording. Likes and subscription changes affect the next scheduled recomputation. Load tests meet the stated latency target.Success measurementIllustrative primary target: a 10% relative increase in recommendation CTR versus the control group. Define CTR as clicks divided by recorded impressions. Plan an initial two-week experiment; estimate sample size before launch and report uncertainty if the result is inconclusive. Monitor seven-day retention and API error rate as guardrails, with acceptable thresholds agreed before launch.RisksWeak relevance, overexposure of already popular posts, and load spikes. Inspect recommendation diversity, provide a fallback, and test capacity before rollout. Technical notes can accompany the PRD, with implementation decisions reviewed during technical refinement. For example, a Go service could use PostgreSQL for source data and Redis for precomputed lists. A draft API might expose GET /recommendations for the authenticated user and POST /feedback for interactions. The team still needs to define schemas, error responses, authorization behavior, and pagination before implementation. Keep release acceptance separate from business success. A correctly implemented feature can fail to improve the chosen metric. That result should inform the next product decision. What if the Product Has No Documentation? Start with the workflow you are changing and the business rules most likely to be misunderstood. Link the relevant code, record open questions, and expand the documentation as the team learns. Documenting the entire system does not need to become a prerequisite for the next useful change. After release, compare outcomes with the original hypothesis. A weak result is a reason to investigate the feature, measurement, and assumptions. Some work provides value through reliability, lower operating costs, or reduced risk, and its evaluation should reflect that purpose. 2. Conduct Technical Refinement Technical refinement, sometimes called technical grooming, connects product expectations with implementation decisions. Its output should be a workable approach and a clear record of remaining questions. Ask a developer or technical lead to review the PRD and identify: Existing constraints and integration dependencies.Performance, security, and operational risks relevant to the change.Decisions that require input from product or another team.Unknowns that need a short investigation or prototype. Bring the relevant people together to resolve those questions. If the implementation cost changes the original assumptions, revisit the scope with the product owner. Keep the Process Proportional Agree on a timebox based on scope and uncertainty. For example, a modest feature might receive two days of initial technical analysis, with a named owner and deadline for each product question. A complex migration may require several investigations. These are planning choices to review as new information appears. Record decisions, tradeoffs, unresolved questions, and owners. There is usually no need to transcribe every discussion. Finish with a small implementation plan: what can run in parallel, what must happen first, and what evidence will show that a risky assumption holds. This process can reduce avoidable rework. It does not guarantee that every issue will be discovered in advance, so leave room to revise the plan during development. 3. Develop Using the Specification-First Principle For work that crosses an API boundary, agree on the contract early. An OpenAPI description can capture HTTP operations, request and response schemas, and other interface details. The contract should also be supported by examples and documented behavior where a schema alone is insufficient. Backend and frontend engineers review the contract together, including empty states, errors, and compatibility expectations.Frontend developers build against mocks that reflect the agreed contract.Backend developers implement the API and verify that its behavior matches the contract.QA and developers prepare API contract checks and derive end-to-end scenarios from the PRD, user flows, and business rules. API contract checks and end-to-end tests serve different purposes. Matching a response schema does not establish that a complete user journey works correctly. How This Reduces Waiting Consider an illustrative recommendation feature. If frontend developers invent a response shape while waiting for the backend, they may need to rewrite rendering and error handling during integration. Agreeing on the response schema, empty-list behavior, and refresh semantics first lets both sides work against the same assumptions. Mocks still need to match the implementation. Run compatibility checks in CI and update the specification, mocks, and tests together when the contract changes. Integrate early enough to expose mismatches before release. Specification-first development also leaves room for exploratory code. A short prototype may be necessary to discover whether a proposed interface is feasible. The aim is to agree on the contract before teams invest heavily in dependent implementations. For more on the approach, see Boost Efficiency With the Specification-First Principle. Keep Documentation Useful as the Project Evolves Documentation helps future team members understand behavior and the reasons behind earlier decisions. DORA's research on documentation quality links high-quality internal documentation with organizational performance and finds that it strengthens the impact of technical practices. Keep a small set of maintained resources close to the work: The PRD and the results of the feature's evaluation.Architecture decisions, API contracts, and relevant data models.Deployment and recovery instructions.Links to changes and the decisions behind them, in GitLab, a README, or another shared system.Test scenarios and acceptance checklists. Assign ownership and update these resources when behavior changes. Outdated documentation can mislead the next person just as missing documentation can leave them guessing. Conclusion These three habits address common sources of delay: Concise feature documentation gives the team a shared goal, scope, and definition of success.Technical refinement surfaces constraints and assigns owners to unresolved questions.Agreed API contracts support parallel development and reduce integration rework. Try them on one feature. Track time spent waiting for decisions, integration rework, and time from an agreed scope to release, alongside quality and product outcomes. Use what you learn to adjust the process. The practical goal is a team that can make decisions, build, and validate changes with less avoidable waiting.

TL;DR: The AI Workflow Inventory Finalizes the A3 Delegation System You probably know your own AI shortcuts: the colleague who drafts the stakeholder update, the nightly job somebody set up before they left, the interview notes that go through a model on demand. However, I want to challenge you: that perceived knowledge can create false confidence, because it feels like knowing the team’s way of working with AI, when it is still only a diffuse understanding of the practice. Ask the team to combine those individual accounts into one list, and you may discover how little of the whole anyone can see. The AI Workflow Inventory is the artifact for that list: one row per recurring task, refined into provisional task classes to prepare the team’s next AI delegation decisions. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translation, proofreading, challenging story arcs and article structures, and summarization. It is a production tool, not a substitute for thinking. Your Own Shortcuts Are the Easy Part When I published the AI Delegation Lifecycle in June 2026, the point before stage 1 was deliberately left without an artifact: the team was supposed to know which work it does with a model, at what frequency, and at what stakes, which I call a forensic analysis of your own workflow. However, as I quickly discovered, this assumption that a team could produce the analysis from memory was wrong, and updating the student reference for the A3 Delegation System earlier this month showed me its design flaw: The Routing Policy assigns an AI model tier per task class,The AI Definition of Done is written per task class (never per task), andThe Delegation Audit opens with a walkthrough of what the team has handed over. All three artifacts consume task classes, but none of the six stages of the A3 Delegation System produces them. The AI Working Agreement is the exception on the other side of the lifecycle: it records the team’s defaults and boundaries and can exist before the inventory is complete. The gap is easy to miss from the inside, because every individual on the team can answer for their own use, and nobody can answer for the sum: who else drafts with a model, which outputs leave the team that way, and which jobs run unattended. That sum, useful, undocumented, and unowned, is what I call “AI Debt,” and it works right up to the moment someone asks who is responsible for it. What One Row Looks Like in the AI Workflow Inventory The AI Workflow Inventory is a one-page canvas with an identity strip (team, owner, date, review date), a panel on where to look, a panel with five refinement rules, and the inventory table itself. Let me show you an example from a Scrum team: Number: I-01Task: Transcribe photos of Retrospective sticky notesProcess: Retrospective facilitationTask class: Transcriptions for internal useHow often: Per SprintWho runs it: Scrum MasterTool: GPT-5.6 SolOutput goes to: Stays in the teamData that enters the model: Photos of handwritten notes and names. The task is written as verb and object, the class is named by output and audience, and “output goes to” has three values: stays in the team, leaves the team, or customer-facing. The number travels to the Workflow Card later, so the row and the respective decision can be matched months apart. Two design choices carry most of the weight: The person and the tool sit in separate columns because people change and tools change while the task stays; a field that says “Anna, ChatGPT” makes it harder to update either one when Anna moves teams or the license changes.The last two fields ask different questions that teams routinely conflate: where the output goes tells you about the stakes on the way out, whereas what enters the model tells you about the exposure on the way in, and an output that “stays in the team” can still have been produced from photographs with names on them. The entry in the data field is a fact to review; it is not permission to keep uploading the names. What the canvas does not have is a life cycle stage or status column, and the omission is deliberate. The inventory records what happens today; the poster and the Workflow Cards show where a workflow stands in the A3 Delegation Life Cycle, and a retired workflow is a row in the Re-classification Log, identified by the same inventory number. A stage column would turn the pre-decision list into a second, competing status board, and agile practitioners already know how that ends from the Product Backlog in Jira and the “real” one in a spreadsheet debate. The AI Workflow Inventory takes work to maintain, but it reduces repeated documentation and decision-making: The team captures each recurring task once.Compatible tasks share one review standard and one routing decision instead of getting one each.The inventory number connects a task to every later decision about it.The life cycle status stays on the poster and in the records that already exist. Update the inventory when the work changes or when the team discovers, corrects, or refines what it has recorded. (I know, it sounds like a database schema, and I started preliminary work on how to design an application for the A3 Delegation System.) Where to Look The canvas names three sources for AI workflows, and the 30-day window matters because memory is a poor inventory tool, while last month’s outputs are evidence: Your meetings come first: which outputs of the events that repeat (planning, pipeline or campaign reviews, stand-ups, monthly reporting, Retrospectives, etc.) did a model draft, summarize, transcribe, or translate?Then your tools: each colleague looks at their own chat history from the last 30 days and brings the prompts, applied Skills, or agent tasks that come back every week or every Sprint, who wrote them, and how often they were rewritten; this is a self-report, and it has to be, since nobody should be reading anyone else’s conversations.Then your outbox: the emails, reports, tickets, and documents that left the team, and the question of which of them started as a model draft. Thirty days is the starting window. Once the obvious rows are on the page, ask about the work no chat history shows: the jobs that run unattended, the quarterly report, the thing that only happens at release time. A nightly test run, such as the one in the example below, may leave no trace in anyone’s recent chat history. The line that decides the session is the last one in that panel: list the shortcuts nobody approved as well, because that is where the AI debt sits. If people read the inventory as a check on whether they followed the rules, those shortcuts stay invisible, and the team ends up with a list that looks tidy and describes a team that does not exist. Therefore, going first with a shortcut of your own helps, as does explaining, before the first row, that recording a task does not approve its use: the session starts with understanding what happens today, and concerns that need immediate attention under the team’s existing boundaries will still be addressed. Provisional Task Classes and the Grouping Test The grouping rule creates a puzzle that a careful reader will spot: it asks whether two tasks share the same AI Definition of Done, the same reviewer, and the same model tier, and those are decisions for later stages of the AI Delegation lifecycle, which the team has not reached. The resolution is that classes in the first session are provisional. Name them by output and audience, test the grouping against whatever review standards and routing decisions the team already has, and where those are missing, leave the grouping provisional and revisit it as the team works through the stages. An AI workflow inventory can carry uncertainty; what it must not carry is a settled-looking decision nobody made. The five rules on the AI Workflow Inventory canvas are short: Capture first, judge later: The A3 decision comes in stage 1, after the row exists, and a row approves nothing.One row per recurring task: Four prompt revisions used for the same weekly update are one task, and the tool goes in its own column.Name the class by output and audience: “Transcriptions for internal use,” “status communication leaving the company.” A class name that fits any task (“AI support”) is too wide to tell anyone which standard applies. (That naming approach is not different from coding.)Same standard, same reviewer, same tier, one class: When one of the three differs, split the class.Every class goes to stage 1; every audit refines the list: A class without an A3 decision is AI debt with a date on it. Three failure patterns are worth watching for: the sanctioned-use list, where only approved use gets written down, and your existing AI debt stays invisible; the prompt or Skill catalog, with a row per prompt/Skill instead of per task, so the classes never form; and the one-time census, filled once and never refined, so that six months later the audit walks a stale list and everyone nods at rows that no longer run. Example: Four Rows From a Scrum Team Here is a full first pass from a fictitious Scrum team of seven: Transcribe photos of Retrospective sticky notes (Scrum Master, GPT-5.6 Sol, stays in the team; input: handwritten notes with names): Internal output can still involve sensitive input.Rewrite customer feedback tickets into Product Backlog item drafts (class: requirement drafts for internal use): A concrete task inside a reusable class.Draft acceptance criteria for new items (same class): A second task that may share that class, as long as reviewer, standard, and tier stay the same.Rewrite interview notes into the hiring feedback template (on demand, Scrum Master, leaves the team; input: candidate names): Informal use that an approved-use-only list would have missed. The last row is on the list only because capture came first; under a rule that lists only approved use, the person running it would have left it out, and the team would have gone into A3’s stage 1 without knowing candidate names were entering a model on demand. From Row to Workflow Card Each decided row of the AI Workflow Inventory becomes a Workflow Card within the A3 Delegation System: workflow, task class, the A3 category, and later the tier, the owner, and the audit dates; the card goes on the poster where the workflow stands today. The word “decided” matters: the row exists before the A3 “Assist-Automate-Avoid” decision; the card exists after it. A row without a card means one of two things, and the remedy differs: either nobody has decided, in which case the class goes to the A3 Framework next, or somebody decided and never recorded it, in which case the team confirms the decision and writes it down. Ask which it is before doing either. For the transcription row, that means: the AI Workflow Inventory gives it a number, a task, and a provisional class; A3’s stage 1 decides whether it is Assist, Automate, or Avoid, and knowing that GPT-5.6 Sol runs it today settles nothing; stage 3 settles the owner, and the Scrum Master who runs it this Sprint does not become the owner by default. Both decisions go on the card. Your First 60 Minutes with the AI Workflow Inventory Book the session, aim for eight rows, and treat it as a first pass. One way to split the time: State the purpose and the boundary (capture, not approval): 10 minutesWalk the three sources, including unapproved shortcuts and unattended jobs: 20 minutesForm provisional task classes with the grouping test: 20 minutesName the inventory owner and the review date, and agree on the next step: 10 minutes The next step is A3’s stage 1: take every class to the A3 decision with the people who run the tasks in the room. If a row needs attention under the team’s existing boundaries before that (candidate names in a model, say), address it; “capture first” gives the decision its own step and does not postpone it. At each Delegation Audit, the list gets two kinds of maintenance. Add the rows that appeared since the last audit, and reconcile the existing ones: does the task still run, does the same person run it, with the same tool, on the same inputs, for the same audience? When you retire a task, record that decision in the Re-classification Log under the same inventory number, and keep the link to its original inventory record. Conclusion Which recurring use would your team’s current list miss? If your team cannot describe where AI already contributes to recurring work, start with a 60-minute AI Workflow Inventory session. The list gives you a shared basis for A3 decisions and helps you find the workflows that individual recollection might miss; then choose where in the A3 Delegation System to apply the method first.

By Stefan Wolpers CORE