Monitoring and Observability

Why Long Chats Need Session-Level Guardrails (CRA) Who this is for: Anyone building chat features, support bots, internal Q&A, coaching tools, RAG assistants. The Usual Setup (and What It Misses) A typical flow: User sends a message.You run moderation, rules, or a small model on that message (sometimes the reply too).If it passes, the big model answers. That is per message. It does not really “remember” the story of the chat. In a long chat: Message 5 looks normal.Message 12 still passes your keyword list.By message 20, something is wrong only if you compare it to how the chat started. So you can pass every single check and still end up with a bad session. That gap is what we call CRA: risk that adds up across turns, not in one obvious line. Figure 1: Each turn can look “green” while the overall thread is not. CRA in Plain English CRA = Conversational Risk Accumulation Idea: Each turn might look okay on its own, but together they break the purpose of the chat or what your company is okay with. What to build: Keep a little session memory (not the full transcript in logs — think IDs, hashes, and scores). After each assistant reply, update a few numbers that describe “how this session feels right now.” Those numbers are hints for dashboards, alerts, and gentle UI — not a courtroom verdict. Three Simple Scores + One Total (Example) We use a small, fixed set of scores and one combined score. Version tag in code: cra_telemetry_v1. Figure 2: Three inputs, one combined CRA score. ScorePlain meaningHow you might compute it (conceptually)S1Topic driftCompare the user’s recent text to how the chat started (or a stated goal). If they wander far from that, S1 goes up.S2Sensitive-looking repliesThe assistant’s answer looks like it contains patterns you care about (fake email shapes, “API key” wording, etc.). This means “flag for review,” not “we proved a leak.”S3Refusal tone shiftingTrack refusal-style phrases in the assistant’s answers over time. If refusals seem to soften late in the thread, S3 captures that shape.CRAOverall session riskA weighted sum of S1, S2, and S3, plus a small extra bump if the user or assistant text looks like prompt injection playbooks. Example weights we used: 35% S1, 45% S2, 20% S3. Rule of thumb: If you cannot explain a score in one short sentence to a product manager, do not use it to auto-block users. Hard Guardrails = Simple, Fast, “No” Hard guardrails are rules, not vibes. They should be cheap and run before you waste tokens. Examples: Max request size – reject giant payloads (HTTP 413).Rate limits – cap requests per IP so one client cannot drain your budget (429).Known-bad phrases – block obvious “ignore all previous instructions” junk (400).“Don’t paste secrets” – block prompts that look like “here is my SSN” (400) with a clear error.Lock down outputs – if your product only allows certain actions, check model output and tool calls against an allowlist before anything runs. These are not CRA. They are basics. CRA sits beside them. Figure 3: Hard = block or validate. Soft = warn, log, nudge. Soft Guardrails = CRA-Friendly, “Heads Up” Soft means: warn, log, maybe show a banner — not silent blocking. After a response, the API can add fields such as: cra_soft_notices – short text for humans (“high drift”, “sensitive-looking wording”, …).cra_signals – numbers for debugging: S1, S2, S3, CRA, turn count. Why start soft: Rules and heuristics misfire. A user might ask for fake email examples for a demo; S2 might spike on purpose. That is why the score is a signal, not proof. Bonus: Cache Duplicate Questions (Save Money) If someone double-clicks Send or retries the same text, do not call the model twice. Cache key idea: Python normalize(question) + mode + endpoint Cache the JSON answer for a few minutes. Mark responses with something like cached: true so the UI can say “from cache.” Browser Tip: Don’t Mix Up “New Chat” and Old Intent If S1 uses “first message of this session” as the anchor, browser storage can fool you: a new tab can look like a new thread while an old “first message” is still stored. Fixes: Store the anchor per session_id, not one global value.Expire or rotate the browser session after idle time so deploys and stale tabs do not reuse the wrong anchor. Telemetry vs. Guardrails (Two Different Jobs) TelemetryGuardrailJobMeasure and learnBlock or change behaviorWhen it hurts youToo many logs, privacyFalse positives, angry usersCRAGood fitUse soft first; hard only after review In logs, avoid raw secrets. Prefer hashes, lengths, and labels (channel, product area). Three Lines for Your Security Reviewer CRA is about conversation behavior over time, not a replacement for database security or tool-permission design.Labels for “bad session” are rare in the real world — use CRA to prioritize review, not as automatic guilt.If weights are public, people might game them — keep basic hard rules and spot checks anyway. Rollout Order (Keep It Boring) Ship hard limits (size, rate, obvious injection, output checks).Add session logging with safe IDs.Show soft notices only inside internal tools first.Tune thresholds on real traffic.Only then add hard session actions (pause tools, re-auth, etc.). Takeaway One-message checks are not enough for long chats. CRA gives you a simple story and a small set of session scores. Hard rules stop obvious abuse; soft CRA helps you see drift before it becomes an incident. Start with telemetry. Add blocking only when you understand the false positives. About the author: Sanjay Mishra is author of two books, The SQL Universe and Oracle Database Performance Tuning: A Checklist Approach. His research spans RAG architectures, NL2SQL, LLM safety, and enterprise AI governance, with work published in IEEE Access, Springer LNNS, and SSRN. He speaks regularly at universities and industry events on applied AI and data engineering. Tags / topics: #LLM #Security #Guardrails #Observability #OpenAI #Architecture #Chatbots

Bug triage on a graphics engineering team is one of those tasks nobody really wants to own. A new crash report comes in, and somebody has to work out whether it looks like a known issue, what the stack trace points at, which subsystem the affected code lives in, and which sub-team should pick it up. The answers exist in the issue tracker, the source repo, and the architecture docs, but pulling them together by hand takes time. And the engineers best at it are the ones you least want spending hours on it. On our team, the archive of resolved bugs had grown to over 1,100 issues. That is a real corpus. It contains the answer to a lot of incoming questions, but only if you can find the right three or four entries quickly. The agent described here does that lookup automatically, combines it with crash log parsing and source code search, and produces a root cause analysis with a confidence score. Triage that used to take hours now takes minutes. This article is about the architecture choices: why AWS Bedrock with Claude, why OpenSearch with HNSW indexing, why DynamoDB for workflow state, and why ECS Fargate. None of these choices is unique. The reasoning behind them is what's portable. What the Agent Actually Has to Do Before the architecture, it's worth being concrete about the work. When a bug report arrives, the agent produces an analysis built on five signals: Historical pattern match against the knowledge base of resolved issues.Source code match against the repositories the trace points into.Crash stack analysis on the trace itself.Log evidence from whatever logs were attached or linkable.Fix ownership, derived from who has historically fixed bugs in the affected components. Each signal contributes to a final confidence score. The combination matters because no single signal is reliable on its own. A stack trace can match a bug that was fixed three releases ago, a source-code hit can be unrelated, and ownership data can be stale. A useful triage answer leans on multiple signals together. That is the work. The architecture exists to support it reliably, repeatedly, and without baking in assumptions that will hurt later. Why RAG, and Why These Pieces The obvious wrong move is to skip retrieval and pass the whole corpus to the model. Context windows aren't the bottleneck people think they are. Even when they're large, signal-to-noise gets bad fast, and cost and latency scale with input size. For any given bug, the relevant slice is small: a few prior tickets, a couple of source files, maybe one architecture doc. Retrieval-augmented generation (RAG) is the right shape because the retrieval layer's job is precisely to find that slice. OpenSearch With HNSW Indexing The knowledge base lives in OpenSearch with vector search over a k-NN HNSW index. HNSW (Hierarchical Navigable Small World) suits corpora in the low thousands to low millions of documents. Query time stays low, and recall stays high without the tuning effort IVF-based indexes demand at smaller scales. OpenSearch was chosen over a dedicated vector database for operational reasons. It runs in the same AWS environment as the rest of the stack, supports keyword and vector search in the same index when you need hybrid retrieval, and doesn't add a new vendor to the diagram. For a team-internal tool, the integration cost of a separate vector DB outweighs the marginal performance gain. Titan Embeddings Embeddings are generated with Amazon Titan. The main reason: the data (bug reports, stack traces, code snippets) never has to leave AWS. That removes a class of compliance questions that come up the moment you start sending source code or internal tickets to an external embedding API. Titan handles technical text well enough for this corpus, and it shares IAM, quotas, and billing with everything else. Claude on Bedrock as the Reasoning Model The reasoning step takes the retrieved context and the parsed crash log and produces the actual analysis. It runs on Claude through Bedrock. Two properties matter here. First, Claude handles long, messy, structured input well: stack traces aren't clean prose, and the surrounding context is a mix of code, logs, and ticket descriptions. Second, it expresses uncertainty rather than picking a confident-sounding wrong answer. For a system whose output a human engineer is going to read and either trust or push back on, that calibration matters more than fluency. The Five-Signal Confidence Score The most consequential part of the system isn't the model call. It's the scoring layer that wraps it. The agent doesn't just say "this looks like a duplicate of bug X." It produces a confidence score, and that score is what triagers use to decide whether to accept the suggestion or dig in themselves. The score is a weighted combination of the five signals listed earlier. Each contributes a sub-score; the weights reflect how predictive each signal has been, in this team's experience, of a correct triage outcome. The interesting design choice is that the weights are not static. Real bug reports don't always include all five signals. Some arrive without attached logs. Some point at code with no clear ownership history. With static weights, missing signals would drag the final score down even when the available signals were strongly aligned. The agent redistributes the weight of any unavailable signal across the available ones, normalized to sum to one. The conceptual shape: Python # Conceptual sketch of dynamic weight adjustment BASE_WEIGHTS = { "historical_match": w1, "source_code_match": w2, "crash_stack": w3, "log_evidence": w4, "fix_ownership": w5, } def adjusted_weights(available_signals): active = {k: v for k, v in BASE_WEIGHTS.items() if k in available_signals} total = sum(active.values()) return {k: v / total for k, v in active.items()} This is a small piece of code that does a disproportionate amount of the work of making the agent's output trustworthy. A given confidence score should mean roughly the same thing whether the bug arrived with logs or without. DynamoDB for Workflow State A triage run is not a single API call. The agent parses the report, retrieves embeddings, runs vector search, fetches matched documents, pulls source code context, calls the reasoning model, computes the score, and writes results back. Each step can fail or be slow independently. Workflow state for each in-flight triage lives in DynamoDB. The schema is intentionally simple: a triage ID as the partition key, a status field, and the accumulated context. Two reasons it's external rather than in-process memory. First, recovery. If the model call fails or times out, the workflow should resume without redoing the embedding and retrieval work. Token costs add up otherwise. Second, observability. The Flask dashboard the team uses to monitor triage operations reads from this same DynamoDB table. That includes real-time status, filterable history, analytics, and the routing view for issues that don't belong to this team. There is no separate event log to maintain. Workflow state is the source of truth, and the dashboard is a view onto it. ECS Fargate for Orchestration The triage workflow runs on ECS Fargate. The choice is shaped by what the workflow looks like: a sequence of calls to external services (Bedrock, OpenSearch, the issue tracker), with the long pole being model latency. Not CPU-heavy, not bursty. Incoming bugs arrive at a steady rate. Fargate handles this shape cleanly. No cold start, no execution time limit, and the operational model is straightforward: container in, container out, IAM and networking inherited from the cluster. The Flask dashboard runs in the same Fargate cluster, sharing the same VPC and observability tooling. The general pattern: short, stateless, bursty work fits Lambda. Orchestrated workflows with slower external calls and a need for predictable behavior fit Fargate. For a team-internal agent that runs continuously, Fargate's properties matter more than its slightly higher baseline cost. Keeping the Knowledge Base Current None of this works if the corpus goes stale. The ingestion pipeline syncs three sources continuously: the issue tracker, where newly resolved bugs become new entries; the documentation repo; and the source code repositories, which provide both file content and ownership signal. The pipeline is fully automated. New content is chunked, embedded with Titan, and indexed in OpenSearch without manual intervention. Ingestion is decoupled from query. They share the index but nothing else, so a slow ingestion run never affects live triage latency, and a problematic batch can be rolled back without touching the query path. What's Worth Taking From This The model layer (Bedrock, Claude, Titan) is interchangeable. Swap them for OpenAI plus their embeddings, or for a self-hosted setup, and the architecture still works. What is not interchangeable, or not easily, is the shape of the rest: Retrieval before reasoning. Don't ask the model to do retrieval against a large corpus. Get the relevant slice with a dedicated retrieval layer, then hand it over with a tight prompt.Multiple signals with dynamic weights. Single-signal confidence scores break under real-world data. Multiple signals with weight redistribution handle the cases where inputs are incomplete.Persist workflow state externally. Even for short workflows, having state in a queryable store pays off in failure recovery and gives the dashboard a single source of truth.Decouple ingestion from query. They have different reliability requirements and should be able to fail independently.Match compute to workload shape. Fargate for orchestrated, latency-tolerant workflows. The wrong choice here shows up later as cold starts, timeouts, or surprise bills. The agent has been doing useful work since it shipped. The thing that took the longest to get right wasn't any single component. It was the scoring layer and the decision to make state external. Those are the parts that determine whether a system like this is something the team relies on or something the team works around.

AWS has been building agentic infrastructure for some time now — Bedrock, AgentCore, Strands — mostly aimed at engineers who want to build their own agent systems from scratch. Amazon Quick is a different layer of the same bet: a ready-to-use agentic workspace that targets teams directly, without requiring custom orchestration code. This article walks through what Quick is, how its components fit together technically, how the MCP integration model works with real code, and where it sits relative to the rest of AWS's agent stack. What Amazon Quick Is Amazon Quick is an AI assistant for work that connects to your existing tools — Slack, Microsoft Teams, Outlook, CRMs, databases, and local files — and gives a unified layer for querying, automating, and acting across them. It launched in preview at AWS's "What's Next with AWS" event on April 28, 2026. The product is aimed at teams, not just individual users. One person can build a custom agent scoped to a specific dataset or workflow, and the whole team benefits from it. Responses from Quick agents are grounded in your actual business data, not the underlying model's training distribution. Under the hood, Quick is built on Amazon Bedrock AgentCore and uses the Model Context Protocol (MCP) as its standard for connecting to external tools. It runs on AWS IAM and VPC, which means it inherits the same security and compliance posture as the rest of your AWS workloads. Components Quick bundles five distinct capabilities. It helps to understand each one separately before thinking about how they compose. ComponentWhat it doesSpacesCollaborative workspaces where teams pool files, dashboards, and data sources. Agents in a Space are grounded in that Space's data.AgentsCustom, domain-scoped agents built on your team's specific data. One person builds, everyone uses.ResearchMulti-source synthesis across internal data, the public web, and third-party datasets. Produces structured reports.Visualize (Quick Sight)Integrated BI layer. Conversational access to dashboards, charts, and forecasting — no separate BI tool required.Automate (Quick Flows)Workflow automation from simple daily tasks to complex multi-step processes with cross-app action execution. Each component is available through the web app, mobile, and a native desktop app (currently in preview for macOS and Windows) that can read local files and calendar context without requiring browser access. Where Quick Sits in the AWS Agent Stack AWS is building in two directions at once. AgentCore is the infrastructure layer for engineers who want to compose their own agent systems — runtime, memory, gateway, observability — with any model and any framework. Quick is the product layer on top: opinionated, team-facing, and deployable without writing orchestration code. The practical implication: if you're an engineer building internal tools or automation pipelines, you'll likely interact with both layers. AgentCore for the infrastructure wiring; Quick as a surface where non-technical teammates interact with the agents you build. The Integration Architecture The core question for any engineer evaluating Quick is: how does it actually connect to external systems, and what does the request path look like? Quick uses MCP (Model Context Protocol) as its primary integration standard. This is significant because MCP is an open protocol — it means Quick agents are not locked into AWS-specific connectors, and any MCP-compatible server can be registered as a tool source. High-Level Request Flow The sequence below shows the full lifecycle of a single agent-triggered tool call — from the moment Quick receives a prompt through to the response returning from a downstream API. Quick acts as the MCP client. Your MCP server exposes tools via listTools and callTool. Quick discovers them at registration time and makes them available to any agent or automation in the workspace. Authentication flows through OAuth 2.0, with support for Dynamic Client Registration (DCR) so Quick can register itself automatically without manual credential setup. Building an MCP Server for Quick Here is a minimal Python MCP server using the mcp SDK that exposes two tools Quick can invoke — get_ticket and list_open_tickets. This pattern works whether you host the server yourself or run it on AgentCore Runtime. Install Dependencies Python pip install mcp[server] httpx uvicorn Server Implementation Python # server.py from mcp.server import Server from mcp.server.sse import SseServerTransport from mcp.types import Tool, TextContent import httpx import json from starlette.applications import Starlette from starlette.routing import Route app = Server("jira-quick-integration") JIRA_BASE_URL = "https://yourorg.atlassian.net" JIRA_TOKEN = "Bearer <your-token>" # in production, load from AWS Secrets Manager @app.list_tools() async def list_tools() -> list[Tool]: return [ Tool( name="get_ticket", description="Retrieve details for a single Jira ticket by issue key.", inputSchema={ "type": "object", "properties": { "issue_key": { "type": "string", "description": "The Jira issue key, e.g. ENG-1234" } }, "required": ["issue_key"] } ), Tool( name="list_open_tickets", description="List open Jira tickets assigned to a given user.", inputSchema={ "type": "object", "properties": { "assignee": { "type": "string", "description": "The Jira username or email of the assignee" } }, "required": ["assignee"] } ) ] @app.call_tool() async def call_tool(name: str, arguments: dict) -> list[TextContent]: headers = {"Authorization": JIRA_TOKEN, "Content-Type": "application/json"} async with httpx.AsyncClient() as client: if name == "get_ticket": key = arguments["issue_key"] resp = await client.get( f"{JIRA_BASE_URL}/rest/api/3/issue/{key}", headers=headers ) resp.raise_for_status() data = resp.json() summary = data["fields"]["summary"] status = data["fields"]["status"]["name"] return [TextContent(type="text", text=f"{key}: {summary} [{status}]")] elif name == "list_open_tickets": assignee = arguments["assignee"] jql = f"assignee={assignee} AND status != Done ORDER BY updated DESC" resp = await client.get( f"{JIRA_BASE_URL}/rest/api/3/search", headers=headers, params={"jql": jql, "maxResults": 20} ) resp.raise_for_status() issues = resp.json().get("issues", []) results = [ f"{i['key']}: {i['fields']['summary']}" for i in issues ] return [TextContent(type="text", text="\n".join(results) or "No open tickets found.")] raise ValueError(f"Unknown tool: {name}") # Wire up SSE transport for Quick compatibility sse = SseServerTransport("/messages/") async def handle_sse(request): async with sse.connect_sse( request.scope, request.receive, request._send ) as streams: await app.run(streams[0], streams[1], app.create_initialization_options()) starlette_app = Starlette( routes=[Route("/sse", endpoint=handle_sse)] ) if __name__ == "__main__": import uvicorn uvicorn.run(starlette_app, host="0.0.0.0", port=8080) A few design constraints to be aware of when building for Quick: Each MCP tool call has a 300-second hard timeout. Operations that exceed this fail with HTTP 424. Keep individual tool calls narrow and fast.The tool list is treated as static after registration. If you add or remove tools on the server, the Quick admin must re-establish the connection to pick up changes.Quick supports both Server-Sent Events (SSE) and streamable HTTP as transports. Streamable HTTP is preferred for new implementations. Registering the MCP Server in Quick Once your server is running and publicly reachable over HTTPS, registration in Quick takes the following path: Shell Quick Console → Integrations → Add Integration → MCP Fields: Server URL: https://your-mcp-server.example.com/sse Auth type: OAuth 2.0 (or Service, or None) Client ID: <from your identity provider> Authorization URL: https://auth.example.com/oauth/authorize Token URL: https://auth.example.com/oauth/token If your identity provider supports OAuth Dynamic Client Registration, Quick will auto-register and you skip the manual client ID step entirely. Quick sends an initial unauthenticated request to the MCP server; if it receives a 401 with a WWW-Authenticate header containing a resource_metadata URL, it fetches the metadata document and proceeds with DCR automatically. Once registered, Quick calls listTools at startup and exposes every discovered tool to agents and automations in the workspace. The AgentCore Gateway Option For teams that don't want to write and operate an MCP server from scratch, Amazon Bedrock AgentCore Gateway provides a managed alternative. You point Gateway at a Lambda function or an OpenAPI spec, and it handles the MCP wrapping, auth, logging, and semantic tool discovery automatically. If you use it, Quick never calls your internal APIs directly — everything flows through Gateway's auth and routing layer, as shown in the sequence diagram above. The semantic search capability is worth noting specifically. When an agent has access to dozens or hundreds of tools, passing the full tool list on every turn wastes context and causes the model to pick the wrong tool. Gateway's built-in x_amz_bedrock_agentcore_search tool lets Quick find the right tool by semantic similarity rather than scanning the entire registry each turn. Practical Considerations A few things worth keeping in mind before integrating: Tool scope matters. When agents are given too many tools simultaneously, selection accuracy degrades — the model reasons over too many options per turn and picks incorrectly more often. Keeping each agent or MCP server to a focused set of 3–5 tools produces better results than exposing everything through one endpoint. This is a known pattern in multi-agent architectures and applies equally to Quick agents. The 300-second timeout is real. Design each tool call to complete a single, bounded operation. Avoid chaining multiple downstream API calls inside a single tool invocation. If you need a multi-step workflow, model it as separate tools and let the agent orchestrate the sequence. Local context on the desktop app. The desktop app reads local files and calendar events directly, without upload. For engineers who work primarily in terminals and local editors, this is a meaningful integration point — meeting context, local documentation, and recent file changes are all available to the assistant without any configuration. MCP interoperability. Because Quick uses MCP as the standard, the same MCP server you build for Quick can also be consumed by Claude Code, Amazon Q Developer, and other MCP-compatible clients. The integration contract is portable. References Amazon Quick — Product overview and featuresIntegrate external tools with Amazon Quick Agents using MCP (AWS ML Blog, Feb 2026)MCP integration — Amazon Quick User GuideAmazon Bedrock AgentCore — Overview and documentationIntroducing Amazon Bedrock AgentCore Gateway (AWS ML Blog)Top announcements of the What's Next with AWS, 2026 (AWS News Blog, Apr 2026)

By Jubin Abhishek Soni CORE

Here is what a production cascade looks like when nobody did anything wrong. An alert fires on a microservice showing elevated latency. The signal is accurate. The automated remediation agent picks it up immediately and does exactly what it was built to do: restart the affected service and reroute traffic. The action is within scope, the credentials are valid, and three seconds later, the platform reports a successful remediation. Then, four dependent services go dark. The postmortem will call it a cascade. The dashboard will show a clean execution on the first incident and a second incident opening 90 seconds later. Nobody will find an error log on the remediation itself because there was none. The agent was not wrong. The action was technically correct. What nobody had built was the ability to ask: given what the system is carrying right now, is this the moment to add more disruption to it? That is not a monitoring gap. Monitoring told everyone exactly what was broken. It is an observability architecture gap — the difference between knowing what is failing and knowing whether the system can safely absorb what you are about to do to fix it. Figure 1: The alert was correct. The instrumentation gap was not in detection — it was in the question asked before acting. The Failure Pattern Is More Consistent Than Teams Expect I ran into this structurally while doing chaos engineering on enterprise SD-WAN infrastructure at Cisco. We were running experiments against production-grade environments across large financial services and telecom customers, and standard chaos tooling kept finding the wrong failures. It was injecting faults into systems whose state had already shifted past the parameters we had set at the start of the experiment. The faults that caused real damage were the ones that chained with conditions already present in the environment — elevated resource utilization, two services over, a background process that had been running for 45 minutes, consuming memory that a restarted service needed, a connection pool sitting at 89 percent because of an unrelated batch job. None of those conditions was hidden. Everyone was instrumented. The problem was that nobody was reading them together as a composite signal before deciding how hard to push the system. We were answering the wrong question. We built a methodology to fix it. Instead of setting static experiment parameters, the engine reads live telemetry before each iteration, derives from that telemetry the system's current capacity to absorb perturbation, and calibrates the intervention intensity accordingly. A feedback loop between the actual impact and the intended impact across successive iterations finds the behavioral boundary without disabling the environment. That methodology became USPTO Patent No. US12242370B2. Patent: https://patents.google.com/patent/US12242370B2/en What we built for SD-WAN infrastructure is the same thing agentic AI deployments need now. The underlying problem is identical: an automated actor is making decisions about whether and how to intervene in a live system, using a signal that accurately describes what is broken but says nothing about what the system can safely absorb in the moment the decision is made. Why AWS FIS and Gremlin Will Not Find This for You Infrastructure fault injection is good at what it does. AWS FIS, Gremlin, and Chaos Toolkit test whether your Lambda survives throttling, whether the event pipeline recovers from a queue outage, and whether the hosting environment holds up under resource pressure. These are legitimate questions, and the tooling answers them well. They just do not test the failure mode that is generating the most expensive incidents as agentic AI deployments scale. An agent's worst production failure is not a cold start timeout or a concurrency breach. It is a clean, successful invocation that executes the wrong sequence — because the combination of inputs, tool call results, and current system state put the agent at the edge of its operational envelope, and nobody built a test that ever got it there. Air Canada's chatbot did not crash. It executed correctly in a scenario the designers never tested. No infrastructure fault injection exercise would have found that boundary because the boundary was not in the infrastructure. The same structure shows up in autonomous remediation. The agent reads a real signal, takes a valid action within its authorized scope, and produces an outcome nobody intended because the action was correct in isolation but wrong given the composite state around it. Standard tooling reports a clean execution. The cascade shows up in the next incident ticket. Finding the behavioral boundary requires a test methodology that reads live system state before calibrating experiment intensity — not one that applies static parameters to a system whose state has already shifted. Static parameters applied to dynamic systems find the failure modes you designed the test to find. They miss the ones that actually hurt. Three Instrumentation Gaps to Close Before Your Agents Hold Production Credentials These did not come from a research paper. They came from postmortems — at Cisco across financial services and telecom customers, and at Splunk across thousands of enterprise observability deployments. The same three gaps show up every time. 1. Concurrent workload state across the dependency graph, not just the service under incident. A service restart that is safe in isolation is frequently dangerous when adjacent services are already running above their normal resource ceilings. The absorb capacity question is a system-level question, not a component-level one. Most runbooks do not include a pre-action resource check across the dependency graph of the service being touched. Automated agents have no reason to be different. What to build: a pre-action query that checks whether any first-degree dependency of the service being remediated is above 80 percent of its 24-hour baseline utilization. One data point. It exists in most observability stacks already. It is almost never surfaced in an incident context. 2. Pending operations competing for the same recovery resources. A recovering service needs I/O headroom during the 60 to 90 seconds after restart while it rebuilds its in-memory state. A background index rebuild consuming 30 percent of available I/O is invisible to the incident response flow because it is not itself failing. It does not show up in any alert. It shows up in the postmortem as a contributing factor. What to build: a pre-action inventory query against active background and scheduled operations on the same infrastructure tier as the remediation target. Not continuous monitoring — just one read before acting. 3. Intervention intensity matched to current system state, not last month's playbook. The remediation that worked last Tuesday was calibrated to last Tuesday's system state. Applying it at the same intensity to a system currently carrying three extra loads is not a reliable practice — it is reusing a number that made sense in a context that no longer exists. Every automated remediation action should answer one question before executing: Is the system in the same absorb capacity range as when this intervention was validated? If it is not, stage the action, reduce intensity, or escalate. This is not complicated engineering. It is a check that almost nobody has built. The automation is not the problem. The automation acting without a pre-action absorb capacity check is the problem. Building that check is a day's work. Not building it is how you get cascades that look like they came from nowhere. "We were validating system health, not output integrity. That experience changed how we define resilience; it is no longer just about systems staying up but about systems staying correct under stress." — John Russo, VP Healthcare Technology Solutions, OSP Labs Which Automated Actions Need This Check and How Urgently Not every intervention carries the same absorb capacity risk. Here is a working classification based on what I have watched produce incidents. The cluster restart and downstream workflow rows are where most of the expensive postmortems come from. Intervention Absorb Risk Minimum Pre-Action Check Automate or Escalate Read-only diagnostics (health checks, metric queries, log pulls) Very Low None Fully automatable, no check needed Traffic rerouting (LB weight shifts, circuit breaker trips) Low to Medium Downstream service vs. 24hr baseline Automate with dependency check; escalate if downstream >75% baseline Single service restart (pod recycle, instance restart) Medium I/O headroom + active background ops on same tier Automate if headroom clear; escalate if background ops active Cluster-level restart (rolling or full, multiple instances) High Full dependency graph resource state + pending ops inventory Stage the restart; never run under pre-existing cross-service stress Config or schema change (feature flags, parameter updates) High to Very High All checks + rollback path validated Human review required outside the nominal absorb capacity range Agent-initiated downstream workflow (external API calls, cross-service triggers) Very High (often irreversible) Intent-execution separation + full pre-action assessment Human authorization unless the action is fully reversible Table 1: The cluster restart and downstream workflow tiers are where most production cascades originate. The check is cheap. The postmortem is not. How to Build the Absorb Capacity Layer Adding absorb capacity as a first-class observable does not mean replacing what you have. Your existing metrics, traces, and logs are doing their job. The gap is not in those signals — it is in the layer that reads them together and produces a single pre-action number before any automated intervention fires. The architecture has three parts. First, a live absorb capacity index: a rolling calculation across the dependency graph of each critical service, reading resource utilization deltas against the 24-hour baseline, shared connection pool saturation, active background operation inventory, and concurrent workload state. Output is a single number per service cluster — current absorb capacity as a percentage of the validated intervention tolerance.Second, an intervention intensity governor that reads that number before any automated remediation executes. If the index is within range, the action proceeds. If not, the governor selects a reduced-intensity variant, stages the action, or sends it to human review. It does not touch the remediation logic. It gates execution.Third, a behavioral boundary testing loop adapted from the intent-based chaos engineering methodology in Patent US12242370B2. Periodic pre-production tests read live telemetry, derive calibrated adversarial pressure from the current absorb capacity model, and use an actual-versus-intended impact feedback loop to keep the model current. Without this loop, the pre-action check is comparing today's system state against a capacity model that was valid when you built it six months ago. Figure 2: The absorb capacity layer sits between existing observability and the autonomous agent. The behavioral testing loop (Patent US12242370B2) keeps the capacity model current as the system evolves over time. The Check That Almost Nobody Has Built Most teams I have worked with have good observability. The signals are there. The alerting is tuned. The dashboards show what is failing in real time. What they have not built is the layer that reads all of it together and answers a different question: not what is broken, but whether the system is in a state that can take what you are about to do to it. Autonomous remediation agents and agentic AI systems make that question urgent in a way it was not when the decision-maker was a human engineer with pattern recognition built over years. The human hesitated. They glanced at adjacent services. They asked the on-call SRE if anything else was running before they pushed the big red button. The agent does not hesitate. It reads the signal, acts within scope, and files the result as success. RL-calibrated infrastructure failures are recoverable. A cluster goes down, the runbook fires, the service comes back. Behavioral failures in systems with real external side effects — agents that trigger downstream workflows, confirm transactions, modify records across services — are not always recoverable in the same way. The damage lands in external systems before any alert fires. Adding absorb capacity as a first-class observable is not a large infrastructure project. The signals you need are already in your stack. The composite read, the pre-action check, the governor that gates execution — none of this requires new technology. It requires deciding to ask the right question before the agent acts, and building the thin layer that makes that question answerable in real time. The observability you have is telling the truth. It is just not telling the whole truth yet.

Large-scale cloud platforms have reached a level of complexity — spanning multi-region Kubernetes clusters, streaming systems like Kafka, and heterogeneous data stores — that often exceeds human cognitive limits. Failures are no longer isolated events; they are emergent behaviors arising from tightly coupled systems where issues propagate across layers such as networking, orchestration, and data pipelines. Even with modern observability stacks, operators must manually correlate signals across dashboards, making incident response slow, inconsistent, and cognitively taxing. Traditional approaches rely heavily on static runbooks and tribal knowledge. These mechanisms do not scale in modern distributed systems. Agentic AI introduces a fundamentally different paradigm. Rather than merely detecting anomalies (as in traditional AIOps), agentic systems use Large Language Models (LLMs) to reason, plan, and act. These systems can iteratively generate hypotheses, validate them using real data, and execute multi-step remediation workflows. The result is not just faster detection, but a closed-loop system capable of autonomous diagnosis and recovery. This article expands on how to architect a production-grade SRE agent that can safely and effectively automate cloud incident response. The system is organized into three layers: Perception (data ingestion), Cognition (multi-agent reasoning), and Action (guarded execution), all operating over a shared knowledge graph. Establish a Cloud Knowledge Graph At the core of any intelligent SRE agent is context. Raw telemetry alone is insufficient; the system must understand how components relate to each other. This is achieved through a domain-specific cloud knowledge graph. The graph models: Nodes: Services, pods, clusters, regions, gateways, Kafka topics, and databasesEdges: Traffic flows, deployment relationships, data lineage, ownership, and failover pathsAttributes: SLOs, capacity limits, configuration history, and prior incidents This structure transforms observability data into a causal reasoning substrate. Instead of treating metrics independently, the agent can traverse dependencies and infer propagation paths. For example, a spike in API latency can be traced through upstream gateways to downstream services and eventually to a throttled database. This graph is not static — it evolves continuously with infrastructure changes and incident learnings. Over time, it becomes a living system model enriched with historical context, enabling better hypothesis generation and faster root-cause analysis. In practice, maintaining graph freshness is critical. You should integrate it with service registries, deployment pipelines, and configuration management systems to ensure it reflects real-time topology. Build the Perception Layer (Observability Pipeline) The Perception Layer acts as the sensory system of the agent, continuously ingesting telemetry across the stack. This includes: Metrics: CPU, memory, I/O, network utilization, Kafka consumer lagLogs: Structured and semi-structured application and infrastructure logsTraces: End-to-end request paths across microservices However, raw ingestion is only the first step. The real value lies in transforming this data into structured, actionable signals. A stream-processing pipeline should: Normalize data across heterogeneous sourcesDetect anomalies using statistical methods and thresholdsEmit structured events tied to entities in the knowledge graph These events act as triggers for the Cognition Layer. Importantly, they should already be enriched with context (e.g., “Service A in region us-east-1 exceeds latency SLO”), reducing the reasoning burden on downstream agents. A critical design consideration is balancing sensitivity and noise. Excessive alerting leads to “signal overload,” a well-known issue where operators — and agents — struggle to prioritize meaningful events . Techniques such as event deduplication, correlation, and temporal aggregation are essential to ensure high-quality inputs. Architect a Multi-Agent Cognition Layer Instead of using a single massive prompt, build a Cognition Layer utilizing a multi-agent LLM architecture (using GPT-5 or Claude-Opus class models) orchestrated by a control plane (e.g., a serverless orchestration layer). Assign specialized roles to different agents: Detector Agent: Monitors the anomaly events and groups related alerts into candidate incidents based on the knowledge graph's dependency structure.Hypothesis Agent: Proposes potential root causes by analyzing the graph and recent telemetry data.Validator Agent: Acts as the investigator by issuing targeted queries back to the observability tools and cloud APIs to confirm or reject the hypotheses based on hard evidence.Planner Agent: Synthesizes an actionable remediation plan. This plan should be an ordered list of operations, complete with preconditions, postconditions, and explicit rollback triggers.Critic (Governance) Agent: Reviews the remediation plan against organizational safety policies before execution, ensuring constraints are not violated. Implement a Guarded Action Layer The Action Layer is what separates an active agent from a passive AIOps recommendation engine. It executes the Planner Agent's steps via the Kubernetes API (scaling, restarting pods) and Cloud Provider APIs (toggling failovers, adjusting traffic weights). Safety is paramount. You must wrap this layer in a strict governance framework: Enforce hard limits on scaling factors and failover scopes.Implement canary rollouts, applying changes to a single zone before expanding.Build auto-rollback mechanisms that trigger immediately if Service Level Objectives (SLOs) deteriorate after an action.Require explicit human-operator approval for high-risk operations like region-wide failovers. Rollout and Optimization Strategies When deploying your SRE agent, start in a "shadow" or assist mode. Allow the agent to observe incidents, propose hypotheses, and draft plans while human operators retain full control and execute the final decisions. As confidence in the system grows, gradually grant it autonomy for low-risk, routine actions. To manage operational costs and latency: Optimize prompts: Externalize static system descriptions into retrieved documents.Caching: Cache intermediate inferences for reuse across similar recurring incidents.Batching: Batch non-urgent tool calls and defer low-impact infrastructure checks to background tasks. Conclusion Agentic AI represents a shift from reactive monitoring to proactive, autonomous operations. By combining a real-time observability pipeline, a continuously evolving knowledge graph, and a multi-agent reasoning system, you can build an SRE agent capable of end-to-end incident management. Using this framework can significantly reduce Mean Time To Recovery, improve root-cause accuracy, and decrease reliance on human escalation — all while maintaining strict safety guarantees. More importantly, these systems create a virtuous cycle: every incident enriches the knowledge graph, improves agent reasoning, and strengthens operational resilience. As cloud systems continue to grow in complexity, agentic SRE architectures will likely become a foundational component of modern reliability engineering.

AI agents have come a long way. They aren’t just answering simple questions, but they’re handling order checks, summarizing support tickets, updating records, routing incidents, approving requests, and even calling internal tools. As these agents slip deeper into real business workflows, just peeking at model logs isn’t enough. Teams need to see everything: what the agent did, why it did it, which systems it poked, and whether the end result actually helped the business. Agent Observability That’s where agent observability comes in. Traditional observability lets teams watch over their apps, APIs, databases, and infrastructure. Agent observability goes a step further. It shines a light on the whole AI workflow: it connects the dots from the user’s request to the agent’s decisions, the tools it touches, the systems it interacts with, and all the way to the final outcome. Let’s see a customer support example. Say a customer messages, “My subscription renewal failed, but I got charged twice.” A human rep checks the account, payment history, billing rules, refund policy, and ticket history before answering. Now, an AI agent might do that job automatically. It’ll spot the billing problem, look up the customer record, call the billing system, check for duplicate payments, and either resolve the issue or escalate it if things get too messy. On the surface, this whole thing just looks like a simple chat. However, under the hood, it’s a full-on workflow. If you want good observability, you need that behind-the-scenes view: Why bother? Because the final response doesn’t tell you the whole story. If the customer comes back unhappy, you need to nail down whether the agent checked the right account, used the right billing tool, hit an error, misread the request, or escalated when it couldn’t help. Don’t just watch the answer: Follow the whole journey When you break down agent interactions, a few basic layers show the full picture. First, track the user request. What did the user ask? Was it urgent, fuzzy, sensitive, or bound to a customer contract? Second, watch the agent’s action. Did it answer straight away, ask a follow-up question, search a knowledge base, use a tool, or hand off to a human? Third, note the context. What sort of information did it use? Did it pull a help article, customer details, invoice, ticket, policy, or product data? Fourth, log tool usage. Did the agent call billing APIs, CRM systems, databases, incident tools, or an approval workflow? Did those calls work, or did they fail? Lastly, look at the result. Did the agent fix the customer’s problem? Was the ticket reopened? Did a human have to clean up after the agent? Without these layers, you’ll know when something was slow or incorrect, but not why. Maybe the context was off, a tool call failed, it lacked permissions, the prompt changed, or something further downstream broke. Use a Single ID to Track Everything One of the easiest fixes is to tag the whole workflow with a tracking ID. Let that ID travel with the request, from the interface through the agent, tools, APIs, and your business systems. Now, if a support ticket gets botched, the team can retrace every step: what the customer asked, what the agent understood, which account it checked, what the billing system said back, and why the agent chose to close or escalate. It’s not just for support. Maybe your SRE team uses an AI agent to help dig into a production alert. The agent scans logs, checks recent deployments, reviews database metrics, and suggests the likely cause. That same tracking ID means you’ll know exactly which systems the agent checked and whether it missed anything crucial. Don’t ignore tool calls; they’re real actions Here’s where things get serious. When an agent calls a tool, it’s taking action. Looking up customers, updating records, approving requests, creating tickets, and kicking off workflows need to be watched closely. For each tool call, capture details like tool name, how long it took, success or failure, retries, permission results, error messages, and what actually happened. Take a finance workflow. Say the agent reviews vendor invoices by extracting details, matching with a purchase order, checking taxes, and routing exceptions to finance. If an invoice gets approved by mistake, did the agent misread the invoice? Match it with the wrong purchase order? Miss a policy update? Or did the finance system return incomplete info? That’s why tracking tool calls is critical. A wrong answer in chat is one thing, but a wrong move in your business system can lead to trouble such as money lost, operations disrupted, and even compliance issues. Understand Agent Decisions, But Protect Privacy Teams need to understand what the agent did, but you don’t want to log every single “thought” it had; it’s just unnecessary noise. Instead, record decision details in a structured way. Example: Intent: billing disputeConfidence: mediumTool: billing lookupReason: account verification neededPolicy result: escalateFinal action: handoff to human Now you have enough to debug the workflow and for reporting, without exposing raw thought streams. You can spot how often agents escalate from low confidence, where tools fail, or if policy rules stop an action. Connect Observability to Business Outcomes Don’t just track the tech stuff; what really matters is whether the agent gets the job done. Watch business metrics like: Resolution timeEscalation rateWorkflow completion rateTool failuresCost per workflowSLA hits or missesReworkHow often humans step in If you’ve got an e-commerce agent helping buyers pick products, check inventory, apply discounts, and guide checkout, you want to know: did the customer actually buy the item? If checkout drops after you tweak a prompt, find out why. Did the agent push out-of-stock items? Apply discounts wrong? Use the wrong tool? Lose customers with confusing answers? Observability at this level helps both engineering and business teams get answers, fast. Build Dashboards for Different Audiences Everyone’s got different needs. SREs care about latency, failed tools, retries, issues with dependencies, and expensive cost spikes. Security teams focus on policy denials, suspicious tool actions, sensitive data flags, or prompt injection attempts. Product owners want completion rates, escalations, customer satisfaction, and abandoned workflows. Engineers need to see how agent behavior shifts after you change the model, prompt, workflow, or deployment. Business folks need throughput, SLAs, cost savings, and improvements to customer experience. Take security operations. Say an agent checks suspicious logins, identity logs, privilege changes, and endpoint activity. Security needs to know: did the agent just review info, or did it try to lock an account? If it got blocked, you want that visible, too. Alert on AI-Specific Failures AI agents fail in new ways. Teams need alerts for things like sudden spikes in tool denials, fallback responses, unexpected tool usage, cost blowups, prompt injection attempts, completion drops, or escalating cases. If an agent suddenly goes wild with refund actions, it could mean a prompt is off, a policy is weak, or something’s getting abused. If fallback responses shoot up, maybe the knowledge base is broken. Costs spike? Maybe the agent is stuck looping, retrying, or making unnecessary expensive calls. Tie alerts to deployments, too. Agents change behavior after you update a prompt, switch models, change schema, adjust policies, or edit a workflow. Teams should compare how the agent behaved before and after. A Simple Way to Grow Observability Observability matures in steps. Basic logs: prompts, responses, errors, timestampsTool visibility: what got used, if it worked, how long it tookEnd-to-end traces: follow the user request through the agent, tools, APIs, systemsBusiness-level result tracking: resolution, escalation, completion, rework, cost, SLAAutomated alerts: regressions after updates, anomalies, unusual patterns Observability is more about making sense of the whole workflow and visibility. Teams need to know what users wanted, what the agent decided, which info it used, which tools it grabbed, which systems it touched, and whether business value was delivered. As AI agents settle into production, observability has to cover more than just servers and app logs. The teams that win will be the ones who trace agent behavior end to end, spot failures early, explain what happened, and keep improving safely.

By Srinivas Chippagiri CORE

What This Series Is About This is Part 2 of a two-part series on building a Slack bot that answers natural language questions about a GitHub repository using AWS Bedrock (Claude) and GitHub's official Model Context Protocol (MCP) server. Part 1 covered the why: most AI tools suggest wrapping GitHub's REST API and feeding the response to a model. That approach works, but it produces brittle glue code that grows with every new question type and every new data source. MCP offers a fundamentally better pattern — a tool registry that the model queries at runtime, making routing decisions autonomously. The result is a 150-line bot that answers questions you never anticipated and extends to new data sources with four lines of configuration. If you have not read Part 1, it is available here: https://dzone.com/articles/build-a-github-slack-bot-with-aws-bedrock-and-mcp. The full project code is on GitHub: https://github.com/sangharshcs/slack-github-mcp-bot. This article covers the implementation — the four key architectural pieces, how to get it running, how to extend it to new MCP servers, and the production lessons from running it on a real engineering team. How It Is Built — The Four Key Pieces The bot has four distinct components. Understanding each one separately makes the whole system easier to reason about and extend. 1. The MCP Request Function All communication with GitHub's MCP server goes through a single function. GitHub MCP returns Server-Sent Events (SSE) rather than plain JSON, so the function handles both response types transparently. It also checks HTTP status and surfaces MCP-level errors cleanly — without this, a 401 or 500 from the server fails silently. The function signature accepts the endpoint and headers as parameters, not hardcoded values. This is the detail that makes the whole system extensible: the same function routes to GitHub today and to any other MCP server tomorrow. 2. The Tool Registry At startup, the bot calls tools/list on every connected MCP server and records which server owns each tool. This registry — a simple JavaScript object mapping tool name to endpoint and auth headers — is the entire routing mechanism. When Claude calls a tool, the bot looks up its origin and sends the request there. Adding a new MCP server means calling the same loadServer() function with the new URL and credentials. The registry grows. The agent loop never changes. This four-line pattern is the extensibility mechanism Eric described as worth expanding on: JavaScript // Same pattern for every MCP server you add: const myServiceHeaders = { Authorization: `Bearer ${process.env.MY_SERVICE_TOKEN}`, 'Content-Type': 'application/json', Accept: 'application/json, text/event-stream', }; await loadServer(process.env.MY_SERVICE_MCP_URL, myServiceHeaders); // Then add routing guidance to your system prompt. // The agent loop below does not change. 3. The Agent Loop The loop sends the question to Claude with the full tool list. Claude selects tools, the bot executes them via the registry, results return to Claude, and the cycle repeats until Claude has enough to answer — typically 3 to 8 tool calls. The loop is generic: it does not know whether it is answering a bug or a PR question. The system prompt configures the behavior. The same code handles every question type, present and future. 4. The System Prompt The system prompt is the highest-leverage piece in the entire system. The difference between a bot your team uses daily and one they quietly stop using is almost always prompt quality, not code quality. Three rules matter most: Explicit Slack markdown syntax. Claude defaults to standard Markdown. Without being told otherwise, it uses **bold** and [text](url), which Slack renders as raw asterisks and broken links. The prompt must specify *bold*, <url|text>, no # headings, no markdown tables.High-volume handling. Without a rule, asking 'list all open issues' on a large repo returns an unusable wall of text. The prompt should specify: if results exceed 15 items, summarise by category and show the top 10.Tool routing for multiple servers. When you add a second MCP server, the prompt tells Claude which questions map to which server. This reduces unnecessary tool calls and keeps responses fast.The complete index.js, package.json, and .env template are in the project repository at https://github.com/sangharshcs/slack-github-mcp-bot. Getting It Running Setup involves three external services — Slack, GitHub, and AWS Bedrock — each requiring a token. Rather than reproducing the full step-by-step here (that lives in the project README at https://github.com/sangharshcs/slack-github-mcp-bot), here is what each token is and where to get it. The Slack bot token (xoxb-...) comes from creating a Slack app at api.slack.com/apps with Socket Mode enabled. Socket Mode is what lets the bot run from your laptop without a public URL — it connects outbound over WebSocket. You also need an App-Level Token (xapp-...) for the socket connection itself, and a Signing Secret from Basic Information. The bot needs these scopes: app_mentions:read, chat:write, im:history, im:write.The GitHub token is a fine-grained personal access token from github.com/settings/tokens. Select your target repository and grant read access to Issues, Pull Requests, Contents, and Metadata. No write access is needed.The Bedrock API key comes from the AWS console under Amazon Bedrock → API keys. Enable the Claude Sonnet 4.6 model under Model access first. One detail that catches everyone: Claude 4.x models require a cross-region inference profile prefix. Use us.anthropic.claude-sonnet-4-6, not anthropic.claude-sonnet-4-6. The bare ID returns "on-demand throughput isn't supported". With credentials in .env, npm install and node index.js is all it takes. The bot logs the number of GitHub tools loaded and is ready to receive mentions. Extending to Other MCP Servers loadServer() is the entire extension mechanism. Call it with any MCP-compatible service. The registry grows, Claude discovers the new tools, and you add one line to the system prompt describing when to use them. MCP Server URL What it adds Linear mcp.linear.app/mcp Issues, projects, cycles, roadmaps Cloudflare mcp.cloudflare.com/mcp Workers, analytics, DNS, R2 storage Stripe mcp.stripe.com/mcp Payments, customers, subscriptions Custom @modelcontextprotocol/sdk Any internal REST API as MCP tools What We Ran Into in Production We have been running this bot on a busy engineering repository for several months. Before sharing the limitations we documented, it is worth saying that none of them were showstoppers — but they were real, and ignoring them would leave you unprepared. The biggest adjustment was latency. Complex queries that trigger 6 to 8 tool calls take 15 to 30 seconds. We handled this with the thinking-indicator pattern — post a placeholder message immediately, then update it when the answer is ready — which kept the experience feeling responsive even when the underlying calls were slow. Debugging took more work than expected. When a traditional API client gives a wrong answer, the fix is obvious. When an agentic loop gives a wrong answer, you need to know which tools Claude chose, what they returned, and how Claude reasoned over the results. We solved this by logging every tool call — name, input, result, timestamp — and shipping those logs to our observability platform. That log became our primary debugging tool for agent behavior. Prompt quality turned out to be load-bearing in a way we did not fully anticipate. Early versions of the bot would return raw asterisks in Slack, generate unusable walls of text for large result sets, and occasionally pick the wrong tool. Each of these was a prompt fix, not a code fix. Investing time in the system prompt before going live would have saved us several rounds of iteration. Cost is worth monitoring at scale. A query triggering 8 Bedrock calls costs meaningfully more than a single response. For an internal team tool used dozens of times a day, the cost is negligible. At higher volume, it warrants attention. The productivity gain from not maintaining API clients has outweighed all of these constraints at the scale we operate. The right framing is not "is MCP perfect?" but "is MCP better than the alternative?" For our team, the answer has consistently been yes. Conclusion The architecture across these two articles is intentionally small. A tool registry, a generic agent loop, and a system prompt that configures behavior — that is all of it. The 150 lines in the repository are a starting point your team can run today and grow as your toolchain does. Start with GitHub MCP. Get it answering questions in Slack. Test it with your team. Then look at what they ask most often and which data sources those questions touch. The next MCP server to register will be obvious. The code to add it is four lines. If you landed on Part 2 first and want to understand the architectural reasoning — why MCP is a fundamentally better pattern than the conventional REST API wrapper approach, and why it matters especially for SRE and platform teams — Part 1 covers all of that and is the recommended starting point. Part 1: Why MCP Changes Everything About AI Tool Integration. Full project code: https://github.com/sangharshcs/slack-github-mcp-bot.

(Note: A list of links for all articles in this series can be found at the conclusion of this article.) In the previous installments of this series, we traced the arc from raw compliance intent — regulations such as NIST 800-53, FedRAMP, PCI DSS, EU AI Act — all the way to machine-readable OSCAL artifacts managed via GitOps pipelines and Trestle-powered automation. The central thesis has been that treating compliance artifacts as code, subject to the same versioning, testing, and review disciplines as software, is the only sustainable path to continuous assurance at scale. Part 3 of this series explored the collaboration topology: Regulators publishing OSCAL catalogs, Control Providers authoring component definitions, System Owners assembling SSPs, and Assessors generating SAPs and SARs — all mediated by Trestle's markdown-to-OSCAL round-trip. The friction was always the same: every persona still needed CLI fluency or IDE comfort to engage productively with OSCAL JSON. That friction is now removable. The Model Context Protocol (MCP) brings a standardized, AI-agent-ready interface to compliance tooling — and compliance-trestle-mcp, the first OSCAL-native MCP server from the OSCAL Compass community, makes every Trestle operation invocable by any MCP-compliant AI client: Claude, Roo Code, GitHub Copilot Workspace, or a custom agentic pipeline. Compliance-as-Code Game Changer With MCP The Model Context Protocol, incubated under the Linux Foundation and now an industry-wide open standard, provides a JSON-RPC layer by which AI models discover and invoke "tools" — discrete, typed operations exposed by servers. Think of it as the USB-C port for AI agents: standardized, self-describing, composable. Once an MCP server is registered, any compliant client can call its tools without custom integration work. For compliance workflows, this changes the architecture of engagement fundamentally. Today, driving Trestle to resolve a NIST 800-53 profile, generate SSP markdown, and assemble the resulting OSCAL JSON requires CLI invocations with precise arguments — work that falls to the Trestle-literate members of a compliance team. With compliance-trestle-mcp, those same operations become natural-language-addressable: an AI assistant executes the correct Trestle command sequence, validates the output, and surfaces results in whatever interface the persona is already working in. Compliance-trestle-mcp: Architecture and Capabilities The server is published on PyPI as compliance-trestle-mcp (v0.1.2, February 2026) and registered on the Official MCP Registry at registry.modelcontextprotocol.io under the identifier io.github.oscal-compass/compliance-trestle-mcp. Status is Active. Source: https://github.com/oscal-compass/compliance-trestle-mcp. Figure 1: compliance-trestle-mcp listed as Active on the Official MCP Registry (registry.modelcontextprotocol.io), v0.1.2. Tool Surface Six tools are currently exposed by the server, each wrapping a core Trestle operation: toolwhat it does trestle_init Initialize a Trestle workspace, creating the OSCAL folder hierarchy (catalogs, profiles, component-definitions, system-security-plans, etc.) trestle_import Import an existing OSCAL model (catalog, profile, SSP, component definition) from a local file or remote URL into the active workspace trestle_author_catalog_generate Generate per-control Markdown files from a catalog JSON, enabling human-readable editing without touching raw OSCAL trestle_author_profile_generate Generate Markdown documentation for the controls selected by a profile, preserving parameter overrides and guidance additions trestle_author_profile_resolve Resolve a layered OSCAL profile to a flat resolved-profile catalog, collapsing all imports and modifications trestle_author_profile_assemble Assemble edited Markdown controls back into a valid OSCAL Profile JSON, completing the round-trip Installation (One Liner) Add the following stanza to your agent's MCP configuration file (e.g., .roo/mcp.json for Roo Code or the Claude Desktop config): JSON { "mcpServers": { "trestle": { "command": "uvx", "args": [ "--from", "compliance-trestle-mcp", "trestle-mcp" ] } } } Personas Revisited: Now With an AI Co-Pilot Part 3 of this series established the canonical compliance-as-code collaboration model: five personas, each with distinct artifacts, editing interfaces, and OSCAL expertise levels. The MCP layer transforms each persona's relationship with those artifacts. Regulator Regulators publish security regulations and standards (NIST 800-53, GDPR, HIPAA) typically as PDFs. With compliance-trestle-mcp, a Regulator's technical team can instruct an AI agent to call trestle_import against a raw OSCAL catalog URL (e.g., the NIST GitHub releases), then trestle_author_catalog_generate to produce reviewable Markdown. Editorial cycles that previously required Trestle CLI expertise are now conversational. The AI handles the workspace plumbing; the domain expert focuses on control prose accuracy. Compliance Officer/CISO Compliance Officers author organizational overlays — parameter tailoring, guidance additions, control selections — expressed as OSCAL profiles layered on a regulatory catalog. With the MCP server, the AI can be prompted to "resolve the FedRAMP Moderate profile against the NIST 800-53 Rev5 catalog and generate the delta markdown for my SSP authoring queue." The agent chains trestle_author_profile_resolve→ trestle_author_profile_generate autonomously, surfacing the output for human review. This eliminates manual multi-step CLI orchestration and radically compresses profile maintenance cycles. Control Provider (Component Author) Control Providers — the engineers maintaining component definitions that map control implementations to policy-as-code rules — have traditionally needed both OSCAL fluency and DevSecOps context simultaneously. Now, an AI agent can assist by importing existing component definitions, generating Markdown stubs for unmapped controls, and prompting the engineer for implementation prose inline in the chat. The component definition round-trip (JSON → Markdown → edit → trestle_author_profile_assemble → JSON) is fully MCP-orchestrated. System Owner/SSO The System Owner assembles SSPs from profiles and component definitions — historically the most labor-intensive and error-prone step. With compliance-trestle-mcp, an AI agent can be directed to initialize the workspace, import all upstream artifacts, resolve the applicable profile, and generate the SSP Markdown scaffolding in a single conversational exchange. What once required mastery of four distinct Trestle sub-commands and careful argument threading is reduced to a natural-language instruction sequence. Assessor Assessors generating Security Assessment Plans (SAPs) and Reports (SARs) need to trace every selected control back through the SSP to the component definition and the originating catalog. With the MCP server, an AI agent can navigate that traceability chain on demand, resolving profiles and surfacing control implementation status, evidence links, and outstanding POA&M items — all without the assessor ever touching Trestle directly. The Emerging OSCAL MCP Ecosystem compliance-trestle-mcp is the first OSCAL-native MCP server from an established open-source compliance project, but it is not alone. A brief survey of the emerging ecosystem: serveroriginfocus compliance-trestle-mcp OSCAL Compass / CNCF Sandbox Full Trestle workflow: init, import, catalog/profile generate-assemble-resolve. First CNCF OSCAL MCP server. Registered at registry.modelcontextprotocol.io. mcp-server-for-oscal AWS Labs (awslabs) OSCAL schema introspection, model listing, and reference resource retrieval. Optimized for AI agents needing authoritative OSCAL structural guidance rather than authoring workflows. OSCAL MCP UI Apps Atelier Logos / Community Visual MCP UI layer for FedRAMP and HIPAA OSCAL workflows; interactive SSP visualization and compliance gap analysis via agentic app runtime. The AWS Labs server (github.com/awslabs/mcp-server-for-oscal) serves a complementary purpose: where compliance-trestle-mcp is workflow-centric (authoring and assembly), the AWS server is schema-centric (introspection and reference), providing AI agents with authoritative answers about OSCAL model structure, valid element sets, and use-case patterns. Together, they cover both the "what is OSCAL" and "do OSCAL" dimensions of agent-assisted compliance. NIST's Vision and the CSWP 53 Horizon The timing is not coincidental. NIST CSWP 53 ("Charting the Course for NIST OSCAL," December 2025 initial public draft) explicitly names agentic AI and digital twins as the next integration frontier for OSCAL — autonomous risk reasoning and continuous assurance driven by AI agents operating on machine-readable compliance artifacts. The compliance-trestle-mcp server is a concrete early instantiation of exactly that vision, with the CNCF Sandbox project providing governance and sustainability guarantees that standalone tools lack. What Comes Next for compliance-trestle-mcp The v0.1.2 release covers the catalog and profile authoring surface. The roadmap naturally extends toward the full OSCAL lifecycle for AI-assisted System Security Plan and MCP resource exposure — surfacing OSCAL documents as MCP resources (not just tool outputs) so AI clients can reason over live workspace state. Conclusion Compliance as Code has always promised to make compliance automation as natural as software development. The MCP layer removes the final adoption barrier: the requirement for personas to learn Trestle directly. With compliance-trestle-mcp, every compliance stakeholder — from the Regulator drafting a new catalog overlay to the Assessor closing out a FedRAMP SAR — can now engage with OSCAL artifacts through natural language, mediated by an AI agent that understands both the domain and the toolchain. The server is live, registered, and installable in seconds. The OSCAL ecosystem is building out MCP coverage rapidly, with NIST's own roadmap pointing in the same direction. The gap between compliance intent and continuous machine-readable assurance has never been smaller. References and Learn More [1] OSCAL Compass / compliance-trestle-mcp GitHub. https://github.com/oscal-compass/compliance-trestle-mcp [2] Official MCP Registry — io.github.oscal-compass/compliance-trestle-mcp. https://registry.modelcontextprotocol.io [3] AWS Labs mcp-server-for-oscal. https://github.com/awslabs/mcp-server-for-oscal [4] COMPASS Part 3: Artifacts and Personas (DZone). https://dzone.com/articles/compliance-automated-standard-solution-compass-part-3-artifacts-and-personas [5] NIST CSWP 53: Charting the Course for NIST OSCAL (Dec 2025 IPD). https://csrc.nist.gov/pubs/cswp/53/charting-the-course-for-nist-oscal/ipd [6] Building Visual MCP UI Apps for FedRAMP & HIPAA with OSCAL (Atelier Logos, Jan 2026). https://www.atelierlogos.studio/blog/2026-01-08-using-the-aws-mcp-server-for-oscal [7] OSCAL Hub — Open-Source OSCAL Platform (RegScale / OSCAL Foundation). https://regscale.com/blog/introducing-oscal-hub/ [8] Model Context Protocol Roadmap (Linux Foundation, updated Mar 2026). https://modelcontextprotocol.io/development/roadmap Below are the links to other articles in this series: Compliance Automated Standard Solution (COMPASS), Part 1: Personas and RolesCompliance Automated Standard Solution (COMPASS), Part 2: Trestle SDKCompliance Automated Standard Solution (COMPASS), Part 3: Artifacts and PersonasCompliance Automated Standard Solution (COMPASS), Part 4: Topologies of Compliance Policy Administration CentersCompliance Automated Standard Solution (COMPASS), Part 5: A Lack of Network Boundaries Invites a Lack of ComplianceCompliance Automated Standard Solution (COMPASS), Part 6: Compliance to Policy for Multiple Kubernetes ClustersCompliance Automated Standard Solution (COMPASS), Part 7: Compliance-to-Policy for IT Operation Policies Using AuditreeCompliance Automated Standard Solution (COMPASS), Part 8: Agentic AI Policy as Code for Compliance Automation With Prompt Declaration LanguageCompliance Automated Standard Solution (COMPASS), Part 9: Taking OSCAL-Compass to Industry Complexity LevelCompliance Automated Standard Solution (COMPASS), Part 10: How OSCAL Mapping Paves the Way for Continuous Compliance Scalability

I set out to build a simple Slack bot that could answer questions about our GitHub repository — open bugs, pending PRs, and recent releases. Straightforward enough. It turned into 400 lines of API glue code. When I asked Claude, ChatGPT, Gemini, and several coding assistants for architecture advice, they all converged on the same conventional pattern: What every AI suggestedWhat it means in practice1. Slack receives the mentionWrite a GitHub REST client2. Bot calls GitHub REST APIRouting logic per question type3. Feed response into Claude/GPTPagination per endpoint4. Model formats the answerMaintain API versions5. Bot posts back to SlackRepeat for every new data source This works. I built it. Three days, 400 lines of API client code, and it answered perhaps 60% of the questions my team asked. Questions like "Are any critical bugs related to PRs merged this week?" required custom correlation logic across multiple endpoints. Every new question type meant new code. Adding error monitoring as a second data source meant a separate integration entirely. After digging deeper into how AWS Bedrock handles tool use, I discovered the Model Context Protocol. I rebuilt the same bot in an afternoon — 150 lines, answering a far wider range of questions, and adding a new data source is a handful of lines in a single function. This article explains what changed and why it matters. The core insight: don't build an API client that feeds a model. Build a model that calls tools. These are fundamentally different architectures. The Architecture: Three Layers, One Loop The system is built in three layers. Each has exactly one responsibility and hands off cleanly to the next: Slack (Socket Mode) User types @mention → question received ↓ question passed to agent AWS Bedrock — Claude (Agent Loop) Reason → decide tools → call → read results → repeat ↓ tool calls routed via registry MCP Servers (GitHub + any other) 40+ tools per server — issues, PRs, releases, code search… ↓ tool results → reasoning → formatted answer → Slack Slack receives the @mention and passes the question down. Bedrock runs the agent loop — Claude reasons about which GitHub MCP tools to call, executes them, reads the results, and loops until it has enough data to answer. The tool registry routes each call to the correct MCP server automatically. The answer travels back up to Slack. Before vs. After: A Real Question To understand why this matters, consider a specific question a developer might ask in Slack: "Are any critical bugs related to PRs merged this week?" On the surface, this seems simple. But answering it correctly requires data from two separate GitHub API endpoints — the issues API for bugs, and the pull requests API for recent merges — and then correlation logic to match issue references in PR descriptions. If you are writing a traditional bot, you need to anticipate this question, write the two API calls, handle pagination on each, and write the join logic. Now imagine a dozen different question types. Each one is a new coding task. Traditional approachMCP approach1. Search GitHub for critical bugsClaude calls list_merged_prs (this week)2. Search for PRs merged this weekClaude calls search_issues (critical bugs)3. Write correlation logic across bothClaude calls get_issue for each candidate4. Handle pagination on each endpointClaude cross-references links in PR bodies5. Feed combined data to model to formatClaude returns correlated, formatted answer6. New question? Write new logic.New question? Model figures out new tools. What makes the MCP approach powerful is not just the line count — it is what the model is doing. Claude receives the full JSON Schema for every available GitHub tool at startup. When the question arrives, it reasons over those tool descriptions, selects the relevant ones, calls them in the right order, and then reasons over the combined results to produce an answer. It does not need to be told: "for bug questions, use search_issues". It reads the tool description and figures that out. The result is that the model can handle questions you never anticipated. "Show me PRs merged this week still linked to open bugs" — a slightly different framing of the same question — works without any code changes, because Claude adapts its tool selection to the new phrasing. Example Slack response: Plain Text :rotating_light: *Critical Bugs Linked to Recent PRs* • <https://github.com/org/repo/issues/1234|#1234> — Payment processing failure (linked to <https://github.com/org/repo/pull/5678|PR #5678>, merged Apr 14) • <https://github.com/org/repo/issues/1290|#1290> — Auth token timeout on mobile (linked to <https://github.com/org/repo/pull/5691|PR #5691>, merged Apr 15) Summary: 2 critical bugs found. Both linked to PRs merged this week. 6 tool calls: list merged PRs, search critical issues, get_issue per candidate. What the Model Context Protocol Does MCP is an open standard that lets AI models discover and call external tools through a uniform interface. Every MCP server exposes a tools/list endpoint returning every available action as a full JSON Schema. The model loads these at startup and reasons over them autonomously. Your application code never routes a single query. GitHub's official MCP server at api.githubcopilot.com/mcp/ exposes 40+ tools — issues, PRs, releases, code search — and a single GitHub token is all the authentication required. The shift is architectural, not cosmetic. The conventional model is a formatter — it receives data you fetched. The MCP model is a reasoning agent — it decides what to fetch, fetches it, and synthesizes the results. The first scales with the API code you write. The second scales with the MCP ecosystem. Why SRE and Platform Teams Should Care This bot started as a developer productivity tool. But when our SRE and platform engineering teams reviewed the architecture, they saw something broader: a pattern that could eliminate an entire category of operational toil. Platform teams spend considerable time maintaining integrations — every API change means updating a client, every new data source means a new integration project. The MCP pattern changes that calculus entirely. Integration toil. MCP server owners maintain compatibility with their own APIs. When GitHub updates its REST API, GitHub's MCP server absorbs that change. You own zero API client code.API drift. Traditional bots silently degrade when response schemas change. With MCP, the server owner tracks those changes — your bot keeps working.Correlation complexity. Linking deploys to errors, PRs to bugs, incidents to changesets — this logic is brittle in code and breaks constantly. Models do this naturally by reasoning across tool results in context.Platform rebuilds for new capabilities. Each new MCP server extends the bot without touching the agent loop. The loop is infrastructure. The servers are plugins. New team joins? New tool added? It is configuration, not development.The compounding effect matters most: every new MCP server registered is immediately available for any question the model asks. Traditional integrations accumulate glue code. MCP integrations accumulate capabilities. Conclusion The conventional approach to building AI-powered developer tools is not wrong — it works, and many teams run it successfully. But it carries a hidden cost: every new capability requires new code, every new data source requires a new integration, and every API change requires maintenance. Over time, that cost compounds. The Model Context Protocol eliminates that cost. By exposing tools through a uniform interface that the model discovers at startup, MCP shifts the integration burden away from your codebase and onto the ecosystem. The model reasons about which tools to call. You reason about what questions to answer. Part 1 has covered the why — the architectural distinction, the before/after comparison on a real question, and why this matters especially for SRE and platform teams. Part 2 puts it into practice with the complete implementation, step-by-step setup, and production lessons that make it reliable for daily use. Continue to Part 2: Implementation, Setup, and Production Patterns. Full project code on GitHub: https://github.com/sangharshcs/slack-github-mcp-bot.

The Pipeline Did Not Fail Cleanly Most pipeline failures don't look like "the job failed." Consider a common scenario. A Glue job reads overnight event files, applies business rules, and writes to an Iceberg curated table. The job runs at its scheduled time and errors out partway through. The control table shows SUCCESS for the previous batch and FAILED for the current one, which is what you'd expect. The problem is what happened between those two states: the job wrote nine of the day's twelve partitions to the staging table before failing. A downstream report ran on its own schedule, picked up the partial data, and the discrepancy didn't surface until a downstream consumer noticed records were missing. By the time someone looks at the failure, the question is no longer "Why did the job fail?" It's "Is it safe to rerun, and what's already corrupted downstream?" That's where debugging gets messy. CloudWatch logs, Glue run metadata, the source S3 path, record counts, data quality results, target table state, and Iceberg snapshots. An experienced engineer can connect those signals, but it takes time, and a less experienced engineer often misses one. In a busy production environment that delay leads to blind reruns, duplicate records, overwritten partitions, or worse. The frustrating part is that the evidence existed. The pipeline just had no structured way to explain itself. That's the gap a triage layer can fill. Not by fixing the pipeline. Not by changing schemas. Not by restarting jobs. By observing the evidence already produced, classifying the failure, explaining what likely happened, and recommending what to do next. What Agentic Observability Means The word "agentic" gets misused a lot right now, especially in data engineering. It's worth being precise. An agentic observability layer is not an LLM with permission to control production. It's a controlled workflow that collects pipeline evidence, builds incident context, classifies the failure against known categories, and produces a structured recommendation. The loop is observe, classify, explain, recommend, and that's where it stops. Everything past "recommend" stays with engineers, deterministic rules, or approval workflows. The difference from normal alerting is the depth of the output. A normal alert says "Glue job daily_customer_interactions failed." An agentic observability layer should produce something closer to: "The job failed because the input contains a new column not present in the curated schema. The staging write started before the failure, so a blind retry will create duplicate records. Quarantine the batch, review the schema contract, and rerun with the same batch_id after validation." That difference is what saves time during an incident. The goal isn't replacing engineers. It's reducing the manual triage work needed before someone can make a real decision. Reference Architecture This does not need to start as a new platform. The triage layer can sit beside existing Glue pipelines and consume signals that already exist. Figure 1. Agentic observability flow for AWS Glue pipelines. Pipeline evidence is collected, converted into structured context, analyzed by an LLM triage layer, and returned as a structured incident output. The component that matters most here is the incident context builder. The LLM should never receive a raw dump of ten thousand log lines. That produces noisy, low-confidence output and burns tokens. The collector should pull a curated set of signals: Glue job name and run ID, status and duration, batch ID, source path, target table, the last fifty error log lines, data quality results, record counts, attempt count, recent deployment version, table snapshot or commit ID, and control table status. That's enough context to analyze the failure without guessing from disconnected log lines. Where This Fits Before going further, one thing worth being honest about: this pattern depends on the platform already having its house in order. The agent can only work with the observability that the platform already has. It is not a substitute for basic pipeline hygiene. It works when the platform tracks batch IDs, clear source paths, data quality results, structured logs, table commits, deployment versions, and ownership mapping. Without those signals, the agent has very little to reason over. If a pipeline doesn't track batch IDs, the agent can't reliably tell whether a run is a retry or a new batch. If quality results aren't stored, it can't reason about input validity. If table commits aren't tracked, it can't tell whether the failure happened before or after a write. LLMs don't create observability. They summarize and reason over the observability that already exists. The teams that get the most out of this pattern are the ones with disciplined data engineering underneath. Failure Categories Manual debugging takes time, partly because every failure looks unique at first glance. Most don't stay unique once you classify them. A small fixed set of categories makes the output easier to review, compare, and route. Failure categoryCommon signalsRecommended actionSchema driftNew column, missing column, cast failure, contract mismatchQuarantine the batch and review the schema contractData skewLong-running tasks, shuffle spill, uneven partitionsRepartition or isolate skewed keysSmall file pressureHigh file count, slow planning, frequent commitsCompact affected partitionsSource delayMissing input path, low record count, late file arrivalWait, retry later, or mark the batch delayedCode regressionRecent deployment plus transformation errorRoll back or compare with the previous runPermission issueAccess denied, catalog failure, IAM or Lake Formation errorFix access policy before retryingPartial write riskFailure after write startedCheck staging and control tables before rerunUnknownWeak or conflicting evidenceEscalate to an engineer with summarized context The category list isn't only documentation. It's part of the system contract. The agent picks from this list rather than inventing categories on each run, which makes downstream routing tractable. Schema drift can go to the data contract owner. Permission issues route to the platform team. Source delays go to the ingestion owner. Partial write risk triggers a manual review workflow rather than auto-retry. This is what makes the system more useful than a chatbot that summarizes logs. Structured Incident Output The output should also be structured. Free-form summaries help humans skim, but they're hard to store, compare, or evaluate over time. JSON works better because it can be written to an incident table and consumed by Slack, Teams, Jira, or ServiceNow without parsing prose. JSON { "pipeline_name": "daily_customer_interactions", "job_run_id": "jr_2026_05_02_001", "status": "FAILED", "failure_category": "SCHEMA_DRIFT", "likely_root_cause": "Input file contains a new column named device_type that is not defined in the curated table schema.", "affected_source_path": "s3://raw/events/date=2026-05-02/", "affected_table": "curated.customer_interactions", "safe_to_retry": false, "recommended_action": "Quarantine the batch, update the schema contract, and rerun with the same batch_id after validation.", "confidence": 0.87 } A structured output gives engineers a quick summary, and it gives downstream tools something reliable to use. If safe_to_retry is false, the orchestrator blocks automatic retry. If failure_category is PERMISSION_ERROR, the issue routes to the platform queue. If confidence is low, the system asks for human review. If the same failure category recurs across runs, dashboards can track it over time. One important framing point: the LLM is not the system of record. The control table, logs, table metadata, and quality checks remain the source of truth. The agent is a reasoning layer that produces structured evidence on top of that. Implementation Sketch A simple implementation starts with assembling the incident context. The example below is intentionally simplified. In production, the LLM call should use structured outputs or schema-validated responses rather than free-form text parsing. Python def build_incident_context(job_run, control_record, dq_results, recent_logs): return { "job_name": job_run["JobName"], "job_run_id": job_run["Id"], "status": job_run["JobRunState"], "started_on": str(job_run["StartedOn"]), "completed_on": str(job_run.get("CompletedOn")), "batch_id": control_record.get("batch_id"), "source_path": control_record.get("source_path"), "target_table": control_record.get("target_table"), "attempt_count": control_record.get("attempt_count"), "control_status": control_record.get("status"), "data_quality_results": dq_results, "recent_error_logs": recent_logs[-50:] } The classifier receives a fixed category list and explicit rules about what it shouldn't recommend. Python def classify_failure(llm_client, incident_context): prompt = f""" You are analyzing a failed data pipeline run. Classify the failure into one of these categories: SCHEMA_DRIFT, DATA_SKEW, SOURCE_DELAY, PERMISSION_ERROR, CODE_REGRESSION, PARTIAL_WRITE_RISK, SMALL_FILE_PRESSURE, UNKNOWN. Return only valid JSON with: failure_category, likely_root_cause, safe_to_retry, recommended_action, confidence. Rules: - Do not recommend a retry if there is partial write risk. - Do not recommend schema changes without human review. - Do not recommend permission changes without platform approval. - Use UNKNOWN when evidence is weak or conflicting. Incident context: {incident_context} """ return llm_client.invoke(prompt) In a real implementation, this prompt should be paired with a strict response schema (failure_category as an enum, likely_root_cause as a string, safe_to_retry as a boolean, recommended_action as a string, confidence as a float between 0 and 1), and the system should reject any output that doesn't match. In production, structured outputs are the better choice when the API supports them. The free-form prompt above is illustrative. The result gets stored, not acted on: Python def store_incident_summary(summary, incident_table): incident_table.put_item( Item={ "pipeline_name": summary["pipeline_name"], "job_run_id": summary["job_run_id"], "failure_category": summary["failure_category"], "safe_to_retry": summary["safe_to_retry"], "recommended_action": summary["recommended_action"], "confidence": summary["confidence"], "created_at": current_timestamp() } ) The agent writes an explanation. Other systems decide what to do with it. What the Agent Should Never Decide This boundary is the most important design choice in the whole pattern, and it's worth being explicit about. An observability agent helps engineers understand a failure. It does not control production data systems. Even at high confidence, certain actions stay out of scope: Changing table schemasGranting IAM or Lake Formation permissionsDeleting dataMarking a partially written batch as successfulOverriding data quality failuresPromoting quarantined dataRewriting production tablesTriggering cross-pipeline backfillsCompacting or expiring table snapshots without approval These actions move from observability into production control, and that line should stay clear. In regulated or business-critical environments, the safest design lets the agent produce structured evidence and recommendations while deterministic rules, approval workflows, or engineers decide whether anything actually executes. An agent saying "this looks like schema drift, the batch is not safe to retry" is useful. The same agent updating the curated table schema on its own is not. It's a future incident waiting to happen. Same with permissions: the agent flagging an IAM issue is useful; the agent granting itself access is a security violation. The trade-off here is real. Letting the agent take action would reduce the mean time to recovery. But the cost of a confident wrong action (silently corrupted data, an unauthorized permission grant, a dropped partition) is much higher than the cost of a few extra minutes of human review. In a regulated data environment, that trade-off is usually easy to justify. This matters as teams move toward self-healing pipelines. Before a pipeline can safely fix itself, it has to first explain itself reliably, at scale, with measurable accuracy. That bar isn't met yet in most production environments. Evaluating the Triage Layer A triage layer should be evaluated like any other production component. "The summary looks good" is not an evaluation. To check whether the pattern behaves reasonably, a small synthetic evaluation can be assembled across common Glue failure modes. Each scenario includes a short set of log lines, control-table state, data quality results, and table metadata, and the agent is scored on two things: whether it picks the correct failure category, and whether the safe_to_retry decision is appropriate. This is a starter evaluation, not a benchmark. Ten synthetic scenarios are enough to sanity-check the design. A real production rollout needs hundreds of labeled historical incidents, edge cases, and human-reviewed outcomes. Anything less should be treated as an early prototype, not production validation. ScenarioExpected categoryAgent categorySafe-to-retry decisionMissing source pathSOURCE_DELAYSOURCE_DELAYCorrectNew column in inputSCHEMA_DRIFTSCHEMA_DRIFTCorrectAccess denied on catalog tablePERMISSION_ERRORPERMISSION_ERRORCorrectShuffle spill and one long taskDATA_SKEWDATA_SKEWCorrectFailure after staging writePARTIAL_WRITE_RISKPARTIAL_WRITE_RISKCorrectToo many small filesSMALL_FILE_PRESSURESMALL_FILE_PRESSURECorrectRecent code deployment plus null pointerCODE_REGRESSIONCODE_REGRESSIONCorrectLow record count, no hard errorSOURCE_DELAYUNKNOWNConservative escalationCast failure due to bad input valueSCHEMA_DRIFTSCHEMA_DRIFTWrong, recommended retryConflicting log signalsUNKNOWNUNKNOWNCorrect escalation In a small evaluation like this one, a well-designed classifier should pick the expected category in most scenarios and, more importantly, get the safe-to-retry decision right in nearly all of them. The illustrative results above show eight correct retry decisions, one conservative escalation (the agent returns UNKNOWN rather than guessing), and one wrong call. That wrong call is the most instructive. On the cast failure, the agent classifies the issue correctly as schema drift but recommends cleanup-and-retry instead of quarantine-and-contract-review. A wrong root cause is inconvenient. A wrong retry recommendation can corrupt data. Safe-retry precision should be weighted higher than classification accuracy when evaluating this kind of system, and that weighting should be reflected in the prompt rules and in the validation rubric. The metrics worth tracking in production: MetricWhy it mattersClassification accuracyWhether the agent identifies the right failure typeSafe-retry precisionWhether retry recommendations are actually safeFalse confidence rateConfident-but-wrong recommendationsMean triage timeReduction in manual debugging timeHuman override rateHow often engineers reject the recommendationCost per incidentLLM and log-processing cost per failed run False confidence rate deserves attention. A low-confidence wrong answer is manageable because engineers know to scrutinize it. A high-confidence wrong answer is dangerous because teams stop scrutinizing. Confidence belongs in the output, but it should never be treated as truth. It's one signal among several in the routing decision. Closing Glue job failures aren't hard because the logs are long. They're hard because the evidence is scattered across logs, run metadata, data quality results, control tables, and table commits, and an engineer has to assemble it before deciding what to do next. An agentic observability layer turns that scattered evidence into a structured incident summary. The safest version of this pattern is controlled triage, not autonomous repair: observe, classify, explain, recommend, and stop there. Deterministic rules, approval workflows, and engineers decide what happens next. Before pipelines can fix themselves, they need to explain themselves. That's the work worth doing first.