DZone Spotlight

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or retry after connectivity changes while the original execution is still progressing. Apple explicitly distinguishes safe retry behavior by HTTP method and notes that URLSession can retry requests in some connection-loss cases, waitsForConnectivity can also cause the system to continue a request when connectivity returns. The dangerous state is therefore not “request failed,” but “completion is unknown.” If that request starts an agent that charges an account, reserves inventory, sends a message, or invokes an MCP tool, a second submission can become a second side effect. The Retry Boundary Is the Real Transaction Boundary “Exactly once” is too strong for a workflow crossing an iPhone, HTTP, an agent runtime, an MCP server, Kafka, a database, and an external API. Kafka can provide exactly-once guarantees within defined Kafka processing boundaries, but those guarantees do not atomically include arbitrary remote tool effects. The practical target is effectively-once behavior, and retries are expected, but every effect is guarded by a stable operation identity and converges on one committed outcome. Kafka’s idempotent producer suppresses duplicate records caused by producer retries, while transactional producers can atomically publish across Kafka partitions; the producer documentation also limits idempotence guarantees to a producer session and requires read_committed consumers for end-to-end transactional visibility. The operation identity must exist before the first network attempt. An iOS client can create an operationId when an action becomes durable local intent, persist it, and reuse it across transport retries. Transport material such as a server challenge may change, but the business ID must not. The server treats (subjectId, operationId) as a uniqueness boundary and stores a canonical payload hash with it. PostgreSQL unique constraints enforce row uniqueness, while INSERT ... ON CONFLICT provides an atomic conflict path under concurrency. SQL INSERT INTO agent_operation(subject_id, operation_id, payload_hash, status) VALUES (:subject, :operationId, :payloadHash, 'ACCEPTED') ON CONFLICT (subject_id, operation_id) DO NOTHING; A conflict with the same payload hash returns the existing operation; a different hash rejects key reuse. The record should exist before agent execution starts, and the accepted response should expose the durable operation identity. Let LangGraph Resume Without Repeating Effects LangGraph persistence is useful precisely because durable execution can replay code. With a checkpointer, LangGraph saves state at super-step boundaries; if execution resumes after a failure, an affected node can run again from the beginning. Official guidance consequently requires idempotent node logic, and task results can be checkpointed so completed task work can be reused during resume instead of recomputed. Replaying from an earlier checkpoint can also re-trigger later LLM calls and API requests. A stable business operation should therefore map to a stable LangGraph thread, while every effectful tool boundary receives the same operation ID. Python config = {"configurable": {"thread_id": operation_id} result = graph.invoke( {"operation_id": operation_id, "command": command}, config ) Checkpointing reduces recomputation but does not replace downstream idempotency. A reservation can succeed before its task result is durably checkpointed. LangGraph’s functional API therefore recommends idempotent tasks because an incomplete task can execute again during resume. Python @task def reserve_inventory(operation_id, sku, quantity): return mcp.call_tool("reserve_inventory", { "operationId": operation_id, "sku": sku, "quantity": quantity }) The significant property in this snippet is not the decorator. The important part is that the business identity crosses the graph boundary and reaches the tool implementation. A downstream inventory service can then use that identity to return a previously committed reservation rather than creating another one. MCP Tasks Are Durable Handles, Not Deduplication Keys The current MCP Tasks design is especially relevant to long-running agent tools. In the July 28, 2026 protocol revision, Tasks moved into the io.modelcontextprotocol/tasks extension. A server can return a durable task handle, and the client can poll with tasks/get, provide input with tasks/update, or request cancellation with tasks/cancel. The task is durably created before its handle is returned, which allows polling after a disconnect. That durability solves result retrieval after task creation, but it does not by itself deduplicate the request that creates the task. The task ID is server-generated. If the server creates task A, the response disappears, and the original tools/call is sent again, a naïve implementation can create task B. Therefore, the business operationId must be part of the tool arguments or equivalent application metadata, and task creation must first look up an existing operation. This follows directly from MCP’s server-generated task-ID model combined with retry ambiguity at the HTTP boundary. The MCP server can return an existing task handle for the same authenticated subject, operation ID, and payload hash, and later return the stored terminal result. Cancellation should also be idempotent because MCP defines it as cooperative rather than a guarantee that underlying work stops immediately. Keep Kafka Guarantees Inside Kafka Kafka is most valuable after the operation has been claimed. A database transaction can persist operation state with an outbox row carrying the same ID. Kafka producer idempotence protects against duplicates caused by producer retries, while consumers can still use the operation ID for application-level deduplication. Kafka transactions can atomically cover Kafka writes, but they do not extend over an MCP server or payment API. The event contract should preserve causality rather than inventing a new identity at each hop. JSON { "operationId": "8E7B6D9E-...", "type": "AgentToolCompleted", "tool": "reserve_inventory", "status": "SUCCEEDED" } A consumer can enforce uniqueness on (consumerName, operationId, eventType) or make the state transition conditional. Kafka delivery guarantees and application idempotency then reinforce each other instead of being treated as interchangeable. Bind Retry Identity to App Attest Without Blocking Legitimate Retries App Attest addresses a different failure mode: whether a request comes from a legitimate app instance and whether signed request material has been replayed or altered. Apple’s current guidance uses a server-provided challenge for assertions and requires the server to validate a strictly increasing assertion counter; that counter is specifically an anti-replay signal. Assertions are generated locally on the device after key attestation. The App Attest assertion must not become the business idempotency token. A legitimate retry should obtain fresh challenge material and generate a fresh assertion while retaining the original operation ID. The data hashed for the assertion can bind the server challenge, operation ID, and canonical payload hash together. Swift let payloadHash = SHA256.hash(data: body) let clientData = challenge + operationID.data + Data(payloadHash) let clientDataHash = Data(SHA256.hash(data: clientData)) let assertion = try await service.generateAssertion( keyID, clientDataHash: clientDataHash ) Apple recommends server-controlled challenges, server-side validation, and assertion-counter tracking as assertions are generated on demand without a round trip to Apple’s servers. The server verifies App Attest, checks that the challenge binds the operation ID and payload, then performs the idempotency lookup. A fresh assertion can retry the same operation; a replayed assertion fails anti-replay validation; an altered payload fails the hash check. Effectively-Once Behavior Is a Composition Property Reliable agent execution does not come from asking iOS to retry less often or from labeling a Kafka pipeline “exactly once.” It comes from carrying one durable business identity across every retry and every boundary, claiming that identity atomically before execution, making LangGraph effects idempotent under resume, using MCP Tasks as durable result handles rather than creation-time deduplication keys, restricting Kafka’s exactly-once guarantees to Kafka’s transactional domain, and using App Attest to prove request integrity without confusing anti-replay state with business deduplication. When those boundaries align, a lost mobile response can cause another HTTP attempt, another graph invocation, or another poll, but it does not cause another business effect. That is the operational meaning of effectively once.

By Uthej Mopathi CORE

Security teams usually describe an application through the assets they know about. This includes the production domain, documented APIs, the services currently in use, and the repositories connected to the latest release. But applications leave things behind as they change. A staging environment created for an old release may still be online months later, alongside an API version that was supposed to be retired. Other forgotten parts of the application can surface through DNS records, certificate data, or information left in client-side code. Most of these assets were created for perfectly legitimate reasons. Trouble starts when the work moves on, but the infrastructure doesn't. Ownership becomes unclear, security controls fall behind, and eventually an internet-facing component can remain active without appearing in the inventory used by the team responsible for it. I use unindexed attack surface to describe the gap between the application a team actively manages and the parts of it that are still reachable. Unindexed Does Not Mean Inaccessible It is easy to assume that an application resource is relatively safe when it doesn't appear in search results or isn't linked from the main application. That assumption falls apart once someone discovers the address. The same idea comes up when explaining the deep web. A large amount of online content sits outside conventional search indexes while remaining accessible through a direct URL, login, or other route. Application infrastructure can end up in a similar position. A staging host or old endpoint may be absent from the normal user journey and still be exposed to the internet. Finding these assets doesn't always require sophisticated techniques. During reconnaissance, an attacker can piece together clues from certificate records, DNS data, JavaScript, public repositories, and documentation. One discovery can lead to another until parts of the application that developers rarely think about become visible. robots.txt is a simple example. It tells compliant crawlers what they should avoid crawling, but it doesn't prevent someone from requesting those paths directly. OWASP's web security testing guidance includes reviewing web server metadata, identifying application entry points, and mapping execution paths during reconnaissance. Development and security teams can use similar techniques to see what their application exposes from the outside. Modern Delivery Creates Assets Faster Than Inventories Can Follow Modern development makes it easy to create infrastructure quickly. A pull request may generate a temporary preview environment, while a migration can leave /api/v1/ running as clients move to /api/v2/. During an incident, a debugging endpoint might be created and never removed afterward. Even a short-lived cloud experiment can leave behind a hostname outside the infrastructure account monitored by security. Keeping track of all of this gets harder as the application changes. Cloud platforms, API gateways, deployment configurations, and DNS may each hold a different piece of the inventory. Something created for one sprint can still be reachable several releases later. OWASP addresses this problem in API9:2023 Improper Inventory Management. Its guidance covers outdated API versions, exposed hosts, and missing documentation that can leave older parts of an application running without the security attention given to current services. Multiple development teams make that inventory harder to maintain. An environment may remain active after the developer who created it has moved to another project. Unless deployment and retirement update the inventory along with the infrastructure, these assets can stay online far longer than anyone intended. Forgotten Assets Often Retain Real Trust An old endpoint can outlive its original purpose without losing the access it was given. An earlier API version, for example, may still connect to the production database even though it hasn't received the authorization checks, rate limits, or input validation added to the current version. Staging environments can have the same problem when they use production-like data or continue running with an identity configuration that hasn't been reviewed in some time. Once an environment falls outside normal development work, security updates and monitoring are easier to miss. OWASP gives a useful example in its guidance on improper inventory management. A beta API host exposes the same password-reset capability as the production API, but the beta version lacks the rate limiting applied to production. Anyone who discovers the older host gets another route to the same function with fewer protections. The same situation can appear elsewhere in an application. A preview deployment might contain credentials left in an old build, while a forgotten administrative interface could still be reachable through a load balancer. These assets become even harder to manage when logging is no longer checked, or alerts still point to a team that has stopped owning the service. The longer an asset sits outside normal development and security workflows, the easier it is for its permissions, dependencies, and controls to fall behind the rest of the application. The Frontend Can Reveal the Backend’s Missing Map The browser often reveals more about an application than teams realize. It needs enough information to communicate with backend services, so production JavaScript can include API URLs, route names, environment identifiers, feature flags, and references to functionality users no longer see. This becomes interesting when the frontend has moved on, but the backend hasn't. A feature may disappear from the interface while its endpoint continues to respond. Commenting out a button or removing a route from the visible application doesn't remove the server-side functionality behind it. Source maps can make this easier to investigate. They help browsers reconstruct optimized JavaScript into something closer to the original source, making debugging easier. MDN's documentation explains how the SourceMap header and sourceMappingURL annotation point developer tools to these files. When source maps are publicly available in production, they can reveal original filenames and make the application's client-side structure easier to follow. That exposure doesn't automatically mean the application is vulnerable. Problems arise when an old endpoint is still reachable, authorization depends too heavily on what the frontend displays, or functionality exposed through the client was never included in the team's current security review. One useful check is to compare what appears in production JavaScript, browser traffic, and available source maps with the routes the team expects to have deployed. Unexpected endpoints deserve a closer look, especially when nobody can immediately explain why they are still there. Make Inventory Part of the Delivery Process Finding forgotten assets starts with looking past the list provided to the vulnerability scanning tool. If that list is incomplete, even a successful scan may leave parts of the application untouched. It is important to continuously verify the accuracy of the inventory against the discoverable assets outside the organization. This means collecting information from cloud accounts, deployment configurations, API gateways, DNS and certificate records, and exploring all hosts/endpoints that are reachable but cannot be placed anywhere in those records. The inventory needs to be closely tied to the deployment process as well. Whenever a service gets deployed, information should be collected about who owns the service, where it runs, what environment it belongs to, and whether it is public or private. This can be accomplished through a CI/CD pipeline as infrastructure is created or changed rather than having someone manage the spreadsheet manually. The same applies to API documentation. OWASP recommends generating API documentation automatically and including it in the CI/CD process. Teams can also compare deployed routes to the approved specification of the API so that an unexpected endpoint becomes visible while the application is still being worked on. From there, a few controls can catch problems early: Require ownership of public services before deploying them to production.Put expiration dates on preview and temporary environments.Flag unexpected new DNS or certificate records outside of the expected deployment process.Track deprecated API versions until they are removed.Keep production data in non-production environments only if it is absolutely necessary. Whenever something unexpected gets detected, assign it to someone who can determine why it is still running. Active services must receive the same level of security attention as the rest of the application. The ones that have served their purpose should be removed. Make Sure Retired Assets Are Actually Gone Removing a service from a repository or architecture diagram doesn't mean the service has disappeared from the internet. Old DNS records can remain, gateway routes may still forward traffic, and credentials created for the service can continue working after the team considers the project finished. Before shutting anything down, check whether it is still receiving traffic. An old API version may have clients nobody remembered, and immediately removing it could break an integration that is still in use. If the service needs to stay online, it should remain under the same monitoring, patching, and access controls as other active systems until those dependencies are dealt with. Once the service is retired, verify the result from outside the environment. Confirm that its hostname no longer resolves where it shouldn't, old routes no longer respond, credentials have been revoked, and any associated storage or third-party integrations have been removed. This step is easy to overlook during migrations or team changes, when responsibility for older infrastructure can become unclear. A service nobody considers active can still be reachable months later if no one checks that the shutdown actually happened. Conclusion Applications change constantly, and some of the infrastructure created along the way will eventually be forgotten. Problems begin when those old hosts, endpoints, and environments remain reachable without anyone checking whether they still need to exist. The inventory needs to change with the application. Build discovery into the delivery process, keep ownership clear, and verify that retired assets are actually gone. If something connected to your application is still reachable from the internet, your team should know why it is there and who is responsible for it.

By Igboanugo David Ugochukwu CORE

Why "Ask the Company" Beats "Ask Around" At a startup, knowledge lives everywhere and nowhere — a Slack thread here, a Notion doc there, a decision buried in an old email thread that only one person remembers. New hires spend their first few weeks just learning where things are, and even tenured employees waste hours pinging teammates for answers that already exist somewhere in the company's systems. This article walks through a lean, production-ready architecture for making your company's knowledge queryable through AI, scoped specifically for teams that don't have a platform engineering org to lean on. The goal isn't to build the most sophisticated system possible — it's to build the smallest system that reliably answers real questions, and grow it from there. Section 1: Start With the Problem, Not the Model 1.1 Define What "Queryable" Actually Means for You Before picking a vector database, decide what questions people should actually be able to ask: onboarding FAQs? Product specs? Customer support history? Internal policy? "Make the company queryable" sounds like one project, but it's really dozens of smaller ones bundled together. Scope creep is the single biggest killer of these initiatives — teams try to boil the ocean, burn a quarter on infrastructure, and never ship anything a normal employee actually uses. Start with one high-value, narrowly defined use case and resist the urge to expand until it works. 1.2 Audit Your Knowledge Sources List every place knowledge currently lives — your wiki, Slack, Google Drive, CRM, ticketing tool, even that one spreadsheet everyone secretly relies on — and rank each by query value versus integration effort. Most startups discover that 80% of the value comes from just two or three sources. Resist the temptation to connect everything on day one; each new source adds ingestion complexity, permission mapping, and another way for stale data to creep in. 1.3 Set a Success Metric Up Front Decide what "working" looks like before you write a line of code. Build a small golden test set — twenty to fifty real questions with known-correct answers — and measure retrieval and answer accuracy against it. Without this, you're shipping based on vibes, and vibes don't survive contact with a skeptical exec asking why the bot gave a wrong answer in a company all-hands. Section 2: A Lean Architecture for Small Teams 2.1 Ingestion Without an Engineering Team Use off-the-shelf connectors — Airbyte, Unstructured.io, or native APIs from the tools you already use — instead of building custom scraping pipelines you'll have to maintain forever. Startups should buy or borrow this layer wherever possible; the engineering hours saved here are better spent on the parts of the system that are actually differentiated, like retrieval quality and access control. 2.2 Chunking That Preserves Meaning Chunk documents by semantic boundary — headers, paragraphs, natural thread breaks in Slack conversations — rather than arbitrary token counts. A chunk that cuts a policy explanation in half mid-sentence produces answers that are technically retrieved but practically useless. Attach metadata to every chunk: source system, author, last-updated date, and access level. This metadata feels like overhead early on, but it becomes essential the moment you need to filter results by permission or debug why a stale answer surfaced. 2.3 Picking a Vector Store You Won't Outgrow (or Overpay For) For most startups, a managed option like Pinecone or Qdrant Cloud is enough, and if you're already running Postgres, the pgvector extension can get you surprisingly far without adding a new piece of infrastructure to operate. Skip self-hosted vector databases until scale genuinely demands them — the operational overhead isn't worth it at startup query volumes, and premature infrastructure investment is one of the most common ways these projects stall out before launch. 2.4 Hybrid Retrieval: Don't Rely on Vectors Alone Combine semantic (vector) search with keyword or BM25 matching so exact terms — ticket numbers, product SKUs, customer names — aren't lost in embedding space. Pure semantic search is great at conceptual similarity but surprisingly bad at exact-match lookups, which are often exactly what employees are searching for. Section 3: The Part Startups Skip (And Regret) 3.1 Access Control From Day One If someone can't see a document in Drive, they shouldn't be able to surface its contents through the AI system either. Filter retrieval using the same permissions as the source system, ideally at query time using per-chunk ACL metadata. Retrofitting access control after launch is painful, risky, and in the worst case turns into a security incident — this is the single most common way these projects go wrong, and it's far cheaper to design for it upfront than to patch it later. 3.2 Logging and Auditability Log every query submitted, and every document surfaced in response. This matters for three reasons: debugging why an answer was wrong, building trust with skeptical stakeholders, and — as you scale — satisfying compliance requirements you may not be thinking about yet but will eventually need. 3.3 Building Trust With Citations Always show sources alongside generated answers. Startups that skip this consistently see low adoption, because people don't trust an answer they can't verify, and one confidently wrong answer without a source is often enough to sour a team on the whole tool. Section 4: Orchestration — Letting AI Choose Where to Look 4.1 Why Blind Search-Everything Doesn't Scale As you connect more sources, querying all of them for every single question gets slow, noisy, and expensive. An orchestration layer lets the system reason about which source is actually relevant to a given question — checking the ticketing system for a support question, the wiki for a policy question — rather than brute-forcing a search across everything indexed. 4.2 Using MCP (Model Context Protocol) as the Connective Tissue MCP standardizes how your AI system calls out to different tools and data sources, which makes it significantly easier to add or swap sources later without rewriting your core retrieval logic each time. For a startup, this matters less for elegance and more for maintainability — you want to be able to plug in a new tool in an afternoon, not rearchitect a subsystem. 4.3 Structured + Unstructured Together Combine retrieval-augmented generation (RAG) over unstructured documents with direct queries to structured data — a SQL database, a CRM API — so the system can answer both "what's our refund policy" and "how many tickets did customer X file last month" in the same interface, without forcing users to know which system holds which kind of answer. Section 5: Rollout Without Breaking Trust 5.1 Start With a Pilot Team Roll out to one team first — support or onboarding are usually good candidates because their questions are repetitive and well-documented — and gather real usage data before attempting a company-wide launch. A contained pilot also gives you a safe space to catch access-control or accuracy issues before they become visible to the whole company. 5.2 Build a Feedback Loop Let users flag wrong or unhelpful answers directly in the interface, with a single click. This becomes your evaluation dataset for continuous improvement, and it signals to early users that the tool is actively maintained rather than a one-off experiment that will quietly degrade. 5.3 Plan for Staleness Knowledge changes constantly, and a system that was accurate at launch can quietly become wrong within weeks if nothing is re-indexed. Set a re-indexing cadence — daily batch jobs or webhook-triggered updates for high-churn sources — so the system doesn't erode the trust you just spent your pilot phase building. Section 6: What This Costs a Startup (Realistically) 6.1 Where the Money Actually Goes Embedding and LLM API calls, not infrastructure, tend to dominate cost at startup scale. Managed vector stores are comparatively cheap, especially at the query volumes most early-stage companies see. Budget accordingly — don't over-provision infrastructure while under-budgeting for the ongoing API costs that will actually show up on your bill every month. 6.2 Cheap Wins to Control Spend Cache frequent or repeated queries, batch embedding jobs instead of running them one document at a time, and use smaller, cheaper models for retrieval-adjacent tasks like reranking or query rewriting — reserving your best (and most expensive) model for final answer generation, where quality matters most. Conclusion: Ship Small, Prove Value, Then Expand The startups that succeed with this don't try to index everything on day one. They pick one painful knowledge gap, solve it well for one team, prove the value with real usage data, and expand from there — with access control and citations built in from the start, not bolted on after an incident. The systems that fail tend to fail for predictable reasons: too broad a scope at launch, no success metric to measure against, and access control treated as an afterthought. Avoid those three mistakes, and the rest is largely a matter of good engineering hygiene.

By Balaji Venkatasubramaniyar CORE

Three weeks. That's how long it took my team to wire Claude into our internal ticketing system last year. Not because the API was hard. Because every layer of the stack was speaking a different dialect — custom function schemas on one side, brittle REST wrappers on the other, and a Python shim in the middle that I was too embarrassed to commit without a comment that said: "don't look at this." We shipped it. It worked. For about four days, until the vendor updated their response payload and our parser silently swallowed the change. Tickets started routing to the wrong queue at 2 AM on a Tuesday. I learned about it from Slack, not monitoring. That experience is why Model Context Protocol (MCP) landed so differently for me than it did for the people writing blog posts about it from a fresh MacBook. This wasn't "interesting new protocol." It was a direct answer to a specific, grinding pain. Stop Calling It a Framework The USB-C analogy gets repeated so often it's starting to lose meaning. Let me make it concrete. USB-C solved a problem the tech industry had been ignoring for a decade: every device spoke a slightly different power/data dialect, and the combinatorial explosion of adapters was genuinely slowing things down. USB-C collapsed that N×M adapter problem into a single connector. One port. Any cable. Any device. You still need to negotiate speeds and capabilities over the wire — but the physical contract is shared, which means you can stop thinking about connectors and start thinking about what you're actually moving. MCP does exactly that for AI tool integration. Before it, connecting an LLM to a tool meant writing a custom schema for that LLM's function-call format, a custom parsing layer for that tool's response shape, and — if you wanted to switch providers — starting over. Six integrations across three LLM providers meant eighteen combinations to maintain. The N×M problem. MCP's answer is a shared protocol layer: one JSON-RPC 2.0 contract, negotiated at initialization, that any compliant client can speak to any compliant server. Tools become server capabilities, not one-off function schemas. The LLM doesn't care whether it's talking to a Salesforce connector or a PostgreSQL server — both speak MCP, both expose the same tool-call lifecycle, both fail in predictable ways. That last part matters more than people give it credit for. The Protocol Stack, Actually Explained Most articles stop at "MCP uses JSON-RPC 2.0." That's true, but it's like saying "HTTP uses TCP." Correct. Not sufficient. Layer 1: JSON-RPC 2.0 Messaging JSON-RPC 2.0 is a stateless, lightweight remote procedure call protocol. It predates AI by over a decade — Ethereum uses it, Ethereum Classic uses it, VS Code's Language Server Protocol is built on it. Anthropic's team made a smart choice borrowing from LSP specifically, because LSP proved that you could build richly typed, bidirectional tooling protocols on top of a dead-simple message format. Every MCP message is one of three shapes: JSON // Request (client → server) { "jsonrpc": "2.0", "id": 42, "method": "tools/call", "params": { "name": "search_tickets", "arguments": { "query": "priority:high assignee:me" } } } // Response (server → client) { "jsonrpc": "2.0", "id": 42, "result": { "content": [{ "type": "text", "text": "Found 3 tickets..." }], "isError": false } } // Notification (no id — fire and forget, no response expected) { "jsonrpc": "2.0", "method": "notifications/tools/list_changed" } The id field is doing important work there. Requests have IDs; notifications don't. The client matches responses to requests by ID — which means you can pipeline multiple concurrent requests without ordering guarantees. That's relevant once you start running parallel tool calls, which is exactly what modern agent orchestrators do. Layer 2: Transport Options MCP supports two transports, and picking the wrong one is one of the most common production mistakes I see. stdio is for developer tooling. Cursor uses it. Claude Desktop uses it for local servers. The server runs as a child process, stdin/stdout are the pipe. Zero network overhead, instant startup, trivially secure. Wrong choice for anything multi-tenant or horizontally scaled. Streamable HTTP is what you deploy to production. Single HTTPS endpoint, HTTP POST for client-to-server, optional SSE (Server-Sent Events) stream for server-to-client pushes. The March 2025 spec update replaced the earlier dedicated SSE-only transport — importantly, Streamable HTTP added support for stateless operation, which is the feature that makes real horizontal scaling possible. More on that in a minute. HTTP # Client → Server: negotiate capabilities POST /mcp HTTP/1.1 Content-Type: application/json Authorization: Bearer eyJhbGc... { "jsonrpc": "2.0", "id": 1, "method": "initialize", "params": { "protocolVersion": "2025-11-25", "capabilities": { "tools": {} }, "clientInfo": { "name": "my-agent", "version": "1.4.0" } } } # Server → Client: confirm supported capabilities HTTP/1.1 200 OK Content-Type: application/json Mcp-Session-Id: a3f9-c2d1-8b04 # only in stateful mode { "jsonrpc": "2.0", "id": 1, "result": { "protocolVersion": "2025-11-25", "capabilities": { "tools": { "listChanged": true } }, "serverInfo": { "name": "ticketing-mcp", "version": "2.1.0" } } } Notice Mcp-Session-Id. That header only appears in stateful mode. In stateless mode — which you want for any horizontally scaled deployment — there's no session header. Every request is self-contained. Critically, that means load balancers can route requests to any instance without sticky sessions. That's the architectural unlock. Layer 3: The Three Primitives MCP servers expose exactly three types of capabilities. This is deliberate. The constraint is the feature. The distinction between Tools and Resources isn't cosmetic. Tools can have side effects. Resources can't. The November 2025 spec update formalized tool annotations — you now declare whether a tool is read-only, destructive, or idempotent in the schema itself. That annotation is what lets your gateway apply different rate limits and audit policies per tool class without building bespoke middleware. OAuth 2.1: Why It's Here and What It Costs You Auth was technically optional in early MCP. The community paid for that decision: trojanized packages, unauthenticated community servers running wide open in local dev environments, and at least one incident report I've seen from an enterprise pilot that I won't name where an MCP server was reachable from a public IP with no credentials required. The November 2025 spec update made OAuth 2.1 the recommended standard for remote servers. In practice, if you're deploying Streamable HTTP in a production environment, treat it as mandatory. A few things worth knowing before you implement this: OAuth 2.1 drops implicit flow entirely. If you have legacy client code that used implicit — and plenty of older enterprise apps do — you're rewriting that before you go live. Plan a sprint.PKCE is mandatory for public clients even with authorization code flow. The spec doesn't give you a waiver for this.Server discovery at /.well-known/oauth-authorization-server is how clients find your token endpoint without hardcoding. Don't skip implementing this. Dynamic client registration makes onboarding new agent clients 10x less painful.Tokens are per-user context, not per-MCP-server. Your gateway needs to thread the right token to the right downstream server. That routing logic is where I've seen the most production bugs — specifically, token scope mismatches that silently returned empty results instead of 403s. Production Architecture: The Full Stack Here's what the actual architecture looks like once you move past single-developer demos. The Stateless Scaling Model This is the piece most tutorials gloss over, and it's the piece that will bite you at 3 AM. The original MCP spec used session IDs. Every client got pinned to a server instance via Mcp-Session-Id. That's great for local development. For Kubernetes? It means sticky sessions, broken pod rollouts, and a load balancer that has to track which client is where. The November 2025 spec update added stateless operation as a first-class option — no session IDs; every request carries all context it needs. Stateless vs. Stateful: The Decision Tree Choose stateless(no session ID) if your tools are side-effect-free queries or short-lived mutations. Your load balancer routes freely, Kubernetes rolling deployments work cleanly, horizontal scale is trivial. Choose stateful(session pinned) only when you genuinely need server-side context across calls — browser automation, long-running file operations, or multi-step transactions where partial state lives on the server. For stateful deployments, you need Redis-backed session storage and sticky session config at the ingress level. Python from fastmcp import FastMCP from fastmcp.server.auth import BearerAuthProvider import httpx, os # All state lives in downstream systems. Zero server-side session state. mcp = FastMCP( "ticketing-mcp", auth=BearerAuthProvider( jwks_uri="https://auth.corp.example/.well-known/jwks.json", required_scopes=["mcp:ticketing:read"], ), ) @mcp.tool( description="Search tickets by JQL query. Read-only.", annotations={"readOnlyHint": True, "idempotentHint": True}, ) async def search_tickets(query: str, max_results: int = 20) -> list[dict]: # Auth context injected per-request by the BearerAuthProvider. # No session object. No global state. Safe for any pod to handle. async with httpx.AsyncClient() as client: resp = await client.get( f"https://jira.corp.example/rest/api/3/search", params={"jql": query, "maxResults": max_results}, headers={"Authorization": f"Bearer {os.environ['JIRA_API_TOKEN']}"}, ) resp.raise_for_status() return resp.json()["issues"] Where MCP Actually Breaks I've been building on this protocol for over a year. Here's the honest list of failure modes nobody talks about until they've hit them. The SSE timeout one catches almost everyone. You configure Streamable HTTP, everything works in dev, you push to prod, and suddenly long-running tool calls are silently dying. The load balancer's idle connection timeout — usually 60 seconds — kills the SSE stream before your database export finishes. The fix is simple once you know it: push heartbeat notifications every 30 seconds, and bump your ingress idle timeout to at least 5 minutes. The discovery process is not simple. MCP vs. the Alternatives: An Honest Comparison The column that matters most in that table is the one everyone argues about least: LLM portability. Right now you might be locked into Claude or GPT-4o. Six months from now, there'll be a model from a lab you've never heard of that outperforms both on your specific task. If your tool integrations are written against a provider's function-call schema, you're rewriting them. If they're MCP servers, you're updating a client config file. The Migration Playbook: 3 Days to 11 Minutes Here's exactly how we did our migration. Not the sanitized version. The version that includes the detour through a broken approach we had to back out. Audit your existing tool integrations – catalog every function schema, every parsing layer, every auth mechanism. We found 14 distinct integrations in our codebase, 6 of which were duplicates with slightly different error handling. Don't migrate duplicates; kill them first.Pick FastMCP, not the raw SDK – We initially tried building directly against the TypeScript SDK for more control. That cost us a week. FastMCP's Python decorator model handles 90% of the scaffolding — schema generation, transport setup, error wrapping. Use it. You can always drop to the raw SDK for edge cases.Deploy your gateway first, before any servers – the gateway is where your auth, rate limiting, and audit logging live. Getting it right before servers come online means you're not retrofitting security. We used a simple FastAPI proxy with httpx for upstream calls. Took two days. Worth every hour.Migrate one server per sprint, not all at once — We tried a big-bang migration on our second attempt. It failed. One server per sprint gives you a working fallback and real production data on how each integration behaves under MCP before you cut over.Instrument tool calls from day one – every tools/call should emit a structured log with: tool name, calling agent, token scope used, response time, and whether it succeeded. That data will save you during the first production incident, which will happen. What's Coming — And What to Plan For MCP governance moved to the Linux Foundation's Agentic AI Foundation in December 2025. That matters because it de-risks the protocol against any single vendor's agenda. OpenAI adopted it in April 2025. Google DeepMind's Vertex AI came on board in March 2026. AWS Bedrock in November 2025. This is no longer Anthropic's protocol. It's infrastructure. The 2026 roadmap has four working-group priorities worth knowing: MCP Server Cards – Machine-readable server manifests at a /.well-known/mcp.json endpoint. Think package.json for your MCP server: capabilities, auth requirements, tool list, rate limits. Enables automatic gateway discovery and policy enforcement without configuration drift.Stateless transport formalization – The current stateless mode is in spec but not yet standardized in behavior across SDKs. The Q2 2026 working group is closing those gaps. Wait for this before going all-in on multi-cloud stateless deployments.A2A protocol integration – Google's Agent-to-Agent protocol handles horizontal agent coordination. MCP handles vertical tool connection. The integration point is where agents hand off tasks to subagents that themselves use MCP. Plan for this architecture now, even if you don't need it yet.Audit extensions – Structured compliance fields for tool calls: user context, data classification, retention tags. If you're building in a regulated industry, this will make your compliance team significantly less anxious. Targeting Q3 2026.

By Dinesh Elumalai CORE

When a mobile application starts an agentic workflow that may run for minutes, the network connection is almost guaranteed to be shorter-lived than the computation. Wi-Fi can disappear, cellular routing can change, the device can lock, and iOS can suspend the process after it moves to the background. Apple explicitly states that backgrounded apps are suspended by default, while background URLSession exists for transfers that must continue when the app is inactive. That distinction matters as a persistent SSE or WebSocket connection can improve foreground responsiveness, but it should not define whether a long-running LangGraph workflow is alive. The reliable design makes workflow execution a backend concern and treats the iOS connection as a detachable observation channel. The Mobile App Cannot Own the Run A fragile implementation ties an HTTP request, an agent run, and a UI progress indicator into one lifecycle. The client sends work, the server begins executing it, and the response remains open until completion. Once the TCP connection disappears, every layer has an ambiguous question: did the workflow fail, did only the stream fail, or did the client simply stop listening? Retrying the original request can be even worse because a second run may be created while the first is still executing. The safer contract separates submission from observation. A start request should create or identify a durable workflow and return a stable handle immediately, commonly with HTTP 202 Accepted, whose semantics explicitly support accepted-but-incomplete processing. The handle can contain an application workflow ID plus the LangGraph thread_id and, where useful, a run ID. An idempotency key prevents the same mobile action from creating duplicate backend work after a timeout. A compact client-side start path can persist that handle before opening any live stream: Swift request.setValue(commandId, forHTTPHeaderField: "Idempotency-Key") let (data, _) = try await URLSession.shared.data(for: request) let handle = try decoder.decode(WorkflowHandle.self, from: data) try workflowStore.save(handle) The important behavior is the ordering. Once the backend accepts the command, the workflow identity is stored locally. A later socket failure loses only live updates, not the ability to locate the computation. For large file inputs, a background URLSession can separately handle upload continuity. Apple documents background downloads and uploads as system-managed transfers that can outlive application suspension, and resumable transfer support can recover from network interruption without restarting all bytes. Durable State Has to Survive the Socket LangGraph fits this model because persistence is built around threads and checkpoints rather than a single uninterrupted request. A checkpointer stores thread-scoped graph state, allowing execution to recover after failure or interruption, and production deployments can use a database-backed checkpointer instead of in-memory state. The thread_id is the stable pointer used to load that state. Python config = {"configurable": {"thread_id": workflow_id} graph.invoke( {"workflow_id": workflow_id, "input": payload}, config=config, ) Checkpointing does not make arbitrary side effects exactly-once. LangGraph saves Graph API state at super-step boundaries, and a node that is interrupted or retried can execute again from the beginning. Current LangGraph guidance explicitly recommends idempotency keys, upserts, or read-before-write checks for effects that can repeat. The Functional API similarly warns that a task that started but did not finish may run again after resume. That behavior should shape node design. External writes belong behind stable operation keys, ideally derived from the workflow and logical step: Python @task def persist_result(workflow_id, result): return results.upsert( key=f"{workflow_id}:final-result", value=result, ) Checkpoints provide durable progress through the graph as idempotency protects the systems touched by that graph. Reliable recovery requires both. Events Make Reconnection Deterministic A durable workflow still needs a durable way to describe progress. Directly forwarding LangGraph tokens or node updates to an iPhone is useful for immediacy, but transient transport data should not be the only record of business-visible state. An event-driven backend can translate meaningful transitions such as accepted, planning, tool_completed, awaiting_approval, completed, and failed into durable events carrying a workflow ID, monotonically increasing sequence, and unique event ID. The transactional outbox pattern is well suited to this boundary. Debezium documents the pattern as a way to avoid inconsistency between database state and events consumed by other services; application state and the outbox record are written together, then change-data capture publishes the event asynchronously. Debezium also describes propagation as at-least-once, which makes consumer deduplication part of the design rather than an optional optimization. A backend transition can keep the state change and event creation in one transaction: Python with db.transaction() as tx: tx.execute( "UPDATE workflows SET status = %s, version = version + 1 WHERE id = %s", ("completed", workflow_id), ) tx.execute( "INSERT INTO outbox(event_id, workflow_id, type, payload) VALUES (%s, %s, %s, %s)", (event_id, workflow_id, "workflow.completed", payload), ) CloudEvents can provide a standard envelope when events cross service boundaries. Its specification defines interoperable event metadata, while the CloudEvents primer states that an event id is unique within an event source. Those semantics map naturally to deduplication keys in downstream consumers. LangGraph’s own deployment streaming already demonstrates the same recovery idea. The streaming API supports reconnection using the last event ID, while the newer event-streaming API assigns sequence values and durable event IDs, replays buffered events after reconnect, and deduplicates replays client-side. The newer API also documents an important limit as its per-run replay buffer is bounded, so early events from a very long run can be evicted. Recovery Must Handle Duplicates and Gaps That bounded buffer is why an application-level status model should exist beside the live LangGraph stream. A reconnecting iOS client should first obtain the authoritative workflow snapshot, including status, output references, and the latest committed sequence. It can then request events after the locally stored cursor. If old events are no longer available, the snapshot repairs the gap as if events are replayed; event IDs or sequence numbers suppress duplicates. This design extends LangGraph’s resumable-stream model with durable application state rather than assuming an in-memory or bounded event buffer is a permanent event log. The foreground recovery path can remain small: Swift let snapshot = try await api.workflow(id: handle.id) apply(snapshot) for try await event in api.events(id: handle.id, after: handle.sequence) { guard event.sequence > workflowStore.sequence(handle.id) else { continue } apply(event) try workflowStore.save(sequence: event.sequence, for: handle.id) } Cursor persistence should occur only after an event has been applied successfully. That ordering converts a crash between receipt and rendering into a harmless replay rather than a silent gap. The same principle applies on the backend, as command acceptance should follow durable recording of the workflow identity and command so that a successful acknowledgment does not refer to state that disappears after a process failure. The outbox pattern applies the same atomicity principle to workflow state and outbound events. iOS background facilities remain useful, but they should complement this protocol rather than replace it. Apple’s Background Tasks framework can grant background runtime, yet Apple also emphasizes that backgrounded applications normally receive no CPU time. Background URLSession is appropriate for long-running network transfers, not as a guarantee that an arbitrary agent event stream remains continuously connected. The Backend Becomes the Workflow Boundary The strongest recovery design changes the meaning of “connection lost.” It no longer means “workflow state unknown.” It means only “the current observer is detached.” LangGraph checkpoints preserve graph progress, idempotent tasks protect external side effects, a transactional outbox makes business transitions publishable without a dual-write race, and durable event identities make replay safe. The iOS application keeps only the minimum recovery coordinates, workflow identity, and the last applied cursor. These responsibilities align with LangGraph’s checkpoint-based persistence model, its replay and idempotency requirements, and event-streaming support for reconnectable consumers.

By Uthej Mopathi CORE

Modern enterprise applications are increasingly running AI inference on-device rather than sending data to a central cloud. Improvements in hardware and model optimization have shifted the balance of compute. As one analysis notes, advances in 5G and edge hardware have made edge AI “a crucial technology for enabling intelligent applications.” Gartner predicts that by 2025 roughly 75% of enterprise data will originate at the edge rather than in traditional data centers. This data gravity, combined with emerging requirements for real-time response, privacy, and resilience, is driving inference tasks out of the cloud. Edge AI Reduces Latency, Bandwidth Costs, and Privacy Risks Running inference at the edge avoids the latency and network costs of cloud round-trips. For latency-critical use cases such as self-driving cars or augmented reality, even a few hundred milliseconds of delay is unacceptable. By processing sensor data locally, an edge device can make sub-10ms decisions for safety and interactivity. Similarly, streaming raw video or IoT sensor feeds to the cloud would incur massive bandwidth use and egress charges. Performing these analytics on-device eliminates that overhead. Local inference also enhances privacy and compliance as sensitive data (for example, images from a security camera or health readings from a wearable) can be analyzed on-premises without ever leaving the device. This helps meet data-sovereignty regulations and avoids exposing private information in transit. Finally, edge models continue to function even during network outages. When connectivity is lost, a device can still operate autonomously, maintaining “offline functionality” and zero downtime for critical tasks. Cloud Training and Edge Inference Enable Real-Time AI These factors have created a clear two-stage AI strategy for many organizations: train large models in the cloud (where vast compute and data are available) but run inference on the edge for real-time and private workloads. For example, one industry advisor observes that companies are already using clouds to develop and train models while then “optimizing, compressing, and deploying [them] to the edge for real-world application. This ensures sub-second decision-making, minimal data transfer, and continuous operation right where the value is delivered.” In practice, this can look like periodically syncing updated models from the cloud to a fleet of edge servers or devices, while daily operation happens locally. In fields like manufacturing, retail, or finance, this hybrid approach yields measurable ROI by applying AI where it matters most. AI Accelerators and Model Compression Make Edge Inference Practical Key technological advances have enabled this shift. On the hardware side, specialized AI accelerators have become commonplace in edge platforms. Mobile SoCs now include NPUs and DSPs for neural networks, while devices like NVIDIA Jetson or Google’s Coral Edge TPU provide GPU-like acceleration for embedded systems. A Qualcomm white paper notes that its chips combine CPUs, GPUs, and “neural processing units” specifically for edge AI, along with optimized frameworks and SDKs to deploy models on devices. These accelerators can execute inference algorithms much faster and more energy-efficiently than a general-purpose CPU. At the same time, neural network architectures have become more compact and efficient. Techniques such as model distillation, quantization, and pruning let developers shrink large models dramatically with little loss in accuracy. In practice, this means today’s “state-of-the-art smaller AI models” can outperform larger models from the cloud era while fitting on a phone or embedded board. For example, Qualcomm reports that many recent large generative models have been distilled down to versions under 100 billion parameters, yet still achieve performance comparable to much bigger models. Quantization (converting weights to 8-bit or 16-bit) and sparse pruning are now routine tools to reduce model size and latency. One survey explains that quantization “lowers power consumption and speeds up operations without significantly sacrificing accuracy, while pruning eliminates unnecessary parameters.” At the software level, lightweight inference frameworks and runtime libraries make deployment easier. TensorFlow Lite, PyTorch Mobile, ONNX Runtime, Intel OpenVINO, and similar toolkits offer optimized kernels for ARM processors, GPUs, and AI accelerators. These frameworks often include mobile-friendly model converters and delegate support for hardware acceleration. For example, TensorFlow Lite lets developers convert a trained TensorFlow model to a flatbuffer, then load it into a mobile app. On-device inference might look like: Java Interpreter interpreter = new Interpreter(modelBuffer); float[][] output = new float[1][NUM_CLASSES]; interpreter.run(inputData, output); This code snippet instantiates a TensorFlow Lite Interpreter with a pre-optimized model buffer and runs it on inputData, producing classification scores in output. Similarly, PyTorch Mobile can serialize a TorchScript model for Android/iOS, and ONNX Runtime can execute models across many hardware targets with reported speedups (Microsoft cites up to 17× faster inference). These mobile runtimes also leverage hardware delegates (GPU or NPU) under the hood when available. Edge AI Requires Careful Resource, Deployment, and Security Management Deploying and managing inference at the edge does introduce new engineering challenges. Resource constraints mean that models must be smaller and less complex than cloud counterparts, as even with quantization, a model that fits on a GPU server might need further compression for a microcontroller. Edge devices have limited memory and power budgets, so operators must balance accuracy against size and speed. The network of devices also requires orchestration, as software like Kubernetes (via lightweight distributions) or IoT platforms (AWS IoT Greengrass, Azure IoT Edge) can roll out updates and monitor health across fleets. For example, an edge deployment might containerize a TensorRT-based inference service and schedule it on Jetson nodes with GPU support, while sending telemetry to a central dashboard. Observability is critical as enterprises often integrate Prometheus/Grafana or cloud IoT logging to capture inference metrics and detect when models drift or hardware issues arise. Security must also be considered, as physical devices at the edge can be vulnerable, so measures like secure boot and authenticated OTA updates are important. Despite these complexities, many enterprises have already benefited. In retail, shops are using in-store edge cameras to detect incidents in real time without sending video to the cloud, saving bandwidth and complying with privacy rules. Industrial plants run anomaly-detection models on local PLCs to spot equipment faults instantly, ensuring operations can continue even if connectivity fails. Financial firms can do fraud checks on transaction terminals with millisecond latency. In all these cases, doing inference on-site is far cheaper and faster than piping every input to a data center. Conclusion In summary, ongoing trends in hardware, model design, and infrastructure are moving inference out of centralized clouds. Edge AI brings compute to the data, cutting latency and cost while meeting privacy requirements. That is not to say cloud AI is obsolete, as it remains essential for training, heavy analytics, and coordination, but the future of inference is local. By 2026, enterprises will likely adopt a hybrid model with cloud resources for development and big data tasks, with optimized models deployed to edge devices for live prediction. This shift requires new patterns of system design and monitoring, but it unlocks real-time intelligence and efficiency that cloud-only architectures can no longer match.

By Uthej Mopathi CORE

Every major AI vendor now supports the Model Context Protocol. The framing is almost always the same: MCP is the universal connector for AI agents in the enterprise. That framing sets up a false choice. MCP, REST/HTTP APIs, and Apache Kafka are not alternatives. They solve different problems at different layers of the architecture. Treating them as competing options produces systems that are fragile exactly where they need to be reliable. These three technologies can and do coexist in the same architecture. The question is not which one to pick. It is which one belongs where, and what the tradeoffs are when more than one could technically do the job. This article maps that decision: what each technology is built for, where the boundaries are, and where the genuine gray areas lie. 1. What Is MCP and What Is It Built For? Anthropic introduced the Model Context Protocol in November 2024 as an open standard for connecting AI assistants to external tools and data sources. Before MCP, every AI model required a custom connector to each external system. Three models, ten systems: thirty custom integrations to build and maintain. MCP collapses that to one standard interface. Any compliant client talks to any compliant server without prior coordination. OpenAI adopted MCP in March 2025. Google DeepMind confirmed support in April 2025. By December 2025, MCP had reached over 97 million monthly SDK downloads across Python, TypeScript, Java, Kotlin, C#, and Swift. Anthropic donated the protocol to the Agentic AI Foundation under the Linux Foundation, with AWS, Google, Microsoft, Bloomberg, and OpenAI as platinum members. MCP is no longer a developer experiment. Signals of enterprise maturity are arriving quickly: AI agents paying for API access autonomously, cross-SDK interoperability between Anthropic and OpenAI converging on MCP Resources, composable enterprise workflows where agents read tool signatures and compose cross-system flows without predefined paths, and an official MCP Registry launched in late 2025 as the community-driven server directory. The 2026 roadmap focuses on scalable transport, agent-to-agent communication, governance maturation, and enterprise readiness covering audit trails and SSO-integrated authentication. MCP handles tool access: how an agent calls an external capability. It does not handle agent-to-agent coordination, which is the domain of protocols like Google's Agent-to-Agent (A2A). MCP and A2A are complementary and address different layers of agentic architecture. The moment MCP is asked to do more than tool access, the architecture starts to break. Security Maturity Is Still Catching Up With Adoption Most incidents disclosed in 2025 and early 2026 are implementation failures, not protocol flaws. An Endor Labs analysis of 2,614 MCP implementations found 82% use file system operations prone to path traversal and 67% use APIs related to code injection. Enterprise-grade authentication with OAuth 2.1 and SAML/OIDC is on the 2026 roadmap but still in progress. The practical controls for today: apply least privilege, limit MCP server access to only the systems and data each tool requires, and monitor tool definitions for unexpected changes. 2. MCP vs. REST/HTTP API MCP and REST/HTTP APIs serve different consumers and should not be treated as interchangeable. REST is an architectural style built on HTTP, widely adopted but with no fixed conventions for discovery, error formats, or method naming. Well-designed REST APIs backed by OpenAPI specifications work well for direct, programmatic data access when a native SDK or versioned API already exists and teams know how to operate it. MCP enforces consistency at the interface level because the consumer is an AI model that cannot tolerate creative API interpretation. MCP standardizes how a tool is called. It does not standardize what the tool returns, how fresh that data is, or whether two agents calling the same tool simultaneously see the same state. For direct data access to vector stores, databases, or business application APIs, a well-governed REST API, native SDK, or Kafka Connect integration is almost always the better choice: lower latency, no protocol overhead, mature tooling. For giving AI agents standardized, discoverable access to a broader set of tools across vendors and frameworks, MCP is the right layer. The two are complementary, not competing. Tool Design Matters as Much as the Protocol Choice One important nuance on tool design: mapping one-to-one from existing APIs to MCP tools rarely works well. What matters is tool granularity, smart metadata, and thoughtful assembly of the MCP layer. An MCP server that exposes well-structured, semantically rich tools lets an AI agent reason about capabilities and compose workflows. This is reminiscent of the composability questions from the enterprise SOA (Service-oriented Architecture) era. SOA promised flexible service composition but delivered integration chaos when governance, metadata quality, and service granularity were treated as afterthoughts. MCP faces the same risk. The protocol is sound; what determines success is the discipline applied to how tools are defined, documented, and assembled. What MCP Does Not Do What MCP does not do matters as much as what it does. It does not manage data, guarantee message delivery, enforce governance, or guarantee consistency across systems. It is an interface layer, not a data pipeline. That boundary becomes even clearer when looking at what Kafka does, which is structurally different from both MCP and REST. 3. Apache Kafka: Event Broker, Decoupling, and the Backbone Role Operational data is the live data that runs business processes: order states, inventory levels, transaction records, customer accounts, risk scores. It originates in systems like SAP, Salesforce, Oracle, and mainframes, and it changes continuously. Kafka is architecturally different from both HTTP and MCP in one way that matters most: it decouples producers and consumers through a persistent, ordered, append-only log. With HTTP or MCP, the caller and the callee are coupled at request time. Every integration is point-to-point. If the target system is slow or unavailable, the caller is directly affected. Kafka breaks that coupling entirely. A producer writes an event once. Any number of consumers read it independently, at their own pace, using their own communication paradigm. One consumer processes records in real time. Another runs nightly batch analytics over the same events. A third powers a stream processing pipeline. A fourth writes results to a data lake via Apache Iceberg. All of them consume the same underlying data product. None of them affects the others. Kafka supports three consumption patterns from a single event stream: streaming, request-response, and batch. The event exists once; each consumer is independent. This is the pub/sub event broker model, and it is what makes Kafka the integration backbone between operational and analytical systems. The diagram below shows this decoupling: a single Kafka topic serving real-time applications, HTTP-based consumers, batch analytics, and MCP agent interfaces simultaneously. Stream Processing With Kafka Streams and Apache Flink Stream processing is a core complement to Apache Kafka, extending the platform from event transport into real-time data processing and decisioning. Kafka Streams is a lightweight Java library embedded in applications. It is well-suited for streaming ETL and simple to medium stateful stream processing without requiring a separate cluster. It integrates closely with existing JVM-based services. Apache Flink is a distributed stream processing engine designed for more complex workloads. It supports Java, Python, and SQL APIs, making it accessible to both application developers and data engineers. Flink runs as a dedicated cluster or in managed environments and is built for high-scale scenarios such as multi-stream joins, event-time processing, large state management, exactly-once semantics, Complex Event Processing (CEP), real-time analytics, and AI model inference. Both approaches extend Kafka with processing capabilities. The choice depends on workload complexity, required deployment model, and preferred programming language, not on replacing Kafka’s role as the event streaming backbone. A detailed comparison is available in the post Apache Kafka and Apache Flink: A Match Made in Heaven. Operational and Analytical Integration, Including the Data Lakehouse Kafka is not only for operational data integration. It serves as the ingestion layer into data lakes, feeds real-time analytical pipelines, enables stream processing with embedded AI models, and connects business applications bidirectionally. A governed data streaming platform provides schema registry, lineage tracking, role-based access control, and exactly-once delivery semantics across all of that. It serves both operational and analytical use cases and acts as the bridge between those two worlds. For how streaming and the lakehouse converge via Apache Iceberg, see Data Streaming Meets Lakehouse. Kafka's append-only commit log is the foundation of data consistency across the enterprise. Every downstream consumer sees the same data in the same order. That is not just a performance feature. It is what prevents the architecture where every system has its own version of the truth. 4. The Tradeoffs: It Is Not Black and White The choice between MCP, REST/HTTP APIs, and Kafka is rarely clean. All three can play a role in the same architecture. REST/HTTP APIs work well for operational data access when volume is moderate and a well-governed API already exists. A REST API backed by a Kafka-derived serving layer can return consistent, current data. The API is the interface; the streaming platform is what makes the data trustworthy behind it. A financial services firm exposing account balances via REST is not doing it wrong, as long as those balances are derived from a governed, consistent data source rather than pulled directly from a source system on every request. Kafka becomes the clear choice when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when the same events need to feed operational applications, analytical pipelines, and AI agents simultaneously. MCP fits best when access is supplementary, loosely coupled, and low-frequency. A support agent looking up a ServiceNow ticket before drafting a response, or a sales assistant pulling the latest slide deck from Google Drive before a call, are good fits. The key test is simple: does it matter if the data the agent receives is a few seconds or minutes old? If yes, MCP should not own that responsibility. If no, MCP is the right interface. SAP: Clean Separation Between ERP Integration and Developer Tooling The boundary between MCP and REST is not a choice between two equivalent options for the same integration. SAP is the clearest example of a clean separation. SAP exposes extensive REST and OData APIs for ERP integration: order management, finance, supply chain, procurement, and HR data flowing bidirectionally between SAP and other enterprise systems. SAP's MCP servers serve an entirely different purpose: developer tooling for ABAP code generation, CAP application development, UI5 and Fiori assistance, and operational tasks like transport validation and incident management. An architect connecting SAP order events to downstream systems uses OData and Kafka Connect. A developer asking an AI coding assistant to generate ABAP code uses the SAP MCP server. Different consumers, different use cases, different data. No overlap. Salesforce and ServiceNow: Same Data, Different Consumer Salesforce and ServiceNow follow a different pattern. Their MCP servers wrap the same underlying REST APIs and expose the same underlying data, but for a different consumer. A developer-written integration calls the Salesforce REST API directly with known endpoints and hardcoded logic. An AI agent calls the Salesforce MCP server, which wraps that same API to make it discoverable and stateful for an agent that cannot read documentation or manage its own session state. The data is identical. The access path differs based on who is consuming it. This is not a free choice between equivalent options. It is the same system serving two different client types through two different interface layers. REST vs. Kafka for Operational Data: The Harder Call The harder boundary is between REST and Kafka for operational data. Both can technically serve it, and that is where the real architectural decision lies. REST is simpler to start with but introduces point-to-point coupling, integration spaghetti at scale, and consistency risks when the same data needs to reach multiple consumers. Kafka is more complex to operate but provides the decoupling, consistency, and governance that enterprise architectures require when the same data needs to reach many consumers reliably. The two are not mutually exclusive. A common and well-proven pattern combines both: Kafka handles the event backbone, decoupling, and consistency, while a REST layer sits on top for synchronous request-response access, API management integration, or compatibility with systems that cannot speak the native Kafka protocol. This is particularly common in mobile applications, legacy system integration, and API gateway architectures. For a detailed look at how REST and Kafka complement each other in practice, see Request-Response with REST/HTTP vs. Data Streaming with Apache Kafka. 5. Decision Framework: MCP, REST/HTTP, or Kafka? Choosing between MCP, REST/HTTP, and Kafka is not a single decision but a set of tradeoffs that depend on data volume, consumer type, consistency requirements, and what is already in production. The comparison table below makes those tradeoffs concrete across eight dimensions. When to Use Which: A Guide to the Decision Tree The decision tree below walks through the same logic as a series of questions, routing to the right choice based on the integration's actual requirements. Use MCP when the integration is supplementary and tool-like: Slack, Google Drive, ServiceNow tickets, internal knowledge bases. The agent needs context to act, not a stream of events to react to. Eventual consistency is acceptable. Apply least privilege, monitor tool definitions for changes, and isolate MCP servers from production systems. Use a REST/HTTP API or native SDK when a well-documented API or SDK already exists and the engineering team knows how to operate it. The access pattern is direct, moderate-volume, and latency-sensitive. REST is also a reasonable choice for operational data when the backend is a governed Kafka-derived serving layer and consistency properties are inherited, not assumed. Use Apache Kafka when data is high-volume or high-velocity, when multiple consumers need the same events, when ordering and exactly-once delivery matter, or when governance, lineage, and auditability are non-negotiable. Kafka is also the right choice when the same data needs to feed operational applications, real-time analytics, data lakes, and AI agents simultaneously. Use the real-time context engine when an AI agent needs current, consistent operational context for autonomous decisions. Kafka and Flink govern the data. MCP provides the agent interface. The consistency guarantee comes from the streaming layer, not from MCP. The practical question is not which protocol to choose. It is whether the data architecture underneath the agents can be trusted. Agents making autonomous decisions about inventory, risk, or customer service are only as reliable as the data they act on. 6. Where MCP and Kafka Work Together: The Real-Time Context Engine There is one pattern where MCP and data streaming complement each other directly: the real-time context engine. Kafka and Flink process and govern the data: ingesting from operational systems, applying transformations and filters, producing real-time materialized views. Those views are then exposed to AI agents through a standardized MCP interface. The streaming platform owns the data, its freshness, and its consistency guarantees. MCP owns the interface to the agent. Neither layer bleeds into the other's responsibility. Data consistency is not delegated to MCP. The streaming platform enforces it upstream before the MCP interface comes into play. The agent calls a tool and receives context that is current, governed, and consistent, not because MCP guarantees it, but because the streaming platform does. Any compliant AI agent, whether Claude, ChatGPT, Amazon Bedrock, LlamaIndex, or CrewAI, can call the context engine and receive current context from operational systems without needing to understand Kafka topics, Flink jobs, or schema evolution. An agent routing shipments from yesterday's inventory, approving transactions against a risk score from three hours ago, or reading an account balance that has not propagated: none of these is reliable. A real-time context engine eliminates this class of error at the source, reduces hallucinations, lowers inference cost, and anchors decisions to current operational reality. From Data Freshness to Agent Governance Enterprise readiness for this pattern also depends on how agents are governed once deployed. Trust, control, and accountability become central once agents start chaining decisions across domains. The context engine is the data layer of that answer. Governance of the agents themselves, covering what they are permitted to do, under what conditions, and with what audit trail, is the other half. This is the dimension enterprise buyers are actively evaluating when selecting agent orchestration platforms. The diagram below shows how the three layers fit together: the streaming platform as the data backbone, the context engine as the governed serving layer, and MCP as the clean interface to agents. 7. Conclusion: One Protocol, One Job MCP has earned its place in the enterprise architecture stack. What it has not yet earned is the role of universal integration layer, and understanding that distinction is what this article has been about. The broader architecture this sits inside connects three interdependent pillars. Event-driven data integration, with Kafka as the backbone, moves data reliably between operational and analytical systems and delivers governed data products to every consumer. Process intelligence is the orchestration layer that determines which decisions to automate, in what sequence, and under what conditions, giving agentic workflows the structure and governance they need to be trustworthy. Trusted agentic AI is where MCP plays its role: the standardized, governed interface through which agents access external tools and context, anchored to real data by the streaming layer beneath it. For a vendor-by-vendor analysis of trust and lock-in across the major AI platforms, see the Enterprise Agentic AI Landscape 2026. For a deeper look at how the three pillars fit together as an enterprise architecture framework, see The Trinity of Modern Data Architecture: Process Intelligence, Event-Driven Integration, and Trusted Agentic AI. One protocol, one job. That is the right way to use MCP.

By Kai Wähner CORE

AI agents become useful when they can do more than generate text. The moment an agent can update a CRM, approve a refund, create a purchase order, change a price, or send a customer response, the architecture must answer a harder question: Which actions should the agent execute automatically, and which should require human approval? That decision sits at the center of production-ready enterprise AI agent architecture. Too little oversight creates operational and compliance risk. Too much oversight turns the system into another approval queue. A well-designed human-in-the-loop system does not place a person behind every action. It uses risk-based approval gates, role-based permissions, auditability, and reversible execution to give AI agents useful autonomy without giving them uncontrolled authority. Full Autonomy Should Not Be the Default Many AI workflow automation projects begin with a simple assumption: if the agent can complete the task, it should be allowed to execute it. That assumption works poorly in enterprise environments. An agent may correctly understand a request but still act on incomplete data, use outdated policy, select the wrong customer record, or apply a technically valid action in the wrong business context. The risk is not limited to hallucination. Production systems also fail because of: Incorrect source dataAmbiguous instructionsPermission errorsDuplicate eventsStale workflow stateIntegration timeoutsDownstream system failures The right goal is therefore not maximum autonomy. It is bounded autonomy: the agent can act independently within predefined limits and escalate when those limits are crossed. Classify Actions by Risk Before designing an AI agent approval workflow, classify the actions the agent may perform. A practical model uses three levels. Low-Risk Actions These are easy to verify and easy to reverse. Examples include: Drafting an emailSummarizing a support ticketCategorizing a documentPreparing a CRM updateGenerating a reportSuggesting the next workflow step These actions can often run automatically, especially when the output remains internal or requires a later human action. Medium-Risk Actions These affect business records or external communication but remain recoverable. Examples include: Updating a CRM fieldScheduling a meetingSending a standard follow-upCreating a draft invoiceAssigning a support ticketUpdating an order status These actions may be automated when confidence is high, and policy conditions are satisfied. Otherwise, they should enter a review queue. High-Risk Actions These create financial, legal, compliance, security, or customer-impacting consequences. Examples include: Issuing a refundApproving a paymentChanging contract termsModifying production accessDeleting recordsChanging pricingSending regulated communications These should require explicit approval unless the organization has defined narrow, well-tested exceptions. The important point is that risk should be assigned to the action, not the model. A highly capable model should not automatically receive broader permissions. Put Approval Gates Before Irreversible Actions An approval gate should sit immediately before the step that creates external or irreversible impact. A common mistake is placing review too early. For example, asking a human to approve the agent’s plan before it has gathered data, validated records, or prepared the final action creates unnecessary work. A better sequence is: Receive the request.Gather relevant data.Validate identity, permissions, and workflow state.Generate the proposed action.Evaluate policy and risk.Request approval when required.Execute.Verify the result.Write to the audit log. This allows the agent to complete the preparation work while reserving human attention for the final decision. The approval screen should show more than a yes-or-no prompt. It should include: The proposed actionThe reason for the actionThe source data usedThe expected impactThe agent’s confidenceRelevant policy checksAvailable alternatives A reviewer should not need to reconstruct the agent’s reasoning from several systems. Use Policy-Based Approval, Not Confidence Alone Confidence scores can be useful, but they should not control approval decisions by themselves. A more reliable approval policy combines several signals: Action typeTransaction valueCustomer or account sensitivityConfidence thresholdData completenessPolicy exceptionsUnusual activityModel or tool failure history For example: Python def requires_approval(action): if action.type in HIGH_RISK_ACTIONS: return True if action.amount > action.auto_approval_limit: return True if action.confidence < 0.90: return True if not action.policy_checks_passed: return True if action.has_unusual_context: return True return False This is intentionally simple. In a production system, the policy engine should remain separate from the language model so that approval rules are deterministic, testable, and version-controlled. The model may recommend an action. The policy layer decides whether the system is allowed to perform it. Apply Role-Based Access Control An AI agent should not have one universal identity with access to every system. Secure AI workflow automation requires least-privilege access. Each agent or workflow should receive only the permissions required for its task. A finance agent may be allowed to prepare invoices but not release payments. A support agent may update ticket status but not alter customer contracts. A procurement agent may create a purchase request but not approve it. Human reviewers also need role-based permissions. An approval is meaningful only when the reviewer has authority over the action. Every approval event should record: Who approved or rejected itThe role usedThe action reviewedThe original proposalAny modificationsThe execution resultThe timestampThe policy version This creates AI agent audit logs that are useful for debugging, compliance reviews, and process improvement. Make Actions Reversible Approval gates reduce risk, but they do not eliminate errors. Where possible, design agent actions as reversible operations. Instead of immediately deleting a record, move it into a recoverable state. Instead of overwriting a value, preserve the previous version. Instead of sending a message without review, allow a delay window for cancellation. Useful patterns include: Soft deletionVersioned recordsCompensating transactionsDelayed executionIdempotency keysStaged updatesRollback workflows Reversibility is one of the most practical AI agent guardrails because it limits the damage from both model errors and system failures. Avoid Creating an Approval Bottleneck A badly designed human-in-the-loop system can be safe but unusable. If every action requires approval, reviewers become overloaded, response times increase, and users begin approving requests without proper inspection. The system should learn operationally, even if the model itself is not retrained. Track: Approval rate by action typeRejection reasonsAverage review timeCommon reviewer editsRepeated low-risk approvalsFalse escalationsIncidents after automatic execution If a category of actions is repeatedly approved without modification, it may be suitable for controlled automation. If a supposedly low-risk action is frequently corrected, its approval policy should become stricter. The goal is to move from broad manual oversight to targeted oversight based on evidence. A Practical Reference Architecture A production-ready design usually includes these components: Agent runtime: Interprets the request and prepares the actionTool layer: Connects the agent to enterprise systemsPolicy engine: Evaluates permissions, risk, and approval rulesApproval service: Presents the proposed action to an authorized reviewerExecution service: Performs approved actions using controlled credentialsAudit store: Records proposals, approvals, tool calls, and resultsMonitoring layer: Detects failures, unusual activity, and policy violations Separating these responsibilities prevents the language model from becoming the policy engine, identity provider, executor, and audit system at the same time. Final Takeaway Human-in-the-loop AI agents should not be designed as autonomous systems with an approval button added later. Approval, permissions, auditability, and reversibility must be part of the architecture from the beginning. The strongest enterprise systems do not ask humans to supervise every step. They automate low-risk work, escalate uncertain or sensitive actions, and preserve clear accountability for every decision. That is what makes an AI agent operationally useful: not unlimited autonomy, but the ability to act safely within well-defined boundaries.

A detector I built was scoring 0.067 recall on temporal errors, meaning it caught about one in fifteen of the wrong dates it was supposed to find. Wrong dates are supposed to be the easy category: extract the years from the claim, extract the years from the source, compare. There is no semantics to get wrong. I assumed the extraction was broken and went looking for the bug. The extraction was fine. The benchmark was the problem, and not in a way that showed up anywhere in the code. The contexts had been written in the wrong voice. That's the part worth passing on, and it has nothing to do with hallucination detection. It applies to anyone who builds a synthetic evaluation set, which by now is most of us. A Detector That Failed Because of Prose Style The setup: a claim, a source context, and a question about whether the claim is supported. The detector is part of HallucinoType, an open-source package I maintain, and its benchmark was built the way most synthetic evaluation sets are built. Take a faithful claim, inject an error of a known type, keep the label. Two hundred fifty pairs, stratified across eight failure categories, thirty-five of them faithful so I could measure false positives. When I wrote the contexts for the date items, I wrote them the way a person naturally writes when they know the claim is wrong. Something like the treaty was signed in 1928, not 1938. Read that as a human, and it is unambiguous. Read it as a program that treats the context as a reference document, and the string 1938 is sitting right there in the source. The detector extracted it, matched it against the claim, found agreement, and passed the item. Every one of the thirty temporal items had this property. Seven of thirty numerical items did too. Rewriting the contexts as ordinary reference prose, the kind a retrieval system would hand you, moved temporal recall from 0.067 to 1.000. I changed no code. The numerical items did not move. They stayed at 0.600, which told me their misses had a different cause and saved me from congratulating myself on a fix that only worked once. The generalization is short enough to put on a sticky note. A context that argues with the claim is not the context a production pipeline supplies. I had unconsciously written my source documents in a fact-checking register, because I was thinking like an annotator rather than like a retrieval index. The register leaked the answer key into the input, and the system read it, exactly as it was built to. What makes this uncomfortable is that nothing about the benchmark looked wrong. The labels were correct, and the errors were real errors; any reviewer would have signed off on it. The defect lived in a stylistic property of the prose that no one thinks to specify, and it moved a headline number by a factor of fifteen. The Same Bug Wearing a Different Hat The same mistake showed up a second time, and I did not recognize it at first. A second detector in the same system checks whether a claim names the wrong person, company, or place. It works by extracting entities from the claim and looking for them in the context. If the entity appears in the context, the detector skips the claim, on the theory that the source confirms it. I upgraded the entity recognizer, the component that reads a sentence and tags which words are people, places, or organizations, from a regular-expression fallback to a proper statistical model. Recall fell from 0.200 to 0.067. A better component made the system worse, which is the kind of result that stops you mid-sprint. The recognizer was not at fault. The skip rule was. A source document can mention a person in a role that has nothing to do with the claim under evaluation, and still mention them truthfully. In one item, the claim misattributed who was second to walk on the Moon, and the context named the substituted astronaut correctly in a different sentence, doing a different thing. The weaker recognizer missed that mention and flagged the error. The stronger one found it, read it as confirmation, and waved the error through. Of twenty-seven items the detector wrongly skipped, twenty-six followed this pattern. The heuristic was never checking the right relationship. It asked whether the entity appears in the document when it needed to ask what the entity is doing in the sentence. Improving the model's ability to answer the wrong question just made it answer the wrong question more reliably. This is a hazard anywhere a rule sits on top of a learned component. Ablating downward, swapping in a deliberately weaker component to confirm the strong one is earning its cost, is something I do routinely. Ablating upward is rarer, and it tells you more: a rule that degrades when its inputs improve is a rule whose logic was wrong all along, and no amount of model quality saves it. A third instance, smaller but the same shape: a pattern for matching units of measurement was absorbing a trailing word, which let bare four-digit years slip past a filter meant to exclude them from numeric comparison. Fixing one regular expression moved numerical precision from 0.857 to 0.947. A lot of apparently semantic behavior turns out to be lexical. What the Headline Number Was Hiding None of these three defects were visible in the metric I would have reported at a demo. On the binary question of whether a claim is unsupported, the full system reached precision 0.991 and recall 0.986: almost nothing it flagged was fine, and almost nothing that was wrong got past it. Those are the numbers that go in an abstract. Averaged across the eight failure categories the system is supposed to distinguish, precision was 0.578 and recall 0.723. One category sat at 0.067 recall. Another fired on nearly everything, reaching 0.960 recall at 0.198 precision, meaning it claimed credit for errors that more specific detectors had already identified correctly. The binary number was not wrong. It was answering a question so coarse that every interesting failure averaged out of it. A system can be excellent at deciding that something is broken and close to useless at saying what broke. If the only number on your dashboard is the first one, you won't find out until the fine-grained output reaches someone who depends on it. None of this is new. It's the same argument as reporting per-class results instead of overall accuracy on an imbalanced dataset. Everyone agrees with it in principle and skips it anyway, because the aggregate is the number that makes the case for the work. Not Getting Fooled by Your Own Corpus Four practices came out of this, all cheap, none of them clever. Write your evaluation inputs in the register your production system receives. If your system reads retrieved documents, your test contexts should read like documents, not like annotations about documents. Voice is a feature your model can see, and the voice of someone who already knows the answer is a particularly dangerous one to hand it.Ablate upward, not just downward. Replace a component with a better one and check that every metric moves in the direction you expect. When something moves the wrong way, the rule sitting on top of that component is making an assumption you have not written down.Report per-stratum results next to the aggregate, always in the same table. Not in an appendix, not on request. If a category is at 0.067, that fact should be as easy to see as the number you are proud of.Hold out items you did not write. A corpus built by the same people who defined the categories will flatter the categories. Mine did. That is the single largest caveat on everything above, and no amount of internal rigor substitutes for a test set authored by someone else. The first one I would not have thought of before it cost me a day, and it's the one I now suspect is quietly wrong in a lot of synthetic eval sets. Injected-error benchmarks are easy to build, and their labels are correct by construction, which makes them feel safer than they are. The label being right does not mean the input is representative. The Register Your System Actually Speaks The failures worth writing up are rarely the ones where the model underperforms. They are the ones where the measuring apparatus was quietly reporting on something other than what you thought. A detector that scores 0.067 because the test data argues with itself is not a model problem. Neither is a rule that gets worse as its inputs get better, or an aggregate that averages away the only result that mattered. A bigger model fixes none of it. What fixes them is reviewing the evaluation harness as carefully as the thing it measures, defects and all. That's unglamorous work, and where most of my debugging time went. Probably where most of yours goes too.

Oracle Database 23ai introduced the powerful DBMS_DEVELOPER package, giving developers and database administrators a streamlined way to access database object metadata in JSON format. This feature represents a significant advancement in how we interact with database schemas, offering a more structured and programmatic way to extract and analyze metadata compared to traditional dictionary views or the older DBMS_METADATA package. In this article, we'll explore the capabilities of DBMS_DEVELOPER, focusing on its GET_METADATA function through detailed examples and practical implementation scenarios. Understanding DBMS_DEVELOPER The DBMS_DEVELOPER package was designed specifically for modern application development patterns, where JSON has become a universal data exchange format. Rather than returning metadata as DDL statements (like DBMS_METADATA), this package returns structured JSON documents that can be easily parsed, processed, and integrated into applications or DevOps workflows. Key Benefits Structured data format: Returns metadata as JSON objects that can be easily parsed Programmatic access: Perfect for integration with applications and automation scripts Versioning capabilities: Built-in ETag mechanism for tracking object changesConfigurable detail levels: Ability to retrieve basic, typical, or comprehensive metadata Setting Up Our Environment Let's set up a sample schema to demonstrate the package functionality: SQL CREATE TABLE customers ( customer_id NUMBER(10) CONSTRAINT pk_customers PRIMARY KEY, first_name VARCHAR2(50) NOT NULL, last_name VARCHAR2(50) NOT NULL, email VARCHAR2(100) CONSTRAINT uk_customer_email UNIQUE, join_date DATE DEFAULT SYSDATE, status VARCHAR2(10) DEFAULT 'ACTIVE' ); CREATE INDEX idx_customer_name ON customers(last_name, first_name); CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email FROM customers WHERE status = 'ACTIVE'; GET_METADATA Basics The core function of the DBMS_DEVELOPER package is GET_METADATA, which returns metadata about database objects in JSON format. Let's start with a basic example: SQL -- Using JSON_SERIALIZE for formatted output SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; The result is a structured JSON document containing comprehensive information about the table, including: Table name and schema Column definitions with data types and constraints Primary key, unique key, and foreign key information Index definitions An etag value representing the current state of the object This structured format makes it significantly easier to extract specific information programmatically compared to parsing DDL statements. NAME and SCHEMA Parameters The NAME and SCHEMA parameters work together to identify the specific database object. These parameters are case-sensitive and must match the object definition in the data dictionary. SQL -- Explicitly specifying schema SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'CUSTOMERS', schema => 'FINANCE') PRETTY) AS metadata; -- Using current schema (implicit) SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA(name => 'CUSTOMERS') PRETTY) AS metadata; When the SCHEMA parameter is omitted, the function uses the current schema. This behavior provides flexibility when working with objects across different schemas in your database environment. OBJECT_TYPE Parameter The OBJECT_TYPE parameter allows you to explicitly specify the type of object you're retrieving metadata for. While often optional (as the database can infer the object type from the name), it becomes necessary in cases where name resolution alone is insufficient. Currently, `DBMS_DEVELOPER` supports three object types: TABLEINDEXVIEW Let's examine metadata for our index and view: SQL -- Retrieving index metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', object_type => 'INDEX') PRETTY) AS metadata; -- Retrieving view metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', object_type => 'VIEW') PRETTY) AS metadata; The OBJECT_TYPE parameter becomes particularly important when dealing with objects that share the same name but have different types, such as packages and package bodies. LEVEL Parameter The LEVEL parameter controls the amount of detail included in the JSON output. Oracle provides three levels: BASIC: Minimal informationTYPICAL: Standard level of detail (default)ALL: Comprehensive metadata This flexibility lets you balance concise output with detailed information based on your needs. SQL -- Basic level metadata SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'BASIC') PRETTY) AS metadata; -- All details SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'IDX_CUSTOMER_NAME', level => 'ALL') PRETTY) AS metadata; The output at the ALL level includes additional attributes such as segment information, compression settings, and physical storage details that aren't present at the BASIC level. ETAG Parameter One of the most powerful features of DBMS_DEVELOPER is the etag mechanism, which provides version tracking for database objects. The etag value changes whenever the object definition changes, making it invaluable for change detection. SQL -- Store the current etag value DECLARE v_metadata CLOB; v_etag VARCHAR2(100); BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA(name => 'ACTIVE_CUSTOMERS'); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; DBMS_OUTPUT.PUT_LINE('Current etag: ' || v_etag); END; / -- Modify the view CREATE OR REPLACE VIEW active_customers AS SELECT customer_id, first_name, last_name, email, join_date FROM customers WHERE status = 'ACTIVE'; -- Check if the object has changed using the stored etag SELECT JSON_SERIALIZE( DBMS_DEVELOPER.GET_METADATA( name => 'ACTIVE_CUSTOMERS', etag => 'A1B2C3D4E5F6G7H8I9J0') -- Previous etag value PRETTY) AS metadata; When you pass an ETag value that matches the current state of the object, the function returns an empty JSON document {}. If the object has changed, it returns the complete metadata with a new ETag value. Practical Scenario: Database Migration and Documentation Let's consider a practical scenario where DBMS_DEVELOPER proves invaluable: a large-scale database migration project with continuous schema changes. The Challenge You're leading a project to migrate a critical application database from on-premises to Oracle Cloud. The development team continues to make schema changes during the migration process, and you need to: Document the current state of all database objectsTrack changes between migration wavesValidate that objects were created correctly in the target environmentGenerate comprehensive documentation for compliance requirements The Solution Using DBMS_DEVELOPER, you can create a robust metadata management system: SQL CREATE TABLE schema_versions ( object_name VARCHAR2(128), object_type VARCHAR2(30), object_schema VARCHAR2(128), capture_date TIMESTAMP, etag VARCHAR2(100), metadata CLOB ); -- Procedure to capture all tables in a schema CREATE OR REPLACE PROCEDURE capture_schema_metadata(p_schema VARCHAR2) AS v_metadata CLOB; v_etag VARCHAR2(100); CURSOR c_objects IS SELECT object_name, object_type FROM all_objects WHERE owner = p_schema AND object_type IN ('TABLE', 'INDEX', 'VIEW'); BEGIN FOR obj IN c_objects LOOP BEGIN v_metadata := DBMS_DEVELOPER.GET_METADATA( name => obj.object_name, schema => p_schema, object_type => obj.object_type ); SELECT JSON_VALUE(v_metadata, '$.etag') INTO v_etag FROM dual; INSERT INTO schema_versions (object_name, object_type, object_schema, capture_date, etag, metadata) VALUES (obj.object_name, obj.object_type, p_schema, SYSTIMESTAMP, v_etag, v_metadata); COMMIT; DBMS_OUTPUT.PUT_LINE('Captured metadata for ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Error capturing ' || obj.object_type || ' ' || p_schema || '.' || obj.object_name || ': ' || SQLERRM); END; END LOOP; END; / This solution provides several key benefits: Efficient change tracking: Using etags to identify exactly which objects have changedStructured documentation: Storing metadata in JSON format for easy extraction of specific attributesHistorical record: Maintaining snapshots of schema evolution over timeValidation capabilities: Comparing source and target schemas during migration During migration, you can extend this system to compare environments: -- Procedure to compare object between environments CREATE OR REPLACE PROCEDURE compare_object( p_name VARCHAR2, p_type VARCHAR2, p_source_schema VARCHAR2, p_target_schema VARCHAR2, p_target_db VARCHAR2 ) AS v_source_metadata CLOB; v_target_metadata CLOB; v_source_etag VARCHAR2(100); v_target_etag VARCHAR2(100); BEGIN -- Get source metadata v_source_metadata := DBMS_DEVELOPER.GET_METADATA( name => p_name, schema => p_source_schema, object_type => p_type ); -- Get target metadata via database link EXECUTE IMMEDIATE 'SELECT DBMS_DEVELOPER.GET_METADATA( name => :1, schema => :2, object_type => :3 ) FROM dual@' || p_target_db INTO v_target_metadata USING p_name, p_target_schema, p_type; -- Extract etag values SELECT JSON_VALUE(v_source_metadata, '$.etag') INTO v_source_etag FROM dual; SELECT JSON_VALUE(v_target_metadata, '$.etag') INTO v_target_etag FROM dual; -- Compare and report IF v_source_etag = v_target_etag THEN DBMS_OUTPUT.PUT_LINE('Objects match exactly'); ELSE DBMS_OUTPUT.PUT_LINE('Objects differ - detailed comparison needed'); -- Further JSON comparison logic could be implemented here END; END; / Conclusion The DBMS_DEVELOPER package represents a significant advancement in Oracle's metadata management capabilities. By providing metadata in JSON format, Oracle has created a more developer-friendly interface that aligns with modern application architecture patterns. Key takeaways include: JSON-based metadata is more programmatically accessible than traditional DDL statements The etag mechanism provides a reliable way to track object changes Multiple detail levels allow you to retrieve just the information you need The package is particularly valuable for documentation, migration, and change tracking While currently limited to tables, indexes, and views, the DBMS_DEVELOPER package has tremendous potential for expansion in future Oracle releases. Database architects and developers should consider integrating this powerful tool into their workflows, particularly for projects involving schema documentation, migration, or programmatic metadata access. As databases continue to evolve toward more autonomous and programmable systems, tools like DBMS_DEVELOPER will become increasingly central to efficient database management practices.

By arvind toorpu CORE