Coding

Most agent projects start the same way: one model, one prompt loop, three or four tools. That's the right place to start, and for a lot of workloads it's also the right place to stop. The trouble shows up later, when the same agent is asked to inspect infrastructure, parse OpenAPI specs, write docs, answer developer questions, and call operational tools. The system prompt turns into a wall of text. Tool selection gets flaky in ways you can't reproduce. Someone tunes the prompt to fix documentation quality and quietly breaks spec analysis, and nobody notices for a week. Splitting the agent is the obvious fix. But the agents were never the hard part. The hard part is the contract between them: how one agent finds another, how it hands over work, where authentication lives, and how you trace a request that crossed three runtimes before it failed. This article builds a small multi-agent system on Amazon Bedrock AgentCore Runtime using the Agent-to-Agent (A2A) protocol: an orchestrator, an OpenAPI spec analyzer, and a documentation generator. Three agents are a toy. The decisions are not. A note on versions. AgentCore and A2A are both moving fast. Everything here was checked against the AWS documentation and AgentCore CLI 0.25.x in August 2026. Pin your CLI and Python dependencies, and re-check the docs before you deploy anything you care about. Is a Second Agent Actually Worth It? Multi-agent isn't a maturity level. Every remote handoff you add buys you latency, an extra auth boundary, and a new class of failure that only shows up under load. Keep one agent when one team owns the workflow, the tools sit inside the same security boundary, and a single prompt can plausibly cover the job. Don't split because the architecture diagram looks better with more boxes in it. Split when responsibilities genuinely diverge. The signals are usually organizational before they're technical: two teams own two capabilities, the specialists want different models, the release cadences don't match, or one capability touches data the other has no business seeing. At that point the agent stops being a prompt and starts being a service, and it needs a service's contract. The arithmetic argument for a shared protocol is the familiar one. Pairwise integrations grow with the square of the number of participants, and each one tends to invent its own payload shape, auth handling, timeout, and error format. A standard protocol doesn't make any of those problems disappear. It just means every participant expresses them in the same place, which is most of the benefit. MCP and A2A Aren't Competing These two get discussed as alternatives. They aren't: they sit on different edges of the architecture. MCP connects an agent to tools and data.A2A connects an agent (or an application) to another agent. A spec analyzer might pull an OpenAPI document out of a repo over MCP, then return its analysis to an orchestrator over A2A. Both, same request. In the code below, the OpenAPI document is passed inline so the A2A flow stays readable. One caveat about "framework-agnostic," since it gets oversold: A2A is framework-agnostic at the protocol level. That doesn't mean two arbitrary SDKs will talk to each other on the first try. Both sides still need a compatible A2A implementation, transport, protocol version, and auth configuration. Once those line up, the internals genuinely don't matter; Strands on one side, LangGraph on the other, fine. What Runtime Gives You Configure an AgentCore Runtime for A2A, and it becomes a transparent proxy in front of your container. Your side of the contract is narrow: a stateless, streamable HTTP server on 0.0.0.0:9000, mounted at the root path, speaking JSON-RPC 2.0. AgentCore passes the JSON-RPC payload straight through without touching it, and adds session isolation, authentication, scaling, and observability around it. Note the port. A2A is 9000 at /, MCP is 8000 at /mcp, plain HTTP is 8080 at /invocations. Getting this wrong is the most common reason a container deploys fine and then never answers. The A2A vocabulary you need for the rest of this: A2A client: whoever is sending work. Here is the orchestrator.A2A server: the agent endpoint receiving it.Agent Card: metadata at /.well-known/agent-card.json describing identity, endpoint, skills, capabilities, and auth requirements. This is how discovery works.Message: one turn, carrying one or more parts (text, structured data).Task and artifact: a task tracks work at one remote agent; artifacts carry its output. An orchestrator fanning out to three specialists creates three separate tasks. For inbound calls, Runtime supports IAM SigV4 or OAuth 2.0 bearer tokens (JWT). A runtime version uses one of them at a time, so pick based on who's calling and where the security boundary sits. If a client authenticates with the wrong one, you get a 403 that looks nothing like an agent error. The Shape of the System The system has a hub and spoke architecture. The orchestrator is an A2A server to the user and an A2A client to the specialists, which is worth stating explicitly because it's easy to forget when you're debugging. Hub and Spoke Achitecture The orchestrator should know what each specialist advertises. It should not know how any of them work. The test: adding a fourth agent a changelog writer, say should mean one new config entry, not edits inside the agents you already shipped. Building a Specialist Prerequisites Python 3.10+Node.js 20+ (the AgentCore CLI ships as an npm package; the A2A tutorial page still says 18, but the CLI's own docs say 20)AWS CDK installed, the CLI deploys through itAWS credentials, plus model access enabled in Bedrocknpm install -g @aws/agentcore If you still have the old Python starter toolkit installed, uninstall it first. Both provide an agentcore command, and you will lose an afternoon to that. Scaffold an A2A project and pick Strands when prompted: Shell bash agentcore create --protocol A2A The generated project gives you the config, the model loader, and the A2A server wiring. The specialist below does one thing: read an OpenAPI document and pull out a compact, honest view of its operations. Python # app/SpecAnalyzer/main.py from typing import Any import yaml from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model HTTP_METHODS = { "get", "put", "post", "delete", "options", "head", "patch", "trace", } def _auth_mode(security: Any) -> str: if security is None: return "unspecified" if security == []: return "none" if isinstance(security, list) and any(item == {} for item in security): return "optional" return "required" @tool def summarize_endpoints(openapi_yaml: str) -> list[dict[str, Any]]: """Return a compact list of operations from an OpenAPI document.""" spec = yaml.safe_load(openapi_yaml) if not isinstance(spec, dict): raise ValueError("The OpenAPI document must contain a YAML object.") paths = spec.get("paths", {}) if not isinstance(paths, dict): raise ValueError("The OpenAPI 'paths' field must be an object.") global_security = spec.get("security") operations: list[dict[str, Any]] = [] for path, path_item in paths.items(): if not isinstance(path_item, dict): continue for method, operation in path_item.items(): if method.lower() not in HTTP_METHODS: continue if not isinstance(operation, dict): continue security = operation.get("security", global_security) operations.append( { "path": path, "method": method.upper(), "operation_id": operation.get("operationId"), "summary": operation.get("summary", ""), "auth": _auth_mode(security), } ) return operations agent = Agent( model=load_model(), system_prompt=( "You are an API specification analyst. " "Call summarize_endpoints before reasoning. " "Report endpoints, authentication behavior, and concrete " "specification risks. Never invent fields that aren't present." ), tools=[summarize_endpoints], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) serve_a2a is the AgentCore SDK helper that does the boring, mandatory parts: /ping, Agent Card serving, AGENTCORE_RUNTIME_URL handling, Bedrock header propagation, and binding to port 9000. The validation inside that tool is the part I'd argue for hardest. OpenAPI path items legally contain keys that aren't HTTP methods parameters, servers, $ref, orsummary. Loop over every key as though it were an operation and you'll either crash on a string or, worse, emit a confident summary containing an endpoint called PARAMETERS. Then the doc generator writes it up, and it ships. Demos never surface this because demo specs are clean. Direct dependencies for both specialists: Shell bedrock-agentcore[a2a] strands-agents[a2a] a2a-sdk PyYAML httpx The a2a extras matter. Without them, serve_a2a and the client classes aren't there. a2a-sdk is what the orchestrator's client code imports from, so it's a direct dependency even though the server side pulls it in transitively. The documentation generator uses the same wrapper with a narrower job: Python # app/DocGenerator/main.py from strands import Agent from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model agent = Agent( model=load_model(), system_prompt=( "You write developer documentation from an API request and a " "specification analysis. Produce Markdown with an overview, " "authentication notes, request examples, error handling, and a " "short Python quickstart. Never invent endpoints, fields, or " "credentials." ), ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) That one's prompt-only on purpose. If your docs have to match a house format, this is where you add output validation or template tools instead of asking a model to remember a style guide. Test It Locally Shell agentcore dev # or, without the inspector UI: python main.py Then send it a JSON-RPC message at the root path: JSON curl -X POST http://localhost:9000/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": "req-001", "method": "message/send", "params": { "message": { "role": "user", "parts": [ { "kind": "text", "text": "Analyze the authentication on this API." } ], "messageId": "12345678-1234-1234-1234-123456789012" } } }' | jq . Pull the Agent Card too, before you deploy anything: Shell curl http://localhost:9000/.well-known/agent-card.json | jq . Thirty seconds of curl catches a missing discovery endpoint or a card whose skills describe an agent you deleted two commits ago. Card problems are miserable to debug from the client side, because what you see is a resolver failure, not a bad card. Deploy the Specialists The CLI defaults to IAM SigV4. Configure bearer-token auth when the caller can't sign requests, the AWS tutorial walks through Cognito for this, though Cognito isn't a requirement; any OIDC provider works. Deploy each specialist as its own runtime: Shell agentcore deploy The CLI packages the code, uploads the artifact to S3, creates the runtime, and hands back an ARN: Shell arn:aws:bedrock-agentcore:us-west-2:<account-id>:runtime/SpecAnalyzer-xyz123 Separate runtimes are half the point of doing this at all. Each specialist scales, releases, and rolls back on its own, without a coordinated deploy across the whole system. The URL Nobody Warns You About Here's the step that eats the most time. The invocation URL isn't the ARN, and it isn't a friendly hostname. It's the ARN, URL-encoded, embedded in a path: https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{url-encoded-arn}/invocations/ Every colon and slash in the ARN gets escaped with quote(arn, safe='') in Python, or let the SDK do it: Python from bedrock_agentcore.runtime import build_runtime_url url = build_runtime_url( "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/SpecAnalyzer-xyz123" ) The Agent Card sits under that path, at .../invocations/.well-known/agent-card.json. Keep the trailing slash on the base URL. A missing slash or an unescaped ARN both produce a 404, which reads like the runtime doesn't exist. Calling a Remote Agent This client assumes bearer auth. For an IAM runtime, sign with SigV4 or use a client that signs for you. Python # app/Orchestrator/client.py import os from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, TextPart DEFAULT_TIMEOUT = 300 def _message(text: str) -> Message: return Message( kind="message", role=Role.user, parts=[Part(TextPart(kind="text", text=text))], message_id=uuid4().hex, ) async def call_remote_agent( runtime_url: str, text: str, session_id: str | None = None, ) -> str: headers = { "Authorization": f"Bearer {os.environ['BEARER_TOKEN']}", "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": session_id or str(uuid4()), } async with httpx.AsyncClient( timeout=DEFAULT_TIMEOUT, headers=headers, ) as http: resolver = A2ACardResolver(httpx_client=http, base_url=runtime_url) card = await resolver.get_agent_card() config = ClientConfig(httpx_client=http, streaming=False) client = ClientFactory(config).create(card) async for event in client.send_message(_message(text)): if isinstance(event, Message): return event.model_dump_json(exclude_none=True) if isinstance(event, tuple) and len(event) == 2: task, _ = event return task.model_dump_json(exclude_none=True) raise RuntimeError("The remote agent returned no A2A result.") With streaming=False, you get exactly one result, which is either a Message or a (Task, UpdateEvent) tuple. Handle both. The tuple form is what trips people up on the first run. Returning raw JSON keeps the protocol visible for a walkthrough, but don't ship it. Normalize messages and artifacts into your own response type at this boundary. Otherwise, every downstream prompt has to understand the full A2A envelope, and you end up with models reasoning about kind fields instead of about your problem. Session IDs deserve a real decision, not a uuid4() you stopped thinking about. Fresh ID for an isolated one-shot task. Same ID when you're continuing a multi-turn conversation with the same remote agent. And keep a separate correlation ID of your own, because one user request may span several runtime sessions and you'll want to stitch them back together in the traces. The Orchestrator Each remote agent becomes a tool. Endpoints come from config; the client resolves the Agent Card at the endpoint before sending work. Python # app/Orchestrator/main.py import os from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model from client import call_remote_agent REMOTE_AGENTS = { "spec_analyzer": os.environ["SPEC_ANALYZER_URL"], "doc_generator": os.environ["DOC_GENERATOR_URL"], } @tool async def delegate(agent_name: str, instruction: str) -> str: """Send work to a named A2A specialist.""" runtime_url = REMOTE_AGENTS.get(agent_name) if runtime_url is None: allowed = ", ".join(sorted(REMOTE_AGENTS)) return f"Unknown agent '{agent_name}'. Available agents: {allowed}." return await call_remote_agent(runtime_url, instruction) orchestrator = Agent( model=load_model(), system_prompt=( "You coordinate two API specialists. Use spec_analyzer for OpenAPI " "parsing and risk analysis. Use doc_generator for documentation and " "SDK examples. When both are needed, analyze the specification " "first, then include that analysis in the documentation request. " "Never assign work outside an agent's advertised capability." ), tools=[delegate], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(orchestrator)) Two small things in there matter more than they look. The tool is async, which avoids wrapping the remote call in asyncio.run() that blows up when the agent is already running inside an event loop, and the traceback doesn't point anywhere near the cause. And returning a friendly string for an unknown agent, rather than raising, gives the model something it can recover from instead of a dead turn. The URLs belong in deployment config, a service catalog, or an agent registry. Not in source control. Let the Model Route, or Hard-Code the Path? The code above lets the orchestrator model decide who to call and in what order. That's genuinely useful when requests vary. It's also the default choice for the wrong reasons. If every documentation request must be analyzed before generation, always, no exceptions encode that sequence in Python and use the model only inside each step. Deterministic orchestration is easier to test, retry, and audit, and it doesn't drift when someone edits a prompt. Save model-driven routing for routes that are actually ambiguous. It shouldn't be there to make a fixed workflow feel more autonomous. Running It With all three runtimes deployed, send the user request to the orchestrator. Something like "Review this customer API spec and create a Python quickstart" should cause it to call the analyzer, feed that analysis to the doc generator, and return one answer. Should. That's expected behavior, not a guarantee the protocol hands you. Whether the right handoffs happen comes down to your orchestrator prompt, your tool descriptions, your eval cases, and your workflow code. A2A standardizes the conversation. It has no opinion about whether your orchestration logic is correct. Before You Call It Production Least privilege per runtime. Each runtime gets permissions for its own tools and data, nothing more. The failure mode to watch for is the orchestrator's role slowly accumulating every permission any specialist ever needed. Distinguish auth failures from agent failures. A2A application errors come back as JSON-RPC error objects, frequently with HTTP 200. Auth and authorization failures come back as native HTTP errors AccessDeniedException is a 403. AgentCore also maps runtime exceptions onto JSON-RPC codes: -32051 for not found (404), -32053 for throttling (429), -32054 for conflict (409), -32055 for a runtime client error (424). Inspect the JSON-RPC shape. A 200 is not a success. Retry deliberately. 429 and retryable-conflict 409 are worth retrying with backoff. A 424 usually means your container failed and the answer is in CloudWatch, not in a second attempt. Never blind-retry non-idempotent work, and pass a task or request identifier so a specialist can recognize a duplicate. Trace every handoff. AgentCore Observability gives you sessions, traces, and spans. Propagate your own business correlation data on top, or you'll have three sets of traces and no way to prove they belong to the same user request. Set lifecycle values on purpose. Defaults are a 15-minute idle timeout and an 8-hour max lifetime, both configurable (60 to 28,800 seconds). Long-running tasks may justify raising them; a bigger number should be a decision, not a leftover. Also watch that your A2A sessions actually go idle — a server that holds a connection open after the client is gone will happily keep a microVM warm on your bill. Cache Agent Cards carefully. Resolving on every call is simple and costs you a round trip per hop. Cache with a short TTL and invalidate when an agent's version or capabilities change. A stale card advertising a skill that no longer exists is a genuinely confusing bug. Keep protocol and SDK versions aligned. Pin dependencies, validate cards, and test a mixed-framework setup rather than assuming it works. Use private connectivity if you need it. Runtime can attach to a VPC for internal resources, and interface VPC endpoints via PrivateLink keep AgentCore API calls off the public internet. A Failure Test Matrix Before you call it done, break the handoffs rather than admiring the happy path: The specialist's Agent Card is unreachable or malformed.The bearer token expires between card resolution and invocation.The analyzer returns a valid task with no usable artifact.The doc generator times out after the analyzer succeeded.The orchestrator picks an agent that doesn't exist, or one it isn't allowed to call.A retry re-runs a task that already had an external side effect. Each of these teaches you something a successful demo can't. They also make the division of labor concrete: Runtime hosts and protects the endpoint, A2A defines the conversation, and workflow correctness is still entirely yours. Closing The value here isn't the agent count. It's that each capability gets a clear boundary, a narrow permission set, and its own release cycle. Runtime provides the managed execution boundary; A2A provides a common way to discover agents and pass work between them. So, start with one agent. Split it when responsibilities, ownership, or security boundaries justify a network hop, and not before. Where the sequence is fixed, keep the orchestration deterministic. Where routing is genuinely variable, let the orchestrator reason over well-described capabilities, and instrument every handoff you add. References AWS: Deploy A2A servers in AgentCore RuntimeAWS: A2A protocol contractAWS: AgentCore Runtime authentication and authorizationAWS: LifecycleConfiguration API referenceAWS: Configure AgentCore Runtime for VPC accessAgentCore CLIAgent2Agent protocol specification

Most machine learning (ML) platforms still hand out accelerators through static quotas, spreadsheet-driven approvals, and hallway negotiation. That model was tolerable when capacity was cheap, and demand was steady. Neither holds for modern AI workloads. This article walks through why static allocation breaks down, the five components of a capacity orchestration layer (intent capture, policy, hardware brokering, continuous reallocation, and goodput accounting), and a 90-day rollout plan that won't destroy your users' trust in the scheduler. The Contradiction You Have Probably Already Measured If you run infrastructure for a large ML organization, you have likely seen a dashboard that says something uncomfortable. The fleet is 90-plus percent allocated. Teams are filing tickets about blocked capacity. And a sampled profile of the same fleet shows a meaningful fraction of devices doing no useful work. Buying more accelerators does not resolve this. It scales the inefficiency. The contradiction comes from a specific architectural choice: treating an accelerator reservation as ownership rather than as a lease with conditions. The industry has spent years optimizing acquisition: supply agreements, multi-year commitments, reserved cloud capacity. Comparatively little engineering has gone into what happens after the hardware lands in the fleet. Why Static Quotas Fail for ML Specifically Conventional service capacity planning works because service demand is autocorrelated. Today's traffic predicts tomorrow's. ML workload demand does not behave this way, for four reasons. Demand is event-driven. An eval result at 2 p.m. can turn a 16-device research project into a 512-device training run by the next morning. No quarterly quota cycle accommodates that. Jobs differ in shape, not just size. A gang-scheduled training job with all-reduce dependencies across 64 devices and a stateless inference replica are both "GPU requests" to a naive scheduler, but they have incompatible placement, topology, and preemption semantics. A fleet fragmented without topology awareness ends up with allocatable devices that no large job can actually use. Reservation and consumption diverge. A job holding 32 devices while blocked on an input pipeline is occupying capacity and producing nothing. Device-level utilization metrics will not reliably catch this, because a device spinning on a synchronization barrier can still report activity. Returning capacity is asymmetrically risky. This one dominates. If reacquiring capacity takes two weeks of negotiation, the rational move for every team is to hold everything they might need at peak, indefinitely. Static allocation rewards hoarding. Each over-request is a locally correct decision inside a badly designed system, which is why exhortations to "be good citizens" never fix it. The Architecture A capacity orchestration layer sits between your workload submission API and the underlying scheduler, whether that is Kubernetes with Kueue or Volcano, Slurm, Ray, or an internal system. It does not replace the scheduler. It feeds the scheduler better information and revisits its decisions after admission. 1. Intent Capture: A Device Count Is an Answer, Not a Requirement When a user requests "32 A100s," they have already collapsed their actual requirement into one number. The scheduler needs the requirement itself: what the workload can substitute, what it can tolerate, and what it cannot give up. In practice that looks like a spec along these lines: YAML apiVersion: capacity.internal/v1 kind: WorkloadIntent metadata: name: ranking-hparam-sweep spec: class: research-batch # drives policy, not placement preferred: accelerator: a100-80gb count: 32 substitutable: - accelerator: h100-80gb count: 20 # based on measured step-time on this model - accelerator: tpu-v5e count: 64 requiresRecompile: true # XLA path validated in CI topology: gangScheduled: true minInterconnect: nvlink-domain elasticity: minReplicas: 8 maxReplicas: 32 resizeCost: 90s # checkpoint + reshard preemption: tolerated: true gracePeriod: 120s checkpointInterval: 600s deadline: "2026-08-14T00:00:00Z" Two fields carry most of the value. substitutable is what makes cross-architecture brokering possible. preemption.tolerated, paired with a real checkpointInterval, is what makes a job safely reclaimable. Interruption tolerance should be verified, not self-declared. Run a preemption drill in CI: kill the job, confirm it resumes from checkpoint within the stated grace period. A self-reported tolerance that has never been exercised will eventually be exercised in production, on a bad day. Be honest about the engineering cost. Portable workload definitions, container images built for multiple backends, and checkpoint formats that survive a device-count change are all real work. Elastic resharding in particular is hard for models with complex parallelism strategies. Treat the full intent spec as something teams grow into. A job that declares only preferred still schedules; it just gets fewer options. 2. Policy Engine: Encode the Argument Once The point of a policy engine is not to remove human judgment. It moves judgment out of per-request negotiation and into rules that can be reviewed, versioned, and revised. On Kubernetes, Kueue's cohort model already expresses the core mechanics through nominal quota, borrowing, and lending: YAML apiVersion: kueue.x-k8s.io/v1beta1 kind: ClusterQueue metadata: name: research-shared spec: cohort: fleet-accelerators preemption: reclaimWithinCohort: Any withinClusterQueue: LowerPriority resourceGroups: - coveredResources: ["nvidia.com/gpu"] flavors: - name: h100-80gb resources: - name: "nvidia.com/gpu" nominalQuota: 256 borrowingLimit: 512 # may burst into idle cohort capacity lendingLimit: 192 # keeps a protected floor of 64 devices Read those preemption lines together with the limits, because the pair is what changes team behavior. lendingLimit guarantees a floor that is never lent out, and reclaimWithinCohort: Any guarantees that lent capacity comes back the moment the owner needs it, by preempting borrowers if necessary. Once returning idle capacity carries no risk of losing it, the economic case for hoarding mostly evaporates. Beyond quota mechanics, useful policy inputs include production criticality, launch dates, queue age, the historical efficiency of the requesting workload, and per-organization fairness floors. Dominant Resource Fairness (Ghodsi et al., NSDI 2011) is a sensible default for multi-resource sharing, with queue aging layered on top so low-priority work eventually runs instead of starving. One requirement is non-negotiable: explainability. Every scheduling decision needs a retrievable answer to "why is my job still queued?" naming the binding policy, the preempting workload, and a projected start time. Without that, engineers conclude the system is arbitrary and route around it. An orchestration layer people distrust is worse than static quotas, because now you have a scheduler plus a shadow economy of private allocations. 3. Hardware Broker: One Portfolio, Not Several Fiefdoms Many organizations run accelerator families as separate silos, each with its own team, toolchain, and queue. The fragmentation hides substitution opportunities and strands capacity on whichever side happens to be idle. A broker matches substitutable declarations against live supply and makes the tradeoff explicit: 32 A100s in four hours, versus 20 H100s now, versus 64 TPU v5e chips now plus a 12-minute recompile. Comparing those options requires measured throughput per accelerator class per model family, which in turn means maintaining a benchmark corpus and refreshing it as software evolves. Compiler-level changes alone can shift the relative ranking of hardware classes over time; our work on compiler autotuning for TPU fleets (CATWILD, MLSys 2026) found that tuning decisions which were optimal at one point degrade as workloads and toolchains drift, and the same staleness problem applies to any performance model a broker relies on. A stale model will confidently route work to the wrong hardware. Scope this realistically. Workloads with hand-tuned CUDA kernels, custom collectives, or architecture-specific memory layouts are not portable, and forcing portability on them wastes effort. The goal is narrower: preserve flexibility where it is technically sound, and make sure the jobs that genuinely are portable are visible to the scheduler as such. 4. Continuous Reallocation: Placement Does Not End at Admission Traditional schedulers decide placement once, at admission, and then stop thinking. AI infrastructure needs a control loop that keeps asking whether each device is still attached to the highest-value feasible work. Conditions the loop should detect and act on: Idle reservations. Allocated devices with sustained near-zero productive compute past a grace threshold get reclaimed.Over-provisioned elastic jobs. When scaling efficiency degrades past a threshold, the marginal device is buying almost nothing. Resize down and return the difference.Blocked jobs. A job stalled on data loading, or an external dependency for longer than its checkpoint interval, gets checkpointed and requeued rather than left squatting.Priority inversions. A launch-critical workload arriving after lower-priority jobs have claimed capacity triggers graceful preemption, per declared policy.Substitution openings. A portable queued job starts now on a different accelerator class instead of waiting for its preferred one. The safety constraints matter as much as the actions. Rate-limit preemptions per team per hour. Never preempt a job whose checkpoint drill has not passed. Cap how many times a single job can be preempted before it gets promoted, or some unlucky workload will never finish. And write every reallocation event into the job's own log with a reason attached. Unexplained preemption is the fastest way to lose organizational support for the whole system. 5. Goodput Accounting: Allocated Is Not Productive You cannot manage what your metrics obscure, and device-level utilization obscures a lot. A layered view separates the questions: LayerQuestionRough definitionAllocation rateIs the fleet handed out?Allocated device-seconds / total device-secondsOccupancyAre allocated devices active?Active device-seconds / allocated device-secondsGoodputIs the activity producing committed progress?Device-seconds contributing to non-discarded work / allocated device-secondsMFUHow close is the work to the hardware roofline?Achieved model FLOPs / peak FLOPs Goodput is the layer that changes behavior. Computing it means joining hardware telemetry with job-level progress signals such as steps committed, checkpoints written, and requests served, so that restart replay, discarded partial work, and stalled collectives count as loss rather than as utilization. A training job that crashes and replays six hours from a stale checkpoint looks fine by occupancy and terrible by goodput, and goodput is telling the truth. Model FLOPs utilization, introduced in the PaLM paper (Chowdhery et al., 2022), is worth tracking but answers a different question. MFU is a compiler and kernel efficiency signal, not an allocation signal. A job can be perfectly scheduled and still leave performance on the floor; that is a different team's bug. The Fairness Constraint A system that optimizes purely for near-term commercial value fails predictably. Exploratory research never runs, because its value is uncertain by definition, and small teams lose every contest against large ones. The safeguards are concrete: a protected research pool that revenue workloads cannot preempt, queue aging that raises effective priority over time, per-team caps within each cohort, and reservation expiry so held capacity gets re-justified periodically. All of these cost measured efficiency, and that cost should be budgeted deliberately. The alternative is an infrastructure layer that quietly defunds the products you have not built yet. A 90-Day Rollout Days 0 to 15: instrument before changing anything. Inventory accelerator types, ownership boundaries, queue mechanisms, and approval paths. Separate reservation time from productive execution time in telemetry, and publish the gap. If you cannot produce a defensible goodput number, everything downstream is guesswork. Expect the first honest measurement to be politically uncomfortable. Days 16 to 45: write the allocation contract. Define priority classes, fairness floors, preemption rules, reservation expiry, and substitution policies, then circulate them with engineering, infrastructure, and product leadership before automating anything. Decide which workloads carry hard guarantees and which are explicitly flexible. This phase is mostly negotiation rather than implementation, and skipping it is the most common way these projects die. Days 46 to 90: automate one bounded pool. Choose a cohort with a contained blast radius. Batch research capacity is usually right, since it tolerates interruption. Enable reclamation, resizing, and substitution there only, and track queue time, end-to-end completion time, goodput, failed migrations, and policy exceptions. Then hold. Do not expand scope until teams trust both the decisions and the explanations behind them. Trust, not scheduler throughput, is the binding constraint on adoption. Closing Accelerator demand is not going to flatten. Models grow, inference spreads into more product surfaces, and experiment volume rises with headcount. Newer hardware improves per-device throughput while doing nothing about contention; if anything, it raises the stakes, since a stranded H100 is a more expensive mistake than the stranded V100 it replaced. Organizations that scale well will stop treating accelerators as assets assigned to teams and start managing them as a continuously rebalanced portfolio governed by explicit, explainable policy. Procurement still matters. Orchestration determines how much of what you procured turns into shipped product. The most valuable accelerator in your fleet is not the next one you buy. It is the one you already own, currently attached to the wrong job.

As Large Language Models (LLMs) continue their rapid trajectory of development, software engineers and cloud architects regularly run into two systemic challenges: The Fixed Knowledge Cutoff: Model intelligence is inherently restricted to its training data window, making it blind to real-time changes. The "Air-Gap" Limitation: Out of the box, LLMs cannot securely interact with external systems or private APIs on their own. Historically, developers bypassed these hurdles by writing fragile, ad-hoc API wrappers or custom orchestrators. Enter the Model Context Protocol (MCP): an open standard designed to standardize how AI applications safely connect to external data sources and execution environments.In this article, we will explore the core architecture of MCP, look at why the Azure MCP Server is a game-changer for cloud engineers, and walk through a step-by-step guide to configuring it inside Visual Studio Code. The Core Architecture of MCP At its heart, MCP establishes a uniform "language" that allows AI applications (Hosts) to talk to external resources (Servers). Rather than building custom integrations for every new LLM or tool, developers can rely on a single, clean architecture: MCP Architecture The standard is built around four fundamental building blocks: MCP Host: The runtime environment or user interface where the AI agent operates (e.g., VS Code, Claude Desktop, Cursor).MCP Client: The architectural component within the host that initiates and maintains the active connection.MCP Server: A lightweight, modular helper service that exposes specific resources, prompts, and tools.Transport Layer: The underlying protocol facilitating communication, typically utilizing JSON-RPC 2.0 over standard input/output (stdio) or HTTPS. MCP Components How the MCP Handshake Works Instead of executing raw, unpredictable bash scripts, the interaction is highly structured: Initialization: The client connects to the server and queries its capabilities. Declaration: The server returns a structured schema listing the specific tools it supports. Execution Request: When an LLM determines it needs external data, the host requests a specific tool execution from the server. Context Injection: The server executes the local process, gathers the result, and returns a semantic JSON response to the host, which is then cleanly formatted for the user. MCP in Action Why Use the Azure MCP Server? If you are managing infrastructure on Microsoft Azure, the Azure MCP Server bridges the gap between your local AI assistant and your active cloud resources. Operating as a secure local process, it natively integrates with the Azure command-line context. The server supports over 40 Azure services and more than 170 tools out of the box, spanning critical cloud primitives: Compute & Containers: Azure Container Apps and Azure Kubernetes Service (AKS). Storage & Resource Management: Azure Storage (blobs and containers) and Azure Resource Groups. Infrastructure as Code: Integrated Azure Terraform Best Practices. Key Advantages Over Raw CLI Executions While you could technically let an AI agent execute arbitrary commands in a terminal, using a dedicated MCP server provides several major structural benefits: Rich Semantics: Instead of parsing messy, unstructured terminal stdout text, the MCP server passes rich semantic data blocks directly back to the LLM. Strict Governance & Scope: You can explicitly configure the server to run in read-only mode or expose only a select subset of tools, preventing the AI from accidentally deleting production infrastructure. Interactive Guardrails: The protocol requires explicit user confirmation before executing tools that touch sensitive data or perform mutative actions. Azure MCP Server Setup and Configuration Modes You can run the Azure MCP server locally across multiple development setups using stdio transport. Below are the three most common configuration schemas. NuGet Configuration For .NET teams, the server can be dynamically fetched and started using the dnx toolchain: JSON { "mcpServers": { "Azure MCP Server": { "command": "dnx", "args": [ "Azure.Mcp", "--source", "https://api.nuget.org/v3/index.json", "--yes", "--", "azmcp", "server", "start" ], "type": "stdio" } } } Node.js Configuration If you are developing in a standard JavaScript/TypeScript ecosystem, you can spin up the server dynamically using the latest npm package via npx: JSON { "mcpServers": { "azure-mcp-server": { "command": "npx", "args": [ "-y", "@azure/mcp@latest", "server", "start" ] } } } Docker Configuration For isolated development environments, you can run the server in a container. Note that you must provide a local environment file (.env) containing your Azure Service Principal credentials: JSON { "mcpServers": { "Azure MCP Server": { "comman { d": "docker", "args": [ "run", "-i", "--rm", "--env-file", "/full/path/to/.env", "mcr.microsoft.com/azure-sdk/azure-mcp:latest" ] } } } Step-by-Step Visual Studio Code Integration A great feature for those who want to work within Visual Studio Code is that they can also manage the Azure MCP Server through an exclusive extension available directly in their editor. The following steps walk you through the setup. Step 1: Install the Extension Search for and install the Azure MCP Server extension directly from the Visual Studio Code marketplace. Install Azure MCP Server Extension Step 2: Initialize and Verify Open the Command Palette (Cmd + Shift + P on macOS or Ctrl + Shift + P on Windows) and search for the extension commands to verify the server is active and running. Select an MCP Server Step 3: Configure Your Tool Accessibility Open your integrated AI chat window and select the Tools icon. Here, you will see a list of all active tools provided by the Azure MCP Server. You can toggle specific permissions on or off—for instance, disabling write operations while keeping read operations active. Select Your Tools Step 4: Interact Natively Now, your AI chat assistant can securely call Azure tools in the background. You can ask complex queries like: "Are there any inactive containers running in my resource group?""Upload our local configuration file directly to our Azure storage container blob storage." Any further interactions that reference an Azure account would use the MCP tools. Confirm Tools and Resources Real-World Engineering Use Cases To see the power of this setup, let’s look at how this changes day-to-day operations: Use Case A: Automated AKS Incident Troubleshooting When an incident occurs in an Azure Kubernetes Service (AKS) cluster, engineers typically run dozens of diagnostic commands. With the Azure MCP server connected, you can simply ask the LLM: "Investigate why the pods in our production namespace are crash-looping." The agent will call the relevant AKS tools, inspect the logs, identify the misconfiguration, and suggest the fix - all in seconds. Use Case B: Continuous Terraform and Compliance Audits Before deploying infrastructure, you can point your local AI agent to your code directory. Because the server incorporates Azure Terraform Best Practices, the agent can audit your configuration files, cross-reference them against your live Azure Resource Groups, and warn you if you are violating security compliance rules or generating drift. Conclusion The Model Context Protocol represents a major step forward in AI-assisted development. By standardizing the communication layer, the Azure MCP Server enables software engineers to transform static, isolated LLMs into active, context-aware cloud operators. Whether you are monitoring active Kubernetes clusters, auditing Terraform configurations, or automating file uploads to blob storage, MCP gives your AI assistant safe and highly scoped access to the "live" Azure ecosystem.

Thirteen thousand internal screenshots from 343 technology companies are sitting in public GitHub repositories because a coding agent could not attach an image to a private pull request. Cybernews reported that the exposed material includes customer records, billing data, payment system screens, and unreleased product features. The agents appear to have completed the task using access they already held, but the route they took exposed data publicly. That last detail is the story. When a private repository could not render the image, the agents created public ones, mostly under employees’ personal accounts. No policy forbade it in a form software could act on, and no control stood between the task and the result. It would be comforting to call this an outlier. A survey of more than 900 executives and technical practitioners says it is closer to the norm. Gravitee’s State of AI Agent Security 2026 report found that only 14.4% of organizations have every AI agent go live with full security and IT approval, while 82% of executives say they feel confident their existing policies protect them from unauthorized agent actions. The sequence is deploy first, approve later, investigate after that. The survey comes from an API management vendor, so treat the figures as directional. Directional is enough. Nobody is reporting that agents wait politely for a security review, and the screenshot leak is what that looks like when the agents are good at their jobs. The gap between AI policy and enforcement Look beneath the confidence. A policy is not a control, and a count of incidents tells you how often something went wrong without telling you whether you could explain it to a regulator. The harder question is a plain one. Can you prove, on demand, who authorized this agent, which data it touched, and under what rule? The same survey puts a number on how far most organizations are from that answer. On average, only 47.1% of an organization’s AI agents are actively monitored or secured. Run that through an audit. An agent that is not actively monitored can leave critical gaps in the record. An agent without its own identity cannot be tied to the person who delegated the work, and agents that share credentials cannot say which of them acted. Each gap removes a link between an action and an accountable human, and an investigator needs every link. If the honest answer is “we would need to reconstruct it,” you do not have a security gap so much as an evidence gap. A detection gap belongs to the SOC. An evidence gap belongs to the CISO and the Chief Compliance Officer together, because one must produce the record and the other must stand behind it when a regulator’s clock starts. When an AI security gap becomes a compliance problem The clock runs in days. Kiteworks Data Security and Compliance Risk: 2026 Annual Survey Report found that 50% of organizations cannot produce a complete AI data access audit record within one business day. Notification windows, customer contract terms, and assessor timelines do not wait for an evidence package that takes weeks. I call the gap between confidence and enforcement governance theater, and I say it without contempt for the executives involved. They are reasoning from the best information they have: a written policy. Nobody has shown them whether that policy is technically enforced at the point where an agent reaches for data. A rule that no system enforces is a statement of intent, and auditors will read it as such. Regulators do not regulate models. HIPAA, PCI DSS, and the financial regulators do not ask whether a clinician or a program disclosed the data, and a screenshot of a billing console in a public repository raises the same questions either way. The obligation attaches to the data, and the evidence requirement attaches with it. That is also why a system prompt is not a compliance control. An instruction telling a model to stay away from certain material can be bypassed or changed, and an assessor will not accept “the agent was told not to” as proof of access control. Enforcement must sit with the data, independent of the model, and it must leave a record. What must change before the next agent ships Start by naming an owner for every agent. Not the team that built it and not the vendor whose model it calls, but one accountable person who can say what the agent may write, publish, and share, and who delegated the work. Ownership of AI security is unsettled in most organizations, and that vacuum is the cheapest gap to close. Next, give agents their own identities and tie each one to the human who authorized it. Humans and agents are two classes of identity that belong under one governance model, with one policy and one audit trail. The goal is not independence for agents. It is attribution for everything they do, with authority scoped to each action rather than inherited whole from a developer’s session. Then inventory every place an agent can write. List each destination an agent might use to make a human see an artifact, record who owns the account and whether the default is public, and refuse or human-gate anything world-readable outside your control. Treat screenshots and recordings as data, because they carry customer records and tokens and slip past rules written for documents. Finally, run the test your auditor will run. Pick a recent agent action and ask your team to produce the complete evidence package covering authorization, data accessed, policy applied, and the log that proves it. Time the answer. If it takes days, that number is your real exposure, and it will persuade a board faster than any survey. The organizations that close the approval gap will not be the ones with the longest policy binders. They will be the ones who can answer the auditor’s question before anyone asks it. Editor’s note: This article originally appeared on our sister publication, TechRepublic.

One of the most interesting parts of my role as a cloud solution architect is working directly with customers as they move from experimentation to production. The conversations change significantly at that point. During an early proof of concept, the questions are usually foundational: Can the model answer the question?Can the agent call the API?Can we connect our enterprise data?Can we build the experience? But when customers start thinking about production, the questions become much harder: Can I trust the agent to take an action?How do I know it followed the business process?How can I prove that it used the right tool?What happens if it gets the right answer but takes the wrong path? I'll use a simplified travel scenario throughout this article. Imagine building an agent that can upgrade an airline passenger when certain business conditions are met. A user asks: “Can you upgrade Alex Johnson's SEA-to-JFK flight AA245 to business class if he is eligible?” The agent responds: “Alex is eligible, and I have successfully submitted the business-class upgrade request.” At first glance, everything looks good. The answer is clear. The request appears to have been completed. The demo works. But I found myself asking a different question: “What did the agent actually do before giving us that answer?” That question became much more interesting than the answer itself. From Evaluating Answers to Evaluating Behavior For a traditional generative AI application, evaluating the final response makes sense. Was it relevant? Grounded? Coherent? Accurate? Those questions don't disappear when we build agents, but agents introduce another dimension. An agent can understand an intent, decide which tool to use, generate tool arguments, execute the tool, inspect the result, choose another tool, take an action, and eventually generate a response. Microsoft Foundry describes this same challenge: production-ready agent applications need evaluation not only of the final output, but also of the quality and efficiency of the workflow that produced it. For our travel scenario, this might be the intended workflow: User request ➔ Look up customer ➔ Check upgrade eligibility ➔ Create upgrade request ➔ Return confirmation Now imagine the agent instead does this: User request ➔ Create upgrade request ➔ Look up customer ➔ Check eligibility ➔ Return confirmation The final answer could still be perfect. But the workflow is absolutely not. That is what I mean by the agentic black box. A Small Version of the Customer Problem When working through an architecture question, I try to make the problem as small as possible first to separate the core issue from enterprise complexity. I created three simple Python functions. The first finds the customer: Python import json def lookup_customer(name: str) -> str: """Find a customer using their name.""" customers = { "Alex Johnson": { "customer_id": "CUST-1001", "loyalty_tier": "Gold" } } customer = customers.get(name) if not customer: return json.dumps({"error": "Customer not found"}) return json.dumps(customer) The second determines whether that customer is eligible for an upgrade: Python def check_upgrade_eligibility(customer_id: str, flight_number: str) -> str: """Check whether the customer can receive an upgrade.""" if customer_id == "CUST-1001": return json.dumps({ "customer_id": customer_id, "flight_number": flight_number, "eligible": True, "reason": "Gold member with available upgrade inventory" }) return json.dumps({ "customer_id": customer_id, "flight_number": flight_number, "eligible": False }) Finally, the tool that performs the business action: Python def create_upgrade_request(customer_id: str, flight_number: str, target_cabin: str) -> str: """Create an airline upgrade request.""" return json.dumps({ "request_id": "UPG-9001", "customer_id": customer_id, "flight_number": flight_number, "target_cabin": target_cabin, "status": "submitted" }) In a real environment, each function could represent a Microsoft Foundry Agent, an Azure Function, an MCP Server, a Logic App, or an enterprise system. But from the agent's perspective, the important question remains: Which tool should I call, with what parameters, and at what point in the workflow? Turning Business Rules into Agent Instructions The customer requirement sounds straightforward: “Do not create an upgrade until eligibility has been confirmed.” That sentence is doing something very important — it is defining a business control. I can expose my Python functions to a Foundry agent as function tools and establish the Foundry project client. Python from typing import Any, Callable, Set from azure.ai.projects.models import FunctionTool, ToolSet from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential import os user_functions: Set[Callable[..., Any]] = { lookup_customer, check_upgrade_eligibility, create_upgrade_request, } functions = FunctionTool(user_functions) toolset = ToolSet() toolset.add(functions) project_client = AIProjectClient( endpoint=os.environ["AZURE_AI_PROJECT"], credential=DefaultAzureCredential(), ) Now we create the agent, converting a business rule into expected agent behavior: Python agent = project_client.agents.create_agent( model=os.environ["MODEL_DEPLOYMENT_NAME"], name="travel-upgrade-agent", instructions=""" You help airline customers request flight upgrades. For every upgrade request: 1. Look up the customer first. 2. Check the customer's upgrade eligibility. 3. Only if the customer is eligible, create an upgrade request. 4. Never create an upgrade request before eligibility is confirmed. 5. Clearly explain the result to the user. """, toolset=toolset, ) Running the Customer Scenario We can create a thread, submit a test request, and execute the run: Python thread = project_client.agents.threads.create() message = project_client.agents.messages.create( thread_id=thread.id, role="user", content="Can you upgrade Alex Johnson's SEA-to-JFK flight AA245 to business class if he is eligible?", ) run = project_client.agents.runs.create_and_process( thread_id=thread.id, agent_id=agent.id, ) If we inspect the output and stop there, we have demonstrated that the agent can work. But we haven't demonstrated that it worked correctly. If an agent can create a ticket, modify a reservation, or provision infrastructure, I want visibility into the trajectory. Did it select the right tool? Did it send correct parameters? Did it take the next action at the right time? Using AIAgentConverter A Foundry execution produces multiple messages and interactions. Parsing all of that manually into an evaluation schema is tedious. That is where AIAgentConverter is useful. It converts a Foundry thread and run into the inputs expected by supported evaluators. Python import json from azure.ai.evaluation import AIAgentConverter converter = AIAgentConverter(project_client) converted_data = converter.convert( thread.id, run.id, ) print(json.dumps(converted_data, indent=2, default=str)) This was the point where the evaluation problem clicked for me. Instead of evaluating only the final string, I now have a normalized representation of the agent interaction that evaluators can inspect. Evaluation Question #1: Did the agent understand the user? "Upgrade the flight if he is eligible" is subtly different from "Upgrade the flight." Using IntentResolutionEvaluator, we can measure whether the agent correctly identified that condition. Python import os from azure.ai.evaluation import IntentResolutionEvaluator model_config = { "azure_deployment": os.environ["AZURE_DEPLOYMENT_NAME"], "api_key": os.environ["AZURE_OPENAI_API_KEY"], "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"], "api_version": os.environ["AZURE_API_VERSION"], } intent_evaluator = IntentResolutionEvaluator( model_config=model_config, threshold=3, ) intent_result = intent_evaluator(**converted_data) Evaluation Question #2: Did it call the right tools? Next, we inspect tool behavior using ToolCallAccuracyEvaluator. Python from azure.ai.evaluation import ToolCallAccuracyEvaluator tool_evaluator = ToolCallAccuracyEvaluator( model_config=model_config, threshold=3, ) tool_result = tool_evaluator(**converted_data) Why does this matter? If the agent selected the correct function but supplied the customer's name ("Alex Johnson") where the API expected an ID ("CUST-1001"), it fails. A language model could still generate an extremely convincing final response to cover this up. Agent behavior is part of the software surface we need to test. Tool Accuracy Is Not the Same as Tool Order Suppose the agent makes these valid calls: lookup_customer ➔ create_upgrade_request ➔ check_upgrade_eligibility. The tools and parameters are correct, but the sequence violates our business process. That is why I wouldn't use Tool Call Accuracy alone. For sequencing, Foundry provides Task Navigation Efficiency, which compares the actual sequence against an expected sequence using three modes: exact_match: Requires the exact same content and order. (Useful for strict payment workflows).in_order_match: Permits extra exploratory steps while preserving the expected order of mandatory steps.any_order_match: Expected steps can occur in any order. (Useful for research tasks). This flexibility proves that evaluation matching isn't just an AI decision—it's a business-process decision. One Successful Demo Is Not Enough A successful demonstration proves the agent worked once. Production readiness asks: How consistently does it work across model upgrades, API changes, and prompt tweaks? AIAgentConverter can prepare thread data for batch evaluation: Python filename = os.path.join(os.getcwd(), "agent_evaluation_data.jsonl") converter.prepare_evaluation_data( thread_ids=[thread_id_1, thread_id_2, thread_id_3, thread_id_4], filename=filename, ) evaluators = { "intent_resolution": IntentResolutionEvaluator(model_config=model_config), "tool_call_accuracy": ToolCallAccuracyEvaluator(model_config=model_config), } from azure.ai.evaluation import evaluate results = evaluate( data=filename, evaluation_name="travel-agent-regression", evaluators=evaluators, azure_ai_project=os.environ["AZURE_AI_PROJECT"], ) My Favorite Evaluation Cases Come From Failures When a customer discovers an edge case — like the agent submitting an action before validating eligibility — I don't just fix the prompt. I turn it into a permanent regression case: JSON { "query": "Upgrade Alex Johnson's flight if he is eligible.", "expected_actions": [ "lookup_customer", "check_upgrade_eligibility", "create_upgrade_request" ] } Over time, your evaluation dataset becomes a history of: "Things we have learned that this agent must never get wrong again." How I Now Think About Agent Evaluation My mental model for architecture discussions has shifted to this layered approach: Plain Text USER INTENT | v +---------+ | AGENT | +----+----+ | +-------------+-------------+ | | | v v v INTENT PROCESS RESPONSE | | | | +------+------+ | | | | | | v v v v v Understand Tool Input Order Quality Choice Intent resolution: Did it understand what the user wanted?Tool selection: Did it choose the right tool?Tool input accuracy: Were the parameters correct?Tool call success: Did the execution succeed?Tool output utilization: Did it correctly use the returned result?Task navigation efficiency: Did it follow the sequence?Response quality: Was the final answer useful and grounded? Where Microsoft Agent Framework Fits While AIAgentConverter is documented in Foundry's classic Agent Service workflow, newer applications built using the Microsoft Agent Framework can integrate Foundry evaluation more directly through FoundryEvals. A simplified current pattern looks like this: Python import os from azure.identity.aio import AzureCliCredential from agent_framework import Agent, evaluate_agent from agent_framework.azure import FoundryChatClient from agent_framework.foundry import FoundryEvals async def evaluate_travel_agent(): credential = AzureCliCredential() chat_client = FoundryChatClient( project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"), credential=credential, ) agent = Agent( client=chat_client, name="travel-upgrade-agent", instructions=( "Help customers with flight upgrades. " "Always verify eligibility before submitting an upgrade." ), tools=[lookup_customer, check_upgrade_eligibility, create_upgrade_request], ) query = "Upgrade Alex Johnson's AA245 flight to business class if he is eligible." response = await agent.run(query) evaluators = FoundryEvals( client=chat_client, evaluators=[ FoundryEvals.INTENT_RESOLUTION, FoundryEvals.TOOL_CALL_ACCURACY, FoundryEvals.TASK_NAVIGATION_EFFICIENCY, ], ) results = await evaluate_agent( agent=agent, responses=response, queries=[query], evaluators=evaluators, ) for result in results: print(f"Status: {result.status}") print(f"Passed: {result.passed}/{result.total}") print(f"Report: {result.report_url}") (Note: Agent evaluation SDKs are evolving rapidly. Validate against the current Microsoft Learn documentation and your installed SDK version before production implementation.) The Real Customer Question Was Trust Looking back, what started this entire line of thinking wasn't really an SDK question. It was a customer asking, implicitly: “How comfortable should I be allowing this agent to take action in my business?” And I realized I could not answer that question simply by looking at the final response. I needed to understand: Plain Text What did it understand? What did it call? What did it send? What came back? What did it do next? Did it follow the process? Did it obey the business rules? That is why trajectory evaluation has become such an important part of how I think about agent architecture. The Path Is Part of the Product For a chatbot, the final answer may be the primary product. For an agent, I increasingly think: The path is part of the product. When an agent starts calling APIs, modifying systems, creating transactions, triggering workflows, or taking enterprise actions, evaluating only the final response is no longer enough. We need to evaluate behavior. For me, that is the real value behind capabilities such as: JSON evaluation_dimensions = [ "Intent Resolution", "Tool Call Accuracy", "Tool Selection", "Tool Input Accuracy", "Tool Output Utilization", "Tool Call Success", "Task Navigation Efficiency"] And it is why I think tools such as AIAgentConverter, the Azure AI Evaluation SDK, Microsoft Foundry evaluators, and Microsoft Agent Framework deserve a place in the architecture conversation much earlier than the final production-readiness review. Because before I tell a customer: “Yes, I think this agent is ready,” I want to be able to answer one additional question: “Do we know what it actually did?” That, to me, is where evaluating agents becomes much more interesting than simply evaluating answers.

QuestDB is well-suited for applications necessitating high-performance ingestion and SQL access to time-oriented data. It is especially useful in financial markets, real-time analytics, observability, telemetry, and operational systems where data arrives continuously and must be queried with low latency. Unlike general-purpose databases, QuestDB is purpose-built with time as a core element of both storage and query processes. QuestDB appeals to enterprise developers by combining a time-series architecture with a familiar SQL interface. This technique reduces the learning curve for teams experienced in relational querying while providing a database optimized for append-heavy, chronological workloads. For systems needing recent-state queries, historical analysis, trend detection, and rapid ingestion, QuestDB delivers a practical solution. Why QuestDB Matters QuestDB excels when applications require both high ingestion rates and fast analytical queries on recent and historical data. This is essential for workloads with continuous data streams where rapid business response is critical, such as market prices, trading activity, telemetry, infrastructure metrics, clickstream data, operational events, and real-time business indicators. In these cases, the database must efficiently manage append-heavy data while supporting intuitive SQL queries across time ranges. As a result, QuestDB is well suited for financial platforms, observability systems, logistics, energy, IoT, and real-time analytics. For example, a trading system can query both the latest prices and historical intervals, an operations platform can analyze recent latency and throughput, and a logistics application can review vehicle or delivery telemetry over time. Its SQL-oriented model is especially valuable for enterprise teams, offering a familiar query language alongside an architecture designed for time-series workloads. Hands-On: Java With QuestDB We will build a simple Java application using QuestDB as the time-series database. For development, the easiest way to begin is with Docker: Shell docker run -d \ --name questdb-instance \ -p 9000:9000 \ questdb/questdb:10.0.1 QuestDB provides its Web Console and HTTP endpoint on port 9000. Once the container is running, you can open http://localhost:9000 in your browser to view the data. Configuring the Application Eclipse JNoSQL uses Jakarta APIs such as CDI and JSON-B, along with Eclipse MicroProfile Config. These APIs are available in popular Java runtimes including Helidon, Quarkus, Payara, and Open Liberty. Add the QuestDB driver dependency: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-questdb</artifactId> <version>${jnosql.version}</version> </dependency> Next, configure the Time Series database and QuestDB connection in microprofile-config.properties: Properties files jnosql.timeseries.database=qdb jnosql.questdb.url=ws::addr=localhost:9000 The qdb value specifies the logical database for the JNoSQL Time Series manager. The QuestDB URL uses the connection format required by the driver. Modeling Sensor Data For this example, we will use a simple sensor reading model: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the measurement time, while the other fields describe the sensor reading at that moment. You can also expose the entity through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate You can now insert a sequence of readings and query both the latest state and recent history: Java public class App { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query retrieves the latest reading for sensor-01. The second retrieves a limited history, skipping the most recent observation. This approach better fits typical time-series use cases than retrieving records by identifier. Using Jakarta Data You can achieve the same functionality using a Jakarta Data repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Notably, QuestDB’s time-series capabilities are available through the same Java programming model as other NoSQL time-series drivers. This lets the application focus on temporal queries such as latest state, ordering, and recent history, while the driver handles database-specific details. Why SQL Matters for QuestDB Adoption QuestDB stands out for its time-series specialization and SQL-oriented approach. When organizations adopt specialized databases, teams often struggle to learn new query languages and mental models. By providing a familiar SQL interface, QuestDB helps reduce these adoption barriers. Teams are already familiar with filtering, ordering, grouping, aggregation, and limiting result sets. Keeping these concepts allows programmers to focus on time-based challenges instead of learning a new query system. This is especially valuable when multiple roles need access to the same data. Financial systems illustrate this well. Market data, exchange rates, trades, and price movements are time-oriented, and many financial professionals already use SQL extensively. QuestDB lets teams leverage existing SQL expertise while gaining from a database optimized for high-frequency ingestion and time-based analysis. This advantage also applies to observability and operational analytics. Teams often need to query metrics for example request latency, throughput, error rates, or customer activity over specific intervals. Using familiar SQL constructs makes it easier to access application data for analysis. QuestDB does not function exactly like a traditional relational database. Its architecture and optimizations target time-series workloads. The result is an approachable query experience combined with a storage engine designed for specialized performance. For enterprises, this combination is powerful: specialized time-series performance without requiring teams to abandon the familiar query model. Conclusion QuestDB is a strong fit for applications that need fast ingestion, recent-state queries, historical analysis, and SQL over continuously changing data. With Eclipse JNoSQL 1.1.18, Java developers can use these capabilities through familiar Jakarta APIs, keeping the application model consistent while QuestDB handles the time-series specialization underneath.

By Otavio Santana CORE

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and requires efficient querying. Typical use cases include monitoring temperature, pressure, vibration, energy usage, machine status, and device telemetry. This approach also applies to manufacturing, smart infrastructure, fleet monitoring, utilities, and edge computing. IoTDB stands out for its focus on large-scale time-series workloads from devices and industrial systems. It is purpose-built for high-frequency data ingestion, historical analysis, and time-based queries. This makes it ideal for applications that require insight into both the current state and historical trends of devices or processes. Why Apache IoTDB Matters Apache IoTDB excels at managing large volumes of device-generated data over time. It addresses issues beyond storage, such as continuous data ingestion, efficient organization by device and timestamp, and fast queries for both recent and historical values. This makes IoTDB well-suited for systems with temporal, device-oriented data models. Common use cases include industrial IoT, smart factories, energy monitoring, connected vehicles, building automation, predictive maintenance, and edge computing. For example, manufacturers can track machinery data, utilities can analyze energy consumption across many meters, and fleet operators can monitor location and engine telemetry. In each scenario, IoTDB enables applications to determine current status, review device behavior over time, and detect measurements outside expected ranges. Hands-On: Java With Apache IoTDB We will build a simple Java application using Apache IoTDB. For simplicity, IoTDB will run locally in Docker. Shell docker run -d \ --name iotdb-instance \ -p 6667:6667 \ -e dn_rpc_address=0.0.0.0 \ apache/iotdb:2.0.11-standalone Next, create the database using IoTDB’s Table SQL dialect: Shell docker exec -it iotdb-instance \ /iotdb/sbin/start-cli.sh \ -h 127.0.0.1 \ -p 6667 \ -u root \ -pw root \ -sql_dialect table \ -e "CREATE DATABASE IF NOT EXISTS jnosql" For this local example, we use the default root/root credentials and disable client redirection. These settings are suitable for single-node development only. In production, use secure credentials and configure redirection based on your IoTDB cluster topology. Configuring the Application Next, configure the connection in microprofile-config.properties: Properties files jnosql.timeseries.database=jnosql jnosql.iotdb.host=localhost jnosql.iotdb.port=6667 jnosql.iotdb.username=root jnosql.iotdb.password=root jnosql.iotdb.enable.redirection=false Eclipse JNoSQL is built on Jakarta APIs such as CDI, JSON-B, and Eclipse MicroProfile Config. These APIs are supported by popular runtimes including Helidon, Quarkus, Open Liberty, and other Jakarta EE-compatible environments. Then add the Apache IoTDB driver: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-iotdb</artifactId> <version>${jnosql.version}</version> </dependency> Modeling Sensor Data A sensor-oriented model aligns well with IoTDB: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the timestamp of the reading. The other fields capture the measurements at that time. The entity can also be exposed through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate With the infrastructure in place, insert several readings and retrieve both the latest value and recent history: Java "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query answers a common IoT question: what is the latest reading from this sensor? The second retrieves its recent history, excluding the latest observation. Using Jakarta Data The same use case can be implemented using the repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Both approaches model the domain in terms of latest state and historical observations. This is where a time-series database like Apache IoTDB is more effective than using a general-purpose database with timestamps as regular fields. Why Apache IoTDB Is More Than Sensor Storage While Apache IoTDB is often associated with IoT, its capabilities go well beyond basic sensor data storage. It excels when organizations must manage large volumes of device-centric data over time while retaining relationships among measurements, devices, and timestamps. As a result, IoTDB is valuable for industrial systems, utilities, transportation, smart infrastructure, manufacturing, and edge computing. For example, in predictive maintenance, a machine may continuously report vibration, temperature, pressure, energy consumption, and operating condition. The true value is not only found in the latest readings, but in studying how these measurements change together over time. Maintenance systems may compare recent data with historical trends, spot anomalies before failures, or correlate changes across multiple devices. This requires more than simple sensor data storage. This approach also benefits energy systems, where smart meters, solar panels, batteries, and grid equipment generate continuous measurement streams that require time-based analysis. In transportation, vehicles produce ongoing data such as speed, location, fuel consumption, battery status, and engine telemetry. Similarly, smart buildings rely on time-series data from HVAC systems, occupancy sensors, and energy meters. IoTDB is especially effective when system architecture is organized around devices and their measurements. It can serve as the historical data layer for operational systems, while other technologies manage transactional or business information. For example, a relational database might store details about a machine, its owner, contracts, or maintenance schedules, while IoTDB manages the extensive measurement data the machine generates. This separation lets enterprise applications treat telemetry as a primary workload, rather than forcing high-frequency device data into models meant for business entities and transactions. Conclusion Apache IoTDB is a strong fit for applications that need to ingest and analyze device-generated data over time, especially in IoT, industrial, energy, and edge scenarios. With Eclipse JNoSQL 1.1.18, Java developers can access these capabilities through familiar Jakarta APIs, keeping the application model consistent while still leveraging IoTDB’s time-series specialization.

By Otavio Santana CORE

The Illusion of a Single Database In a traditional monolithic application, maintaining data consistency is straightforward. If you need to create a new order and update warehouse inventory, you wrap the logic inside a single database transaction: Java @Transactional public void placeOrder(OrderRequest request) { orderRepository.save(request.toOrder()); inventoryRepository.decrementStock(request.getItemId(), request.getQuantity()); } If the inventory update throws an exception, the relational database rolls back the entire transaction. Either both operations succeed, or neither does. In a distributed microservices architecture, that safety net disappears. When your OrderService writes a record to a local PostgreSQL database and immediately publishes an event to an Apache Kafka cluster to notify the InventoryService, you are dealing with two completely independent, non-atomic systems. The Catastrophic Failure Modes When you attempt to write to a local database and publish an event within the same business method, one of two failures will inevitably occur: Scenario A (Database First, Network Fails) [Save to Database: SUCCESS] ──> [Network / Kafka Outage: FAILS] Result: Order exists in database, but downstream services are never notified. Scenario B (Publish First, Database Fails) [Publish to Kafka: SUCCESS] ──> [Database Unique Constraint Violation: FAILS] Result: Downstream services charge payment or pack inventory for an order that was never saved. Distributed two-phase commit (2PC) protocols are notoriously slow, fragile, and rarely supported across modern cloud-native message brokers. To achieve guaranteed consistency without blocking throughput, the industry-standard architecture is the Transactional Outbox Pattern. The Blueprint: The Transactional Outbox Pattern Instead of trying to speak to two external systems at once, the microservice performs all its operations within a single, local database boundary. Plain Text [Incoming Request] │ ▼ ┌───────────────────────────────────────────────────────────┐ │ Local ACID Transaction │ │ ├── 1. Insert into orders table │ │ └── 2. Insert into outbox_events table │ └───────────────────────────────────────────────────────────┘ │ ▼ [Outbox Relay / Change Data Capture (CDC)] │ ▼ [Message Broker: Kafka Topic] Atomic local write: The application saves the domain entity (orders) and a corresponding event payload into an outbox_events table inside the exact same local @Transactional boundary.Asynchronous relay: An independent background process reads the outbox table and publishes the messages to Kafka.Acknowledgment: Once Kafka acknowledges receipt, the relay marks the outbox event as published or removes the row. 1. Database Schema for the Outbox Define a dedicated outbox table designed for high-throughput polling and sequential reads: SQL CREATE TABLE outbox_events ( id UUID PRIMARY KEY, aggregate_type VARCHAR(255) NOT NULL, aggregate_id VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, processed BOOLEAN DEFAULT FALSE, processed_at TIMESTAMP WITH TIME ZONE ); CREATE INDEX idx_outbox_unprocessed ON outbox_events (created_at) WHERE processed = FALSE; 2. The Application Layer: Atomic Persistence The Spring service writes both the domain entity and the outbox event in one atomic operation: Java package com.example.outbox.service; import com.example.outbox.dto.OrderRequest; import com.example.outbox.entity.Order; import com.example.outbox.entity.OutboxEvent; import com.example.outbox.repository.OrderRepository; import com.example.outbox.repository.OutboxRepository; import com.fasterxml.jackson.databind.ObjectMapper; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import java.time.Instant; import java.util.UUID; @Service public class OrderService { private final OrderRepository orderRepository; private final OutboxRepository outboxRepository; private final ObjectMapper objectMapper; public OrderService(OrderRepository orderRepository, OutboxRepository outboxRepository, ObjectMapper objectMapper) { this.orderRepository = orderRepository; this.outboxRepository = outboxRepository; this.objectMapper = objectMapper; } @Transactional public void createOrder(OrderRequest request) { // 1. Persist domain entity Order order = new Order(UUID.randomUUID(), request.getCustomerId(), request.getTotalAmount()); orderRepository.save(order); // 2. Persist outbox event inside the exact same transaction try { String jsonPayload = objectMapper.writeValueAsString(order); OutboxEvent outbox = new OutboxEvent( UUID.randomUUID(), "ORDER", order.getId().toString(), "ORDER_CREATED", jsonPayload, Instant.now(), false ); outboxRepository.save(outbox); } catch (Exception e) { throw new RuntimeException("Failed to serialize outbox event payload", e); } } } 3. The Relay Layer: Reliable Kafka Publishing An asynchronous background worker polls the unprocessed records in the outbox and delivers them to the broker: Java package com.example.outbox.relay; import com.example.outbox.entity.OutboxEvent; import com.example.outbox.repository.OutboxRepository; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; import org.springframework.transaction.annotation.Transactional; import java.time.Instant; import java.util.List; @Component public class OutboxMessageRelay { private static final Logger log = LoggerFactory.getLogger(OutboxMessageRelay.class); private final OutboxRepository outboxRepository; private final KafkaTemplate<String, String> kafkaTemplate; public OutboxMessageRelay(OutboxRepository outboxRepository, KafkaTemplate<String, String> kafkaTemplate) { this.outboxRepository = outboxRepository; this.kafkaTemplate = kafkaTemplate; } @Scheduled(fixedDelayString = "${app.outbox.poll-interval-ms:2000}") @Transactional public void publishPendingEvents() { List<OutboxEvent> pendingEvents = outboxRepository.findTop50ByProcessedFalseOrderByCreatedAtAsc(); for (OutboxEvent event : pendingEvents) { try { // Publish using aggregateId as the Kafka partition key to preserve message ordering kafkaTemplate.send("orders-events", event.getAggregateId(), event.getPayload()) .whenComplete((result, ex) -> { if (ex == null) { event.setProcessed(true); event.setProcessedAt(Instant.now()); outboxRepository.save(event); log.info("Successfully relayed outbox event: {}", event.getId()); } else { log.error("Failed to relay event to Kafka: {}", event.getId(), ex); } }); } catch (Exception e) { log.error("Synchronous dispatch failure for outbox event: {}", event.getId(), e); break; // Halt batch progression to preserve order } } } } 4. Production Engineering Guardrails Preserve Partition Ordering Always use the aggregate_id (such as order_id) as the partition key when publishing the Kafka message. This ensures all state transitions for a single business entity land on the exact same Kafka partition and are processed in strict sequence. Polling vs. Log-Based CDC (Debezium) Scheduled polling is easy to set up and ideal for small-to-medium systems. For high-volume enterprise platforms processing thousands of writes per second, replace database polling with Change Data Capture (CDC) engines like Debezium. Debezium reads the database write-ahead log (WAL) directly, streaming changes to Kafka with zero query overhead on the operational database. Idempotency on the Consumer The Transactional Outbox pattern guarantees at-least-once delivery. If the relay publishes an event to Kafka but crashes before marking the outbox row as processed, it may re-send the message upon restart. Downstream consumers must maintain an idempotency check (e.g., tracking processed message IDs in Redis) to discard duplicates. Architectural Strategy Matrix Dimension Dual-Write (@Transactional + Kafka) Distributed 2PC Transactional Outbox Pattern Data Consistency Broken (Silent inconsistencies) Strong Strong (Eventual consistency) System Latency Low High (Blocking locks) Ultra-low local execution Broker Resilience Fragile (Network crashes drop events) Low High (Decoupled publishing) Operational Simplicity Deceptively simple Complex Straightforward Summary In distributed systems, atomicity cannot cross network boundaries. When you attempt to update a local database and publish a message to an event bus inside the same method, failure is a mathematical certainty over time. By shifting to the Transactional Outbox Pattern, you leverage the battle-tested ACID guarantees of your relational database to capture domain state and outbound events simultaneously. This eliminates the dual-write anti-pattern, guarantees at-least-once delivery, and builds a dependable bridge between relational transactions and event-driven architecture.

OpenSearch is an open-source, distributed search and analytics suite derived as a fork of Elasticsearch and maintained under the Apache 2.0 license. When it comes to memory configuration, the guidance is often reduced to a few rules of thumb: swapoff -a, vm.swappiness=1, or bootstrap.memory_lock, and allocating 50% of available memory to the JVM heap while leaving the rest for Lucene and the filesystem page cache, OpenSearch off-heap caches, network buffers, and other system needs. These recommendations are repeated throughout documentation, blog posts, and operational guides, yet their origins and the mechanisms that justify these specific values are rarely examined. Undoubtedly, they provide a reasonable and safe starting point or a safe upper bound in most of the cases, but a safe default is not necessarily an optimal configuration. All of this raises even more questions. How do these defaults affect cluster performance? What is the optimal JVM heap ratio? Does memory given up by the JVM actually become filesystem page cache, and at what point does that trade-off stop paying off? How do read/write latency correlate with the heap ratio? These questions become particularly important in resource-constrained environments and in the cloud, where long-term contracts may make existing instances significantly cheaper, making horizontal or vertical scaling a difficult decision. In this article, we'll try to answer these questions through benchmarking. This is Act 1 of a two-act series. Act 1 focuses on identifying the cause of the latency problems we observed with the current defaults. Act 2 will explore what other heap-ratio values might look like for read/write loads. Along the way, I'll share the tools and commands used throughout the investigation, making this article a practical reference as well for you and for myself when I inevitably need to retrace the investigation months later. Knowledge Context The story also crosses several boundaries, such as the Kernel VM, the JVM, and Lucene. So, it’s important to outline the concepts mentioned in this part of the article beforehand, both for the context and, optionally, to enrich the AI context if you'd like to summarize everything. AreaWhere memory livesWhy it matters hereLinux page cacheFile-backed RAMLucene relies heavily on it for index data; under memory pressure, these pages can be reclaimed and read again later.Linux swapDisk-backed anonymous memoryAnonymous process memory can be swapped out under pressure. vm.swappiness influences this decision but does not prohibit it.Linux PSIKernel pressure signalShows time tasks spend stalled due to CPU, memory, or I/O pressure. We'll use I/O PSI while investigating latency.JVM heapAnonymous memoryControlled by Xms/Xmx; contains Java objects and several OpenSearch data structures.JVM native memoryAnonymous/file-backed memory outside XmxIncludes code cache, metaspace, stacks, direct buffers, and native allocations. Heap metrics do not account for all of it.OpenSearch cachesHeap/off-heap, depending on cacheTheir sizes may depend on heap size, which becomes important when we change jvm_heap_ratio.OpenSearch indexing bufferHeapIts size depends on heap and therefore becomes an important variable in Act 2.Lucene mmapFile-backed/page cacheLucene index files mapped into the process do not consume JVM heap; resident pages compete for physical RAM. Environment I used Aiven for OpenSearch on Azure, with cluster metrics exported to Thanos. The OpenSearch Benchmark metrics don't provide everything we need to answer our questions, particularly host-level metrics such as Linux PSI and swap activity. Exporting the cluster metrics to Thanos allows us to use PromQL queries later to retrieve the additional metrics needed for the investigation. The cluster consists of 3 nodes: CPUAMD EPYC 7763v (Milan)vCPU / RAM2 vCPU, 8 GiBDisk Size175 GiB per nodeAzure Regionazure-westeuropeAzure DiskPremiumV2_LRSAzure SKUStandard_D2as_v5 OpenSearch Version3.6.0JDKjava-21-openjdk-headlessGCG1GC Act 1. The Latency and an Extra GB The symptom: elevated query latency across the cluster, first reported by the customer after a kernel and Azure image upgrade. The load pattern on OpenSearch itself remained unchanged, as did the cluster configuration and settings. The monitoring panels give us the first clue. In the screenshots below, the green vertical line marks the moment of the upgrade. After that point, the page cache grows by roughly a gigabyte, while I/O PSI, previously close to zero, starts showing significant spikes. Nothing crashed, no alert fired, and from the JVM point of view everything looked normal. So where did that extra gigabyte of page cache come from? Nothing was actually freed. It moved. The interesting part isn't just that memory went to swap; it's which memory. When you have thousands of running clusters, there is always a small fraction of them operating close to the edge: relatively stable, yet sensitive enough that even a small change can noticeably affect performance. Like a star nearing the end of its lifetime, they may look stable right up until something disturbs the balance. The immediate cause of the page-cache change was identified fairly quickly: Azure applies tuning parameters that differ from the Linux kernel defaults, and those parameters were not applied by the older image. Once applied, the larger buffers and read_ahead increased the filesystem cache footprint, putting additional pressure on anonymous memory and eventually pushing some of it to swap. But rather than stopping there, let's use this incident as an opportunity to experiment with the heap ratio and make the behavior of OpenSearch instances explicit and less dependent on such environmental changes. Evidence It Is on Swap; None of It Locked First, find what is going on on a node itself: Shell PID=$(pgrep -f 'org.opensearch.bootstrap.OpenSearch') grep -E 'VmRSS|RssAnon|RssFile|RssShmem|VmSwap|VmLck' /proc/$PID/status Shell VmRSS: 4366408 kB # resident RssAnon: 3329896 kB # heap + anonymous native RssFile: 1036496 kB # resident mmap'd Lucene pages RssShmem: 16 kB VmSwap: 2679496 kB # on swap VmLck: 0 kB # bootstrap.memory_lock=fasle, none locked Shell grep -E 'MemFree|MemAvailable|Cached|SwapFree' /proc/meminfo Shell MemFree: 258216 kB MemAvailable: 3479264 kB Cached: 3403196 kB SwapCached: 854396 kB SwapFree: 4745444 kB The Swap Device Is dm-crypt Then confirm there is somewhere for it to go, and on what kind of device: Shell swapon --show Plain Text NAME TYPE SIZE USED PRIO /dev/dm-4 partition 8G 3.5G -1 The dm-* swap device is the interesting detail to catch and to keep in mind. This is an encrypted device, so once a page is requested it could drive more I/O -> more dm-crypt allocations -> more high-order pressure. A self-reinforcing loop and a good example of read amplification. Paging Is Live, Not Historical The next logical step is to check whether it is live paging or just a stale historical tail. The vmstat 1 5 the Linux Virtual Memory Statistics Tool should give us an exact answer for this, where non-zero swap blocks in and out (marked as si, so): Shell vmstat 1 5 Plain Text procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 1 0 3620928 159716 5864 3704212 113 80 3361 577 3147 12 8 7 84 1 0 0 0 0 3625240 189036 5860 3705260 0 5228 24268 5228 4841 4072 14 12 69 4 0 0 0 0 3625240 166196 5860 3710696 0 0 32 0 3572 2556 21 5 74 0 0 0 0 0 3625240 162668 5860 3716200 8 0 164 0 3729 2619 20 7 73 0 0 0 12 0 3625312 147692 5860 3724288 0 140 9207 524 3679 4024 22 14 63 1 0 0 Non-zero si/so in 4 of 5 samples show the live paging process, swpd also climbing across five seconds, which is good proof. The Swapped Pages Are Anonymous, Not File-Backed Let's also check swap memory consumption for each of the process's mappings, to confirm that swap is heap-related: Shell PID=$(pgrep -f org.opensearch.bootstrap.OpenSearch) awk '/^[0-9a-f]/{h=$0} /^Swap:/{if($2>0)print $2" kB "h}' /proc/$PID/smaps | sort -rn | head -20 Plain Text 1160380 kB 708400000-7ffe00000 rw-p 00000000 00:00 0 62296 kB 7f30c0000000-7f30c3f4b000 rw-p 00000000 00:00 0 61452 kB 7f1fc8000000-7f1fcbc03000 rw-p 00000000 00:00 0 61448 kB 7f1fa8000000-7f1fabc02000 rw-p 00000000 00:00 0 61444 kB 7f2f38000000-7f2f3bc01000 rw-p 00000000 00:00 0 61444 kB 7f24b4000000-7f24b7c01000 rw-p 00000000 00:00 0 61444 kB 7f22ac000000-7f22afc01000 rw-p 00000000 00:00 0 61444 kB 7f2238000000-7f223bc01000 rw-p 00000000 00:00 0 61444 kB 7f216c000000-7f216fc01000 rw-p 00000000 00:00 0 61444 kB 7f2168000000-7f216bc01000 rw-p 00000000 00:00 0 61444 kB 7f2148000000-7f214bc01000 rw-p 00000000 00:00 0 61444 kB 7f1fec000000-7f1fefc01000 rw-p 00000000 00:00 0 61444 kB 7f1fe8000000-7f1febc01000 rw-p 00000000 00:00 0 61444 kB 7f1fcc000000-7f1fcfc01000 rw-p 00000000 00:00 0 61444 kB 7f1fac000000-7f1fafc01000 rw-p 00000000 00:00 0 60772 kB 7f1fc0000000-7f1fc3c14000 rw-p 00000000 00:00 0 59952 kB 7f311e000000-7f3123f41000 rw-p 00000000 00:00 0 59340 kB 7f30b4000000-7f30b7c9b000 rw-p 00000000 00:00 0 57180 kB 7f3110000000-7f3113e3e000 rw-p 00000000 00:00 0 43084 kB 7f3130800000-7f3133d70000 rwxp 00000000 00:00 0 The Largest Swapped Region Is But why is what we are seeing above a heap-related area? There are a few clues for that. The region size 0x708400000 - 0x7ffe00000 is exactly 4,154,458,112 bytes = 3,962 MiB as we use -Xms == -Xmx and the whole thing is committed at startup, and nothing else in a JVM process is a single contiguous ~4 GB anonymous rw-p mapping. Second, It's the lowest mapping in the address space, smaps_rollup [rollup] line starts at exactly 708400000: Shell cat /proc/$PID/smaps_rollup Shell 708400000-7ffd3bb56000 ---p 00000000 00:00 0 [rollup] Private_Dirty: 3213788 kB Swap: 2680532 kB SwapPss: 2679412 kB Locked: 0 kB The JIT Code Cache Is Swapped Too Decoding the top swapped regions: 1160380 kB at 708400000 – is the JVM heap, the compressed‑oops heap base and matches the [rollup] start from smaps_rollup.The dozens of 61444 kB regions – these areas are probably related to native/off‑heap: Netty, JNI, Lucene native, etc.43084 kB marked rwxp – the JIT code cache, also swapped out, a bad sign. Together, these regions account for almost exactly the ~2.5 GB of swapped memory we observed earlier: cold heap regions, native/off-heap allocations, code cache, and possibly thread stacks. Practically, this means two consequences: GC can amplify swap latency. 1 GB of the JVM heap was swapped out. G1 does not necessarily touch all of those pages during a mixed collection, but any GC phase that accesses a swapped page incurs a major fault and has to bring it back through the dm-crypt device. Hence, short GC work can produce significantly longer pauses.A swapped page can also contain executable code. The next call into a swapped-out compiled method can trigger a major fault before the code can run. The resulting latency may land on an otherwise random request and be difficult to attribute directly to GC, index I/O, or the query itself. Evidence of Sustained Anon Memory Churn workingset_refault_anon counts anonymous memory refault events after reclaim; it does not count unique pages. Together with pswpin and pswpout, it shows how much anonymous memory paging has accumulated since boot. Shell grep -E 'workingset_(refault|activate)_anon|pswpin|pswpout' /proc/vmstat Plain Text workingset_refault_anon 89061837 workingset_activate_anon 1856069 pswpin 86690374 pswpout 61618603 These counters are cumulative, so fetching them at 10-minute intervals clearly shows that this wasn't a one-time eviction. There was sustained process: nearly 89 million anonymous pages were repeatedly swapped out and faulted back in. vm.swappiness = 1 and GC Amplification In this story vm.swappiness=1 was set since the cluster's inception. It does what Linux defines it to do, but it doesn't provide the protection we wanted. I suspect this matters particularly in the most resource-constrained deployments. How do we know that? The entire result above is a counterexample. swappiness biases the kernel's choice between reclaiming file-backed and anonymous pages. It does not prevent anonymous pages from being swapped out. Even at 1, this can still happen under sustained memory pressure. On a resource-constrained node whose index is several times larger than its RAM, pressure on the filesystem cache is not an exceptional condition; this is the normal operating state. This has two important consequences: It is not self-healing. Nothing proactively pages anonymous memory back in on a schedule. A swapped-out page returns to RAM only when it is accessed again, and cold memory, as it's defined, may remain untouched for a long time. As a result, the cold JVM memory can remain in swap indefinitely.It may remain invisible until something touches it. At steady state, a cold tail of the heap can remain in swap without producing obvious symptoms. The problem becomes visible when those pages are touched again, causing major page faults and potentially amplifying GC and request latency. Key Takeaways So, the root cause of the latency problems is an oversized heap combined with page cache pressure (the cold heap tail has been swapped out). vm.swappiness=1 did not protect the heap. It biases what gets reclaimed; it does not prevent anonymous memory from being swapped out.vm.swappiness=1 should not be relied on with the other defaults in production. An oversized heap can lead to GC amplification that is difficult to detect.Most of the swapped-out memory wasn't heap at all, but malloc arenas and the JIT code cache, none of it inside Xmx, so heap metrics didn't show it.Swap on dm-crypt exacerbates the issue, resulting in longer GC pauses and random request latency spikes. dm-crypt may be unavoidable in production due to security requirements. The fix isn't another swappiness tweak. It's two things: stop committing heap you don't use, and make the heap you do commit non-evictable. Follow Up In Act 2, we'll answer the remaining questions raised at the beginning of this article and look more closely at the trade-off introduced by bootstrap.memory_lock. This setting makes the heap resident and swap-immune, but it also turns jvm_heap_ratio from a soft default into a permanent memory commitment. The question then becomes: what heap ratio best suits different read and write workloads? There is one more complication to mention in advance: in a resource-constrained environment, merge storms can distort benchmark results, making an otherwise good heap ratio appear poor. See the screenshot below.

Today, every vendor offering BI solutions has incorporated a chat box. Whether you use Copilot or some other natural-language interface that connects you to a data warehouse, just ask a question in simple words, and it will generate SQL automatically. While this is conducive to productivity in other industries, in banking it presents an opportunity for a new attack. It is not enough to simply say that wrong SQL can be produced. It’s that an ungoverned text-to-SQL layer may join tables it shouldn’t, return columns that should have been masked. As a result of a lack of oversight, a marketing analyst could receive a query containing raw account numbers, since none of the components of the stack told the system to do otherwise. Prompt-level guardrails (“please don’t show PII”) are not a security control. They’re just a suggestion, and a model under adversarial pressure ot just a confusing prompt will ignore a suggestion. The issue isn't just about providing a more intelligent prompt; rather, it's about placing the artificial intelligence assistant on the same layer of information as human analysts, meaning an environment where the database manages the relevant security details as per row and column criteria instead of relying on technology. Consequently, if the analyst does not have access to the specific column, there is no justification for the AI assistant to have access to it as well. The diagram below (Figure 1) shows the steps taken to build that layer in BigQuery: a validated semantic layer that allows human dashboards and AI-generated queries to be connected to the same quality definitions. This would allow the assistant to leverage existing security rather than creating it. Figure 1. The Semantic Layer Resolver Step 1: Stop Letting Anyone (Human or AI) Query Raw Tables The initial phase is architectural, not related to AI: there is no query made by a person or a system involving the base tables. Instead, each of the metrics that are accessible to consumers is defined at least once in a BigQuery view, and its calculation logic is embedded in that view. SQL -- Certified metric: Risk-Weighted Assets, defined once, queried everywhere CREATE VIEW analytics.risk_weighted_assets AS SELECT exposure.customer_id, exposure.region, exposure.exposure_class, exposure.outstanding_balance, risk_weights.weight_pct, ROUND(exposure.outstanding_balance * risk_weights.weight_pct / 100, 2) AS rwa_amount, CURRENT_TIMESTAMP() AS calculated_at FROM finance.exposures AS exposure JOIN reference.basel_risk_weights AS risk_weights ON exposure.exposure_class = risk_weights.exposure_class WHERE exposure.status = 'ACTIVE'; The view of "risk-weighted assets" created through a dashboard, a notebook, and an LLM agent is identical. There is no alternative version in a researcher’s spreadsheet, nor can an AI agent "helpfully" recreate the calculation based on exposure tables but use incorrect risk weightings. Step 2: Enforce Security at the Data Layer, Not the Application Layer It is important to ensure that BigQuery includes row-level and column-level security and associates it with the table. This means the principle will work irrespective of the entity making the query. SQL -- Row-level security: a regional analyst only ever sees their region's rows CREATE ROW ACCESS POLICY regional_filter ON analytics.risk_weighted_assets GRANT TO ('group:[email protected]') FILTER USING (region = 'EMEA'); Column masking works the same way, through policy tags rather than per-report logic: YAML # Dataplex policy tag: applied once, enforced everywhere the column is queried taxonomy: financial-pii policyTags: - displayName: "customer-account-number" description: "Masked for all roles except fraud-investigation" - displayName: "customer-ssn" description: "Masked for all roles except compliance-audit" Once a policy tag is applied to a column, a user, or any AI agent acting under that user's identity, who doesn’t have the appropriate fine-grained reader role, will receive either a null value or a hashed value. There is no mistake that the model has to circumvent; it’s simply a different result. This is what makes querying with AI safe, since whatever the query for the AI is, it cannot reveal anything that the column policy prohibits. Step 3: Give the Grounding Layer Metadata to Query Against It is impossible for an LLM to adhere to rules of governance it knows nothing about. Accordingly, a metadata directory is necessary for the semantic layer that contains a description of each certified metric with enough detail for the agent to turn an inquiry posed in natural language into the correct interpretation and filtering process, not simply provide it with a raw schema dump. JSON { "metric_id": "risk_weighted_assets", "display_name": "Risk-Weighted Assets", "view": "analytics.risk_weighted_assets", "owner": "[email protected]", "sensitivity": "internal", "allowed_dimensions": ["region", "exposure_class", "customer_id"], "definition": "Balance times Basel risk weight, summed by class.", "lineage": ["finance.exposures", "reference.basel_risk_weights"], "last_certified": "2026-06-01" } This record is the thing the AI agent actually reads. The document specifies which view will be interrogated, lists the dimensions available for filtering results, and identifies who to contact if something goes wrong. It is worth mentioning that in this record there is no schema given for the finance exposes table, which leaves the model nothing to "discover" about. Step 4: Route Natural-Language Requests Through the Semantic Layer, Not the Warehouse When the certified metrics with their metadata have been obtained, the resolution process consists of transforming the user's natural-language question into a query that uses an allowed view rather than directly referring to the underlying schema. Python class SemanticLayerResolver: def __init__(self, metric_catalog, bq_client): self.catalog = metric_catalog # metric_id -> metadata, from Step 3 self.bq_client = bq_client def resolve(self, nl_request: str, user_identity: str) -> QueryResult: # 1. Map the request to a certified metric, never to a raw table. # A constrained classifier over self.catalog.keys() works better # here than open-ended text-to-SQL against the full warehouse. metric = self.match_metric(nl_request) if metric is None: return QueryResult.refuse("No certified metric found.") # 2. Extract filters, restricted to the metric's allowed_dimensions. filters = self.extract_filters( nl_request, metric["allowed_dimensions"] ) # 3. Build SQL against the certified view only. sql = self.build_query(metric["view"], filters) # 4. Execute as the requesting user, so BigQuery's row/column # security applies exactly as it would for a human query. result = self.bq_client.query(sql, user=user_identity) # 5. Attach lineage and certification metadata to the answer, # so "what the AI said" is auditable like any report. return QueryResult( data=result, metric_id=metric["metric_id"], lineage=metric["lineage"], certified_at=metric["last_certified"], ) The important line is step 4: the query is executed under the requesting user instead of using a shared service account. Hence, all the downstream access control mechanisms are automatically applied. There is no need for a separate permission system for the resolver because it has no access rights that exceed the rights of the requesting user. Step 5: Audit Every AI-Generated Query Like You Would a Human's The governance teams will not agree on a system based on the suggestion of " having faith in the model." What they approve is proof in every case where a resolver has provided information, just as is done when an individual writes a report. Python def log_ai_query(user_identity, nl_request, result: QueryResult): audit_log.write({ "user": user_identity, "request": nl_request, "metric_id": result.metric_id, "lineage": result.lineage, "policy_version": result.certified_at, "row_count": result.row_count, "timestamp": now(), }) One financial institution successfully applied this approach. What used to be a lengthy project in which one would have to analyze whether an AI assistant could access customer information has been transformed into something evaluated right away: the assistant can perform the same functions as a human worker. The financial institution was also measuring the new trend of using a certified semantic layer, not only in regard to the AI being discussed. Conflicts over defining metrics across different business lines practically vanished when the organization no longer had to create a separate “AI-compliant” data model. The Real Insight: Governance Is What Makes AI Fast, Not What Slows It Down It’s easy to assume that the best approach to deal with the LLM and sensitive data combination is to include a review step in which a human sits in on every step of the process, or another model is deployed to analyze the first model’s outputs before they are used. This is not only unscalable, but it also misses the point. Another way is to make sure that the insecure path cannot be taken, rather than simply being shunned. If the data layer implements row-level security, column masking, and certified metric definitions, you can confirm that an AI agent querying the data cannot generate queries that reveal any data previously available to someone with the same role. As a result, there is no need to verify output against constraints, since they were already included in the model. This shift is suggested by this pattern. Governed self-service, the architecture that permits a business analyst to carry out data initiatives safely in the absence of ticket submission, also creates a secure basis for AI-enhanced analysis. But it wasn't the main purpose. It is just a coincidence that it has worked out this way.