Monitoring and Observability

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboard is green enough for everyone to want the incident closed. Then somebody checks the downstream projection, account balance, inventory state, enrichment table, or audit report and finds the real problem: data moved, but state did not converge. That is the failure this article is about. Consumer lag is a progress signal, not a recovery verdict. It proves position movement. It does not prove terminal outcomes, idempotent side effects, owned failures, or source-to-target convergence. If an incident review ends with “lag went back to zero,” the team has proved movement, not recovery. Declare the Recovery Cohort First The first observability decision is not which metric to graph. It is what work the recovery run is supposed to prove. A useful cohort names: the topic, partitions, and offset rangethe event types included in the runthe business-key scopethe expected record countthe recovery reasonthe ownerand the stop conditions A compact contract object makes that scope reviewable: YAML recovery_run: id: rec-2026-09-05-01 reason: projection_drift_after_consumer_restart owner: billing-platform-oncall run_type: incident_repair source: topic: billing-events event_types: - InvoicePosted - CreditApplied offsets: - partition: 8 from: 492100 to: 492881 - partition: 9 from: 388010 to: 388522 business_scope: key_type: account_id selector: incident_account_hashes_2026_09_05 expectations: expected_record_count: 1294 reconciliation: source_of_truth: billing_ledger checks: - account_day_balance_matches - no_missing_decision_events - no_unowned_final_failures safety: dry_run_required: true max_retry_attempts: 3 stop_on_reconciliation_mismatch: true The object does not have to be beautiful. It has to be unambiguous. During an incident, ambiguity is expensive: “replay the billing topic” is not a scope, and “check the dashboard” is not evidence. The cohort also forces a design decision: do you need partition/offset proof, business-key proof, or both? Offsets are excellent for Kafka lineage. Business keys are what downstream systems usually reconcile. A serious recovery plan connects them instead of pretending one replaces the other. Count Terminal Outcomes, Not Processed Records Processed count is a dangerous comfort metric. A record can be processed and still be wrong. It can be skipped correctly. It can be suppressed as a duplicate. It can be retried. It can land in a dead letter queue. It can be quarantined because replaying it would violate a business rule. The recovery run should account for outcomes explicitly: OutcomeTerminal?MeaningAPPLIEDYesThe expected side effect was performed.UPDATEDYesExisting target state was corrected or brought current.SKIPPEDYesThe record was intentionally ignored under a documented rule.DUPLICATE_SUPPRESSEDYesIdempotency prevented a repeat side effect.FAILED_RETRYABLENoProcessing failed and remains inside retry policy.FAILED_FINALYesProcessing failed and needs owner action.QUARANTINEDYesThe record is retained but should not be replayed without domain review. The key equation is simple: expected = applied + updated + skipped + duplicate_suppressed + failed_final + quarantined Retryable failures should not count as terminal until the retry policy is exhausted or the run is stopped. If the math does not close, the recovery has not been proven. Keep Metrics Bounded Recovery runs create tempting labels: run IDs, account IDs, order IDs, offset ranges, incident IDs. Put those blindly into metrics and the observability system becomes the next incident. Use metrics for bounded dimensions and put high-cardinality detail in decision logs, traces, audit tables, or exemplars. MetricBounded labelsPurposekafka_recovery_records_expectedtopic, event_type, run_typeDefines the denominator.kafka_recovery_records_unaccountedtopic, event_type, run_typeShows records not yet terminal.kafka_recovery_records_terminal_totaloutcome, event_type, run_typeShows how records completed.kafka_consumer_duplicate_suppressed_totalconsumer_group, event_type, run_typeProves replay did not duplicate side effects.kafka_recovery_reconciliation_mismatchescheck_name, run_typeBlocks false recovery success. That split matters. Metrics tell you that recovery is unhealthy at a class level. The decision record tells you which run, offsets, business keys, and owners are involved. Emit Decisions You Can Inspect A useful recovery log does not just say that a handler ran. It says why a record took a path. JSON { "recoveryRunId": "rec-2026-09-05-01", "eventId": "evt-1001", "businessKeyHash": "bkh_7f3a", "topic": "billing-events", "partition": 8, "offset": 492881, "eventType": "InvoicePosted", "outcome": "DUPLICATE_SUPPRESSED", "reason": "idempotency_key_exists", "schemaVersion": "v4", "consumerGroup": "usage-aggregator", "runType": "incident_repair", "decidedAt": "2026-09-05T18:42:31Z" } Notice what is missing: raw payloads, customer identifiers, tokens, and private business data. Recovery logs should explain decisions without becoming a second data-leak surface. The Java side can stay small. The following Java 17/Spring/Micrometer sketch is illustrative, not a complete production service. It assumes repository, meterRegistry, objectMapper, and log are class fields: Java record RecoveryDecision( String recoveryRunId, String eventId, String businessKeyHash, String topic, int partition, long offset, String eventType, RecoveryOutcome outcome, String reason, String consumerGroup, String runType, Instant decidedAt ) {} void emit(RecoveryDecision decision) { repository.insert(decision); Tags tags = Tags.of( "event_type", decision.eventType(), "run_type", decision.runType(), "outcome", decision.outcome().name() ); meterRegistry .counter("kafka_recovery_records_decision_total", tags) .increment(); if (decision.outcome() != RecoveryOutcome.FAILED_RETRYABLE) { meterRegistry .counter("kafka_recovery_records_terminal_total", tags) .increment(); } log.info("kafka_recovery_decision={}", toJson(decision)); } String toJson(RecoveryDecision decision) { try { return objectMapper.writeValueAsString(decision); } catch (JsonProcessingException error) { return "{\"serialization_error\":\"recovery_decision\"}"; } } That emitter is not the recovery system. It is the instrumentation boundary. It makes the record decision visible in the durable decision log, the metric stream, and the incident timeline. Reconciliation Is the Verdict Layer The hardest part of recovery observability is deciding what “correct” means. For some domains, a count is enough. For most serious domains, it is not. A count can match while the wrong keys, amounts, versions, or timestamps remain wrong. The source of truth must be named. The tolerance window must be named. Late arrivals, clock skew, eventually consistent sinks, and manual exceptions must be handled explicitly. Otherwise reconciliation becomes another dashboard panel that looks precise while hiding the policy decision. For example, if the recovery cohort is supposed to repair account/day billing state, compare the source ledger to the target projection for the same cohort. This is PostgreSQL-style SQL; use an equivalent pattern for your database: PLSQL WITH expected AS ( SELECT account_hash, business_date, SUM(amount_cents) AS expected_amount_cents FROM billing_ledger WHERE recovery_run_id = :recovery_run_id GROUP BY account_hash, business_date ), actual AS ( SELECT account_hash, business_date, amount_cents AS actual_amount_cents FROM billing_daily_projection WHERE recovery_run_id = :recovery_run_id ) SELECT COALESCE(e.account_hash, a.account_hash) AS account_hash, COALESCE(e.business_date, a.business_date) AS business_date, COALESCE(e.expected_amount_cents, 0) AS expected_amount_cents, COALESCE(a.actual_amount_cents, 0) AS actual_amount_cents FROM expected e FULL OUTER JOIN actual a ON e.account_hash = a.account_hash AND e.business_date = a.business_date WHERE COALESCE(e.expected_amount_cents, 0) <> COALESCE(a.actual_amount_cents, 0); Also check whether every selected event produced a decision: PLSQL SELECT c.event_id FROM kafka_recovery_cohort_event c LEFT JOIN kafka_recovery_decision_log d ON c.recovery_run_id = d.recovery_run_id AND c.event_id = d.event_id WHERE c.recovery_run_id = :recovery_run_id AND d.event_id IS NULL; Those queries are intentionally plain. Recovery confidence should not depend on a heroic debugging session. It should depend on executable checks that can run before anyone is asked to close the loop. Alert on Evidence Failure, Not Just Kafka Symptoms Recovery alerts should catch bad recovery, not only broken Kafka. Page when terminal outcomes do not close after the deadline, reconciliation fails after the check has completed, or an active recovery run stops making progress. Do not page simply because an in-progress recovery still has work remaining. YAML groups: - name: kafka-recovery rules: - alert: KafkaRecoveryUnaccountedRecords expr: | kafka_recovery_records_unaccounted > 0 and on(topic, event_type, run_type) kafka_recovery_deadline_reached == 1 for: 15m labels: severity: page annotations: summary: "Kafka recovery has overdue unaccounted records" - alert: KafkaRecoveryRunTooOld expr: | time() - kafka_recovery_last_progress_timestamp_seconds > 1800 and on(topic, event_type, run_type) kafka_recovery_run_active == 1 for: 5m labels: severity: ticket annotations: summary: "Kafka recovery run has stopped making progress" The alert is not the verdict. It tells the operator where to look. The verdict should still come from the recovery evidence record. Test the Evidence Path Recovery safety should not only be described. It should be testable. At minimum, test the evidence path with a small matrix: TestInjectionExpected proofDuplicate replayProcess the same event_id twice.One APPLIED, one DUPLICATE_SUPPRESSED, no second side effect.Missing decisionSelect an event into the cohort but skip emission.no_missing_decision_events fails.Projection driftChange target state after replay.Reconciliation returns mismatched keys.Poison recordSend a record that violates a domain rule.Outcome is QUARANTINED with owner and reason.Stalled runStop progress after partial terminal outcomes.Unaccounted-record or stale-progress alert fires. Run these tests in two places. First, keep a small contract-test suite around the recovery library or consumer adapter so decision emission cannot regress. Second, run a replay drill in a non-production environment with a deliberately small cohort. The drill should prove that the metrics, decision log, reconciliation query, alerts, and final verdict all agree before the same pattern is trusted during an incident. Recovery Definition of Done Before closing a Kafka recovery run, require evidence for each item: CheckEvidenceCohort declaredTopic, partitions, offsets, event types, expected count, owner, and stop conditions are recorded.Terminal math closesExpected records equal terminal outcomes, excluding retryable in-flight failures.Idempotency is visibleDuplicate suppressions and side-effect decisions are emitted as bounded metrics and durable decision records.Failure ownership existsFinal failures and quarantined records have reasons, owners, and next actions.Reconciliation passedThe named source of truth matches the target state for the approved scope, or exceptions are explicitly owned.Verdict recordedThe run ends as RECOVERED, RECOVERED_WITH_EXCEPTIONS, BLOCKED, QUARANTINED, or FAILED. This full pattern is not free. Structured decision logs add write volume. Reconciliation consumes compute. Audit tables need retention and access control. Quarantine creates human workflow. There are also cases where the full pattern is too much. A stateless consumer that only updates an internal cache may not need decision tables, reconciliation jobs, and formal verdicts. Start with the recovery definition of done, then choose the minimum evidence that proves the business risk you actually have. Kafka gives teams a powerful recovery primitive because the log can be replayed. That does not make recovery safe by itself. Lag tells you the consumer moved. Recovery evidence tells you whether the system became correct again.

An AI inference service makes a model available to an application through an API. When you ask an inference API a question, the application sends your prompt to the service. The service processes it and generates a response, often streaming the answer back in batches. Producing a correct answer is only part of the job. The service also needs to respond quickly, handle many requests, and use GPU compute resources efficiently. Measuring each incoming request helps services understand what users experience and investigate problems when performance bottlenecks occur. AI Inference Metrics Models process input in units called tokens. Hence, common metrics include time to first token (TTFT), time between tokens (TBT), and time per output token (TPOT). Other measurements could be end-to-end latency, input and output token counts, request rate, error rate, etc. Also, TTFT and TBT describe different analyses. A response might start almost immediately but stream slowly, or take several seconds to start and then finish. Measuring only total response time hides that distinction. Averages can also hide problems. Services often examine P95 latency. This helps reveal slower experiences that an average can mask. To have a deeper understanding of such service behavior, a service can emit a simplified log record like the one below when a request finishes. One record explains one request. Millions of such records may be used to analyze the service and answer questions such as “Why did service-side latency increase during a particular time window?” “Did service errors increase with incoming traffic?” etc. This article explores using Parquet, Vortex, and DuckDB for such analytical queries. JSON { "event_time": "2026-09-01T12:30:00", "model": "llama", "input_tokens": 100, "output_tokens": 200, "ttft_ms": 100 } Columnar Formats Most telemetry queries need only a handful of columns; for example, a high-latency investigation may need only one hour of records. Columnar storage lets query engines access selected columns, and data layouts can help them skip irrelevant data. One such columnar format, Parquet, organizes data into row groups, column chunks, and pages. Another format, Vortex, takes a compressed array approach that supports some operations without fully decoding the data. Neither design guarantees faster queries; the outcome depends on the data, query implementation, and work being performed. This article uses DuckDB as the query engine because it can run SQL queries against both Parquet directly and Vortex through its extensions without setting up a database server. Scenario: Investigating High Latency When an AI service feels slow, a common starting point is to examine requests from the affected time window. For example, “How many requests were handled?” “What was the P95 TTFT?” etc. This experiment explores how file format and row ordering affect the time it takes to query one hour of inference latency data. For this purpose, one million inference records are synthetically generated using the structure shown earlier. Each record contains an event timestamp, model name, input and output token counts, and time to first token (TTFT) in milliseconds. All timestamps are interpreted as UTC. A given record is written in two formats, i.e., Parquet with Zstandard compression and Vortex with default writer settings. For each format, time-sorted and shuffled versions are created. This produces the four files below. Plain Text sorted.parquet - Time sorted records, Parquet with Zstandard compression. shuffled.parquet - Shuffled records, Parquet with Zstandard compression. sorted.vortex - Time sorted records, Vortex with default writer settings. shuffled.vortex - Shuffled records, Vortex with default writer settings. The following query calculates request count, average TTFT, and P95 TTFT for a populated one-hour window. The query is then repeated against the shuffled files. Request counts and latency statistics are checked across all four configurations before timing results are compared. SQL SELECT COUNT(*) AS request_count, AVG(ttft_ms) AS avg_ttft_ms, QUANTILE_CONT(ttft_ms, 0.95) AS p95_ttft_ms FROM read_parquet('data/sorted.parquet') WHERE event_time >= TIMESTAMP '2026-09-01 12:00:00' AND event_time < TIMESTAMP '2026-09-01 13:00:00'; For Vortex, only the file reader and path are changed: SQL FROM read_vortex('data/sorted.vortex') Step 1: Generate the Data Files Python from pathlib import Path import duckdb # Refuse to overwrite an existing dataset. Path("data").mkdir(exist_ok=False) con = duckdb.connect() con.execute("SET threads = 1") con.execute("SET preserve_insertion_order = true") con.execute("INSTALL vortex") con.execute("LOAD vortex") print("Writer DuckDB:", duckdb.__version__) con.execute(""" CREATE TABLE logs AS SELECT -- One request per second; timestamps are interpreted as UTC. TIMESTAMP '2026-09-01' + i * INTERVAL '1 second' AS event_time, -- Synthetic model mix: 70%, 20%, and 10%. CASE WHEN i % 10 < 7 THEN 'llama' WHEN i % 10 < 9 THEN 'gemma' ELSE 'qwen' END AS model, -- Different hash constants give each field a distinct sequence. -- Input tokens: 100–8099; output tokens: 100–1099. CAST(100 + hash(i, 11) % 8000 AS INTEGER) AS input_tokens, CAST(100 + hash(i, 22) % 1000 AS INTEGER) AS output_tokens, -- Baseline TTFT: 100–499 ms; add 800 ms during the slow hour. CAST(100 + hash(i, 33) % 400 AS INTEGER) + CASE WHEN event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' THEN 800 ELSE 0 END AS ttft_ms FROM range(1000000) AS t(i) """) formats = { "parquet": "FORMAT PARQUET, COMPRESSION ZSTD", "vortex": "FORMAT VORTEX", } layouts = { "sorted": "event_time", # Timestamp breaks any hash ties, giving a deterministic order. "shuffled": "hash(event_time), event_time", } for layout, order in layouts.items(): for fmt, options in formats.items(): path = f"data/{layout}.{fmt}" con.execute(f""" COPY ( SELECT * FROM logs ORDER BY {order} ) TO '{path}' ({options}) """) print("Created", path) con.close() Shell Writer DuckDB: 1.5.5 Created data/sorted.parquet Created data/sorted.vortex Created data/shuffled.parquet Created data/shuffled.vortex Step 2: Run the Experiment Each configuration receives one warmup, followed by five measured executions. Execution order is randomized across configurations, and DuckDB is configured to use one query worker. File generation, sorting, and extension loading are excluded from query timings. Median query latency is reported alongside the observed minimum and maximum. File size is recorded separately. Two different kinds of latency are involved: TTFT describes the synthetic AI requests, while query latency measures how long DuckDB takes to analyze those records. The experiment measures the latter, including filtering, aggregation, and fetching the small result. Python import csv import math import random import statistics import time from pathlib import Path import duckdb con = duckdb.connect() con.execute("SET threads = 1") con.execute("LOAD vortex") print("Reader DuckDB:", duckdb.__version__) print("Vortex:", con.execute(""" SELECT extension_version FROM duckdb_extensions() WHERE extension_name = 'vortex' """).fetchone()[0]) query = """ SELECT count(*), avg(ttft_ms), quantile_cont(ttft_ms, 0.95) FROM {source} WHERE event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' """ jobs = [ (layout, fmt) for layout in ("sorted", "shuffled") for fmt in ("parquet", "vortex") ] times = {job: [] for job in jobs} rng = random.Random(42) reference = None # Round 0 is warmup; rounds 1–5 are measured. for repeat in range(6): rng.shuffle(jobs) for layout, fmt in jobs: sql = query.format( source=f"read_{fmt}('data/{layout}.{fmt}')" ) start = time.perf_counter() result = con.execute(sql).fetchone() elapsed = (time.perf_counter() - start) * 1000 if reference is None: reference = result assert result[0] == 3600 assert all( math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-6) for a, b in zip(result[1:], reference[1:]) ), "Results differ across configurations" if repeat: times[(layout, fmt)].append(elapsed) rows = [] for (layout, fmt), values in sorted(times.items()): rows.append({ "format": fmt, "layout": layout, "size_mib": Path(f"data/{layout}.{fmt}").stat().st_size / 2**20, "median_ms": statistics.median(values), "min_ms": min(values), "max_ms": max(values), }) with open("results.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=list(rows[0])) writer.writeheader() writer.writerows(rows) print("\nRequests, average TTFT, p95 TTFT:", reference) print("\nFormat Layout MiB Median ms Min–max ms") for r in rows: print( f"{r['format']:9} {r['layout']:9} {r['size_mib']:7.2f}" f" {r['median_ms']:12.2f}" f" {r['min_ms']:.2f}–{r['max_ms']:.2f}" ) con.close() Shell Reader DuckDB: 1.5.5 Vortex: 9a8eb67 Requests, average TTFT, p95 TTFT: (3600, 1100.8336111111112, 1281.0) Format Layout MiB Median ms Min–max ms parquet shuffled 9.40 11.91 11.32–12.49 vortex shuffled 6.60 7.14 6.99–9.14 parquet sorted 6.79 2.75 2.30–2.86 vortex sorted 4.10 1.68 1.38–4.12 Takeaway: File Format vs. Row Ordering The selected hour contained 3600 requests, with an average TTFT of 1100.83 ms and a P95 TTFT of 1281 ms. These values describe the synthetic inference records, and the query times measure how long DuckDB took to calculate that summary. Time-sorted Vortex had the lowest median query time and smallest file size. However, ordering improved query performance substantially for both formats. Note that these findings apply to this synthetic dataset and hour window query after warmup. Timings varied across runs, and the results do not establish a universal format ranking or directly measure how much data was skipped. QuestionRESULTSTakeawayDid ordering matter more than format?Sorting improved median query time by 4.33x for Parquet and 4.25x for Vortex. Switching from Parquet to Vortex improved it by 1.64–1.67x within the same layout.Yes, for this workload. Time ordering had the larger effect.Did the smaller file produce the faster query?Shuffled Vortex was slightly smaller than sorted Parquet (6.60 vs. 6.79 MiB), but its median query time was about 2.6x longer.Not necessarily. File size alone did not predict query speed.Which configuration performed best?Sorted Vortex had the smallest file (4.10 MiB) and lowest median query time (1.68 ms). Sorted Vortex used the least storage and had the fastest median query time in this experiment. Conclusion AI inference services generate telemetry that helps explain token usage, identify latency changes, and investigate high latency issues. Efficient storage and querying can make that data less expensive to retain and faster to analyze. In this synthetic DuckDB experiment, time ordering improved median query performance. Sorted Vortex produced the smallest file and fastest median query, but sorted Parquet outperformed shuffled Vortex, highlighting the importance of organizing telemetry around its access patterns. These local warm-up-based results do not establish a universal format winner or demonstrate faster inference. Instead, they illustrate how format choice and data layout can support more efficient AI telemetry analysis, making it easier to investigate performance problems before deciding how to address them. References Build a DIY AI Model Hosting Platform With vLLMUnderstanding Inference Time ComputeFile Format, Parquet Vortex DocumentationDuckDB Documentation, VortexReading and Writing Parquet Files, DuckDB Docs

By Bhala Ranganathan CORE

Most agent projects start the same way: one model, one prompt loop, three or four tools. That's the right place to start, and for a lot of workloads it's also the right place to stop. The trouble shows up later, when the same agent is asked to inspect infrastructure, parse OpenAPI specs, write docs, answer developer questions, and call operational tools. The system prompt turns into a wall of text. Tool selection gets flaky in ways you can't reproduce. Someone tunes the prompt to fix documentation quality and quietly breaks spec analysis, and nobody notices for a week. Splitting the agent is the obvious fix. But the agents were never the hard part. The hard part is the contract between them: how one agent finds another, how it hands over work, where authentication lives, and how you trace a request that crossed three runtimes before it failed. This article builds a small multi-agent system on Amazon Bedrock AgentCore Runtime using the Agent-to-Agent (A2A) protocol: an orchestrator, an OpenAPI spec analyzer, and a documentation generator. Three agents are a toy. The decisions are not. A note on versions. AgentCore and A2A are both moving fast. Everything here was checked against the AWS documentation and AgentCore CLI 0.25.x in August 2026. Pin your CLI and Python dependencies, and re-check the docs before you deploy anything you care about. Is a Second Agent Actually Worth It? Multi-agent isn't a maturity level. Every remote handoff you add buys you latency, an extra auth boundary, and a new class of failure that only shows up under load. Keep one agent when one team owns the workflow, the tools sit inside the same security boundary, and a single prompt can plausibly cover the job. Don't split because the architecture diagram looks better with more boxes in it. Split when responsibilities genuinely diverge. The signals are usually organizational before they're technical: two teams own two capabilities, the specialists want different models, the release cadences don't match, or one capability touches data the other has no business seeing. At that point the agent stops being a prompt and starts being a service, and it needs a service's contract. The arithmetic argument for a shared protocol is the familiar one. Pairwise integrations grow with the square of the number of participants, and each one tends to invent its own payload shape, auth handling, timeout, and error format. A standard protocol doesn't make any of those problems disappear. It just means every participant expresses them in the same place, which is most of the benefit. MCP and A2A Aren't Competing These two get discussed as alternatives. They aren't: they sit on different edges of the architecture. MCP connects an agent to tools and data.A2A connects an agent (or an application) to another agent. A spec analyzer might pull an OpenAPI document out of a repo over MCP, then return its analysis to an orchestrator over A2A. Both, same request. In the code below, the OpenAPI document is passed inline so the A2A flow stays readable. One caveat about "framework-agnostic," since it gets oversold: A2A is framework-agnostic at the protocol level. That doesn't mean two arbitrary SDKs will talk to each other on the first try. Both sides still need a compatible A2A implementation, transport, protocol version, and auth configuration. Once those line up, the internals genuinely don't matter; Strands on one side, LangGraph on the other, fine. What Runtime Gives You Configure an AgentCore Runtime for A2A, and it becomes a transparent proxy in front of your container. Your side of the contract is narrow: a stateless, streamable HTTP server on 0.0.0.0:9000, mounted at the root path, speaking JSON-RPC 2.0. AgentCore passes the JSON-RPC payload straight through without touching it, and adds session isolation, authentication, scaling, and observability around it. Note the port. A2A is 9000 at /, MCP is 8000 at /mcp, plain HTTP is 8080 at /invocations. Getting this wrong is the most common reason a container deploys fine and then never answers. The A2A vocabulary you need for the rest of this: A2A client: whoever is sending work. Here is the orchestrator.A2A server: the agent endpoint receiving it.Agent Card: metadata at /.well-known/agent-card.json describing identity, endpoint, skills, capabilities, and auth requirements. This is how discovery works.Message: one turn, carrying one or more parts (text, structured data).Task and artifact: a task tracks work at one remote agent; artifacts carry its output. An orchestrator fanning out to three specialists creates three separate tasks. For inbound calls, Runtime supports IAM SigV4 or OAuth 2.0 bearer tokens (JWT). A runtime version uses one of them at a time, so pick based on who's calling and where the security boundary sits. If a client authenticates with the wrong one, you get a 403 that looks nothing like an agent error. The Shape of the System The system has a hub and spoke architecture. The orchestrator is an A2A server to the user and an A2A client to the specialists, which is worth stating explicitly because it's easy to forget when you're debugging. Hub and Spoke Achitecture The orchestrator should know what each specialist advertises. It should not know how any of them work. The test: adding a fourth agent a changelog writer, say should mean one new config entry, not edits inside the agents you already shipped. Building a Specialist Prerequisites Python 3.10+Node.js 20+ (the AgentCore CLI ships as an npm package; the A2A tutorial page still says 18, but the CLI's own docs say 20)AWS CDK installed, the CLI deploys through itAWS credentials, plus model access enabled in Bedrocknpm install -g @aws/agentcore If you still have the old Python starter toolkit installed, uninstall it first. Both provide an agentcore command, and you will lose an afternoon to that. Scaffold an A2A project and pick Strands when prompted: Shell bash agentcore create --protocol A2A The generated project gives you the config, the model loader, and the A2A server wiring. The specialist below does one thing: read an OpenAPI document and pull out a compact, honest view of its operations. Python # app/SpecAnalyzer/main.py from typing import Any import yaml from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model HTTP_METHODS = { "get", "put", "post", "delete", "options", "head", "patch", "trace", } def _auth_mode(security: Any) -> str: if security is None: return "unspecified" if security == []: return "none" if isinstance(security, list) and any(item == {} for item in security): return "optional" return "required" @tool def summarize_endpoints(openapi_yaml: str) -> list[dict[str, Any]]: """Return a compact list of operations from an OpenAPI document.""" spec = yaml.safe_load(openapi_yaml) if not isinstance(spec, dict): raise ValueError("The OpenAPI document must contain a YAML object.") paths = spec.get("paths", {}) if not isinstance(paths, dict): raise ValueError("The OpenAPI 'paths' field must be an object.") global_security = spec.get("security") operations: list[dict[str, Any]] = [] for path, path_item in paths.items(): if not isinstance(path_item, dict): continue for method, operation in path_item.items(): if method.lower() not in HTTP_METHODS: continue if not isinstance(operation, dict): continue security = operation.get("security", global_security) operations.append( { "path": path, "method": method.upper(), "operation_id": operation.get("operationId"), "summary": operation.get("summary", ""), "auth": _auth_mode(security), } ) return operations agent = Agent( model=load_model(), system_prompt=( "You are an API specification analyst. " "Call summarize_endpoints before reasoning. " "Report endpoints, authentication behavior, and concrete " "specification risks. Never invent fields that aren't present." ), tools=[summarize_endpoints], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) serve_a2a is the AgentCore SDK helper that does the boring, mandatory parts: /ping, Agent Card serving, AGENTCORE_RUNTIME_URL handling, Bedrock header propagation, and binding to port 9000. The validation inside that tool is the part I'd argue for hardest. OpenAPI path items legally contain keys that aren't HTTP methods parameters, servers, $ref, orsummary. Loop over every key as though it were an operation and you'll either crash on a string or, worse, emit a confident summary containing an endpoint called PARAMETERS. Then the doc generator writes it up, and it ships. Demos never surface this because demo specs are clean. Direct dependencies for both specialists: Shell bedrock-agentcore[a2a] strands-agents[a2a] a2a-sdk PyYAML httpx The a2a extras matter. Without them, serve_a2a and the client classes aren't there. a2a-sdk is what the orchestrator's client code imports from, so it's a direct dependency even though the server side pulls it in transitively. The documentation generator uses the same wrapper with a narrower job: Python # app/DocGenerator/main.py from strands import Agent from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model agent = Agent( model=load_model(), system_prompt=( "You write developer documentation from an API request and a " "specification analysis. Produce Markdown with an overview, " "authentication notes, request examples, error handling, and a " "short Python quickstart. Never invent endpoints, fields, or " "credentials." ), ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) That one's prompt-only on purpose. If your docs have to match a house format, this is where you add output validation or template tools instead of asking a model to remember a style guide. Test It Locally Shell agentcore dev # or, without the inspector UI: python main.py Then send it a JSON-RPC message at the root path: JSON curl -X POST http://localhost:9000/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": "req-001", "method": "message/send", "params": { "message": { "role": "user", "parts": [ { "kind": "text", "text": "Analyze the authentication on this API." } ], "messageId": "12345678-1234-1234-1234-123456789012" } } }' | jq . Pull the Agent Card too, before you deploy anything: Shell curl http://localhost:9000/.well-known/agent-card.json | jq . Thirty seconds of curl catches a missing discovery endpoint or a card whose skills describe an agent you deleted two commits ago. Card problems are miserable to debug from the client side, because what you see is a resolver failure, not a bad card. Deploy the Specialists The CLI defaults to IAM SigV4. Configure bearer-token auth when the caller can't sign requests, the AWS tutorial walks through Cognito for this, though Cognito isn't a requirement; any OIDC provider works. Deploy each specialist as its own runtime: Shell agentcore deploy The CLI packages the code, uploads the artifact to S3, creates the runtime, and hands back an ARN: Shell arn:aws:bedrock-agentcore:us-west-2:<account-id>:runtime/SpecAnalyzer-xyz123 Separate runtimes are half the point of doing this at all. Each specialist scales, releases, and rolls back on its own, without a coordinated deploy across the whole system. The URL Nobody Warns You About Here's the step that eats the most time. The invocation URL isn't the ARN, and it isn't a friendly hostname. It's the ARN, URL-encoded, embedded in a path: https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{url-encoded-arn}/invocations/ Every colon and slash in the ARN gets escaped with quote(arn, safe='') in Python, or let the SDK do it: Python from bedrock_agentcore.runtime import build_runtime_url url = build_runtime_url( "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/SpecAnalyzer-xyz123" ) The Agent Card sits under that path, at .../invocations/.well-known/agent-card.json. Keep the trailing slash on the base URL. A missing slash or an unescaped ARN both produce a 404, which reads like the runtime doesn't exist. Calling a Remote Agent This client assumes bearer auth. For an IAM runtime, sign with SigV4 or use a client that signs for you. Python # app/Orchestrator/client.py import os from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, TextPart DEFAULT_TIMEOUT = 300 def _message(text: str) -> Message: return Message( kind="message", role=Role.user, parts=[Part(TextPart(kind="text", text=text))], message_id=uuid4().hex, ) async def call_remote_agent( runtime_url: str, text: str, session_id: str | None = None, ) -> str: headers = { "Authorization": f"Bearer {os.environ['BEARER_TOKEN']}", "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": session_id or str(uuid4()), } async with httpx.AsyncClient( timeout=DEFAULT_TIMEOUT, headers=headers, ) as http: resolver = A2ACardResolver(httpx_client=http, base_url=runtime_url) card = await resolver.get_agent_card() config = ClientConfig(httpx_client=http, streaming=False) client = ClientFactory(config).create(card) async for event in client.send_message(_message(text)): if isinstance(event, Message): return event.model_dump_json(exclude_none=True) if isinstance(event, tuple) and len(event) == 2: task, _ = event return task.model_dump_json(exclude_none=True) raise RuntimeError("The remote agent returned no A2A result.") With streaming=False, you get exactly one result, which is either a Message or a (Task, UpdateEvent) tuple. Handle both. The tuple form is what trips people up on the first run. Returning raw JSON keeps the protocol visible for a walkthrough, but don't ship it. Normalize messages and artifacts into your own response type at this boundary. Otherwise, every downstream prompt has to understand the full A2A envelope, and you end up with models reasoning about kind fields instead of about your problem. Session IDs deserve a real decision, not a uuid4() you stopped thinking about. Fresh ID for an isolated one-shot task. Same ID when you're continuing a multi-turn conversation with the same remote agent. And keep a separate correlation ID of your own, because one user request may span several runtime sessions and you'll want to stitch them back together in the traces. The Orchestrator Each remote agent becomes a tool. Endpoints come from config; the client resolves the Agent Card at the endpoint before sending work. Python # app/Orchestrator/main.py import os from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model from client import call_remote_agent REMOTE_AGENTS = { "spec_analyzer": os.environ["SPEC_ANALYZER_URL"], "doc_generator": os.environ["DOC_GENERATOR_URL"], } @tool async def delegate(agent_name: str, instruction: str) -> str: """Send work to a named A2A specialist.""" runtime_url = REMOTE_AGENTS.get(agent_name) if runtime_url is None: allowed = ", ".join(sorted(REMOTE_AGENTS)) return f"Unknown agent '{agent_name}'. Available agents: {allowed}." return await call_remote_agent(runtime_url, instruction) orchestrator = Agent( model=load_model(), system_prompt=( "You coordinate two API specialists. Use spec_analyzer for OpenAPI " "parsing and risk analysis. Use doc_generator for documentation and " "SDK examples. When both are needed, analyze the specification " "first, then include that analysis in the documentation request. " "Never assign work outside an agent's advertised capability." ), tools=[delegate], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(orchestrator)) Two small things in there matter more than they look. The tool is async, which avoids wrapping the remote call in asyncio.run() that blows up when the agent is already running inside an event loop, and the traceback doesn't point anywhere near the cause. And returning a friendly string for an unknown agent, rather than raising, gives the model something it can recover from instead of a dead turn. The URLs belong in deployment config, a service catalog, or an agent registry. Not in source control. Let the Model Route, or Hard-Code the Path? The code above lets the orchestrator model decide who to call and in what order. That's genuinely useful when requests vary. It's also the default choice for the wrong reasons. If every documentation request must be analyzed before generation, always, no exceptions encode that sequence in Python and use the model only inside each step. Deterministic orchestration is easier to test, retry, and audit, and it doesn't drift when someone edits a prompt. Save model-driven routing for routes that are actually ambiguous. It shouldn't be there to make a fixed workflow feel more autonomous. Running It With all three runtimes deployed, send the user request to the orchestrator. Something like "Review this customer API spec and create a Python quickstart" should cause it to call the analyzer, feed that analysis to the doc generator, and return one answer. Should. That's expected behavior, not a guarantee the protocol hands you. Whether the right handoffs happen comes down to your orchestrator prompt, your tool descriptions, your eval cases, and your workflow code. A2A standardizes the conversation. It has no opinion about whether your orchestration logic is correct. Before You Call It Production Least privilege per runtime. Each runtime gets permissions for its own tools and data, nothing more. The failure mode to watch for is the orchestrator's role slowly accumulating every permission any specialist ever needed. Distinguish auth failures from agent failures. A2A application errors come back as JSON-RPC error objects, frequently with HTTP 200. Auth and authorization failures come back as native HTTP errors AccessDeniedException is a 403. AgentCore also maps runtime exceptions onto JSON-RPC codes: -32051 for not found (404), -32053 for throttling (429), -32054 for conflict (409), -32055 for a runtime client error (424). Inspect the JSON-RPC shape. A 200 is not a success. Retry deliberately. 429 and retryable-conflict 409 are worth retrying with backoff. A 424 usually means your container failed and the answer is in CloudWatch, not in a second attempt. Never blind-retry non-idempotent work, and pass a task or request identifier so a specialist can recognize a duplicate. Trace every handoff. AgentCore Observability gives you sessions, traces, and spans. Propagate your own business correlation data on top, or you'll have three sets of traces and no way to prove they belong to the same user request. Set lifecycle values on purpose. Defaults are a 15-minute idle timeout and an 8-hour max lifetime, both configurable (60 to 28,800 seconds). Long-running tasks may justify raising them; a bigger number should be a decision, not a leftover. Also watch that your A2A sessions actually go idle — a server that holds a connection open after the client is gone will happily keep a microVM warm on your bill. Cache Agent Cards carefully. Resolving on every call is simple and costs you a round trip per hop. Cache with a short TTL and invalidate when an agent's version or capabilities change. A stale card advertising a skill that no longer exists is a genuinely confusing bug. Keep protocol and SDK versions aligned. Pin dependencies, validate cards, and test a mixed-framework setup rather than assuming it works. Use private connectivity if you need it. Runtime can attach to a VPC for internal resources, and interface VPC endpoints via PrivateLink keep AgentCore API calls off the public internet. A Failure Test Matrix Before you call it done, break the handoffs rather than admiring the happy path: The specialist's Agent Card is unreachable or malformed.The bearer token expires between card resolution and invocation.The analyzer returns a valid task with no usable artifact.The doc generator times out after the analyzer succeeded.The orchestrator picks an agent that doesn't exist, or one it isn't allowed to call.A retry re-runs a task that already had an external side effect. Each of these teaches you something a successful demo can't. They also make the division of labor concrete: Runtime hosts and protects the endpoint, A2A defines the conversation, and workflow correctness is still entirely yours. Closing The value here isn't the agent count. It's that each capability gets a clear boundary, a narrow permission set, and its own release cycle. Runtime provides the managed execution boundary; A2A provides a common way to discover agents and pass work between them. So, start with one agent. Split it when responsibilities, ownership, or security boundaries justify a network hop, and not before. Where the sequence is fixed, keep the orchestration deterministic. Where routing is genuinely variable, let the orchestrator reason over well-described capabilities, and instrument every handoff you add. References AWS: Deploy A2A servers in AgentCore RuntimeAWS: A2A protocol contractAWS: AgentCore Runtime authentication and authorizationAWS: LifecycleConfiguration API referenceAWS: Configure AgentCore Runtime for VPC accessAgentCore CLIAgent2Agent protocol specification

As Large Language Models (LLMs) continue their rapid trajectory of development, software engineers and cloud architects regularly run into two systemic challenges: The Fixed Knowledge Cutoff: Model intelligence is inherently restricted to its training data window, making it blind to real-time changes. The "Air-Gap" Limitation: Out of the box, LLMs cannot securely interact with external systems or private APIs on their own. Historically, developers bypassed these hurdles by writing fragile, ad-hoc API wrappers or custom orchestrators. Enter the Model Context Protocol (MCP): an open standard designed to standardize how AI applications safely connect to external data sources and execution environments.In this article, we will explore the core architecture of MCP, look at why the Azure MCP Server is a game-changer for cloud engineers, and walk through a step-by-step guide to configuring it inside Visual Studio Code. The Core Architecture of MCP At its heart, MCP establishes a uniform "language" that allows AI applications (Hosts) to talk to external resources (Servers). Rather than building custom integrations for every new LLM or tool, developers can rely on a single, clean architecture: MCP Architecture The standard is built around four fundamental building blocks: MCP Host: The runtime environment or user interface where the AI agent operates (e.g., VS Code, Claude Desktop, Cursor).MCP Client: The architectural component within the host that initiates and maintains the active connection.MCP Server: A lightweight, modular helper service that exposes specific resources, prompts, and tools.Transport Layer: The underlying protocol facilitating communication, typically utilizing JSON-RPC 2.0 over standard input/output (stdio) or HTTPS. MCP Components How the MCP Handshake Works Instead of executing raw, unpredictable bash scripts, the interaction is highly structured: Initialization: The client connects to the server and queries its capabilities. Declaration: The server returns a structured schema listing the specific tools it supports. Execution Request: When an LLM determines it needs external data, the host requests a specific tool execution from the server. Context Injection: The server executes the local process, gathers the result, and returns a semantic JSON response to the host, which is then cleanly formatted for the user. MCP in Action Why Use the Azure MCP Server? If you are managing infrastructure on Microsoft Azure, the Azure MCP Server bridges the gap between your local AI assistant and your active cloud resources. Operating as a secure local process, it natively integrates with the Azure command-line context. The server supports over 40 Azure services and more than 170 tools out of the box, spanning critical cloud primitives: Compute & Containers: Azure Container Apps and Azure Kubernetes Service (AKS). Storage & Resource Management: Azure Storage (blobs and containers) and Azure Resource Groups. Infrastructure as Code: Integrated Azure Terraform Best Practices. Key Advantages Over Raw CLI Executions While you could technically let an AI agent execute arbitrary commands in a terminal, using a dedicated MCP server provides several major structural benefits: Rich Semantics: Instead of parsing messy, unstructured terminal stdout text, the MCP server passes rich semantic data blocks directly back to the LLM. Strict Governance & Scope: You can explicitly configure the server to run in read-only mode or expose only a select subset of tools, preventing the AI from accidentally deleting production infrastructure. Interactive Guardrails: The protocol requires explicit user confirmation before executing tools that touch sensitive data or perform mutative actions. Azure MCP Server Setup and Configuration Modes You can run the Azure MCP server locally across multiple development setups using stdio transport. Below are the three most common configuration schemas. NuGet Configuration For .NET teams, the server can be dynamically fetched and started using the dnx toolchain: JSON { "mcpServers": { "Azure MCP Server": { "command": "dnx", "args": [ "Azure.Mcp", "--source", "https://api.nuget.org/v3/index.json", "--yes", "--", "azmcp", "server", "start" ], "type": "stdio" } } } Node.js Configuration If you are developing in a standard JavaScript/TypeScript ecosystem, you can spin up the server dynamically using the latest npm package via npx: JSON { "mcpServers": { "azure-mcp-server": { "command": "npx", "args": [ "-y", "@azure/mcp@latest", "server", "start" ] } } } Docker Configuration For isolated development environments, you can run the server in a container. Note that you must provide a local environment file (.env) containing your Azure Service Principal credentials: JSON { "mcpServers": { "Azure MCP Server": { "comman { d": "docker", "args": [ "run", "-i", "--rm", "--env-file", "/full/path/to/.env", "mcr.microsoft.com/azure-sdk/azure-mcp:latest" ] } } } Step-by-Step Visual Studio Code Integration A great feature for those who want to work within Visual Studio Code is that they can also manage the Azure MCP Server through an exclusive extension available directly in their editor. The following steps walk you through the setup. Step 1: Install the Extension Search for and install the Azure MCP Server extension directly from the Visual Studio Code marketplace. Install Azure MCP Server Extension Step 2: Initialize and Verify Open the Command Palette (Cmd + Shift + P on macOS or Ctrl + Shift + P on Windows) and search for the extension commands to verify the server is active and running. Select an MCP Server Step 3: Configure Your Tool Accessibility Open your integrated AI chat window and select the Tools icon. Here, you will see a list of all active tools provided by the Azure MCP Server. You can toggle specific permissions on or off—for instance, disabling write operations while keeping read operations active. Select Your Tools Step 4: Interact Natively Now, your AI chat assistant can securely call Azure tools in the background. You can ask complex queries like: "Are there any inactive containers running in my resource group?""Upload our local configuration file directly to our Azure storage container blob storage." Any further interactions that reference an Azure account would use the MCP tools. Confirm Tools and Resources Real-World Engineering Use Cases To see the power of this setup, let’s look at how this changes day-to-day operations: Use Case A: Automated AKS Incident Troubleshooting When an incident occurs in an Azure Kubernetes Service (AKS) cluster, engineers typically run dozens of diagnostic commands. With the Azure MCP server connected, you can simply ask the LLM: "Investigate why the pods in our production namespace are crash-looping." The agent will call the relevant AKS tools, inspect the logs, identify the misconfiguration, and suggest the fix - all in seconds. Use Case B: Continuous Terraform and Compliance Audits Before deploying infrastructure, you can point your local AI agent to your code directory. Because the server incorporates Azure Terraform Best Practices, the agent can audit your configuration files, cross-reference them against your live Azure Resource Groups, and warn you if you are violating security compliance rules or generating drift. Conclusion The Model Context Protocol represents a major step forward in AI-assisted development. By standardizing the communication layer, the Azure MCP Server enables software engineers to transform static, isolated LLMs into active, context-aware cloud operators. Whether you are monitoring active Kubernetes clusters, auditing Terraform configurations, or automating file uploads to blob storage, MCP gives your AI assistant safe and highly scoped access to the "live" Azure ecosystem.

The Foundry model catalog covers a lot of ground, but it doesn't cover everything. If you need a model trained on your own proprietary, tabular data, a churn predictor built on your actual customer history, a fraud score trained on your actual transaction patterns, that's not a model catalog problem. That's a real machine learning problem, and on Microsoft's stack it belongs to Azure Machine Learning, a genuinely separate platform from Foundry with its own SDK, its own workspace concept, and its own deployment model. This is a hands-on build of the whole path: train a model with Azure ML's SDK, register it, deploy it behind a managed endpoint, and then wire that endpoint into a Foundry agent as a function tool, so a conversational agent can call your custom model mid-conversation the same way it would call any other tool. The Mental Model First Azure ML and Foundry don't share a runtime. They share an ecosystem, and a genuine amount of engineering effort has gone into making the seam between them small, but it's worth being precise about where that seam actually is: A command job is how training happens. You point Azure ML's SDK at a script, a compute target, and a set of inputs, and it runs your training code as a tracked, reproducible job.The model registry is where a trained model becomes a named, versioned artifact, independent of the job that produced it. This is what lets you promote a specific version to production without recreating the training run.A Managed Online Endpoint is how a registered model actually serves predictions. It's a real, always-on (or autoscaling) piece of infrastructure with its own URL, its own auth, and its own cost, running inside the Azure ML workspace, not inside Foundry.A Foundry agent's FunctionTool is the bridge. Foundry doesn't know or care that the function it's calling happens to invoke an Azure ML endpoint. From the agent's perspective, it's just a tool with a name, a schema, and a result. Prerequisites An Azure ML workspace and a Foundry project, in the same or different resource groups; it doesn't matter which, since nothing about this integration requires them to share infrastructure.Python 3.9+ with both SDKs installed. Python pip install azure-ai-ml azure-ai-projects azure-identity Python from azure.ai.ml import MLClient from azure.identity import DefaultAzureCredential ml_client = MLClient( DefaultAzureCredential(), subscription_id="<subscription-id>", resource_group_name="<resource-group>", workspace_name="<aml-workspace-name>", ) Step 1: Train the Model A command job wraps a training script, here a plain scikit-learn classifier, and runs it on managed compute. Python from azure.ai.ml import command, Input, Output train_job = command( code="./src", command="python train.py --data ${{inputs.training_data} --model_output ${{outputs.model_output}", inputs={"training_data": Input(type="uri_folder", path="azureml://datastores/workspaceblobstore/paths/churn-training/")}, outputs={"model_output": Output(type="uri_folder")}, environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest", compute="cpu-cluster", display_name="churn-model-training", ) returned_job = ml_client.jobs.create_or_update(train_job) ml_client.jobs.stream(returned_job.name) ml_client.jobs.stream blocks and prints logs until the job finishes, which is worth doing in any script you're actually going to run rather than fire-and-forget, since a training job failing silently in the background is a bad way to find out your pipeline is broken. Step 2: Register the Trained Model Registration turns the job's output into a named, versioned artifact you can reference independently of the job. Python from azure.ai.ml.entities import Model from azure.ai.ml.constants import AssetTypes model = ml_client.models.create_or_update( Model( path=f"azureml://jobs/{returned_job.name}/outputs/model_output", name="churn-classifier", type=AssetTypes.MLFLOW_MODEL, description="Customer churn classifier, trained on 18 months of account history.", ) ) Using MLFLOW_MODEL as the type here isn't incidental. If your training script logs the model with MLflow's autologging, Azure ML's managed endpoints can deploy it with a built-in scoring container, no custom score.py inference script required. That's a real time saver worth designing your training script around from the start rather than discovering after the fact. Step 3: Deploy It Behind a Managed Endpoint Python from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment endpoint = ManagedOnlineEndpoint(name="churn-endpoint", auth_mode="key") ml_client.online_endpoints.begin_create_or_update(endpoint).result() deployment = ManagedOnlineDeployment( name="blue", endpoint_name="churn-endpoint", model=model, instance_type="Standard_DS3_v2", instance_count=1, ) ml_client.online_deployments.begin_create_or_update(deployment).result() endpoint.traffic = {"blue": 100} ml_client.online_endpoints.begin_create_or_update(endpoint).result() The blue deployment name isn't a convention you have to follow, but it's worth keeping, since it sets up the pattern you'll want the first time you deploy a new model version: create a green deployment alongside blue, split traffic between them, and shift fully once you trust the new version, rather than replacing blue outright and hoping. Step 4: Confirm It Works Before Anything Else Touches It Python import json test_input = {"input_data": {"columns": ["tenure_months", "monthly_spend", "support_tickets"], "data": [[14, 89.50, 3]]} response = ml_client.online_endpoints.invoke( endpoint_name="churn-endpoint", request_file=None, deployment_name="blue", input_data=json.dumps(test_input), ) print(response) Get a real prediction back here before wiring anything else into it. Debugging a broken endpoint through the extra layer of an agent's function-calling loop is meaningfully harder than debugging it directly. Step 5: Wrap the Endpoint as a Foundry Agent Function Tool This is the actual bridge. The Foundry agent doesn't call Azure ML directly; your application code does, in response to the agent asking for it. Python import os import requests from azure.ai.projects import AIProjectClient from azure.ai.projects.models import PromptAgentDefinition, Tool, FunctionTool from azure.identity import DefaultAzureCredential def get_churn_score(tenure_months: int, monthly_spend: float, support_tickets: int) -> dict: payload = {"input_data": {"columns": ["tenure_months", "monthly_spend", "support_tickets"], "data": [[tenure_months, monthly_spend, support_tickets]]} resp = requests.post( "https://churn-endpoint.<region>.inference.ml.azure.com/score", headers={"Authorization": f"Bearer {os.environ['AML_ENDPOINT_KEY']}", "Content-Type": "application/json"}, json=payload, timeout=10, ) resp.raise_for_status() return {"churn_probability": resp.json()[0]} func_tool = FunctionTool( name="get_churn_score", description="Predict churn probability for a customer given tenure, spend, and support ticket history.", parameters={ "type": "object", "properties": { "tenure_months": {"type": "integer", "description": "How many months the customer has been active."}, "monthly_spend": {"type": "number", "description": "Average monthly spend in dollars."}, "support_tickets": {"type": "integer", "description": "Number of support tickets in the last 90 days."}, }, "required": ["tenure_months", "monthly_spend", "support_tickets"], "additionalProperties": False, }, strict=True, ) project = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential()) tools: list[Tool] = [func_tool] agent = project.agents.create_version( agent_name="retention-agent", definition=PromptAgentDefinition( model="gpt-4.1-mini", instructions="Help the team assess churn risk. Call get_churn_score whenever specific customer numbers are provided.", tools=tools, ), ) Step 6: Run It End to End Python import json from openai.types.responses.response_input_param import FunctionCallOutput openai_client = project.get_openai_client() conversation = openai_client.conversations.create() response = openai_client.responses.create( input="A customer's been with us 14 months, spends about $90/month, and filed 3 tickets recently. Churn risk?", conversation=conversation.id, extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}, ) for item in response.output: if item.type == "function_call" and item.name == "get_churn_score": result = get_churn_score(**json.loads(item.arguments)) follow_up = openai_client.responses.create( input=[FunctionCallOutput(type="function_call_output", call_id=item.call_id, output=json.dumps(result))], conversation=conversation.id, extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}, ) print(follow_up.output_text) The model decides whether the question warrants calling get_churn_score at all; your code executes the actual HTTP call to Azure ML when it does, and the result flows back into the same conversation for the model to finish answering. Nothing about the agent definition or the Responses API calls changes based on what the function actually does behind the scenes, whether it hits a database, a Lambda function, or, as here, a completely separate Azure ML workspace. Where This Fits in the Bigger Picture Worth being explicit about something the SDKs don't make obvious on their own: Azure ML's managed endpoint and the Foundry project endpoint are two different resources with two different auth boundaries, and nothing about this integration merges them. The Azure ML endpoint has its own key or managed identity, scoped to that workspace. The Foundry project has its own credential, scoped separately. FunctionTool doesn't create a trust relationship between the two; it's just a name and a schema the model uses to ask your code to do something. The actual authentication to Azure ML happens entirely in your own function implementation, same as it would if that function called any other external API. That's a feature, not a limitation. It means the Azure ML side of this can be owned, secured, and rotated independently of the Foundry side, by a different team if that's how your organization is structured, without either side needing write access to the other's resources. Production Considerations Before You Commit Process expires 10 minutes after a function call is issued. Submit the tool's output back to the conversation before that window closes, or the run fails. If get_churn_score's HTTP call to Azure ML is slow, that's a hard deadline, not a soft one; budget for it.Prefer a managed identity over a static endpoint key once this is more than a prototype. A key baked into an environment variable is fine for local testing and a real liability in anything that runs unattended. Grant the identity running your function tool code the AzureML Data Scientist or a more narrowly scoped custom role against just this endpoint, not the whole workspace.Blue/green the endpoint; don't overwrite it. Standing up a new deployment alongside the old one and shifting traffic gradually is the only way to catch a regression in a newly trained model before it's serving 100% of real requests.Treat strict=True on the FunctionTool schema as a correctness feature, not boilerplate. It's what keeps the model from calling get_churn_score with a malformed or partial argument set that would otherwise fail inside your function rather than being caught before the call.Online endpoint compute is billed whether or not it's actively scoring. Unlike a serverless model call, a Managed Online Endpoint with instance_count=1 running around the clock costs money at idle. If call volume is low and bursty, look at scale-to-zero options or batch endpoints instead of defaulting to always-on instance count 1.Model drift doesn't announce itself. Nothing in this pipeline retrains automatically when the real-world distribution shifts away from what the model was trained on. Log the inputs and outputs of every get_churn_score call somewhere you can actually review, and revisit training data on a real cadence, not only when someone notices predictions have gotten worse. Where This Leaves You Foundry's model catalog is the right tool for the overwhelming majority of generative AI work, and for teams that never need a model trained on their own proprietary data, it's genuinely the whole story. But the moment the problem is "predict something specific about our own customers, from our own historical data," that's Azure ML's job, and pretending otherwise means either forcing a language model to approximate a task it was never built for, or building your own training and serving infrastructure by hand. The actual integration effort here is small: a training job, a registered model, a managed endpoint, and one FunctionTool definition. The two platforms don't need to merge for that to work. They just both need to keep doing the one thing each is actually good at. References Microsoft Learn. "Train models with the Python SDK v2." Azure Machine Learning. learn.microsoft.com/en-us/azure/machine-learning/how-to-train-model?view=azureml-api-2Microsoft Learn. "Deploy machine learning models to online endpoints." Azure Machine Learning. learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?view=azureml-api-2Microsoft Learn. "Use function calling with Microsoft Foundry agents." learn.microsoft.com/en-us/azure/foundry/agents/how-to/tools/function-callingMicrosoft Learn. "Explore Microsoft Foundry Models in Azure Machine Learning." learn.microsoft.com/en-us/azure/machine-learning/foundry-models-overview?view=azureml-api-2Microsoft Learn. "Get started with Microsoft Foundry SDKs and endpoints." learn.microsoft.com/en-us/azure/foundry/how-to/develop/sdk-overview

By Jubin Soni, FBCS CORE

Enterprise context — Acme FinServ. SOC 2 CC7 (system monitoring) requires that Acme can detect and investigate anomalous activity. When an agent-driven workflow touches customer data at 2 AM, "we have logs somewhere" is not an answer an auditor accepts. The distributed trace built in this part is the forensic evidence trail: a single trace ID that ties the Goose prompt to every agentgateway policy decision and every Quarkus tool call, so a post-incident review can reconstruct exactly which agent did what, in what order, and how long each governed hop took. The Core Problem In Part 1, we built a Quarkus MCP tool server. In Part 2, we secured it with agentgateway's JWT authentication, RBAC, and ExtMCP guardrails. The architecture works — but when something goes wrong in production, you're flying blind. Agentic workflows are fundamentally different from traditional request-response APIs. A single user prompt like "Debug customer CUST-4091" triggers a multi-round-trip loop: Goose calls tools/list to discover available toolsThe LLM selects getCustomerStatus and Goose sends tools/callThe LLM reads the response, sees primaryRegion: US-EAST-1, and chains a second tools/call to getZoneHealthLogsThe LLM correlates both results and generates a diagnostic summary Each of these hops crosses process boundaries: Goose → agentgateway → Quarkus. Without distributed tracing, you see four isolated HTTP requests in your access logs. You cannot tell they belong to the same agentic workflow. When step 3 takes 12 seconds instead of 200ms, you have no waterfall to pinpoint whether the latency came from agentgateway policy evaluation, Quarkus bean validation, or a slow downstream call. This creates black holes in telemetry dashboards — the exact gap that autonomous agents exploit to degrade silently. The Solution: W3C Trace Context Across All Three Layers The fix is standard distributed tracing, applied to the MCP transport layer: agentgateway exports spans for every proxied MCP request and propagates traceparent headers to the backend.Quarkus with quarkus-opentelemetry picks up the incoming traceparent, creates child spans for tool execution and bean validation, and exports them to the same Jaeger instance.Jaeger correlates both sides into a single trace waterfall — one view from agent prompt to tool result. Prerequisites Everything from Parts 1 and 2, plus: Podman – for running Jaeger (podman compose) Verify Podman is available: Shell podman --version Step 1: Launching the Observability Backend We use Jaeger v2 as both the OTLP collector and the trace UI. A single container accepts traces from agentgateway on port 4317 (OTLP gRPC) and from Quarkus on port 4318 (OTLP HTTP), and serves the query UI on port 16686. Shell cd part3-observability podman compose up -d This starts Jaeger v2 with OTLP collection enabled by default. Verify it's running: Shell curl -sf http://localhost:16686/ > /dev/null && echo "Jaeger UI is ready" Open http://localhost:16686 — you'll see an empty Jaeger UI. We'll populate it with MCP traces in the following steps. Production Alternative: Grafana Tempo For production deployments, replace Jaeger with Grafana Tempo backed by object storage (S3/GCS). The OTLP endpoint stays the same — only the compose.yml changes. Grafana provides richer dashboards, alerting, and long-term trace retention. Step 2: Enabling OpenTelemetry in Quarkus Add the quarkus-opentelemetry extension to Part 1's pom.xml: Properties files <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-opentelemetry</artifactId> </dependency> Configure the exporter in application.properties: Properties files # OpenTelemetry quarkus.otel.service.name=customer-tools quarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4318 quarkus.otel.exporter.otlp.traces.protocol=http/protobuf quarkus.otel.traces.sampler=always_on quarkus.otel.traces.suppress-non-application-uris=false PropertyPurposeservice.nameIdentifies this service in Jaeger's service dropdowntraces.endpointOTLP HTTP receiver — Jaeger's port 4318 (base URL only; Quarkus appends /v1/traces)traces.protocolhttp/protobuf — Quarkus uses its Vert.x-based HTTP exportertraces.sampleralways_on — sample every span (reduce in production)suppress-non-application-urisfalse — include MCP endpoint spans (they'd be filtered otherwise) When no OTLP collector is running (Parts 1 and 2 without Jaeger), Quarkus logs a connection warning, but the MCP server works normally. When the collector IS running (Part 3), traces flow automatically. Zero code changes to the MCP tools. Rebuild Part 1: Shell cd part1-quarkus-mcp mvn package -DskipTests What Quarkus Auto-Instruments With quarkus-opentelemetry on the classpath and the SDK enabled, Quarkus automatically creates spans for: LayerSpan NameWhat It CapturesHTTP serverPOST /mcpInbound MCP request with method, status, latencyCDI beansCustomerServiceTools.getCustomerStatusTool execution time within the MCP handlerBean ValidationHibernateValidatorParameter validation before tool logic runsREST clientOutbound HTTP callsAny downstream API calls (future extensions) No @WithSpan annotations needed. The Quarkus OpenTelemetry extension instruments the reactive pipeline automatically. Step 3: Configuring W3C Trace Context in agentgateway agentgateway supports native OpenTelemetry trace export. Add a tracing block to the gateway configuration: Properties files config: adminAddr: localhost:15000 tracing: otlpEndpoint: http://localhost:4317 otlpProtocol: grpc randomSampling: 1.0 FieldPurposeotlpEndpointOTLP receiver — Jaeger's port 4317otlpProtocolgrpc for OTLP/gRPC (also supports http)randomSamplingSample 100% of traces (reduce to 0.01–0.1 in production) How Trace Propagation Works When agentgateway receives an MCP request: Creates a root span for the proxy operation (e.g., agentgateway.mcp.proxy)Injects a traceparent header into the forwarded request to Quarkus:traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01Quarkus reads the traceparent, creates a child span under the same trace ID, and records tool executionBoth spans export to Jaeger via OTLP, where they appear as a single correlated trace This is standard W3C Trace Context propagation — the same mechanism used across all OpenTelemetry-instrumented services. Configuration Files Part 3 provides two agentgateway configurations: ConfigUse Caseconfig-traced.yamlTracing only — proxy + OTLP export, no security layersconfig-traced-guardrails.yamlTracing + ExtMCP guardrails — observe the guardrail evaluation spans too Step 4: Running the Interactive Demo Start all services with the one-command script: Shell cd part3-observability ./start-all.sh The script starts Jaeger, Quarkus (with OTel enabled), and agentgateway (with trace export), then launches the demo SPA on :8890. Open the MCP Observability Console at http://localhost:8890/index.html and walk through the three demo steps: Initialize – Establishes an MCP session through agentgateway. The architecture diagram animates the trace propagation: root span creation in agentgateway, traceparent injection, child span in Quarkus, and OTLP export to Jaeger.List Tools – Discovers all 5 tools through the traced proxy. The trace waterfall panel shows the agentgateway proxy span and the Quarkus HTTP span side by side with timing.Multi-Tool Workflow – Simulates Goose's multi-turn reasoning: getCustomerStatus (finds region US-EAST-1) → getZoneHealthLogs (checks zone health) → getSLACompliance (correlates SLA metrics). Each step generates a full trace with waterfall visualization. The stat tiles track traces generated, spans collected, and Jaeger status. Click Open Jaeger to view the real trace waterfalls in the Jaeger UI at http://localhost:16686. Step 5: Generating Traces via CLI To generate additional traces manually, simulate a multi-turn agentic workflow: Shell # Step 1: Initialize MCP session export MCP_SESSION_ID=$(curl -s -D - http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' \ | grep -i "mcp-session-id:" | sed 's/.*: //' | tr -d '\r') # Step 2: Discover tools curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 3: Agent calls getCustomerStatus (first tool invocation) curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"getCustomerStatus","arguments":{"customerId":"CUST-4091"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 4: Agent chains getZoneHealthLogs based on the region from step 3 curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"getZoneHealthLogs","arguments":{"zoneId":"US-EAST-1"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 5: Agent fetches SLA compliance for correlation curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"getSLACompliance","arguments":{"serviceId":"api-gateway"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . Each of these requests generates a trace that flows through agentgateway into Quarkus and lands in Jaeger. Step 6: Visualizing the Trace Waterfall in Jaeger Open http://localhost:16686 in your browser. Finding Traces In the Service dropdown, select customer-tools (Quarkus) or agentgatewayClick Find TracesClick on any trace to open the waterfall view Reading the Waterfall A typical tools/call trace shows the following span hierarchy: Shell agentgateway.mcp.proxy [12ms] └─ POST /mcp [8ms] ← Quarkus HTTP server └─ CustomerServiceTools.getCustomerStatus [2ms] ← CDI tool execution SpanServiceWhat It Tells Youagentgateway.mcp.proxyagentgatewayTotal proxy overhead including policy evaluationPOST /mcpcustomer-toolsQuarkus HTTP handling time for the MCP requestgetCustomerStatuscustomer-toolsPure tool execution time (business logic) What to Look For Proxy overhead: The gap between the agentgateway span and the Quarkus span shows network + policy evaluation time. If this grows, check guardrail server latency.Validation time: Bean Validation spans appear before tool execution. Regex-heavy patterns like ^CUST-[0-9]{4,8}$ are fast, but complex validators on large payloads can add latency.Multi-turn correlation: When Goose chains multiple tool calls (e.g., getCustomerStatus → getZoneHealthLogs), each appears as a separate trace. The mcp-session-id tag lets you filter all traces belonging to one agent session.Error traces: Failed validations (invalid customer ID format) or guardrail rejections (blocked poison payloads) produce error spans with exception details. Connecting Goose for Real Traces Launch Goose pointed at agentgateway and prompt a multi-tool workflow: Shell goose session "Debug customer CUST-4091 — check their account status, then pull health logs for their region and SLA compliance for api-gateway." This generates a burst of correlated traces in Jaeger showing Goose's multi-turn tool orchestration from the proxy layer down to individual tool execution spans. What We Achieved Starting from the secured architecture in Part 2, we added full observability without changing any MCP tool code: LayerWhat We AddedConfig ChangeQuarkusquarkus-opentelemetry dependencypom.xml + application.propertiesagentgatewaytracing block in config YAMLconfig-traced.yamlObservability backendJaeger all-in-one via Podman Composecompose.yml The entire stack runs locally with a single ./start-all.sh command and produces end-to-end trace waterfalls in Jaeger. Production Considerations ConcernLocal (this tutorial)ProductionTrace backendJaeger all-in-one (in-memory)Grafana Tempo + object storageSampling rate100% (default: 1.0)1-10% or adaptive samplingTrace retentionContainer lifetimeDays/weeks in durable storageAlertingManual Jaeger inspectionGrafana alerting on span latency/error rateMetricsTraces onlyAdd Prometheus + quarkus-micrometer for RED metrics Coming Up in Part 4 With tracing in place, you can now see every MCP tool call flowing through the system. In Part 4, we will move beyond single-agent tool calls to multi-agent orchestration — using the Agent-to-Agent (A2A) protocol to coordinate autonomous agents that can delegate work, enforce governance via AGENTS.md, and call back into our MCP tool services.

By Daniel Oh CORE

Throughout my career, observability and monitoring systems have played a very important role in system stability, availability, and scalability when architecting, whether a system handles millions or billions of events per day. Having architected core infrastructure across communication platforms, enterprise compliance pipelines, and high-throughput transaction systems, I have repeatedly seen how background instrumentation can silently become a bottleneck under heavy production load. Telemetry and observability come with their own challenges. Instrumenting telemetry without degrading live business application performance, increasing request latency, or inflating cloud costs is often deprioritized initially but becomes very important as the system scales. Furthermore, it is a much harder problem to solve than processing domain events themselves in production. When an event-driven platform scales past 15 billion events per day, peaking between 200,000 and 400,000 TPS (transactions per second), the infrastructure costs and performance overhead of logging and tracing libraries create a severe operational bottleneck known as the Telemetry Tax. This article explains why addressing this telemetry tax is important and also provides you with brief insight into how you can start handling it with architectural changes at the initial level. The Multiplier Effect In a very scalable, high-throughput microservices architecture, business transactions rarely execute as isolated operations. An incoming event arriving at an API gateway typically passes through five to ten downstream services, caching layers, database instances, queues, message brokers (e.g., Kafka), etc. If every microservice hop generates three telemetry events in addition to the core business logic events, assuming two metric updates and a single log line, the resulting metric data scales exponentially: Regular workload (Core business Logic events): 15,000,000,000 events per dayMetrics and logs: 100,000,000,000 signals per dayStorage estimation: 15+ TBs of uncompressed JSON payloads daily Without specialized memory management and proper architecture, this massive volume of background instrumentation introduces critical failures in live production systems. Example: Garbage collection: Short-lived heap allocation for JSON log stringsThread Lock Contention: Sync metric exporters hold mutex locks while completing the request to send telemetry metrics, adding 10-20ms of latencyCascading downstream failures: Failure in downstream telemetry collector or failure in processing existing logs and metrics causes issues or backlog in upstream transaction workers, which may result in queue back-pressure and API timeouts. Engineering a Zero-Allocation Telemetry Pipeline To eliminate garbage collection pauses and thread lock delays, high-throughput microservices must decouple metric and log generation from core business logic, utilizing a middleware concept with pre-allocated memory pools and asynchronous buffers. Under heavy execution loads, standard string concatenations from logging and JSON marshaling for telemetry signals escape to the heap during Go's escape analysis. When millions of goroutines allocate short-lived objects on the heap per second, Go's runtime is forced into frequent concurrent sweep phases, consuming 15-20% of available CPU purely on garbage collection. By utilizing sync.Pool, we pre-allocate backing byte arrays that survive across the request lifecycle. When a worker goroutine finishes formatting a payload, the buffer pointer is reset (buf[:0]) and returned to the pool without invoking runtime.newobject . This keeps memory allocation in the critical execution path at zero bytes. Go package telemetry import ( "context" "sync" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" ) var tracer = otel.Tracer("zero-alloc-telemetry") // BufferPool pre-allocates memory var bufferPool = sync.Pool{ New: func() interface{} { b := make([]byte, 0, 1024) return &b }, } type RingBufferPipeline struct { telemetryChannel chan *trace.Span } func NewPipeline(bufferSize int) *RingBufferPipeline { return &RingBufferPipeline{ telemetryChannel: make(chan trace.Span, bufferSize), } } func (p *RingBufferPipeline) InstrumentEvent(ctx context.Context, eventID string) { // Retrieve pre-allocated byte slice from pool bufPtr := bufferPool.Get().(*[]byte) buf := (*bufPtr)[:0] defer func() { *bufPtr = buf bufferPool.Put(bufPtr) }() // Non-blocking trace span initialization ctx, span := tracer.Start(ctx, "ExecuteTransaction", trace.WithAttributes(attribute.String("event.id", eventID))) defer span.End() buf = append(buf, []byte("event_processed:")...) buf = append(buf, eventID...) // Async non-blocking dispatch select { case p.telemetryChannel <- span: default: // Add metric here to protect API response SLOs } } How to Control Ingestion Costs With Tail-based Adaptive Sampling Collecting 100% of telemetry traces across 100+ billion daily events leads to unsustainable tool ingestion costs (Regardless of the tool you use, e.g., Datadog, OpenTelemetry, Splunk). Standard Head-based sampling (deciding whether to keep a trace at the start of the request) drops fatal error traces while keeping millions of redundant HTTP 200 success traces. By deploying OpenTelemetry collectors configured with Tail-based adaptive sampling, trace spans are held in a 500 ms sliding memory buffer prior to routing: HTTP 200 / Successful Executions: Sampled at 0.1% to maintain baseline latency metricsHTTP 5xx / Latency exceptions (>200ms) / Server Errors: Retained at 100% for debugging, or any other analysis Production Performance Benchmark (Approx.) Refactoring telemetry infrastructure from sync logging to zero-allocation ring buffers and tail-based adaptive sampling helps with substantial performance improvements. (Note: The numbers in the table below are rough estimations based on high-scale modeling and past operational experience. Actual metrics may vary depending on your system and other aspects of architecture) Performance MetricSync TelemtryZero-allocation Adaptive TelemetryIngestion Throughput35k events/sec400k events/secp99 API response latency200ms40msAverage Monthly Cost$35-45k+$15-20kCPU overhead15-20% total CPU time spent on GC2-3% total CPU time spent on GC These estimations illustrate that the telemetry tax is not an inevitable cost of scale, but a consequence of applying synchronous, allocation-heavy architectural patterns to hyper-scale architectures. By refactoring memory management at the application level and applying adaptive filtering at the collector layer, engineering teams can achieve deep operational visibility while protecting system performance and cloud infrastructure budgets. Key Takeaways for System Architects Decouple the path: Never allow telemetry exporters to execute synchronously on the core business logic pathPre-allocate memory: Use memory buffer pools to eliminate garbage collection pauses during high-throughput event processingSample at the tail: Evaluate trace retention based on the execution outcome rather than making static decisions at the request ingress.

Most AWS migration projects don't fail because of technical complexity. They fail because teams treat migration as a single activity rather than a set of distinct strategies applied to different workloads. AWS defines seven migration strategies — the 7Rs — that determine how each application moves to the cloud. The decision of which strategy applies to which workload has more impact on project cost, timeline, and outcome than any architectural choice you'll make after. Yet in practice, most teams default to "lift-and-shift everything" without evaluating whether that's appropriate. This article presents a practitioner's framework for classifying workloads into the 7Rs, based on delivering 50+ AWS migrations across fintech, SaaS, healthcare, and e-commerce. The 7R Strategies Retire Not every workload deserves migration. During discovery, you will invariably find applications that are redundant, unmaintained, or replaceable. In a typical enterprise portfolio of 20–40 applications, 10–20% qualify for retirement. Decision criteria: No active users, duplicate functionality already covered by another system, or maintenance cost exceeds business value. Common mistake: Teams skip this step because retiring applications requires stakeholder conversations. The result is migrating dead applications that consume compute budget indefinitely. Retain Some workloads shouldn't migrate in this wave. Applications with deep hardware dependencies, pending end-of-life within 12 months, or complex regulatory constraints that require legal review before cloud deployment are candidates for retention. Decision criteria: High migration complexity combined with low business urgency, or external constraints that prevent cloud deployment within the project timeline. Retain is not "never migrate." It's "not now." Document these workloads with a future migration path and trigger conditions. Rehost (Lift-and-Shift) Moving applications to EC2 or containers without code modifications. AWS Application Migration Service (MGN) automates this by continuously replicating servers and orchestrating cutover with minutes of downtime. Decision criteria: Application has a short remaining lifespan (1-2 years), speed of migration matters more than optimization, or the application is a black box with no available source code. Timeline: Days to weeks per workload. Trade-off: You gain cloud elasticity and pay-as-you-go pricing immediately, but you inherit all existing architectural inefficiencies. A poorly designed monolith on-premises becomes a poorly designed monolith on EC2. Relocate Hypervisor-level migration, primarily for VMware workloads moving to VMware Cloud on AWS. The OS, application, and configuration remain untouched. Decision criteria: Large VMware estate, tight data center exit deadline, and applications that cannot tolerate any configuration change. Replatform Migration with targeted adaptations to managed services. The application architecture stays intact, but you replace self-managed infrastructure components with AWS equivalents: Self-ManagedAWS ManagedOperational BenefitSelf-hosted PostgreSQLRDS for PostgreSQLAutomated backups, patching, failoverCron jobs on EC2EventBridge + LambdaNo server to maintain, pay-per-invocationSelf-managed RedisElastiCacheAutomatic failover, scalingNginx load balancerApplication Load BalancerManaged TLS termination, WAF integrationSelf-hosted ElasticsearchOpenSearch ServiceManaged cluster scaling, snapshots Decision criteria: Application is well-structured but operationally expensive. The team spends significant time on database maintenance, patching, backup verification, or scaling. Timeline: 2-4 weeks additional per workload compared to rehost. Trade-off: Moderate additional effort (schema compatibility testing, connection string changes) in exchange for a 40-60% reduction in ongoing operational cost. For most mid-complexity applications, replatforming represents the optimal balance between migration effort and long-term benefit. Refactor (Re-Architect) Rebuilding applications for cloud-native patterns: microservices decomposition, containerization (ECS/EKS), serverless (Lambda), event-driven architecture (EventBridge, SQS, SNS, Step Functions). Decision criteria: The application is a core business asset that needs capabilities the current architecture cannot deliver — true horizontal scaling, independent service deployments, multi-region active-active, or zero-downtime deployments. Timeline: Months. Budget accordingly. Trade-off: Highest upfront investment, but delivers the best long-term results in terms of deployment velocity, fault isolation, and scaling capability. Reserve this for 2-3 applications maximum in a migration portfolio. Repurchase Replacing custom-built software with a commercial SaaS product. The application doesn't move to AWS; it moves to a vendor. Decision criteria: The in-house application solves a problem that is not a core competency and commercially available alternatives have matured to cover your requirements. Common candidates: CRM, HR systems, monitoring, project management. The Decision Framework Classification should happen during the assessment phase, before any infrastructure work begins. For each workload, evaluate four dimensions: 1. Business Value How critical is this application to revenue generation or core operations? High: Core product, customer-facing, revenue-generatingMedium: Internal operations, supports core processesLow: Legacy, rarely used, or duplicate functionality 2. Technical Complexity How difficult is it to migrate given current architecture, dependencies, and state management? High: Stateful, tightly coupled, hardware dependencies, proprietary protocolsMedium: Standard web application with database, some external integrationsLow: Stateless, containerizable, standard protocols 3. Team Capacity Does your engineering team have the skills and bandwidth to support a complex migration approach? High capacity: Can support re-architecting alongside other workLimited capacity: Can handle replatforming with some external supportMinimal capacity: Rehost or retain is the only realistic option 4. Time Constraint How quickly must this workload be operational on AWS? Immediate (weeks): Data center exit, contract expiryStandard (1-3 months): Planned migration within a programFlexible (3-6 months): Can wait for deeper optimization Mapping Dimensions to Strategy Business ValueComplexityCapacityTimeRecommended StrategyLowAnyAnyAnyRetire or RepurchaseAnyHighLowImmediateRehost (with future replatform plan)MediumMediumMediumStandardReplatformHighMedium-HighHighFlexibleRefactorAnyAnyAnyBlockedRetain A Practical Example Consider a portfolio of 15 applications for a mid-size SaaS company: Plain Text ┌─────────────────────────────────────────────────────┐ │ RETIRE (3) │ │ - Legacy admin panel (replaced by new one 2024) │ │ - Internal wiki (moved to Confluence) │ │ - Prototype service (never went to production) │ ├─────────────────────────────────────────────────────┤ │ RETAIN (1) │ │ - Hardware security module integration │ │ (requires legal review for cloud deployment) │ ├─────────────────────────────────────────────────────┤ │ REHOST (4) │ │ - Backoffice tools (low traffic, stable) │ │ - Legacy reporting engine (EOL in 18 months) │ │ - Monitoring collector agents │ │ - Staging environment clone │ ├─────────────────────────────────────────────────────┤ │ REPLATFORM (5) │ │ - Main API (PostgreSQL → RDS, cron → Lambda) │ │ - Worker services (EC2 → ECS Fargate) │ │ - File processing pipeline (S3 + Lambda) │ │ - Authentication service (→ElastiCache for sessions│ │ - Notification service (→ SES + SQS) │ ├─────────────────────────────────────────────────────┤ │ REFACTOR (1) │ │ - Core product platform (monolith → microservices) │ ├─────────────────────────────────────────────────────┤ │ REPURCHASE (1) │ │ - Custom CRM (→ HubSpot) │ └─────────────────────────────────────────────────────┘ This distribution — 20% retire, 7% retain, 27% rehost, 33% replatform, 7% refactor, 7% repurchase — is representative of what I see in practice. The replatform bucket is almost always the largest. Migration Tooling Alignment Each strategy maps to specific AWS tooling: StrategyPrimary ToolsRehostAWS Application Migration Service (MGN), Migration HubReplatformDMS (databases), manual adaptation, Terraform/IaCRefactorECS/EKS, Lambda, Step Functions, custom developmentRelocateVMware Cloud on AWS AWS Migration Hub provides a unified tracking dashboard across all strategies. For database migrations specifically, AWS Database Migration Service (DMS) handles both homogeneous and heterogeneous migrations with continuous replication (CDC), enabling near-zero-downtime cutovers. Common Anti-Patterns "Rehost everything, optimize later." Teams that plan to rehost first and replatform in a second phase rarely execute phase two. The urgency disappears once applications are running, and the team moves to other priorities. If replatforming is the right strategy, do it during migration. "Refactor everything for cloud-native." The opposite extreme. Not every application needs microservices. A well-structured monolith running on ECS Fargate can serve thousands of requests per second with simpler operations than a distributed system. "One strategy for all workloads." Every application in the portfolio has different characteristics. The decision framework exists because one size does not fit all. Conclusion The 7R classification exercise takes 3-5 days for a typical portfolio. It requires involvement from engineering leads, product owners, and sometimes finance (for retire/repurchase decisions). The output - a workload-by-workload strategy map - becomes the foundation for accurate timeline estimates, resource planning, and budget allocation. Without it, you're building infrastructure for workloads that might not need to exist. For a comprehensive breakdown of migration costs, the full 6-phase delivery process, and AWS tooling details, see my complete AWS cloud migration guide.

Most "RAG over PDFs" pipelines have a step nobody talks about much: something has to turn a scanned invoice, a multi-column contract, or a photographed receipt into text a model can actually reason over. On Microsoft's stack, that something is usually the Document Intelligence SDK, formerly Form Recognizer, and it's worth understanding on its own terms rather than treating it as a black box that happens before the interesting part starts. This is a hands-on deep dive into that SDK specifically. Not a tour of every Foundry Tools SDK — Vision and Speech and Content Safety each deserve their own treatment, but a real build using Document Intelligence: extracting layout as clean markdown, pulling structured fields out of a known document type, classifying documents before routing them, and training a custom extraction model on your own labeled data. The Mental Model First Two clients, and three kinds of model, cover almost everything this SDK does: DocumentIntelligenceClient runs analysis. Every call goes through one method, begin_analyze_document, and a model_id parameter decides what kind of analysis happens. It's a long-running operation, so every call returns a poller.DocumentIntelligenceAdministrationClient manages models. This is where you build custom extraction models and classifiers, list what's already been trained, and delete what you don't need anymore.Prebuilt models (prebuilt-layout, prebuilt-invoice, prebuilt-receipt, prebuilt-idDocument, prebuilt-read, and others) handle common, well-known document shapes out of the box. No training required.Custom extraction models, trained on your own labeled documents, handle document types nobody prebuilt a model for: your specific contract template, your specific intake form.Classifiers solve a different problem entirely: given a document of unknown type, which model should even look at it? This matters more than it sounds like it should, since most real document pipelines receive a mix of types, not one known shape. Prerequisites A Document Intelligence resource (or a multi-service Foundry resource, which includes it), giving you an endpoint and either an API key or Entra ID access.Python 3.9+ with the SDK installed. Python pip install azure-ai-documentintelligence azure-identity Python from azure.ai.documentintelligence import DocumentIntelligenceClient from azure.core.credentials import AzureKeyCredential endpoint = "https://YOUR-RESOURCE.cognitiveservices.azure.com" client = DocumentIntelligenceClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY")) For anything past local experimentation, swap the key for DefaultAzureCredential and an RBAC role scoped to the resource, the same pattern every other Foundry-adjacent SDK in this series has used. Step 1: Layout Extraction, Straight to Markdown This is the single most useful call in the whole SDK if your end goal is feeding documents into a RAG pipeline. prebuilt-layout doesn't just extract text; it understands headings, tables, and section structure, and it can hand all of that back as GitHub-flavored markdown instead of a flat text blob. Python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentContentFormat with open("contract.pdf", "rb") as f: poller = client.begin_analyze_document( "prebuilt-layout", AnalyzeDocumentRequest(bytes_source=f.read()), output_content_format=DocumentContentFormat.MARKDOWN, ) result = poller.result() print(result.content[:500]) result.content is now a markdown string, headings as #, tables as GFM pipe tables, page structure preserved. That matters more than it sounds like it should: a table flattened into plain text loses its row and column relationships, and a model reasoning over that text has to reconstruct structure it was never actually given. Markdown output keeps the structure intact. Step 2: Pulling Structured Fields From a Known Document Type For document types Document Intelligence already knows, invoices are the clearest example; you get named fields back with a confidence score per field, not just raw text. Python with open("invoice.pdf", "rb") as f: poller = client.begin_analyze_document("prebuilt-invoice", AnalyzeDocumentRequest(bytes_source=f.read())) result = poller.result() for doc in result.documents: vendor = doc.fields.get("VendorName") total = doc.fields.get("InvoiceTotal") if vendor: print(f"Vendor: {vendor.value_string} (confidence: {vendor.confidence:.2f})") if total: print(f"Total: {total.value_currency.amount} (confidence: {total.confidence:.2f})") That confidence score isn't decoration. It's the field you should actually branch on in production code; more on that in the production section below. Step 3: Add-On Capabilities You'll Want More Often Than the Docs Suggest A few optional capabilities aren't on by default, since they add processing cost, but are worth turning on deliberately rather than discovering you needed them after the fact: Python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentAnalysisFeature with open("shipping-label.pdf", "rb") as f: poller = client.begin_analyze_document( "prebuilt-layout", AnalyzeDocumentRequest(bytes_source=f.read()), features=[DocumentAnalysisFeature.BARCODES, DocumentAnalysisFeature.FORMULAS], ) BARCODES extracts barcode and QR code payloads directly, useful for shipping labels and inventory documents where the barcode carries the actual identifier the text doesn't repeat. FORMULAS pulls out mathematical expressions as LaTeX, relevant if you're processing scientific or financial documents where a formula matters more than the surrounding prose. There's also a high-resolution mode for documents where small print matters, at the cost of slower processing. Step 4: Build a Classifier to Route Mixed Document Types Real intake pipelines rarely receive one document type. A classifier solves the "what am I even looking at" problem before you commit to an extraction model. Python from azure.ai.documentintelligence import DocumentIntelligenceAdministrationClient from azure.ai.documentintelligence.models import ( BuildDocumentClassifierRequest, ClassifierDocumentTypeDetails, AzureBlobContentSource, ) admin_client = DocumentIntelligenceAdministrationClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY")) poller = admin_client.begin_build_classifier( BuildDocumentClassifierRequest( classifier_id="support-doc-classifier", doc_types={ "invoice": ClassifierDocumentTypeDetails( azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-invoices-container>") ), "contract": ClassifierDocumentTypeDetails( azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-contracts-container>") ), }, ) ) classifier = poller.result() You need at least five sample documents per category to train a classifier at all, and more than that for anything you'd trust in production. Once it's built, classifying an incoming document is a single call: Python with open("unknown.pdf", "rb") as f: poller = client.begin_classify_document("support-doc-classifier", AnalyzeDocumentRequest(bytes_source=f.read())) result = poller.result() for doc in result.documents: print(f"Classified as: {doc.doc_type} (confidence: {doc.confidence:.2f})") Step 5: Build a Custom Extraction Model for Your Own Document Type When a document type isn't invoices, receipts, or any of the other prebuilt shapes, train your own. This needs a set of labeled training documents in Blob Storage, produced through the labeling tool in Foundry's document intelligence studio or programmatically. Python from azure.ai.documentintelligence.models import ( BuildDocumentModelRequest, AzureBlobContentSource, DocumentBuildMode, ) poller = admin_client.begin_build_document_model( BuildDocumentModelRequest( model_id="acme-service-agreement-v1", build_mode=DocumentBuildMode.TEMPLATE, azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-training-container>"), description="Extraction model for Acme's standard service agreement template.", ) ) model = poller.result() Two build modes matter here, and they're not interchangeable. TEMPLATE mode is faster to train and works well when your documents follow a consistent visual layout, the same form filled out differently each time. NEURAL mode handles structural variation better, different layouts that still represent the same document type, at the cost of needing more training examples and longer build time. Start with TEMPLATE unless your documents genuinely vary in structure, not just content. One naming constraint worth knowing before you hit it: a custom model ID can't start with prebuilt-, since that prefix is reserved for Microsoft's own models across every resource. Where This Fits in the Bigger Picture This is the detail that trips people up once they've also worked with the Foundry SDK or Agent Framework elsewhere in this series: Document Intelligence doesn't go through your Foundry project endpoint at all. It has its own resource, its own endpoint (resource.cognitiveservices.azure.com), and its own authentication scope. That's what "Foundry Tools SDK" actually means as a category, prebuilt AI services with tool-specific endpoints, distinct from the Foundry SDK's unified project endpoint that Agent Framework and the Responses API build on. The practical upshot is the pipeline most teams actually want: run prebuilt-layout over incoming documents, get markdown back, and hand that markdown to a Foundry IQ Knowledge Base as a File Knowledge Source. Document Intelligence handles turning the PDF into clean, structured text. Foundry IQ handles chunking, embedding, and retrieval on top of it. Neither service needs to know the other exists; they just happen to compose well because Markdown is a reasonable interchange format for both. Production Considerations Before You Commit Don't trust a field just because it came back. A field with a confidence score of 0.41 should not silently flow into a downstream system as if it were as reliable as one scored 0.98. Set a threshold, route low-confidence extractions to human review, and log the confidence distribution over time so a model quietly degrading on a document template change doesn't go unnoticed.Classifier training minimums are a floor, not a target. Five documents per category is what the service requires to build at all. It is not enough to trust a classifier's accuracy in production. Budget for real evaluation data, held out from training, before routing real documents based on classifier output.TEMPLATE vs NEURAL is a real tradeoff, not a default to leave unexamined. Picking NEURAL by default because it sounds more capable means slower training and a higher training-data bar for a benefit you may not need if your documents are already visually consistent.Preview API versions and regional availability move independently of the SDK version. A given SDK release doesn't guarantee every feature is available in every region. Check current regional availability for newer capabilities (certain add-ons, newer prebuilt models) before designing around them.Markdown output is currently scoped to prebuilt-layout. Don't assume other prebuilt or custom models will hand back the same content format; check per-model support before building a pipeline that assumes Markdown everywhere.Cost scales with pages and capability, not just call count. Add-on features like high-resolution mode and custom model training both carry their own cost beyond the base per-page analysis price. Model this before committing to a design that turns on every add-on by default. Where This Leaves You The Document Intelligence SDK is easy to undersell because the interesting part of most AI applications feels like it's happening somewhere else, in the model, in the retrieval layer, in the agent's reasoning. But the quality ceiling of everything downstream is set right here, at the point where a physical or scanned document either does or doesn't become text a model can actually use well. Layout extraction to markdown, confidence-aware field extraction, classifiers for mixed intake, and custom models for your own document shapes cover the large majority of real document-processing needs, and all four are a few lines of SDK code once you know which one you need. The judgment call was never really about the API. It's about matching the right one of these four tools to what's actually in your inbound documents. References Microsoft. "azure-ai-documentintelligence README." Azure SDK for Python. github.com/Azure/azure-sdk-for-python/blob/main/sdk/documentintelligence/azure-ai-documentintelligence/README.mdMicrosoft Learn. "Document Intelligence layout model." learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layoutMicrosoft. "Migration guide, azure-ai-documentintelligence." Azure SDK for Python. github.com/Azure/azure-sdk-for-python/blob/main/sdk/documentintelligence/azure-ai-documentintelligence/MIGRATION_GUIDE.mdMicrosoft Learn. "Get started with Microsoft Foundry SDKs and endpoints." learn.microsoft.com/en-us/azure/foundry/how-to/develop/sdk-overviewMicrosoft Learn. "What is Foundry IQ?" learn.microsoft.com/en-us/azure/foundry/agents/concepts/what-is-foundry-iq

By Jubin Soni, FBCS CORE

A sidecar is a container that runs alongside another container as part of the same deployment unit. Just because two containers are in the same cluster or deployed around the same time doesn't make one a sidecar. There are two things that make a sidecar. First is that they share a network namespace, so they can reach each other over localhost rather than a network address. Second, they share a lifecycle. This means that they are created together, scaled together, and by default torn down together. Neither container has an existence independent of the other. The problem it solves is giving a specific concern its own boundary. For example, it can have its own filesystem, its own memory space, and often its own permissions or dependency set, without giving up the simplicity of deploying and operating one unit. You get isolation without paying for the operational overhead of running and coordinating a fully separate service. The test that defines the pattern across all of these is this: does it live and die with its partner container as one unit of deployment? If yes, it's a sidecar. If you have to reach it by hostname, through service discovery, or via a queue, it isn't one anymore. That is a separate service that happens to sit next to the first. That test matters because two adjacent patterns get called "sidecar" when they aren't: Decoupled worker/microservice. A separately deployed container, reached over the network, scaled on its own. A web application offloading work to Celery workers via Redis is a common instance of this: the app enqueues a job (send this signup email), a pool of workers pulls jobs off the queue independently, and neither side shares a network namespace or a lifecycle with the other. The workers scale on queue depth, not on how many web replicas are running, and a web app restart doesn't take queued or in-flight jobs down with it. n8n has its own version of the same shape: "queue mode," where a main node accepts webhooks and separate worker nodes pull jobs off a Redis queue. It's tempting to call either of these a sidecar relationship since the worker and the web app do feel paired, but neither qualifies: they don't share a deployment unit, and killing one doesn't touch the other.Ambassador/adapter. A container that proxies or translates traffic on its parent's behalf, like the Envoy example above, is actually this, more precisely. Structurally it's still a sidecar; it just gets a more specific name for what it does. Using n8n to Understand It What n8n Is n8n is a workflow automation platform like Zapier, but self-hostable and node-based rather than form-based. A handful of components make up a running instance: The editor/UI, where workflows are built visually as a graph of nodes.The main process, which serves that UI, listens for webhooks, and orchestrates workflow execution. The workflow execution decides what runs next, passing data between nodes and recording results.Nodes, the individual units of a workflow: trigger nodes (a webhook arrives, a schedule fires), action nodes (call an API, write to a database, send an email), and the Code node. The code node lets you drop in arbitrary JavaScript or Python to transform data however the built-in nodes can't. The code node is relevant in this article. The database, where workflow definitions, credentials, and execution history persist. In this article, Postgres is used. For most of what n8n does, the main process is the only thing doing work: routing a webhook, calling an API, writing a database row. The exception is the Code node, and that exception is the whole reason task runners exist. The Task Runner Feature and Its Use Case By default, a Code node's JavaScript or Python executes inside n8n's main process. This main process holds the database connection, the encryption key, and every credential stored in every workflow you've built. That's fine for trusted, well-understood scripts. It becomes a real problem the moment the code in that node is untrusted, third-party, or arbitrary enough that you can't fully audit it before it runs. By the way, that is how most Code nodes are used in practice. Task runners exist to solve exactly that use case: run Code node logic somewhere the main process's credentials and connections aren't reachable from it, without turning "write some JavaScript to reshape this JSON" into a separately deployed microservice every time. Going Deep on the Task Runner Feature n8n ships two modes for this: Internal mode (the default) runs Code nodes inline, in-process. No isolation. This is the fastest to set up, but the weakest boundary.External mode moves execution into a separate runner process entirely. That process connects back to the main n8n instance over a broker (an authenticated connection the main process listens on) and receives individual tasks to execute rather than having any standing access to n8n's internals. The runner never touches the database connection, the encryption key, or stored credentials directly; it only ever sees the specific input data for the task it's been handed. External mode goes further than just "a different process," too. The runner's own configuration (the n8n-task-runners.json file built in Phase 4) sets explicit allowlists — which environment variables the runner process can see at all, and which JavaScript built-ins or Python modules it's permitted to import, standard library and third-party tracked separately. So the boundary isn't just "different memory space," it's "different memory space, plus a declared, auditable list of exactly what this process is allowed to touch." That's a specific concern (arbitrary code execution) given its own boundary, without turning it into a fully independent service you have to deploy, discover, and monitor separately. It's the sidecar problem, stated exactly: external mode gives you the isolation; running the external runner as its own container in the same task definition is what makes that isolation a sidecar rather than just a separate process sharing a machine. Why This Needs to Scale Independently and Why "In the Same Container" Isn't Enough Most n8n deployment guides run n8n with task runners in internal mode, or with the external runner living inside the same container as the main process. For example, you will see guides about deploying n8n on a single EC2 instance, Render, DigitalOcean, or any platform's basic tier. That gets you the process isolation, which solves the security half of the problem. It doesn't solve the other half, which is that a runner sharing a container with the app can't be scaled, resourced, or restarted independently of it. That stops mattering the moment Code-node execution becomes the actual bottleneck rather than webhook handling or UI traffic. Imagine workflows doing heavy data transformation in Python, running numpy/pandas operations across large payloads, or executing many Code nodes concurrently. If the runner is bundled into the main container, giving it more CPU means giving the entire n8n instance more CPU, whether the UI and webhook layer need it or not. There's no way to say "the runner needs 2 more vCPUs, n8n itself is fine". Why AWS Fargate's Task Definition Is the Right Fit A Fargate task definition lets each container in the task carry its own CPU and memory reservation, its own health check, and its own essential flag governing what happens if it fails while still keeping every container in the task on one shared network interface. That's the sidecar promise made literal: isolation and independent resourcing for the runner, without losing the operational simplicity of one task, one deploy, one thing to scale as a unit when you do want to scale both together. The rest of this guide deploys exactly that: one Fargate task, two containers, wired together the way the definition above requires. Each infrastructure decision below gets tied back to a specific part of what's laid out here, so that by the end, the concept isn't something read once at the top, but it's something built. Prerequisites AWS account with billing enabledA domain you control, with DNS accessDocker installed locally, with docker buildx availableAWS CLI configured (aws configure) with permissions for ECR, ECS, RDS, ACM, and IAMThe runner image source (Dockerfile + n8n-task-runners.json) — built in Phase 4 Architecture Markdown User's Browser (HTTPS) | [Application Load Balancer] <- Certificate Manager (SSL Cert) | (Port 5678, HTTP internal) [ECS Fargate Task] |-- Container: n8n (main) <-- shared network namespace --> Container: n8n-runner (sidecar) | (Port 5432, PostgreSQL) [RDS PostgreSQL Database] The load balancer and RDS layers are ordinary AWS plumbing. The box in the middle is where the sidecar relationship actually lives. There is one task and two containers, each with its own resourcing. Phase 1: RDS PostgreSQL RDS Console → Create database → Standard create → Engine: PostgreSQLDB instance identifier: n8n-db. Master username: postgres. Generate and save a strong master password.Instance size: db.t4g.microStorage: 20 GB gp3, autoscaling on, max 100 GBConnectivity: the VPC you'll use throughout. Public access: No. New security group: n8n-db-sg, left empty for now.Additional configuration → Initial database name: n8n. Skip this and n8n fails on first connect with "database does not exist" — the DB instance identifier names the server, this field names the database inside it.Create, wait for "Available," copy the endpoint from Connectivity & security. Phase 2: ACM Certificate n8n requires HTTPS for webhooks to function Certificate Manager, in the same region you'll deploy the Load Balancer in → Request a public certificateDomain name: n8n.yourdomain.comValidation method: DNS validationCreate the CNAME record ACM provides at your registrar. If your registrar auto-appends your domain to the Host field, paste only the portion before your domain — the full string duplicates it and validation never completes.Wait for status: Issued Phase 3: Security Groups Two connections need rules: Security groupInbound rulePurposen8n-alb-sg443 from 0.0.0.0/0Public HTTPSn8n-ecs-sg5678 from n8n-alb-sgALB → n8n containern8n-db-sg (edit existing)5432 from n8n-ecs-sgn8n container → RDS Phase 4: Build and Push the Runner Image Dockerfile: Dockerfile FROM n8nio/runners:1.121.0 USER root RUN cd /opt/runners/task-runner-javascript && pnpm add moment uuid adm-zip RUN cd /opt/runners/task-runner-python && uv pip install numpy pandas pydantic requests boto3 certifi COPY n8n-task-runners.json /etc/n8n-task-runners.json ENV N8N_RUNNERS_CONFIG_FILE=/etc/n8n-task-runners.json USER runner It starts from n8n's own n8nio/runners base (containing the launcher and both runner processes), adds only the dependencies workflows actually need, and drops back to a non-root user once the root-only install steps finish. n8n-task-runners.json is where the isolation described above stops being architectural and becomes enforced: JSON { "task-runners": [ { "runner-type": "javascript", "health-check-server-port": "5681", "allowed-env": ["PATH", "GENERIC_TIMEZONE", "NODE_OPTIONS"], "env-overrides": { "NODE_FUNCTION_ALLOW_BUILTIN": "crypto,zlib", "NODE_FUNCTION_ALLOW_EXTERNAL": "moment,uuid,adm-zip" } }, { "runner-type": "python", "health-check-server-port": "5682", "env-overrides": { "N8N_RUNNERS_STDLIB_ALLOW": "json,zipfile,io,base64,datetime,re,math,random,statistics", "N8N_RUNNERS_EXTERNAL_ALLOW": "numpy,pandas,pydantic,requests,boto3,certifi" } } ] } allowed-env restricts which environment variables the runner process can see; N8N_RUNNERS_STDLIB_ALLOW / EXTERNAL_ALLOW restrict which Python modules it can import, stdlib and third-party separately. One container, two runner processes — the launcher inside n8nio/runners spawns both. Build and push: Shell docker buildx build -t n8nio/runners:custom . aws ecr create-repository --repository-name n8n-runners --region us-east-1 aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com docker tag n8nio/runners:custom <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom --username AWS is a fixed literal, not your actual username — ECR auth always uses it. The password piped via --password-stdin is a short-lived token generated by the CLI, not your account password. Phase 5: The Task Definition This is where the two containers become an actual sidecar pair, and where the independent-resourcing argument from the introduction becomes a real field rather than a claim. JSON { "family": "n8n-task", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "executionRoleArn": "arn:aws:iam::<account-id>:role/n8n-task-execution-role", "containerDefinitions": [ { "name": "n8n", "image": "n8nio/n8n:1.121.0", "essential": true, "entryPoint": ["sh", "-c"], "command": [ "mkdir -p /home/node/certs && wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /home/node/certs/rds-ca.pem && /docker-entrypoint.sh" ], "portMappings": [{ "containerPort": 5678, "protocol": "tcp" }], "environment": [ { "name": "DB_TYPE", "value": "postgresdb" }, { "name": "DB_POSTGRESDB_HOST", "value": "<rds-endpoint>" }, { "name": "DB_POSTGRESDB_PORT", "value": "5432" }, { "name": "DB_POSTGRESDB_DATABASE", "value": "n8n" }, { "name": "DB_POSTGRESDB_USER", "value": "postgres" }, { "name": "DB_POSTGRESDB_SSL_CA", "value": "/home/node/certs/rds-ca.pem" }, { "name": "DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED", "value": "false" }, { "name": "WEBHOOK_URL", "value": "https://n8n.yourdomain.com/" }, { "name": "GENERIC_TIMEZONE", "value": "Africa/Lagos" }, { "name": "N8N_RUNNERS_ENABLED", "value": "true" }, { "name": "N8N_RUNNERS_MODE", "value": "external" }, { "name": "N8N_RUNNERS_BROKER_LISTEN_ADDRESS", "value": "0.0.0.0" }, { "name": "N8N_RUNNERS_BROKER_PORT", "value": "5679" } ], "secrets": [ { "name": "DB_POSTGRESDB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/db-password" }, { "name": "N8N_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/encryption-key" }, { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n" } } }, { "name": "n8n-runner", "image": "<account-id>.dkr.ecr.<region>.amazonaws.com/n8n-runners:custom", "cpu": 512, "memory": 1024, "essential": false, "dependsOn": [{ "containerName": "n8n", "condition": "START" }], "environment": [ { "name": "N8N_RUNNERS_TASK_BROKER_URI", "value": "http://localhost:5679" } ], "secrets": [ { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:5680/healthz || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 20 }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n-runner" } } } ] } Five fields here map directly back to the introduction: Per-container cpu/memory on n8n-runner. This is the independent-resourcing argument made literal. The runner gets its own 512 CPU units and 1024 MB, carved out of the task total, separate from whatever n8n is allotted. If Code-node execution turns out to be the bottleneck, this is the number you raise without touching the main container's allocation at all. That's the exact thing a same-container runner can't offer you. networkMode: awsvpc is the mechanical basis of "shared network namespace." Every container in the task gets one elastic network interface between them. This is the setting that makes Phase 3's missing security group rule make sense. There's one network surface, not two. N8N_RUNNERS_TASK_BROKER_URI: http://localhost:5679 only works because of the line above. The runner reaches n8n over localhost because they are the same task. If this pointed anywhere else, you would have built the decoupled-worker pattern from the introduction instead, no matter what you called the container. A shared N8N_RUNNERS_AUTH_TOKEN, pulled from Secrets Manager by both containers. Sharing a network namespace means the runner is reachable by anything else in the task. The isolation the whole pattern exists for still needs a trust boundary at the process level, not just the network level. A plaintext token here would defeat that, since task definitions are readable by anyone with ecs:DescribeTaskDefinition. essential: false on the runner. This governs how tightly the two containers' lifecycles are actually coupled. essential: true would mean a runner crash tears down the whole task, main container included. false means the runner can crash and recover independently: Code-node executions fail until it's back, but the UI and webhooks keep serving. The pattern doesn't mandate one answer; it just means this has to be a decision, not a default you inherited. The health check on port 5680 hits the launcher's own endpoint, separate from the per-runner-type ports (5681 JS, 5682 Python) set in Phase 4's config file. ECS is checking the supervisor, not each runner process individually. Register it: aws ecs register-task-definition --cli-input-json file://n8n-task-def.json Phase 6: Cluster, Service, and Load Balancer ECS → Create cluster → n8n-cluster → Infrastructure: AWS FargateCreate a service inside it: Task definition: n8n-task, latest revisionDesired tasks: 1Networking: your VPC, at least two subnets across AZs, security group n8n-ecs-sg, public IP onLoad balancing: Application Load Balancer, listener on 443 using the Phase 2 certificateTarget group: HTTP, port 5678, health check path /healthzCreate, wait for steady state. Notice the target group and health check only ever reference the n8n container. It did not mention n8n-runner at all. The n8n-runner container doesn't get a port that maps to the load balancer, doesn't get its own listener, doesn't get its own DNS entry. Everything that makes it reachable from outside the task goes through n8n . Phase 7: DNS At your registrar, add a CNAME: Host n8n, Value = your Load Balancer's DNS name. Confirm with nslookup n8n.yourdomain.com once it propagates. Verifying the Sidecar Relationship Visiting https://n8n.yourdomain.com and completing owner setup confirms the main container and database are working. To confirm the runner specifically: Create a workflow with a Code node (JavaScript or Python), and run it.Pull CloudWatch logs for both streams (/ecs/n8n-task, prefixes n8n and n8n-runner). The n8n-runner stream should show the launcher starting both runner processes and reporting a broker connection. The n8n stream should show the Code node's execution dispatched out rather than run inline. If the workflow completes but nothing appears in n8n-runner's logs, check N8N_RUNNERS_MODE=external on the main container first. That's the setting that actually hands execution off instead of running it in-process regardless of what else is configured.