DZone Spotlight

Saturday, September 19 View All Articles »

Stop Blaming Executor Memory: The Real Reasons Your Spark Jobs Are Slow

After a decade of building and debugging large-scale data pipelines across financial services, payments processing, and analytics platforms, I can tell you that almost every slow Spark job I've investigated had the same root cause — and it wasn't the one the team thought it was. The default response when a Spark job is slow is to add more executor memory, increase the number of executors, or bump spark.sql.shuffle.partitions. Sometimes that helps. Usually it doesn't. What I've found, consistently, is that the real problems are structural — a join strategy mismatch that silently multiplies your intermediate dataset by ten times, a single slow task on a degraded node that holds an entire stage hostage, or a decrypt chain that re-reads source data six times when it only needed to read it once. This article is organized around five patterns I keep seeing across teams. Each one looks different on the surface but traces back to a misunderstanding of how Spark actually executes your code. For each pattern, I'll describe what it looks like, when it bites you, the failure mode, and how to fix it. Pattern 1: The OR Join That Quietly Multiplies Your Data What It Looks Like A join condition with an OR clause. Usually introduced when a business requirement adds a secondary matching rule — match on primary card number, or if the transaction is a virtual card transaction, match on the underlying physical PAN. The SQL looks reasonable. The engineer tests it on a sample, and it returns the right rows. When It Bites You At scale. With 100 million transaction rows and 50 million account rows, this query starts running for hours. The output size is also wrong — much larger than expected before DISTINCT trims it down. The Failure Mode Spark cannot use a hash join or sort-merge join when the join condition contains OR. It falls back to BroadcastNestedLoopJoin — for every row in the left table, scan every row in the right table. That's O(n x m). On real datasets, this produces an intermediate result in the hundreds of GB before any downstream filter runs. I've watched a pipeline that should produce 8 GB of output generate 400 GB of intermediate data because of exactly this pattern, taking a 20-minute job to 4 hours. You can verify this in 30 seconds: run df.explain(formatted) and look for BroadcastNestedLoopJoin in the physical plan. If you see it on a join involving any table over a few million rows, it's almost certainly unintentional. The Fix Split the join into two equi-join legs and UNION ALL the results: SQL -- Leg 1: primary match (equi-join — uses SortMergeJoin or BroadcastHashJoin) SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.card_number = acct.card_number UNION ALL -- Leg 2: fallback match, filtered scope only SELECT txn.*, acct.* FROM transactions txn JOIN accounts acct ON txn.fpan = acct.physical_pan WHERE txn.transaction_type = 'VIRTUAL' Each leg is a proper equi-join. Apply DISTINCT at the end to deduplicate rows that matched both. The performance difference is routinely an order of magnitude. Pattern 2: The Straggler Task That Nobody Notices Until It's Too Late What It Looks Like A stage that should take 10 minutes takes 3 hours. The Spark UI shows nearly all tasks completed quickly. One or two tasks are still running with a disproportionately long duration. When It Bites You Jobs running on shared YARN or cloud infrastructure where any node can have a bad disk, a noisy neighbor, or degraded network throughput. Also common in stages that call external services per partition — one slow API response can cause a single partition's tasks to take 100x longer than the others. The Failure Mode A stage doesn't complete until the last task completes. Not the median. Not p95. The absolute last one. If 2,200 tasks finish in under 2 minutes and one takes 3 hours and 7 minutes, the stage takes 3 hours and 7 minutes. The other 2,199 executors sit idle. This is the straggler problem, and it's distinct from data skew. The diagnostic: in the Stage detail view, check the task duration distribution. If MAX is dramatically higher than p99, that's a straggler (hardware or external service issue). If p75 is already much higher than p50, that's skew (data distribution issue). They require different fixes, and many teams treat them identically. The Fix For stragglers caused by degraded infrastructure, enable Spark speculation: Properties files spark.speculation=true spark.speculation.multiplier=3 # task must be 3x slower than median spark.speculation.quantile=0.9 # wait for 90% completion before speculating Speculation re-launches slow tasks on a different executor and uses whichever copy finishes first. The caveat: don't use this on stages that write to non-idempotent sinks. For read-heavy or compute-heavy stages — including external decryption calls — it's often the single most impactful config change you can make. Pattern 3: The df.rdd Decrypt Chain That Recomputes Everything Six Times What It Looks Like A pipeline that calls an external encryption or decryption service per record, implemented as a series of df.rdd.mapPartitions() calls, one per column that needs to be processed. When It Bites You When you have multiple columns to decrypt. Each .rdd call creates a new computation starting from the original DataFrame — Spark re-reads from source, re-executes all upstream joins and filters, and then runs the decryption for that column. With six columns to decrypt, you're doing that six times. The Failure Mode Two distinct sub-problems compound each other. First, going to RDD bypasses Catalyst entirely — no predicate pushdown, no column pruning, no Tungsten execution. Second, without a persist checkpoint before the chain, every decrypt call lineages all the way back to the source. I've seen this double the runtime of a job compared to the same pipeline with a single persist() before the decrypt chain. On top of that, the external call latency per partition is dominated by the number of HTTP round trips, not the payload size. Cutting your batch size in half doubles your request count and roughly doubles your wall-clock time for that stage. Most teams set an initial batch size and never revisit it. The Fix Two changes, applied together: Persist the input DataFrame before starting the decrypt chain. This means the join and filter logic runs once, and each decrypt call reads from the cached result.Increase the batch size for external calls. Test at several sizes — going from 20,000 to 40,000 records per batch often cuts stage time by 30-50% with no change to correctness. Scala val base = rawDf.filter(...).join(key1, ...).persist(StorageLevel.MEMORY_AND_DISK) val step1 = decryptColumn(base, secret1) // reads from cache val step2 = decryptColumn(step1, secret2) // reads from cache val step3 = decryptColumn(step2, secret3) // reads from cache Without persist, step2 re-executes everything step1 did from source. With persist, each step reads from the in-memory result of the previous. Pattern 4: The shuffle.partitions Setting That Nobody Updates What It Looks Like A job that works fine in staging — where data volumes are 10% of production — but runs slowly, spills to disk, or produces thousands of tiny output files in production. When It Bites You When the default spark.sql.shuffle.partitions=200 is left unchanged. 200 partitions made sense as a default for medium datasets but is almost always wrong at production scale — either too few (huge partitions, memory pressure) or too many (tiny partitions, scheduling overhead, small files problem). The Failure Mode Too few partitions means each executor handles a disproportionately large chunk of data. With 200 partitions on a 1 TB shuffle, each partition is 5 GB. That will spill to disk. Too many partitions means thousands of 1 MB tasks — the scheduling overhead becomes significant, and your output has thousands of tiny files that hurt downstream readers. With Adaptive Query Execution (AQE) enabled in Spark 3.2+, this problem largely manages itself. AQE merges small post-shuffle partitions automatically and can handle modest skew. But AQE can't help if it's disabled, and it can't fix the upstream causes of extreme skew. The Fix Enable AQE if you're on Spark 3.2+: Properties files spark.sql.adaptive.enabled=true spark.sql.adaptive.coalescePartitions.enabled=true spark.sql.adaptive.skewJoin.enabled=true If you need to set shuffle.partitions manually, target roughly 128-256 MB per partition post-shuffle. For a 500 GB shuffle, that means 2,000-4,000 partitions. Set it high and let AQE coalesce down — that's cheaper than setting it low and getting OOM errors. Pattern 5: The Incremental Job That Degrades Silently Over Time What It Looks Like A job that runs in 15 minutes when first deployed and runs in 4 hours six months later. No code changes. No obvious data quality issues. The team attributes it to data growth. When It Bites You When the job fails a few times in a row, and the recovery accumulates multiple windows' worth of data. Or when the watermark logic was designed for small windows but nobody anticipated that the underlying join tables would grow significantly. The Failure Mode Two separate causes, often confused. First, if the watermark is a single timestamp and the job has been failing, recovery runs can accumulate large backlogs. A job that normally processes 2 hours of data may need to process 48 hours on first successful recovery, with no change to the resource configuration. Second, growth in reference data (like an accounts table or lookup table used in a join) increases the size of every run regardless of whether the incremental input grew. I've seen a 30-minute job become a 3-hour job purely because the accounts table grew from 10 million rows to 80 million rows over 18 months, while the OR join condition (see Pattern 1) meant that growth was amplified into the intermediate result. The Fix Two design principles that pay off over the lifetime of the pipeline: Track processed partitions explicitly rather than using a single timestamp watermark. This makes recovery granular — you can replay specific missing partitions without re-processing everything after them.Add a fast-path no-op check before initializing the full Spark session. Check whether any new partitions exist first. A 5-second check that exits early is much better than a 2-minute executor startup that discovers there's nothing to process. For the reference table growth problem: if your lookup table grows significantly, revisit whether it can be broadcast (small enough to fit in executor memory) or whether the join itself needs to be redesigned. Quick Diagnostic Reference Use this table to map what you observe in the Spark UI to the likely pattern and first action to take: WHat you observeLikely patternconfirm withfirst action MAX task duration >> p99 Straggler (Pattern 2) Task timeline in Stage UI Enable spark.speculation p75 >> p50 task duration Data skew Input bytes per task Repartition on join key; AQE skewJoin BroadcastNestedLoopJoin in explain() OR join (Pattern 1) df.explain( formatted) Rewrite as UNION of equi-joins Stage runtime grows week on week; no code change Incremental accumulation or reference table growth (Pattern 5) Input bytes trend in History Server Audit watermark logic; check reference table size OOM errors or heavy disk spill Too few shuffle partitions (Pattern 4) Spill metrics in Stage UI Enable AQE or increase shuffle.partitions The Common Thread Every pattern here traces back to the same underlying issue: Spark is executing something different from what the engineer intended. The OR join was intended as a flexible matching rule; Spark turned it into a nested loop. The decrypt chain was intended as six independent transformations; Spark turned it into six full re-reads of source data. The incremental job was intended to process one window of data; without proper watermark design, it occasionally processes twelve. The Spark UI has everything you need to see this — task distribution, input and output sizes, physical plans, spill metrics. Most teams open it when something breaks and close it once they find the obvious error. Opening it proactively, forming a hypothesis, and then confirming or refuting it in the metrics is the practice that separates engineers who consistently improve pipeline performance from those who add executor memory and hope for the best. The mistake isn't choosing the wrong config. It's not understanding what Spark is actually doing with your code. More

Federated MCP Control Plane: Policy-Aware Access to Multi-Backend Tool Servers

By Harish Gaggar

A federated gateway provides secure, policy-aware access to tool servers. The thing that made me stop and rethink our whole approach to agentic tooling was a text file. An engineer on one of our platform teams had wired an AI coding assistant up to our internal source control. To do it, they had pasted a personal access token into a local MCP server config in their home directory. It worked. That also meant a long-lived credential with broad repository scope sat in plaintext in a file the agent could read, on a laptop, with no audit trail and no expiry. Multiply that by every engineer who wants their assistant to see internal code, artifacts, docs, and warehouse tables, and you have hundreds of copies of your crown-jewel credentials distributed across endpoints you do not control. That is the real problem with Model Context Protocol adoption in an enterprise. MCP itself is a good protocol. The failure mode is topological: the default deployment story puts the server, the credentials, and the client on the same machine, which is exactly where you least want them in a network-isolated environment. What we built instead was a federated control plane. One gateway, many backend tool servers, and a thin local connector that holds no secrets at all. The Three-Hop Topology The pattern is simple to state, and most of the engineering effort goes into the seams: Plain Text Connector -> Gateway -> Server The connector runs locally next to the IDE or agent. It speaks stdio to the client, because that is what most assistants expect, and streamable HTTP outbound to the gateway. It is deliberately dumb. It knows one URL and how to complete a browser-based login. It stores no client secret, no API key, no PAT. The gateway is the control plane. It terminates authentication, brokers OAuth on the user's behalf, resolves which backend server should handle a given request, enforces policy, and emits telemetry. It is the only component that ever touches a credential. The backend servers are the actual MCP implementations: source control, artifact repository, documentation search, static analysis, browser automation, warehouse metadata. Each is a separate deployment with its own least-privilege identity. They live in-cluster, on the internal network, with no default egress to the public internet. The property that matters is that the trust boundary sits at the gateway, not at the laptop. A compromised developer machine yields a session, not a credential. The Gateway as an OAuth Broker This is the part people underestimate. The gateway does not proxy the user's token; it exchanges an authenticated session for a narrowly scoped downstream credential, per backend, per request. Concretely, when a request arrives, the gateway resolves the caller's identity from the session, looks up the target server, and mints or fetches a downstream token with only the scopes that server is registered to need: Python async def broker(request: MCPRequest, session: Session) -> MCPResponse: server = registry.resolve(request.server_id) if server is None: raise PolicyError("unregistered_server") if not policy.allows(session.principal, server, request.method): audit.deny(session.principal, server.id, request.method) raise PolicyError("not_permitted") # Client secrets are held by the gateway only; never sent downstream # to the connector and never written to a client-side config. token = await broker_pool.token_for( principal=session.principal, provider=server.auth_provider, # e.g. saml_scm, google scopes=server.least_privilege_scopes, # e.g. ["repo:read"] ttl_seconds=900, ) return await transport.forward(server, request, bearer=token) Two design choices are worth calling out. First, least_privilege_scopes is a property of the registered server, not of the user's login. A developer authenticating once through the gateway does not thereby grant every backend the union of their permissions. A documentation server gets read scope on docs and nothing else, even if the same human has admin rights elsewhere. Second, we deliberately started with a static client registration model backed by the platform's own secret store, with a migration path to Dynamic Client Registration. DCR is where this should end up, but shipping a working broker with rotating short-lived tokens beat waiting for the spec ecosystem to settle. Secrets are created by CI/CD from a managed secret store; no human hands a production secret to a running workload. Guardrails Against Tool Poisoning Once agents can call tools, tool descriptions become an attack surface. A malicious or compromised server can return a tool definition whose description instructs the model to exfiltrate context, or can silently mutate a description after initial approval. Rate limiting alone does not help here. We enforce validation at the gateway in both directions of the exchange: Python POISON_PATTERNS = [ r"ignore (all )?(previous|prior) instructions", r"do not (tell|inform|mention to) the user", r"<\s*(system|assistant)\s*>", ] def validate_tool_manifest(server_id: str, manifest: dict) -> None: for tool in manifest["tools"]: blob = f"{tool['name']} {tool.get('description', '')}" for pattern in POISON_PATTERNS: if re.search(pattern, blob, re.IGNORECASE): quarantine(server_id, tool["name"], reason=pattern) raise PolicyError("suspect_tool_description") # Descriptions are pinned at review time. Drift requires re-approval. if sha256(blob) != registry.approved_digest(server_id, tool["name"]): raise PolicyError("manifest_drift") The digest pinning is the load-bearing control. Pattern matching catches the naive cases; pinning catches the case where an approved server changes its behavior after review. Any drift takes the tool out of rotation until a human re-approves it. On top of that: per-principal and per-server rate limits, an explicit allow/block list of methods, and argument validation before forwarding. We mapped these controls to published guidance for AI system risks so the security review had something concrete to assess rather than a narrative. Observability Is Not Optional Here When something goes wrong in an agentic workflow, the user's report is usually "the assistant got confused." That is not debuggable. Centralizing traffic through one gateway means you get, for free, the telemetry that makes it debuggable: latency percentiles per server and per method, error rates by status code, MCP method distribution, transport breakdown between stdio and streamable HTTP, and per-principal activity. Two things surfaced from that data that we would never have found otherwise. One backend was returning successful responses with empty payloads for a large share of calls, which looked healthy on an error-rate dashboard and terrible to users. And tool usage was heavily concentrated: a small number of servers and a small number of engineers accounted for most traffic, which told us where to spend reliability effort instead of guessing. Making It Self-Service, or It Dies A control plane that requires a platform engineer in the loop becomes the bottleneck it was meant to remove. The onboarding path we settled on is a scaffolded repository from an internal portal, image build and promotion through CI, infrastructure-as-code deployment via pull request, automated vulnerability scanning, and auto-registration into the gateway registry on merge. New server idea to registered production service is one pull request and two approvals. The lesson I would pass on: solve the credential topology first, then the ergonomics. Teams that start with developer convenience end up retrofitting security onto a distributed pile of local configs, and that retrofit is far more expensive than getting the trust boundary right on day one. More

How to Test Web Accessibility Using Playwright and Axe-Core

By Sidharth Shukla

Common Pitfalls in RAG Applications: What to Avoid When Using Vector Search and Embeddings

By Seshendranath Balla Venkata

Refcard #291

Code Review Core Practices

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Refcard #267

Getting Started With DevSecOps

By Akanksha Pathak

CORE

The New API Contract Is Probabilistic: Building Reliable Systems Around Unreliable Model Outputs

For decades, API design rested on a reassuring assumption: given valid input and a stable dependency, software should return a predictable result. Large language models break that assumption without breaking the API. A request can receive HTTP 200, perfectly valid JSON, and a confidently wrong answer. That distinction matters. The network contract may still be deterministic, but the semantic contract is now probabilistic. A model endpoint does not promise one correct output; it samples a likely output from a distribution shaped by the prompt, context, model version, retrieval results, and decoding process. Machine-learning engineer Chip Huyen puts it plainly: “LLMs are stochastic; there’s no guarantee that an LLM will give you the same output for the same input every time.” Reliable AI systems begin when developers stop treating that behavior as an exception. Redefine What the Contract Guarantees The traditional contract defines fields, types, status codes, and errors. The AI contract, however, has to specify the acceptable behavior: what kind of evidence the model can accept, what failure categories are allowed, when to abstain, what latency and costs budget are in place, and how to proceed in case there is not enough confidence. Confidence has to be measured using evidence coverage, validation outcomes, or classifier calibration, but not the self-assessment of the model. Thus, the goal changes from "function returns correct value" to something measurable: for a certain slice of traffic, the system passes some quality criteria at an acceptable frequency. Different tasks require different criteria. For example, summarizing movies does allow for some awkward sentences. Changing a customer's credit limit does not allow any inventions or ambiguities. Put a Deterministic Envelope Around the Model The model should be one component inside ordinary software, not the authority at the center of it. The surrounding application should normalize inputs, constrain outputs, validate results, and choose whether to accept, retry, fall back, or escalate. Structured generation is the first layer. Use a JSON Schema, enums, required fields, and explicit null states instead of asking for “JSON” in a prompt. OpenAI, for example, reported 100% schema adherence for one model in its complex JSON Schema evaluation. That solves a parsing problem, not a truth problem. A fabricated invoice number can still be a perfectly valid string. Semantic validation must follow structural validation. Check identifiers against source systems, dates against business rules, citations against retrieved passages, and calculated values with deterministic code. Treat every generated field as untrusted input. The control flow should be explicit: Plain Text generate -> validate schema -> verify evidence -> apply policy -> accept | bounded retry | fallback | human review The important output is not merely the model’s answer. It is a typed system decision such as accepted, rejected, or needs_review, accompanied by evidence and a machine-readable failure reason. Keep Side Effects Behind a Transaction Boundary Probabilistic text becomes dangerous when it can directly create a refund, delete a record, or send a message. Separate proposing an action from committing it. Let the model select only from allow-listed tools and produce typed arguments. Then let deterministic code authenticate the user, authorize the operation, verify current state, and enforce limits. Add idempotency keys so a retry cannot repeat a payment or ticket creation. For high-impact actions, show a preview or require human approval. This architecture also limits prompt injection. Untrusted content may influence a proposal, but it should never grant the model new permissions. Make Retries a Policy, Not a Reflex Retries can repair malformed output or a transient timeout. They can also multiply cost, latency, and side effects while reproducing the same semantic error. Set a small attempt budget and retry only failures that may be recoverable. Feed validation errors back in a structured form, use exponential backoff for provider faults, and stop when the remaining time or token budget is insufficient. If the evidence is missing, another generation is not a remedy; retrieval, clarification, or abstention is. Fallbacks should match the risk. A smaller model, cached result, or rules engine may preserve availability. A safe refusal or human queue may be the correct degraded mode when correctness matters more than speed. Test Distributions, Not Favorite Prompts A handful of convincing demos proves little. Build an evaluation set from real tasks, known edge cases, adversarial inputs, and failures observed in production. Run each important case multiple times when sampling variability matters, and report pass rates with confidence intervals rather than one aggregate score. Evaluation practitioners Hamel Husain and Shreya Shankar offer excellent advice: “Start with error analysis, not infrastructure.” Review traces with domain experts, classify concrete failures, then automate the checks that matter. Prefer deterministic assertions for schema, policy, and executable code; reserve model-based judges for qualities that rules cannot capture, and calibrate those judges against human labels. Version the entire behavior-producing system: model identifier, prompt, schema, retrieval corpus, tool definitions and safety rules. Run regression suites and canary traffic before changing any of them. In production, monitor validator failures, abstentions, retries, latency, cost, and user corrections. Store redacted traces where privacy permits, because averages alone rarely explain why a system failed. Reliability Moves Outward The model does not need to become deterministic for the product to become dependable. Databases still fail, networks still partition, and users still submit hostile input; engineering makes those systems useful by containing uncertainty. Generative AI demands the same discipline, applied at the semantic boundary. Define measurable behavior, constrain the output, verify claims, isolate side effects, test continuously, and fail safely. Google’s site reliability literature opens with a durable warning: “Hope is not a strategy.” With probabilistic APIs, it is not a contract either.

By Micheal Chukwube

A Senior Engineer’s Guide to Foundry IQ, MCP, and the OpenAI Agents SDK

Most Foundry writeups assume you're all in on Microsoft's stack end to end: the Agent Framework for orchestration, the Foundry Agent Service for hosting, and the Responses API wrapped in Microsoft's own client. That's a reasonable default, but it's not the only shape this can take. Microsoft Foundry hosts OpenAI's own models behind an OpenAI-compatible endpoint, and Foundry IQ exposes every knowledge base as a plain MCP server. Put those two facts together, and you get a genuinely different setup: OpenAI's own Agents SDK, unmodified, orchestrating a model that happens to be running on Foundry, grounded by a knowledge base that happens to be Foundry IQ, with MCP as the only thing that has to agree between them. This is a hands-on guide to building exactly that. Not because you should always prefer OpenAI's SDK over Microsoft's own tooling, but because knowing this path exists changes how you think about lock-in. If your orchestration layer is a thin, protocol-based client, swapping the model host or the knowledge layer underneath it is a config change, not a rewrite. The Mental Model First Three pieces, from three different places, held together by two protocols: An OpenAI model, hosted on Microsoft Foundry. Foundry deploys OpenAI's models behind an endpoint that speaks the same wire format as OpenAI's own API, including the Responses API. Point any OpenAI-compatible client at that endpoint with a different base_url and it has no idea it's not talking to OpenAI directly.A Foundry IQ Knowledge Base, doing the same job it always does: chunking, embedding, indexing, and agentic retrieval over your sources. What matters here is that every knowledge base speaks MCP natively. It doesn't care what called it.The OpenAI Agents SDK, running as your orchestration layer, in your own process, not inside Foundry at all. It calls the Foundry-hosted model for reasoning and generation, and calls the Foundry IQ Knowledge Base as an MCP tool for grounding. Neither call requires Microsoft-specific code. The thing worth sitting with here: nothing about this setup is a workaround or an unsupported hack. Foundry explicitly documents the OpenAI SDK as the recommended client when you want maximum OpenAI compatibility or the lowest latency path to a Foundry-hosted model. Foundry IQ explicitly exposes MCP as a first-class interface, not an afterthought. This guide is just connecting two things that were each already built to be connected this way. Prerequisites You'll need: A Microsoft Foundry project with an OpenAI model deployed (a gpt-5.1 or similar deployment, created through the Foundry portal or the Foundry SDK).A Foundry IQ Knowledge Base already built and populated. If you haven't done this before, the short version is a Knowledge Source pointed at your data plus a knowledge base wrapping it, both created through azure-search-documents. The full walkthrough, chunking strategy, semantic configuration, and all, is worth its own read if you're starting from zero.Python 3.10+ with the OpenAI SDK and the Agents SDK installed. Shell pip install openai openai-agents azure-identity Step 1: Point a Plain OpenAI Client at Your Foundry Deployment Before bringing the Agents SDK into it, confirm the basic connection works with the plain OpenAI client. This is the part that trips people up the least, but it's worth isolating as its own step, because if it doesn't work here, nothing built on top of it will either. Python from openai import OpenAI from azure.identity import DefaultAzureCredential, get_bearer_token_provider token_provider = get_bearer_token_provider( DefaultAzureCredential(), "https://ai.azure.com/.default" ) client = OpenAI( base_url="https://YOUR-RESOURCE-NAME.services.ai.azure.com/api/projects/YOUR-PROJECT/openai", api_key=token_provider, ) response = client.responses.create( model="gpt-5.1", input="Say hello in one sentence.", ) print(response.output_text) Two things to get right here. The base_url is your Foundry project endpoint with /openai on the end, not the raw resource endpoint, and not the older /openai/v1/ Azure OpenAI-specific path (that one still works for Azure OpenAI resources, but the project endpoint is the current recommended shape for Foundry). And api_key accepts a callable token provider, not just a string, which is how Entra ID authentication slots in without you having to manually refresh anything. Step 2: Swap in the Token Provider and Hand the Client to the Agents SDK The Agents SDK doesn't have its own concept of Azure authentication. It just needs an AsyncOpenAI client, and it doesn't care where that client points. Python from openai import AsyncOpenAI from agents import set_default_openai_client, set_tracing_disabled async_client = AsyncOpenAI( base_url="https://YOUR-RESOURCE-NAME.services.ai.azure.com/api/projects/YOUR-PROJECT/openai", api_key=token_provider, ) set_default_openai_client(async_client) One gotcha worth flagging immediately: the Agents SDK ships with built-in tracing that exports run traces to OpenAI's own platform dashboard by default. That's a sensible default when you're calling OpenAI directly, but it's an odd one once your model calls are routed through Foundry instead, since your traces would still be leaving through a separate, OpenAI-direct path that doesn't share your Foundry project's auth or data boundary. If that matters for your compliance posture, disable it or point it at your own collector: Python set_tracing_disabled(True) # or, to keep tracing but redirect it, register a custom trace processor instead This is easy to miss because nothing breaks if you leave it on. It just quietly sends run metadata somewhere your Foundry-hosted setup otherwise never touches. Step 3: Connect to the Knowledge Base Over MCP Every Foundry IQ Knowledge Base exposes itself at a predictable MCP endpoint. The Agents SDK's MCPServerStreamableHttp class is built for exactly this kind of self-managed, HTTP-based MCP server. Python from agents.mcp import MCPServerStreamableHttp, create_static_tool_filter kb_server = MCPServerStreamableHttp( name="foundry-iq-kb", params={ "url": "https://YOUR-SEARCH-SERVICE.search.windows.net/knowledgebases/team-kb/mcp?api-version=2026-05-01-preview", "headers": {"api-key": "YOUR-SEARCH-ADMIN-KEY"}, "timeout": 15, }, cache_tools_list=True, tool_filter=create_static_tool_filter(allowed_tool_names=["knowledge_base_retrieve"]), ) cache_tools_list=True is worth defaulting to here. A knowledge base publishes exactly one tool, knowledge_base_retrieve, and that isn't going to change between requests, so there's no reason to pay a tools/list round trip on every single agent turn. The tool_filter is mostly redundant given there's only one tool to begin with, but it's cheap insurance if the knowledge base ever grows to a second tool you don't want this particular agent touching. Step 4: Build the Agent and Run It With the client and the MCP server both wired up, the agent itself is short. Python import asyncio from agents import Agent, Runner async def main(): async with kb_server as server: agent = Agent( name="support-agent", instructions=( "Answer questions using the knowledge_base_retrieve tool. " "Always call it before answering. Preserve [ref_id:N] citations " "from the tool's response in your final answer." ), model="gpt-5.1", mcp_servers=[server], ) result = await Runner.run(agent, "What's our current rate limit on the export API?") print(result.final_output) asyncio.run(main()) The model string here is the Foundry deployment name, not an OpenAI model ID, since every call now routes through the client you registered in Step 2. If you want to stream the response instead of waiting for the full turn, Runner.run_streamed gives you the same event-based streaming interface regardless of which backend is actually generating the tokens: Python result = Runner.run_streamed(agent, "What's our current rate limit on the export API?") async for event in result.stream_events(): if event.type == "raw_response_event" and hasattr(event.data, "delta"): print(event.data.delta, end="", flush=True) Nothing in either of these two blocks is Foundry-specific or Azure-specific. That's the point. The vendor-specific work all happened in Steps 1 through 3, in the client and connection setup, not in how you define or run the agent. Where the Credentials Actually Live Two separate credentials are doing two separate jobs here, and it's worth being precise about which is which, because they fail differently. The model credential is whatever you passed as api_key on the AsyncOpenAI client, a Foundry project token from DefaultAzureCredential, or a static API key if you're using key-based auth on the resource. This is checked on every responses.create() call the Agents SDK makes internally when the agent reasons or generates a final answer. Scope this to the project, not the whole Foundry resource, using the same RBAC roles you'd use for any other Foundry SDK client (Cognitive Services User is usually sufficient for inference-only access). The knowledge base credential is the api-key header on the MCP server's params, and it's checked independently by Azure AI Search when the knowledge_base_retrieve tool gets called. These two credentials can be, and generally should be, scoped to completely different principals. A key that can call your Foundry model deployment shouldn't automatically be able to query every knowledge base on your Search service, and the reverse is just as true. If you're building anything past a prototype, put each behind its own least-privilege identity rather than reusing one Foundry project's admin key for both. If your knowledge base sits over permission-sensitive content, this is also where the on-behalf-of pattern from Foundry IQ's own permission model applies unchanged: thread the requesting user's token through as an additional header on the MCP params, since the KB's enforcement of ingestionPermissionOptions doesn't know or care that the caller this time is the OpenAI Agents SDK instead of the Foundry Agent Service. Production Considerations Before You Commit Decide on tracing deliberately, not by default. Leaving the Agents SDK's tracing on means run metadata leaves through an OpenAI-direct path that bypasses your Foundry project's boundary entirely. Turn it off or replace it with a custom processor as a first-day decision, not something you notice in a security review months later.Cache the tool list, but know when to invalidate it. cache_tools_list=True avoids a redundant round trip, but if you ever change what a knowledge base exposes, which is rare but not impossible as Foundry IQ's MCP surface evolves, a long-lived process holding a stale cached tool list will keep calling the old shape until it's restarted.Separate the model deployment's quota from the knowledge base's query load. These are billed and throttled independently. A burst of retrieval-heavy queries against the knowledge base won't show up as pressure on your model deployment's tokens-per-minute limit, and the reverse is also true, so alert on both rather than assuming one is a proxy for the other.Pin the MCP API version. The api-version=2026-05-01-preview query parameter on the knowledge base's MCP URL is still a preview surface as of this writing. Track it the same way you'd track any other preview dependency, and don't assume a bare /mcp URL without a version pin will behave identically across a Foundry IQ update.Keep the instructions honest about tool use. The Agents SDK does not force a tool call. If your instructions say "always call knowledge_base_retrieve" but the model decides a question doesn't need it, you'll get an ungrounded answer with no error. Log whether the tool was actually invoked on each run, not just what the final answer said, if grounding is a correctness requirement rather than a nice-to-have. Where This Leaves You The interesting thing this setup demonstrates isn't that OpenAI's SDK can technically reach a Foundry endpoint. It's that both vendors built their integration points — an OpenAI-compatible inference endpoint on one side, an MCP-native knowledge base on the other — generally enough that they compose without either one knowing the other exists. That's a genuinely different bet than the usual platform story, where the value proposition is staying inside one vendor's tooling end to end. If you're already committed to the OpenAI Agents SDK for orchestration, whether for its tracing, its handoff model, or just team familiarity, you don't have to give that up to use Foundry-hosted models or Foundry IQ's retrieval layer. The protocol boundary is the only thing that has to hold, and both sides are already built to it. References Microsoft Learn. "Get started with Microsoft Foundry SDKs and endpoints." learn.microsoft.com/en-us/azure/foundry/how-to/develop/sdk-overviewMicrosoft Learn. "Use the Azure OpenAI Responses API." Microsoft Foundry. learn.microsoft.com/en-us/azure/foundry/openai/how-to/responsesMicrosoft Learn. "How to migrate from Azure AI Inference SDK to OpenAI SDK." Microsoft Foundry. learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migrationOpenAI. "Configuration." OpenAI Agents SDK documentation. openai.github.io/openai-agents-python/configMicrosoft Learn. "What is Foundry IQ?" learn.microsoft.com/en-us/azure/foundry/agents/concepts/what-is-foundry-iq

By Jubin Soni, FBCS

CORE

Understand the Sidecar Pattern by Deploying n8n to AWS Fargate

A sidecar is a container that runs alongside another container as part of the same deployment unit. Just because two containers are in the same cluster or deployed around the same time doesn't make one a sidecar. There are two things that make a sidecar. First is that they share a network namespace, so they can reach each other over localhost rather than a network address. Second, they share a lifecycle. This means that they are created together, scaled together, and by default torn down together. Neither container has an existence independent of the other. The problem it solves is giving a specific concern its own boundary. For example, it can have its own filesystem, its own memory space, and often its own permissions or dependency set, without giving up the simplicity of deploying and operating one unit. You get isolation without paying for the operational overhead of running and coordinating a fully separate service. The test that defines the pattern across all of these is this: does it live and die with its partner container as one unit of deployment? If yes, it's a sidecar. If you have to reach it by hostname, through service discovery, or via a queue, it isn't one anymore. That is a separate service that happens to sit next to the first. That test matters because two adjacent patterns get called "sidecar" when they aren't: Decoupled worker/microservice. A separately deployed container, reached over the network, scaled on its own. A web application offloading work to Celery workers via Redis is a common instance of this: the app enqueues a job (send this signup email), a pool of workers pulls jobs off the queue independently, and neither side shares a network namespace or a lifecycle with the other. The workers scale on queue depth, not on how many web replicas are running, and a web app restart doesn't take queued or in-flight jobs down with it. n8n has its own version of the same shape: "queue mode," where a main node accepts webhooks and separate worker nodes pull jobs off a Redis queue. It's tempting to call either of these a sidecar relationship since the worker and the web app do feel paired, but neither qualifies: they don't share a deployment unit, and killing one doesn't touch the other.Ambassador/adapter. A container that proxies or translates traffic on its parent's behalf, like the Envoy example above, is actually this, more precisely. Structurally it's still a sidecar; it just gets a more specific name for what it does. Using n8n to Understand It What n8n Is n8n is a workflow automation platform like Zapier, but self-hostable and node-based rather than form-based. A handful of components make up a running instance: The editor/UI, where workflows are built visually as a graph of nodes.The main process, which serves that UI, listens for webhooks, and orchestrates workflow execution. The workflow execution decides what runs next, passing data between nodes and recording results.Nodes, the individual units of a workflow: trigger nodes (a webhook arrives, a schedule fires), action nodes (call an API, write to a database, send an email), and the Code node. The code node lets you drop in arbitrary JavaScript or Python to transform data however the built-in nodes can't. The code node is relevant in this article. The database, where workflow definitions, credentials, and execution history persist. In this article, Postgres is used. For most of what n8n does, the main process is the only thing doing work: routing a webhook, calling an API, writing a database row. The exception is the Code node, and that exception is the whole reason task runners exist. The Task Runner Feature and Its Use Case By default, a Code node's JavaScript or Python executes inside n8n's main process. This main process holds the database connection, the encryption key, and every credential stored in every workflow you've built. That's fine for trusted, well-understood scripts. It becomes a real problem the moment the code in that node is untrusted, third-party, or arbitrary enough that you can't fully audit it before it runs. By the way, that is how most Code nodes are used in practice. Task runners exist to solve exactly that use case: run Code node logic somewhere the main process's credentials and connections aren't reachable from it, without turning "write some JavaScript to reshape this JSON" into a separately deployed microservice every time. Going Deep on the Task Runner Feature n8n ships two modes for this: Internal mode (the default) runs Code nodes inline, in-process. No isolation. This is the fastest to set up, but the weakest boundary.External mode moves execution into a separate runner process entirely. That process connects back to the main n8n instance over a broker (an authenticated connection the main process listens on) and receives individual tasks to execute rather than having any standing access to n8n's internals. The runner never touches the database connection, the encryption key, or stored credentials directly; it only ever sees the specific input data for the task it's been handed. External mode goes further than just "a different process," too. The runner's own configuration (the n8n-task-runners.json file built in Phase 4) sets explicit allowlists — which environment variables the runner process can see at all, and which JavaScript built-ins or Python modules it's permitted to import, standard library and third-party tracked separately. So the boundary isn't just "different memory space," it's "different memory space, plus a declared, auditable list of exactly what this process is allowed to touch." That's a specific concern (arbitrary code execution) given its own boundary, without turning it into a fully independent service you have to deploy, discover, and monitor separately. It's the sidecar problem, stated exactly: external mode gives you the isolation; running the external runner as its own container in the same task definition is what makes that isolation a sidecar rather than just a separate process sharing a machine. Why This Needs to Scale Independently and Why "In the Same Container" Isn't Enough Most n8n deployment guides run n8n with task runners in internal mode, or with the external runner living inside the same container as the main process. For example, you will see guides about deploying n8n on a single EC2 instance, Render, DigitalOcean, or any platform's basic tier. That gets you the process isolation, which solves the security half of the problem. It doesn't solve the other half, which is that a runner sharing a container with the app can't be scaled, resourced, or restarted independently of it. That stops mattering the moment Code-node execution becomes the actual bottleneck rather than webhook handling or UI traffic. Imagine workflows doing heavy data transformation in Python, running numpy/pandas operations across large payloads, or executing many Code nodes concurrently. If the runner is bundled into the main container, giving it more CPU means giving the entire n8n instance more CPU, whether the UI and webhook layer need it or not. There's no way to say "the runner needs 2 more vCPUs, n8n itself is fine". Why AWS Fargate's Task Definition Is the Right Fit A Fargate task definition lets each container in the task carry its own CPU and memory reservation, its own health check, and its own essential flag governing what happens if it fails while still keeping every container in the task on one shared network interface. That's the sidecar promise made literal: isolation and independent resourcing for the runner, without losing the operational simplicity of one task, one deploy, one thing to scale as a unit when you do want to scale both together. The rest of this guide deploys exactly that: one Fargate task, two containers, wired together the way the definition above requires. Each infrastructure decision below gets tied back to a specific part of what's laid out here, so that by the end, the concept isn't something read once at the top, but it's something built. Prerequisites AWS account with billing enabledA domain you control, with DNS accessDocker installed locally, with docker buildx availableAWS CLI configured (aws configure) with permissions for ECR, ECS, RDS, ACM, and IAMThe runner image source (Dockerfile + n8n-task-runners.json) — built in Phase 4 Architecture Markdown User's Browser (HTTPS) | [Application Load Balancer] <- Certificate Manager (SSL Cert) | (Port 5678, HTTP internal) [ECS Fargate Task] |-- Container: n8n (main) <-- shared network namespace --> Container: n8n-runner (sidecar) | (Port 5432, PostgreSQL) [RDS PostgreSQL Database] The load balancer and RDS layers are ordinary AWS plumbing. The box in the middle is where the sidecar relationship actually lives. There is one task and two containers, each with its own resourcing. Phase 1: RDS PostgreSQL RDS Console → Create database → Standard create → Engine: PostgreSQLDB instance identifier: n8n-db. Master username: postgres. Generate and save a strong master password.Instance size: db.t4g.microStorage: 20 GB gp3, autoscaling on, max 100 GBConnectivity: the VPC you'll use throughout. Public access: No. New security group: n8n-db-sg, left empty for now.Additional configuration → Initial database name: n8n. Skip this and n8n fails on first connect with "database does not exist" — the DB instance identifier names the server, this field names the database inside it.Create, wait for "Available," copy the endpoint from Connectivity & security. Phase 2: ACM Certificate n8n requires HTTPS for webhooks to function Certificate Manager, in the same region you'll deploy the Load Balancer in → Request a public certificateDomain name: n8n.yourdomain.comValidation method: DNS validationCreate the CNAME record ACM provides at your registrar. If your registrar auto-appends your domain to the Host field, paste only the portion before your domain — the full string duplicates it and validation never completes.Wait for status: Issued Phase 3: Security Groups Two connections need rules: Security groupInbound rulePurposen8n-alb-sg443 from 0.0.0.0/0Public HTTPSn8n-ecs-sg5678 from n8n-alb-sgALB → n8n containern8n-db-sg (edit existing)5432 from n8n-ecs-sgn8n container → RDS Phase 4: Build and Push the Runner Image Dockerfile: Dockerfile FROM n8nio/runners:1.121.0 USER root RUN cd /opt/runners/task-runner-javascript && pnpm add moment uuid adm-zip RUN cd /opt/runners/task-runner-python && uv pip install numpy pandas pydantic requests boto3 certifi COPY n8n-task-runners.json /etc/n8n-task-runners.json ENV N8N_RUNNERS_CONFIG_FILE=/etc/n8n-task-runners.json USER runner It starts from n8n's own n8nio/runners base (containing the launcher and both runner processes), adds only the dependencies workflows actually need, and drops back to a non-root user once the root-only install steps finish. n8n-task-runners.json is where the isolation described above stops being architectural and becomes enforced: JSON { "task-runners": [ { "runner-type": "javascript", "health-check-server-port": "5681", "allowed-env": ["PATH", "GENERIC_TIMEZONE", "NODE_OPTIONS"], "env-overrides": { "NODE_FUNCTION_ALLOW_BUILTIN": "crypto,zlib", "NODE_FUNCTION_ALLOW_EXTERNAL": "moment,uuid,adm-zip" } }, { "runner-type": "python", "health-check-server-port": "5682", "env-overrides": { "N8N_RUNNERS_STDLIB_ALLOW": "json,zipfile,io,base64,datetime,re,math,random,statistics", "N8N_RUNNERS_EXTERNAL_ALLOW": "numpy,pandas,pydantic,requests,boto3,certifi" } } ] } allowed-env restricts which environment variables the runner process can see; N8N_RUNNERS_STDLIB_ALLOW / EXTERNAL_ALLOW restrict which Python modules it can import, stdlib and third-party separately. One container, two runner processes — the launcher inside n8nio/runners spawns both. Build and push: Shell docker buildx build -t n8nio/runners:custom . aws ecr create-repository --repository-name n8n-runners --region us-east-1 aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com docker tag n8nio/runners:custom <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom --username AWS is a fixed literal, not your actual username — ECR auth always uses it. The password piped via --password-stdin is a short-lived token generated by the CLI, not your account password. Phase 5: The Task Definition This is where the two containers become an actual sidecar pair, and where the independent-resourcing argument from the introduction becomes a real field rather than a claim. JSON { "family": "n8n-task", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "executionRoleArn": "arn:aws:iam::<account-id>:role/n8n-task-execution-role", "containerDefinitions": [ { "name": "n8n", "image": "n8nio/n8n:1.121.0", "essential": true, "entryPoint": ["sh", "-c"], "command": [ "mkdir -p /home/node/certs && wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /home/node/certs/rds-ca.pem && /docker-entrypoint.sh" ], "portMappings": [{ "containerPort": 5678, "protocol": "tcp" }], "environment": [ { "name": "DB_TYPE", "value": "postgresdb" }, { "name": "DB_POSTGRESDB_HOST", "value": "<rds-endpoint>" }, { "name": "DB_POSTGRESDB_PORT", "value": "5432" }, { "name": "DB_POSTGRESDB_DATABASE", "value": "n8n" }, { "name": "DB_POSTGRESDB_USER", "value": "postgres" }, { "name": "DB_POSTGRESDB_SSL_CA", "value": "/home/node/certs/rds-ca.pem" }, { "name": "DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED", "value": "false" }, { "name": "WEBHOOK_URL", "value": "https://n8n.yourdomain.com/" }, { "name": "GENERIC_TIMEZONE", "value": "Africa/Lagos" }, { "name": "N8N_RUNNERS_ENABLED", "value": "true" }, { "name": "N8N_RUNNERS_MODE", "value": "external" }, { "name": "N8N_RUNNERS_BROKER_LISTEN_ADDRESS", "value": "0.0.0.0" }, { "name": "N8N_RUNNERS_BROKER_PORT", "value": "5679" } ], "secrets": [ { "name": "DB_POSTGRESDB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/db-password" }, { "name": "N8N_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/encryption-key" }, { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n" } } }, { "name": "n8n-runner", "image": "<account-id>.dkr.ecr.<region>.amazonaws.com/n8n-runners:custom", "cpu": 512, "memory": 1024, "essential": false, "dependsOn": [{ "containerName": "n8n", "condition": "START" }], "environment": [ { "name": "N8N_RUNNERS_TASK_BROKER_URI", "value": "http://localhost:5679" } ], "secrets": [ { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:5680/healthz || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 20 }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n-runner" } } } ] } Five fields here map directly back to the introduction: Per-container cpu/memory on n8n-runner. This is the independent-resourcing argument made literal. The runner gets its own 512 CPU units and 1024 MB, carved out of the task total, separate from whatever n8n is allotted. If Code-node execution turns out to be the bottleneck, this is the number you raise without touching the main container's allocation at all. That's the exact thing a same-container runner can't offer you. networkMode: awsvpc is the mechanical basis of "shared network namespace." Every container in the task gets one elastic network interface between them. This is the setting that makes Phase 3's missing security group rule make sense. There's one network surface, not two. N8N_RUNNERS_TASK_BROKER_URI: http://localhost:5679 only works because of the line above. The runner reaches n8n over localhost because they are the same task. If this pointed anywhere else, you would have built the decoupled-worker pattern from the introduction instead, no matter what you called the container. A shared N8N_RUNNERS_AUTH_TOKEN, pulled from Secrets Manager by both containers. Sharing a network namespace means the runner is reachable by anything else in the task. The isolation the whole pattern exists for still needs a trust boundary at the process level, not just the network level. A plaintext token here would defeat that, since task definitions are readable by anyone with ecs:DescribeTaskDefinition. essential: false on the runner. This governs how tightly the two containers' lifecycles are actually coupled. essential: true would mean a runner crash tears down the whole task, main container included. false means the runner can crash and recover independently: Code-node executions fail until it's back, but the UI and webhooks keep serving. The pattern doesn't mandate one answer; it just means this has to be a decision, not a default you inherited. The health check on port 5680 hits the launcher's own endpoint, separate from the per-runner-type ports (5681 JS, 5682 Python) set in Phase 4's config file. ECS is checking the supervisor, not each runner process individually. Register it: aws ecs register-task-definition --cli-input-json file://n8n-task-def.json Phase 6: Cluster, Service, and Load Balancer ECS → Create cluster → n8n-cluster → Infrastructure: AWS FargateCreate a service inside it: Task definition: n8n-task, latest revisionDesired tasks: 1Networking: your VPC, at least two subnets across AZs, security group n8n-ecs-sg, public IP onLoad balancing: Application Load Balancer, listener on 443 using the Phase 2 certificateTarget group: HTTP, port 5678, health check path /healthzCreate, wait for steady state. Notice the target group and health check only ever reference the n8n container. It did not mention n8n-runner at all. The n8n-runner container doesn't get a port that maps to the load balancer, doesn't get its own listener, doesn't get its own DNS entry. Everything that makes it reachable from outside the task goes through n8n . Phase 7: DNS At your registrar, add a CNAME: Host n8n, Value = your Load Balancer's DNS name. Confirm with nslookup n8n.yourdomain.com once it propagates. Verifying the Sidecar Relationship Visiting https://n8n.yourdomain.com and completing owner setup confirms the main container and database are working. To confirm the runner specifically: Create a workflow with a Code node (JavaScript or Python), and run it.Pull CloudWatch logs for both streams (/ecs/n8n-task, prefixes n8n and n8n-runner). The n8n-runner stream should show the launcher starting both runner processes and reporting a broker connection. The n8n stream should show the Code node's execution dispatched out rather than run inline. If the workflow completes but nothing appears in n8n-runner's logs, check N8N_RUNNERS_MODE=external on the main container first. That's the setting that actually hands execution off instead of running it in-process regardless of what else is configured.

By Iyanuoluwa Ajao

Stop Overfeeding Your AI Agent's Context Window

Picture a checkout page throwing the dreaded 500 error at 2 a.m. Someone opens an AI agent and asks it to fix things. The instinct is to be generous. Paste in the runbooks. Drop in three dashboards. Attach a pile of customer complaints. Let the model sort it out. More context should mean a smarter answer. Right? Not really. Researchers who studied how language models actually use long inputs found something inconvenient for anyone who pastes first and thinks later. "Performance can degrade significantly when changing the position of relevant information" Source: Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," arXiv:2307.03172 In plain English, an AI agent buried under ten documents isn't automatically wiser than one working with the two documents that actually matter. The fix for agentic AI isn't a bigger context window. It's giving the agent the right kind of knowledge, delivered the right way, at the right moment. There are four main ways to do that: skills, MCP, RAG, and memory. Each one solves a different problem. Mixing them up is where a lot of enterprise AI projects quietly go wrong. Skills: The Onboarding Manual Nobody Wrote A skill is a set of instructions for doing one specific job, sometimes with a bit of code attached. Think of it as the manual you'd hand a sharp new hire on day one. Check the error rate first. Then check recent deployments. If neither explains the problem, stop guessing and escalate to a human. Without that manual, a capable model will improvise, and improvisation is exactly what you don't want during an outage. A good skill doesn't just list steps. It carries judgment about when to follow them and when to stop. The trick is that skills only load when they're relevant, which is what keeps them cheap. Anthropic, which built this pattern into Claude, puts it plainly. Only relevant content occupies the context window at any given time (Source: Anthropic, Agent Skills documentation, ) A library of fifty skills doesn't cost fifty skills worth of context. It costs one: the one the agent actually needed for this task. But a skill can't reach outside itself. It can tell an agent to check the error rate. It can't get the agent to the dashboard. MCP: Giving the Agent Hands That's where the Model Context Protocol, or MCP, comes in. MCP is a standard way for an agent to connect to outside systems: logging tools, databases, ticketing systems, whatever a company already runs. The agent is the "host." Each connected system sits behind an "MCP server" that knows how to talk to it. Before MCP, wiring an AI assistant into five internal tools meant writing five custom integrations, then doing it again for the next assistant. Anthropic built MCP to close exactly that gap. An open standard that enables developers to build secure, two-way connections (Source: Anthropic, "Introducing the Model Context Protocol," ) Back to the checkout error. With MCP wired up, the agent doesn't just know it should check the error rate. It can go pull the number from the logging stack and the metrics dashboard itself. That solves the access problem. It doesn't solve judgment. MCP hands the agent raw numbers. It has no opinion about whether last month's number was normal for this particular system. RAG: The Library Card That's where retrieval-augmented generation, or RAG, earns its keep. Instead of stuffing every manual and dependency map into the prompt up front, RAG lets the agent search a collection of documents and pull back only the passages that match the question, using semantic search rather than a keyword match. The original RAG paper, published by Facebook AI researchers back in 2020, was blunt about the problem it set out to solve. Language models are good at sounding confident. They're less reliable at being precise. Their ability to access and precisely manipulate knowledge is still limited (Source: Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks") For the checkout bug, RAG might point the agent to the exact paragraph in the payment gateway's integration guide that explains why timeouts spike under load, buried in a document nobody on the team has fully read in a year. RAG is knowledge a person deliberately wrote down and filed away. It's only as good as what gets filed. Memory: What the Agent Remembers From Last Time Memory looks a lot like RAG from a distance. Both retrieve relevant information on demand. The difference is where the knowledge comes from. RAG pulls from documents a person wrote. Memory pulls from what the agent picked up itself while working. Say this exact 500 error happened three months ago, and the real cause turned out to be a stale feature flag nobody had documented anywhere. Memory is what lets the agent recall that the hard way this time, then write the fix back down for next time. A recent academic survey framed the shift well. Memory is increasingly the substrate through which agents self-evolve (Source: "A Survey of Agent Memory in the Second Half") That's the piece a pure RAG setup or a static skill file can't give you. An agent that gets measurably better at your specific systems the longer it works on them, instead of starting fresh every time. Four Tools, One Simple Rule None of these four replace each other. None of them do much alone beyond a demo. A rough rule of thumb, borrowed from how production teams tend to use them: Knowledge someone wrote down on purpose: RAGKnowledge the agent picked up from experience: memoryA repeatable procedure with judgment attached: a skillA way to reach the outside world without custom glue code: MCP Most serious agentic systems end up using all four together. The skill tells the agent what steps to follow and when to escalate. MCP gets it into the logging and metrics tools. RAG surfaces the relevant page from the documentation nobody memorized. Memory remembers that this exact error showed up before, and what actually fixed it. Skip any one layer and the agent falls back to guessing. That's the same problem as throwing everything into the context window in the first place, just with better manners. Choosing Which One to Build First The four rarely get built at once, and deciding where to start is one of the harder calls in enterprise AI architecture. Get the skill and the connective tissue right before reaching for a memory layer, and the sequencing tends to hold up. Build memory first, on top of a shaky procedure, and the agent will remember the wrong lessons very efficiently. This is the kind of tradeoff Faisal Feroz works through regularly as a Chief Technical Architect and Fractional CTO, helping enterprise teams turn legacy platforms into AI-first, event-driven systems. Readers weighing the same decisions on their own stack can find more of his writing on enterprise AI architecture at fferoz.medium.com, or connect with him on LinkedIn at linkedin.com/in/faisalferoz to talk through where skills, MCP, RAG, or memory actually fit.

By Faisal Feroz

Context Engineering: The Missing Piece in Agentic Systems

Context engineering is becoming essential as AI agents take on more software development work. An agent can plan, code, test, investigate incidents, trigger CI, and help deploy software. But none of that matters if it is operating without the right information. This is the main problem I keep seeing. We connect an LLM to a few tools, give it a good prompt, and expect magic. Then the agent has to figure out which service we mean, who owns it, what repository it belongs to, whether it is healthy, what incidents are open, and whether a deployment is safe. That is a lot of disconnected information to reconstruct every single time. Context engineering is the discipline of structuring, surfacing, and governing the information an AI agent needs to act reliably. It is how we give agents the right facts, rules, tools, and boundaries so they can make better decisions without hallucinating or wasting time hopping between systems. Key Takeaways Context engineering gives AI agents structured access to instructions, knowledge, memory, examples, tools, and guardrails.A context layer reduces tool switching and prevents agents from wasting effort interpreting disconnected SDLC data.Service catalogs, reusable skills, and human approval gates make agentic workflows more reliable and governable.Deployment recommendations should be grounded in visible evidence such as ownership, health, test coverage, runbooks, and incidents. Step 1: Understand What Context Engineering Actually Means AI agents are powered by LLMs. The LLM is basically the brain, but it does not automatically know the current state of your engineering organization. It does not know your service ownership, deployment history, runbooks, incident status, infrastructure, or internal policies unless you provide that information. That is where Context Engineering comes in. Instead of leaving an agent to guess, I give it access to relevant, organized context. This helps it plan properly, use tools properly, and take actions with much more accuracy. A simple way to think about it is this: Without context: An agent guesses what a service is, where its data lives, and what action is safe.With context: An agent can retrieve the service record, ownership, health, repository, runbook, deployment data, and guardrails before responding or acting. Context engineering is not just about putting more tokens into a prompt. It is about making the right information accessible at the moment an agent needs it. The goal is grounded actions, not longer conversations. Step 2: Identify Why Your Engineering Team Needs a Context Layer Most engineering ecosystems are distributed by design. Source code might live in GitHub, documentation in Notion, incidents in PagerDuty, conversations in Slack, infrastructure in AWS, observability in Datadog, and deployments in Kubernetes. Each tool is useful. The issue is that the knowledge is fragmented. For a developer, that fragmentation creates constant context switching. To understand one service, I may need to open a repository, find its owner, inspect deployment history, search incident records, locate the runbook, and check infrastructure health. That slows down delivery and increases the chance of missing something important. For an AI agent, the problem becomes even bigger. If I ask it to analyze bottlenecks, delivery velocity, quality gaps, and patterns across the SDLC, it may need to fetch and interpret data from every one of those disconnected systems. It spends tokens trying to understand the environment before it can solve the actual task. A context layer sits between the engineering ecosystem and the agent. It connects services, teams, workflows, documentation, policies, and operational data in one place. With that layer in place, context engineering can improve: Deployment speed and confidenceAccuracy in agent responses and actionsSecurity and policy enforcementOperational reliabilityCollaboration across teamsDeveloper productivity by reducing tool switching The point is not to replace every engineering tool. The point is to let humans and agents access the relevant context without manually rebuilding the story every time. Step 3: Fix Engineering Chaos Before It Turns Into Agentic Chaos The software development lifecycle has many stages: planning, coding, building, testing, securing, deploying, operating, learning, and improving. Teams commonly introduce specialized tools at every stage. Over time, that creates tool sprawl, duplicate data, fragmented workflows, and unclear ownership. I call that engineering chaos. It hurts quality, security, compliance, productivity, and operational excellence. Now add AI agents on top of that environment. If every agent is independently connected to different tools and given incomplete instructions, the chaos gets multiplied. Agents may have no shared visibility, no human approval points, inconsistent decisions, and no meaningful safeguards. This is why context engineering should begin with a simple question: What does an agent need to know before it can safely answer or act? For example, if I ask an agent, “Analyze my SDLC data and surface bottlenecks, velocity, quality gaps, and interesting patterns,” the agent needs more than a prompt. It may need: Repository and pull request data from GitHubInfrastructure context from AWSIncident data from PagerDutyOperational discussions from SlackDeployment state from Kubernetes Without a unified context model, the agent must interpret isolated facts from every system. That consumes tokens and can lead to weak, incomplete, or incorrect conclusions. Context Engineering gives that agent a better starting point. Step 4: Build the Six Types of Agent Context When I design context for an AI agent, I think in six categories. Each category answers a different part of the agent’s decision-making problem. 1. Instructions Instructions define rules, goals, and boundaries. They tell an agent what its job is and what it should not do. For example, an incident investigation agent may be instructed to gather evidence, summarize findings, and avoid triggering production actions. 2. Knowledge Knowledge includes documents, architecture diagrams, service metadata, domain data, repositories, and runbooks. This is the factual material an agent needs to understand the environment. 3. Memory Memory holds session logs, previous decisions, and persistent state. It lets an agent maintain continuity across multi-step workflows rather than treating every action as a completely new task. 4. Examples Examples provide short demonstrations and reference patterns. They show an agent what a useful output or a correct workflow looks like. This is especially useful when a task needs a consistent format. 5. Tools Tools include APIs, scripts, CI systems, and external services. Tools turn an agent from a chat interface into something that can retrieve current data and execute approved tasks. 6. Guardrails Guardrails are the hard constraints: safety rules, checklists, policy requirements, and approval gates. They are critical when an agent can do more than just answer a question. Instructions, knowledge, and memory are generally more static forms of context. Examples, tools, and guardrails are dynamic because they can change with the workflow, the service, and the current situation. Effective Context Engineering brings all six together instead of relying on a single prompt. Step 5: Separate Prompt Engineering From Context Engineering Prompt engineering and context engineering work together, but they solve different problems. Prompt engineering is about what to say. It focuses on the instructions and examples used to guide an interaction. It is useful for optimizing a single request or response. Context engineering is about what the agent gets to see. It focuses on managing accessible information across a workflow: the service data, connected tools, policies, history, ownership, and real-time status the agent needs. A great prompt cannot compensate for missing operational facts. If an agent does not know the owning team, service tier, runbook, open incidents, or deployment policy, no clever wording will make its production decision trustworthy. Step 6: Create a Service Catalog That Gives Agents Grounded Context To make Context Engineering practical, I need a system that represents the services in my environment and connects their information. In the demo, I use Port.io as a context layer for an agentic SDLC. A service catalog can hold details such as: Service name and identifierEnvironment, such as staging or productionOwning teamRepository associationRunbook URLSlack channelService tier and visibilityObservability linksOn-call rotation status Once this context is registered, an agent can answer a question like “Share everything about the shipment service” by retrieving a unified service overview. In the example, that overview includes the owning team, language, repository, branch, recent code activity, runbook, on-call status, health information, deployments, pull requests, and scorecard data. This is the practical value of context engineering. Instead of manually gathering facts from several tools, I can ask once and get a contextual answer built from the connected service record. Step 7: Turn Repeated Agent Instructions Into Reusable Skills Agents often perform repeated tasks: investigate an incident, assess deployment risk, review a pull request, measure DORA metrics, run CI, or deploy a service to production. Repeating the full instructions every time is not scalable. That is where agent skills are useful. A skill packages the context and logic needed for a repeatable operation. For example, I can define skills for: Incident responsePort readiness checksRunning CIDeploying a serviceDeploying to production When I ask an agent to run CI for the shipment service, it can load the relevant CI skill and combine it with the shipment service context. The agent is not starting from zero. It knows the service, the intended workflow, and the constraints around execution. This makes Context Engineering reusable. Skills reduce repeated setup work, standardize workflows, and help agents perform the same task in a predictable way across services. Step 8: Add Human Gates to Agentic SDLC Workflows Automation does not mean removing human control. In an agentic SDLC workflow, agents can gather requirements, plan work, generate code, test changes, and run continuous integration. But important actions should still include approval or rejection points. For example, a workflow can fetch service context first, then proceed through: Requirements gatheringPlanningCodingTestingContinuous integrationHuman approval before sensitive actions Human gates are part of good context engineering because they provide governance. The agent can recommend, prepare, and trigger approved workflows, but a person can still decide whether a proposed action should proceed. Step 9: Use Context to Make Better Deployment Decisions The final demo makes the value very clear. A simple application loads context for a selected service and gives a production-readiness verdict. For a healthy payment service, the context shows a clear picture: ownership is assigned, the Slack channel is configured, the runbook is documented, on-call rotation is active, test coverage is 94%, health status is healthy, the service was deployed recently, and there are no open incidents. Based on that connected information, the service is marked ready to deploy. For another service, the result is completely different. It is marked as not ready because key context is missing. There is no owning team, no runbook, and several other readiness requirements are incomplete. The system identifies the gaps instead of making a blind recommendation. That is what a production decision should look like. Not “yes” or “no” based on a vague prompt, but a verdict grounded in explicit evidence: Identity and ownershipHealth and operational statusRunbook availabilityOn-call coverageTest coverageRecent deployment historyOpen incidentsRequired scorecard checks When the context indicates risk, the result can say to proceed with caution and explain why. This is far more useful than an agent giving an unverified deployment recommendation. Step 10: Treat Context Engineering as an Engineering Discipline Context engineering is important because AI agents are only as reliable as the environment they can understand. If an agent has scattered data, unclear ownership, missing policies, and unrestricted tools, it will struggle no matter how advanced the model is. The practical path is straightforward: Map the tools and data sources that define your SDLC.Define the service-level context agents need to retrieve.Centralize ownership, health, repositories, runbooks, incidents, and policies.Create reusable skills for common workflows.Use tools for live data and approved execution.Add guardrails and human approvals around consequential actions.Make agent verdicts explainable through visible context. That is how I move from disconnected AI experiments to reliable agentic engineering workflows. Context Engineering reduces unnecessary token use, reduces confusion, and gives agents the facts they need to help build, test, operate, and deploy software with more control. Context Engineering FAQs What Is Context Engineering for AI agents? Context Engineering is the practice of organizing and governing the information an AI agent can access, including instructions, service data, memory, tools, examples, and safety constraints. It helps the agent make grounded decisions rather than guessing. How Is Context Engineering Different From Prompt Engineering? Prompt engineering focuses on how to phrase instructions for an interaction. Context Engineering focuses on the information the agent can retrieve and use throughout a workflow, such as ownership, repositories, incidents, deployment data, and policies. What Context Should an SDLC Agent Have? An SDLC agent should have the context needed for its task, which can include service ownership, repository details, environment, runbooks, on-call status, deployment history, test coverage, incident status, relevant tools, and hard safety rules. Why Are Human Approval Gates Important for AI Workflows? Human gates preserve control over consequential actions. Agents can retrieve context, prepare work, and recommend or trigger an approved workflow, while a person retains the ability to approve or reject sensitive changes.

By Pavan Belagatti

CORE

AI Architectures That Drive Real Business ROI

In this article, I'll try to give practical insights for choosing the right AI architecture for impact, not just experimentation. Companies are spending heavily on AI. Many are still struggling to show clear business returns. The most common reason is not the model; it is the architecture. Teams often jump straight to multi-agent systems or "autonomous AI" because those terms sound advanced. In reality, a well-designed decision intelligence system or a focused single-agent architecture often delivers faster, more reliable ROI than a complex multi-agent setup that no one can debug or govern. This article maps the five AI architectures that are actually driving measurable business value. For each one, you will see: What the architecture looks likeWhen you should use itWhy it works from a business perspectivePractical risks and success factors The goal is simple: help you choose the right level of architectural complexity for the outcome you need. 1. AI Decision Intelligence Architecture What it is: This is the classic "data -> insight -> decision -> action" loop, now powered by stronger models. Data from operational systems flows into an analytics layer, an AI model produces predictions or scores, a decision engine applies business rules and thresholds, and actions are triggered (often still with human oversight). When to use it: Strategy, forecasting, pricing, demand planning, risk scoring, inventory optimization, and any domain where the primary value is better decisions at scale. Why it works: It directly connects data to decisions that affect revenue, cost, or risk. The architecture is relatively mature, easier to govern, and usually has clear KPIs (forecast accuracy, reduction in stock-outs, improved conversion, lower credit losses, etc.). Practical notes: Success depends more on data quality, feature engineering, and decision policy design than on the latest foundation model. Many organizations already have 70% of this architecture in place and only need to modernize the model and decision layers. 2. AI Personalization Engine Architecture What it is: User data and behavioral tracking feed a feature store. An AI model (recommendation, ranking, or generative) produces personalized outputs: product recommendations, content, offers, or next-best-action. The system continuously learns from engagement. When to use it: Marketing, e-commerce, media, customer experience, and any product surface where relevance directly drives engagement and revenue. Why it works: Personalization has one of the most proven ROI profiles in AI. Even modest lifts in click-through, conversion, or average order value compound quickly at scale. The architecture is well understood and has mature tooling (feature stores, real-time inference, experimentation platforms). Practical notes: The biggest failures come from poor cold-start handling, lack of real-time features, or treating personalization as a pure model problem instead of a full-stack system (data-> features -> model -> delivery -> feedback). 3. Single-Agent AI Architecture What it is: A single agent receives a goal, maintains memory, reasons about the next step, uses tools, and executes. It operates in a loop until the task is complete. This is the architecture behind many of today’s coding assistants, research helpers, and internal automation agents. When to use it: Task automation, structured multi-step workflows, coding, document processing, customer support escalation, and any problem that can be owned by one competent agent with good tools. Why it works: It handles multi-step work with context and logic in a way that pure predictive models or simple RPA cannot. It is significantly simpler to build, observe, and govern than multi-agent systems, while still delivering real autonomy on well-scoped tasks. Practical notes: Most organizations should master single-agent systems before moving to multi-agent. The limiting factors are usually tool quality, memory design, evaluation harnesses, and clear task boundaries, not the choice of foundation model. Key insight: A reliable single-agent system with excellent tools and evaluation often outperforms a poorly coordinated multi-agent system in both speed of delivery and actual business results. 4. Multi-Agent AI Architecture What it is: A planner (or meta-agent) decomposes a complex user goal into sub-tasks. Specialized task agents execute those sub-tasks, often in parallel, using shared or private memory. Results are aggregated into a final output. This is the architecture used in advanced research systems and complex enterprise workflows. When to use it: Complex workflows that genuinely require different skills (research + analysis + writing + coding), long-horizon projects, or situations where parallelism and specialization produce clear gains in quality or speed. Why it works: It distributes cognitive load. Different agents can be optimized (or even use different models) for different sub-problems. When designed well, the system scales in capability without making any single agent monolithic. Practical notes: Coordination cost is real. Handoff failures, inconsistent memory, and unclear ownership of the final result are common. Multi-agent systems require stronger observability, evaluation, and governance than single-agent systems. Do not adopt this architecture just because it sounds more advanced. 5. Autonomous AI System Architecture What it is: A closed-loop system: Input -> Perception-> Reasoning-> Planning-> Execution -> Feedback. The system continuously senses its environment, updates its understanding, plans, acts, and learns from outcomes with minimal human intervention. This is the most ambitious architecture on the spectrum. When to use it: End-to-end automation of well-understood business processes, self-optimizing systems, and domains where continuous operation without constant human oversight is both possible and desirable (certain supply-chain, infrastructure, or trading systems, for example). Why it works: When the feedback loops are high-quality and the environment is sufficiently stable or well-modeled, the system can improve over time and operate at a scale and speed humans cannot match. Practical notes: This is the highest-risk architecture. Failures can be expensive and hard to contain. Most organizations should treat full autonomy as a long-term destination, not a starting point. Strong guardrails, human oversight points, and kill switches are mandatory. How to Choose the Right Architecture ArchitectureComplexityTime to ValueBest ForMain RiskDecision IntelligenceLow–MediumFastForecasting, optimization, riskPoor data or unclear decision policiesPersonalization EngineMediumFast–MediumEngagement, conversion, CXWeak feedback loops or cold startSingle-AgentMediumMediumTask automation, coding, researchBad tools or weak evaluationMulti-AgentHighSlowerComplex multi-skill workflowsCoordination and observability failuresAutonomous SystemVery HighSlowestFully automated closed-loop processesUncontrolled behavior and high blast radius Simple decision rules: If the primary value is better decisions from data, then start with decision intelligence.If the primary value is relevance at scale, then build a personalization engine.If you need multi-step task completion with tools, then master single-agent first.Only move to multi-agent when you have clear specialization and coordination benefits.Treat autonomous systems as a maturity goal, not a first project. Common Mistakes That Destroy ROI Jumping to multi-agent or autonomous too early: complexity without corresponding process maturity.Treating architecture as a model problem: the model is rarely the bottleneck; tools, data, evaluation, and governance usually are.No clear success metrics: if you cannot define what "good" looks like in business terms, you cannot steer the system.Ignoring observability: agentic and autonomous systems that cannot be inspected become impossible to improve or trust.Building technology in search of a problem: the architecture must serve a real workflow and a real economic outcome. Closing The organizations that extract real ROI from AI are not necessarily the ones using the most advanced architecture. They are the ones that match the architecture to the problem, keep the design as simple as the use case allows, and invest heavily in data quality, tools, evaluation, and governance. Start with the architecture that solves the actual business problem with the least unnecessary complexity. Prove value. Then, and only then, increase architectural sophistication where the returns justify the cost and risk. Decision intelligence and personalization still deliver some of the clearest and fastest returns. Single-agent systems are currently the highest-leverage step-change for knowledge work and automation. Multi-agent and fully autonomous systems are powerful... but only when the organization is ready to operate them with discipline. Choose deliberately. Measure ruthlessly. Scale what works.

By Ram Ghadiyaram

CORE

The Trinity of Modern Data Architecture: Process Intelligence, Event-Driven Integration, and Trusted Agentic AI

Most enterprises have all three pieces. A process automation layer. A data integration strategy. An AI initiative. Yet critical decisions still fail, agents still hallucinate, and workflows still run on yesterday's data. The investments exist. The convergence does not. The problem is not a lack of technology, but a lack of architectural thinking. Process intelligence, event-driven integration, and trusted agentic AI are being built in isolation, by different teams, with different goals, on different timelines. The result is an architecture that looks complete on a slide and breaks in production. This post argues that these three capabilities form a Trinity. They only deliver their full value when they are designed to work together. Three Capabilities, One Architectural Commitment Process intelligence, event-driven integration, and trusted agentic AI each solve a real problem. Each one also creates new risks when it operates alone. The following architecture shows how the three layers connect into a single, converged system. Process Intelligence: The Layer That Gives Agentic AI Its Boundaries Process intelligence is the evolution of classic Business Process Management (BPM) into something adaptive, event-aware, and AI-ready. It is the layer where technology maps directly to business value. Every workflow connects to a concrete business outcome: a loan approved, a shipment rerouted, a fraud case resolved. Process mining observes how business processes actually run, identifies where decisions fail, and surfaces where automation would deliver the most value. Vendors like Celonis have built entire platforms around this capability. Process orchestration executes workflows, enforces business rules, and produces the audit trails that compliance teams depend on. Camunda is a leading example. Agentic process orchestration goes one step further: it allows AI agents to participate directly in workflow execution, taking autonomous actions within defined boundaries while the process layer maintains control. Automation is the business driver. Organizations adopt process intelligence to automate more, faster, with less manual intervention, while keeping humans in control of the decisions that matter. But agentic automation only works safely when the process layer defines the operational envelope: what the agent can decide alone, what requires human approval, and what must be escalated regardless of what the model recommends. This is where guardrails live in practice. Not as theoretical constraints inside a model, but as concrete workflow gates that stop, route, or escalate before an action is executed. Process intelligence is what makes automation trustworthy at scale. Event-Driven Integration: From Scheduled Batches to Live Events Event-driven integration is the architectural principle that connects operational systems continuously, based on what happens rather than when a scheduler runs. An event from a payment system, a sensor, a CRM update, or a logistics platform travels in real time to whatever system needs to act on it. Apache Kafka has become the de facto standard for event-driven integration at enterprise scale. Other options exist, including cloud-native messaging services and specialized event brokers, but Kafka is where the ecosystem has converged. What matters in any case is the commitment to events as the primary integration primitive to ensure true decoupling, scalability, and data consistency across real-time and batch systems. The market reflects this shift. Process orchestration engines have rearchitected their core runtimes to be event-driven from the ground up, built for real-time throughput and horizontal scale. Camunda's Zeebe is a leading example. Zeebe is itself an event-driven engine, which means organizations can implement event-driven workflows and lightweight integration patterns without Kafka as a prerequisite. For broader enterprise integration at scale, Apache Kafka complements the process orchestration layer, connecting the full landscape of operational systems, SaaS platforms, and data infrastructure into a single event-driven backbone. Core business applications and SaaS platforms followed. SAP S/4HANA, Salesforce CRM, and ServiceNow have all added eventing interfaces and Change Data Capture (CDC) capabilities alongside their traditional API-based request-response integrations. The direction is clear: even systems that were designed around synchronous HTTP are moving toward event-driven models. Process engines receive live state. Agentic AI systems receive current context. Decisions are made on what is actually happening, not on what happened last night. Trusted Agentic AI: Safety Is an Architecture, Not a Setting Trusted agentic AI is an architectural property, not a product feature. Agentic AI systems do not just generate responses. They take actions, trigger workflows, and interact with operational systems. That autonomy is what makes trust and safety an architectural concern rather than a model configuration. It operates at two levels. The first is the model itself. Vendors like Anthropic and Mistral build alignment, constitutional constraints, and refusal behaviors directly into their models. This provides a baseline. The second level is the process intelligence layer. A well-aligned model can still be manipulated through prompt injection or adversarial inputs. It can still hallucinate when the surrounding data is stale or incomplete. Model-level safety defines how the agent behaves within a given context. Process-level safety defines the operational envelope: what the agent is allowed to do, which decisions require human approval, and what the fallback is when the agent is wrong. Both levels are necessary. Neither is sufficient alone. When the Trinity Splits: Three Agentic AI Failure Scenarios Three short failure scenarios make this concrete. Process intelligence without event-driven integration. A workflow engine automates a credit decision. The data feeding it comes from a nightly batch export. The process runs correctly. The decision is based on a customer's financial state from 18 hours ago. The automation worked. The outcome was wrong. Event-driven integration without process intelligence. Transaction data flows in real time across systems. An agentic AI system detects an anomaly and flags a potential fraud case. But there is no process intelligence layer defining what happens next. There is no approval gate, no escalation path, no audit trail. The agent acts, or it does not, and nobody can explain which or why. Trusted agentic AI without the other two. The agent is aligned, tested, and governed at the model level. But it receives context from a batch pipeline, so its reasoning is grounded in outdated information. And no process intelligence layer enforces boundaries on what it can do next. The agent behaves well in the lab. It causes problems in production. The Trinity in Action: Process Intelligence and Agentic AI Across Three Industries The following three scenarios show this architectural model working across industries. Each one is different. The pattern is the same: an event triggers a process, an agentic AI system acts within it, and process intelligence defines the boundary between automation and human control. Financial services. A transaction event triggers an agentic AI fraud risk assessment in real time. The risk score flows into a case management workflow. Below a defined threshold, the process is automated. Above it, the process intelligence layer routes the case to a human analyst before any account action is taken. The guardrail is not inside the model. It is inside the process. Healthcare. A patient monitoring system emits a deterioration signal. The event reaches a care pathway engine, which initiates the appropriate clinical workflow. An agentic AI system recommends an intervention. The process intelligence layer requires clinician confirmation before that recommendation becomes an order. The agent informs. The human decides. The process enforces that boundary every time. Supply chain. A supplier sends a disruption signal. The event reaches the process engine before the procurement team opens their inbox. An agentic AI system analyzes inventory, evaluates alternative suppliers, and proposes rerouting options. The process intelligence layer defines which decisions the agent can execute autonomously and which require sign-off. Speed comes from the event-driven layer. Governance comes from process intelligence. Trust comes from both working together. Build the Trinity, Not the Parts This Trinity is not a new product category. It is a way of thinking about a converged architecture that most enterprises have not yet adopted. Event-driven integration ensures that every process and every agentic AI system works on current reality. Process intelligence ensures that automation stays within governed, auditable boundaries. Trusted agentic AI ensures that agents behave reliably within the context they are given, and that the process intelligence layer catches what the agent cannot. The following architecture maps the complete picture across all three layers: Organizations that invest in all three separately will keep getting the results they are getting today. Organizations that design them to converge will build something qualitatively different: infrastructure that moves fast, governs well, and earns the trust of the business. The technology exists. The architectural commitment is what is missing.

By Kai Wähner

CORE

Why Real-Time Data Pipelines Are Becoming the Foundation of Industrial AI

I spent the first six months of a project convinced we had a model quality problem. Our anomaly detection system for manufacturing telemetry was missing obvious defects; things a human operator would catch in seconds. We tried bigger models, better embeddings, more training data. Nothing moved the needle. Then one afternoon, while tracing a specific false negative, I noticed the timestamp. The sensor reading that would have triggered a correct alert had arrived 47 seconds after the decision window closed. The model never saw it. Not because the model was bad. Because the pipeline delivered the data too late for the model to act on it. That's when I stopped thinking about model architecture and started obsessing over data delivery. And honestly, everything I've built since has been shaped by a simple realization: in industrial AI, the pipeline IS the product. The model is just the last mile. How Generative AI Changed the Conversation (But Not the Bottleneck) Everyone's building AI assistants, intelligent search, predictive analytics, autonomous workflows. The conversation focuses on foundation models, prompt engineering, inference optimization. Makes sense; that's the exciting part. But in industrial environments (semiconductor fabs, energy plants, discrete manufacturing), the bottleneck isn't model capability. It's whether the right data reaches the model at the right time, in the right shape, with the right lineage attached. I've watched teams spend months fine-tuning a model that was getting stale sensor readings. Months. The model was perfectly capable. It was just blind. This is why I've come to believe that industrial AI success is a data architecture problem first and a model problem second. The reason is not because models don't matter. Instead, it is because a brilliant model on bad plumbing produces confidently wrong answers, which is worse than no answer at all. What Semiconductor Fabs Taught Me About "Real-Time" Here's where my background in semiconductor manufacturing gives me a perspective most streaming architects don't have. In a modern fab (say, a 300mm facility running at 5nm or 3nm process nodes), a single wafer passes through 500+ process steps. Each step generates telemetry: gas flow rates, chamber pressure, plasma power, temperature profiles, film thickness measurements, overlay alignment data. Multiply that by 50 wafers per lot, dozens of lots per day, and you're looking at billions of data points daily. The fab doesn't batch-process this data overnight. It can't. A wafer worth $10,000+ is moving through the line continuously. If a process parameter drifts out of spec and you don't catch it until the nightly ETL job runs, you've potentially scrapped an entire lot. That's half a million dollars gone because your pipeline was "fast enough for batch." Fabs solved this decades ago with a discipline called Fault Detection and Classification (FDC). Every equipment run is analyzed in real-time (within milliseconds of completion). Statistical models compare current sensor traces against known-good profiles. If something looks off, the system raises an alarm before the next wafer enters the chamber. This isn't some exotic research concept. It's running in every leading-edge fab on the planet right now. And the architecture behind it looks remarkably like what we're trying to build in enterprise streaming: FAB FDC ARCHITECTUREenterprise streaming equivalent Equipment sensor streams (SECS/GEM protocol) Apache Kafka / Apache Flink event streams Real-time trace comparison Stream processing with windowed aggregations SPC control charts with Western Electric rules Anomaly detection on feature pipelines Recipe parameter adjustment (APC) Automated model retraining triggers Lot genealogy / WIP tracking Data lineage and event provenance The patterns are the same. The fab version just had higher stakes, forcing better discipline earlier. Why Streaming Isn't "Faster Batch." It's a Different Mental Model. This distinction tripped me up for a while. I kept thinking of streaming as "batch that runs every second instead of every hour." That's wrong, and it leads to bad architecture. Batch assumes data is static until the next scheduled update. You collect, then process, then serve. Streaming assumes data is continuously evolving. Events flow through the platform as they occur. Applications subscribe and react while the underlying process is still unfolding. The practical difference is enormous: Batch thinking: "We'll retrain the model on last night's snapshot." Result: the model is always 8-24 hours behind reality. In a manufacturing context, that's thousands of wafers processed with stale parameters. Streaming thinking: "The feature pipeline receives fresh observations as events arrive." Result: the model's context is minutes old, not hours. Decisions happen while outcomes can still be influenced. Apache Kafka, Apache Flink, and event-driven frameworks like Apache Pulsar make this architecturally possible today. The tooling has matured. The question isn't whether streaming works; it's whether your organization has made the mental shift from "collect then analyze" to "analyze as it flows." The Hidden Engineering Nobody Wants to Talk About Building industrial AI involves way more plumbing than anyone admits during the planning phase. Behind every successful deployment lies a data platform responsible for ingesting, validating, enriching, governing, and distributing information from dozens of independent systems. In manufacturing environments specifically: Equipment comes from multiple vendors (Applied Materials, Lam Research, Tokyo Electron; each with different telemetry formats).Sampling frequencies vary wildly (100ms for some sensors, 1Hz for others, event-based for yet others).Some systems generate structured events while others produce semi-structured logs.Data quality fluctuates depending on operating conditions (a chamber during maintenance produces garbage telemetry that looks like anomalies to a naive model). Before AI can analyze any of this, the platform must reconcile these inconsistencies into a unified representation. Schema registry (Confluent Schema Registry, Apicurio), data quality frameworks (Great Expectations, dbt tests), and format standardization (Apache Avro, Protocol Buffers) do this work. It's unglamorous. Nobody writes blog posts about schema reconciliation. But I've seen more AI projects die from bad plumbing than from bad models. The ratio isn't even close. Why Data Governance Isn't Compliance Anymore. It's Model Quality. This shift snuck up on me. I used to think of governance as something the compliance team worried about: data classification, retention policies, access controls. Important, but not my problem as an architect. Then I watched a machine learning model produce wildly inconsistent predictions because it was consuming two different versions of the same feature; one from the real-time pipeline (current) and one from a batch backfill (stale). No governance framework flagged this because nobody had defined "which version should the model use?" as a governance question. In industrial AI, governance questions become engineering questions: Where did this data originate? (Lineage: Apache Atlas, OpenLineage)Has it been validated? (Quality gates in the pipeline itself.)Which version should the model use? (Catalog: Apache Iceberg's time-travel, Delta Lake's versioning.)Can this information cross regional boundaries? (Compliance-as-code in the streaming layer.) The strongest architectures I've seen integrate governance directly into the event pipeline. Metadata travels with data. Access policies apply at the stream level. Lineage is preserved through every transformation. Not as a separate process; as part of the infrastructure itself. Why RAG Quality Is a Pipeline Problem (Not a Prompt Problem) Retrieval-augmented generation has become the default architecture for enterprise GenAI. Makes sense; you ground the language model in your proprietary knowledge rather than relying solely on its training data. But here's what I keep seeing: teams spend weeks optimizing prompts and chunking strategies while their knowledge base quietly goes stale. Documents update, but embeddings don't re-index. Permissions change, but the retrieval layer doesn't reflect them. Metadata drifts from reality. The language model still generates fluent responses. They're just increasingly grounded in yesterday's (or last month's) context. RAG quality, in my experience, depends more on the freshness and accuracy of the retrieval pipeline than on the generation model sitting on top. A well-maintained knowledge pipeline with a mid-tier model outperforms a frontier model drinking from a stale index. This means treating your RAG pipeline like a streaming system: continuous ingestion, continuous re-indexing, continuous validation. Not a one-time "load the docs and forget." Building for Scale Without Burning Money Industrial AI platforms process enormous event volumes. Millions of messages per minute. Thousands of assets generating telemetry simultaneously. Multiple AI services consuming overlapping datasets. Scaling this naively (just add more brokers, more compute, more storage) gets expensive fast. What I've found works better: Process at the edge when possible. In semiconductor manufacturing, FDC analysis often runs on edge compute at the equipment level (15ms response time vs 800ms round-trip to a centralized system). The same principle applies to any industrial streaming architecture: if the decision can be made locally, don't pay the latency and cost of a centralized round-trip. Tiered storage with hot/warm/cold patterns. Real-time features stay in low-latency stores (Redis, Apache Druid). Recent history lives in columnar formats (Apache Parquet on object storage). Deep history moves to cold archives. Apache Iceberg handles this elegantly with its metadata layer. Backpressure instead of over-provisioning. Rather than provisioning for peak load 24/7, build systems that gracefully handle bursts through buffering and backpressure mechanisms. Kafka's consumer group model does this naturally when configured properly. Observability across the entire pipeline. Not just the model; the pipeline itself. OpenTelemetry for tracing, Prometheus for metrics, distributed tracing that follows an event from sensor to prediction. When something goes wrong (and it will), you need to know where the failure point is in seconds, not hours. What I'd Tell Myself Two Years Ago If I could go back to the start of that project where we spent six months blaming the model: 1. Instrument the pipeline first. Before deploying any model, measure data freshness at every stage. Know exactly how old your model's context is at inference time. If it's stale, the model doesn't matter yet. 2. Treat streaming as a prerequisite, not an optimization. For industrial AI that needs to influence real-time outcomes, batch architectures aren't "good enough for now." They're architecturally incompatible with the goal. 3. Invest in schema discipline early. It's painful and boring. It pays for itself within months. Every team I've talked to that skipped this step regretted it when they tried to add a second or third data source. 4. Governance is architecture, not documentation. If governance policies don't enforce themselves automatically in the pipeline, they don't exist in practice. They're just PDFs nobody reads. 5. The pipeline IS the AI product. The model is important but replaceable. The data infrastructure that feeds it is the durable competitive advantage. Invest accordingly. Industrial AI is maturing quickly. The teams shipping reliable systems aren't the ones with the best models. They're the ones with the best plumbing. And honestly, that's encouraging because plumbing is engineering, and engineering is what we do. I'd love to hear what's worked (or spectacularly failed) in your streaming architectures for AI. The patterns are still emerging, and I think the best ideas are coming from practitioners who've felt the pain firsthand.

By Ajay Kumar Govindaram

The AI Evolution in Software Testing: A QA Manager's Blueprint for Staying Irreplaceable

Just a few months back, I observed a test suite with a self-healing feature “fixing” a failed selector three different times during the same sprint cycle. In each instance, the fix performed its function perfectly well; however, it didn’t address the real problem of a shipped UI regression, as its sole focus was on keeping the test green. No one on the team became aware of the situation until one of the customers discovered it. That was the point when my perception of AI in Quality Assurance changed dramatically more than any keynote or LinkedIn post. Today, if you are a QA Engineer, chances are high that you have come across similar headlines as well: manual testing is dying, autonomous agents create and fix test scripts in seconds, and your career is going to be at risk soon. The fear is justified, and I do not find it irrational as a Senior QA Manager who sees new testing tools appearing in our workflows every quarter, or even faster than we can develop any governing principles. What I see happening is quite different: not only is the position not going away, but it is becoming increasingly difficult to fake. AI excels at internalizing all the mechanical, low-context tasks that previously made up the bulk of a QA Engineer’s workload, leaving only those tasks that have never been mechanical or low-context and involve judgment, risk assessment, and determining what quality looks like for a particular product. This significantly narrows the number of people qualified for the position. 1. Stop Writing Tests. Start Auditing Them. For decades, a significant proportion of QA time was spent on the technicalities: automation scripts, manual click-through of UI workflows, and broken selectors caused by someone renaming a div tag. AI is really good at these types of jobs, and pretending otherwise is just a waste of time. You should approach AI-generated tests as you would a junior engineer’s pull request: they are useful and efficient, but require your review before implementation. Learn to feed it real context, not vibes. The difference between a useless AI-generated test and a genuinely good one almost always comes down to whether you gave it the actual acceptance criteria, edge cases, and business rules, or just a vague prompt. This is a real skill, and most QA teams haven’t invested in it yet.Get comfortable with self-healing tools, and stay suspicious of them. Self-healing automation is very valuable for handling cosmetic churn in your user interface. In addition, as shown by my story above about selectors, it can silently hide the very bugs it's supposed to detect.Your value moves from writing to verifying. That’s no downgrade. The ability to check that 100 automatically created tests are useful, as opposed to just being syntactically correct, is more difficult than having written 50 by hand. 2. Learn to Test the Thing That Doesn’t Give the Same Answer Twice Every product I am working on is trying to add AI, and none of the QA processes I have seen have been designed with the requirements of such a task in mind. In conventional software, the deterministic factor is the key component; in other words, whatever the input, the output will always be the same. But with the use of AI, there is nothing like that, since the same query asked twice yields two different answers. This opens up an actual underserved field of skills, known as AI Trust, Risk, and Security Management, and, to be honest, what you call it is less important than the brawn behind it. Areas where one could actually develop some skills: Bias and fairness testing. Learning to actually evaluate whether a model’s outputs skew unfairly across groups, not just whether the demo looks fine.Hallucination detection. Building repeatable ways to check whether an LLM’s output is grounded in real data or confidently making things up. This is genuinely hard and genuinely valuable; most teams are doing it on an ad hoc basis right now.Adversarial and prompt-injection testing. Deliberately trying to break an AI system’s guardrails before someone outside your company does it for you. I’ll be honest about the caveat here: this field is young enough that best practices are still being written in real time, including by people learning on the job. Nobody has fifteen years of AI-TRiSM experience, because it didn’t exist fifteen years ago. That’s exactly why it’s a good place to plant a flag now rather than waiting for it to mature. 3. Protect the Part of Quality AI Genuinely Can’t Do AI is just a statistical machine. It doesn’t have any firsthand knowledge of being frustrated with a difficult checkout process, any cultural knowledge to know why something that works well in one place doesn’t feel right somewhere else, and no sense of that hard-to-pin-down friction that you can’t specify. AI is responsible for functional testing; however, when it comes to the people-oriented aspect of quality, that’s where I would focus my efforts: Exploratory testing following a hunch. The best bugs I’ve ever discovered came from getting a slight feeling that something was amiss and investigating it, rather than through a written test case. The hunch does not stem from any particular model.Accessibility and usability should be top priorities rather than something ticked off a box prior to deployment. Is the product actually good to use? The algorithm will tell you whether the button meets the contrast ratio requirement. It cannot tell you whether the user flow around the button is confusing.Being there when risk is being discussed. AI will tell you whether the feature complies with the specification. AI has no way of knowing if the specification itself is incorrect for your market, your users, and the regulations. This discussion has to involve a human who knows the business, not the ticket. 4. Let Production Data Tell You Where to Look AI is based on data, and therefore, your testing approach should be too. The QA engineers who actually derive useful insights from AI do not test everything equally; instead, they let the data drive them. Analyze the real usage pattern of your application and prioritize automation accordingly; focus on testing those paths that users actually use, not those that were expected according to the initial requirements specification. Close the loop with your DevOps team about what is really breaking in production. If you find out that there are constantly recurring errors of one type or another, this information is directly relevant to the testing priorities of your AI solution, not something that you talk about separately.Understand what the data pipeline looks like, at least at a high level. A significant amount of “AI testing” in the future will involve testing the data pipeline that feeds into the AI algorithm, not just the outputs. A 90-Day Plan, If You Want One If you’d rather have a concrete starting point than a philosophy, here’s roughly how I’d sequence it: A 90 Day qa plan Days 1–30 Learn the tools Bring an AI coding assistant into your actual daily automation work, not a sandbox exercise, and pay attention to where it’s confidently wrong. Days 31–60 Expand the domain Take a real course on ML fundamentals or AI testing methodology, not just a vendor’s product training. Days 61–90+ Make it visible Propose one concrete AI-driven improvement on your current team, whether that’s AI-assisted test data generation or a pilot of self-healing UI tests with a defined review process attached. Regarding tooling: this landscape moves fast enough that my suggestions for product names will be obsolete by the end of the first year, but remember that it is the categories which are meaningful, not individual product names. Natural language test generation, self-healing test execution, visual testing, and AI security testing are just a few of the currently meaningful categories. Tools in any of those categories worth considering are those that let you see and adjust the AI's decisions. The Bottom Line Anxiety about AI in testing often arises from conflating two distinct concepts. Testing is a technical task, while QA is a mindset focused on protecting the user experience. AI excels at technical tasks but cannot replicate the QA mindset. In practice, automation is removing repetitive tasks, leaving the core responsibility of defining quality for each product and user group, and identifying issues beyond a model’s reach. This results in a more meaningful, though more demanding, role.

By Kiran Paul Kanikaram

Event-Driven AI Systems With Kafka and Autonomous Agents

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish decisions back into operational workflows. In that setting, event streaming is not simply middleware. It becomes the record of how intelligent behavior unfolds over time. Kafka is designed to read, write, store, and process streams of events across distributed systems, while Kafka Streams adds joins, aggregations, windowing, event-time processing, and exactly once support for stateful stream applications. At the same time, modern agent runtimes have shifted toward durable execution, persistence, and human-governed control flows rather than single-turn prompting alone. That convergence makes Kafka a strong coordination layer for autonomous agents that need to react continuously instead of responding once and disappearing. That architectural change also alters the role of the model. In an API-centric design, the model is often treated as a synchronous dependency behind a request. In an event-driven design, the model becomes one participant in a larger decision pipeline. Observations arrive as events, context is assembled from topics and state stores, agent steps are logged, and decisions are emitted as new events for downstream systems. Because Kafka topics can be replayed and reprocessed, the same stream can feed planners, validators, enrichment services, audit consumers, and human-review workflows without creating hard coupling between those components. The resulting system is easier to inspect, easier to recover, and easier to evolve than a chain of tightly bound remote calls. Turning Kafka Into the Coordination Layer The most important benefit is not only scale. It is the replacement of brittle request chains with an append-only coordination layer. A payment event, support ticket update, equipment alarm, or fraud signal can be published once and then consumed independently by retrieval components, compliance checks, planners, and execution agents. Kafka consumer groups divide partitions across consumers in the same group, and each partition is consumed by a single consumer within that group, which preserves ordering at the partition level while still allowing horizontal scale. For agentic systems, that detail is central. If all events for the same case, customer, or device are keyed consistently, one partition becomes the serialized timeline for that entity, and the agent no longer has to reconstruct order from racing HTTP callbacks. The event log also becomes a durable memory boundary. Kafka log compaction retains the latest value for each key, which makes compacted topics useful for task state, policy snapshots, approval status, or tool metadata that must survive restarts and recover quickly. On the runtime side, agent frameworks persist checkpoints and thread-scoped state so interrupted flows can resume from a saved step instead of starting over. Used together, those layers create a pragmatic split of responsibilities, such as Kafka preserves externally visible state transitions, and the agent runtime preserves internal execution context between steps, pauses, and failures. That is exactly the kind of separation needed when autonomous behavior must remain observable without being reduced to stateless prompt calls. Designing Agent Loops Around Events Once Kafka becomes the backbone, the agent loop changes shape. The entry point is no longer a prompt alone. It becomes a domain event that is enriched, correlated, and converted into a bounded task. Research on ReAct showed the value of interleaving reasoning and acting, and current agent frameworks translate that idea into practical workflows with durable execution, interrupts, and resumable state. The production version of an autonomous agent is therefore less like a chat session and more like a state machine that reasons, uses tools, emits intermediate facts, and pauses when a policy boundary requires approval. A concise stream processor can prepare that task before the model loop begins: Java builder.stream("order-events", Consumed.with(Serdes.String(), orderSerde)) .selectKey((key, event) -> event.customerId()) .join(customerTable, this::mergeContext) .mapValues(this::toAgentTask) .to("agent-tasks"); This pattern keeps context assembly close to the log instead of scattering it across synchronous service calls. Records are keyed by stable business identity, joined with the latest customer state, and emitted as small agent-tasks messages that the runtime can consume directly. Kafka Streams is explicitly intended for stateful processing with joins, event-time semantics, and exactly-once guarantees, so the enrichment stage remains deterministic, replayable, and independent from the model-serving layer. The execution boundary can remain equally narrow: Java @KafkaListener(topics = "agent-tasks", groupId = "claims-agent") @Transactional public void handle(AgentTask task) { AgentDecision decision = agentRuntime.run(task); kafkaTemplate.send("agent-decisions", task.taskId(), decision); } A compact runtime method can express the control flow without hiding it: Java public AgentDecision run(AgentTask task) { AgentState state = stateStore.load(task.taskId()); PlanStep step = planner.next(state, task); if (step.requiresApproval()) return AgentDecision.pause(task.taskId(), "manual-review"); ToolResult result = toolExecutor.execute(step.tool(), step.arguments()); return planner.complete(task, state, result); } This arrangement matters because the runtime receives a prepared task and emits an explicit decision event instead of mutating external systems invisibly. When transactions are enabled, Spring for Apache Kafka supports exactly-once semantics for the read-process-write sequence, and Kafka itself uses idempotent producers plus transactions so retries do not create duplicate log entries. External side effects still need idempotent design when they happen outside Kafka, but the event pipeline itself becomes much more predictable and auditable. Reliability and Control in Production Reliability in event-driven AI systems is usually lost at the edges rather than inside the model call. Kafka’s exactly-once features matter because an autonomous agent often emits decisions that trigger downstream actions, compensations, or audits. Kafka Streams supports exactly-once v2, and exactly-once flows configure consumers with read_committed isolation so aborted transactions do not leak into downstream processing. The event contract matters just as much as the delivery contract. Schema Registry centralizes schemas, validates them, and enforces compatibility modes so producers and consumers can evolve independently. In practice, a stable AgentDecision schema with explicit action type, confidence, explanation reference, and approval status is usually more valuable than a loosely structured JSON envelope because it can be consumed safely by analytics jobs, rule engines, operational systems, and auditors maintained by different teams. Operational control also has to assume malformed input, tool failure, and policy limits. Kafka Connect supports dead letter queues for records that cannot be processed successfully, and Spring Kafka supports dead-letter handling for repeated listener failures. Kafka also supports SASL-based authentication and ACL-driven authorization, which matters when planners, tool executors, and audit services must have different permissions over topics and consumer groups. Combined with interrupt-driven approval workflows from modern agent runtimes, those controls allow autonomous agents to operate inside explicit safety and governance boundaries instead of as opaque background processes. Where This Architecture Fits Best This architecture is strongest when work is asynchronous, stateful, and externally observable. Fraud triage, claims handling, supply chain exception management, field-service coordination, and security operations are better fits than chat-only assistance because the hard problem is not generating a sentence. The hard problem is reacting to a changing stream of facts, correlating them by entity and time, and making bounded decisions with replayable outcomes. Event-driven AI systems with Kafka and autonomous agents are compelling because they treat intelligence as part of an operational stream rather than as an isolated endpoint. The most effective implementations keep the log authoritative, keep schemas explicit, keep agent state durable, and keep irreversible actions observable and governable. That combination produces systems that are not only responsive, but also replayable, auditable, and resilient enough for enterprise use, which is ultimately the threshold that separates a convincing demo from a production architecture.

By Uthej Mopathi

CORE