AI/ML

Connecting with our community is one of my favorite parts of the job! Because of this, our team has started a Member Spotlight series to further highlight what makes DZone’s developers unique. First in the series is Pavan Belagatti. After nearly a decade of contributing to DZone, I took some time to learn more about the person behind the articles — from what first sparked his interest in tech to how he stays current and what he enjoys outside of work. What first got you interested in technology? "I come from a small town in India. I saw a computer for the first time at one of my friend’s houses when I was 16, and I was fascinated by how things worked. We used to take the whole computer apart to see what was inside and then assemble everything back properly. That’s when I started getting attracted to technology." What’s one tool you couldn’t work without? "I use Canva a lot since I create a lot of technical content and often need to create technical diagrams. Developers really like seeing these kinds of technical architecture diagrams, so Canva has become an essential tool for me. Other than Canva, I like playing with our own platform, Port.io, to automate developer workflows." What’s your favorite way to keep your technical skills current? "Since I am a developer evangelist, I need to be active on social media, follow a lot of tech influencers, and keep track of what they are sharing. I also follow news publications and developer communities like DZone. All of these things help me understand what skills are trending and what’s happening in the industry. Then I use ChatGPT, Claude, or Gemini along with YouTube videos to learn and upskill myself." In your free time, what do you like to do? "I like going to the gym, swimming, and exploring different cafes near where I stay (there are a lot of cafes around here). I also enjoy hanging out with friends and sometimes just going out for long drives." I see you create content on YouTube. Is there a particular video you'd like to share with our community? "Sure... this video is all about building a software factory, where I am showing a simplified software automation factory. With the adoption of AI agents, we can now integrate them across our SDLC and automate the entire software development and delivery pipeline. Of course, there will be human gates throughout the process to approve or reject the outcomes at most of these SDLC stages, while keeping guardrails and security best practices in place." To see more of Pavan's content, here's the link to his DZone profile.

Traditional vector RAG retrieves by embedding the question and finding semantically similar chunks, often augmented with lexical search, filtering, or reranking. This approach works when the answer is explicitly described in one or more chunks. However, it breaks down when the answer depends on relationships between facts. The question "Does my application depend on a compromised package?" illustrates this limitation. The vulnerable package may be several layers deep in the dependency tree, and no single chunk contains the answer. The answer emerges by following a chain of dependencies, but similarity search can struggle because the answer is distributed across multiple relationships rather than represented as a single semantic concept. GraphRAG addresses this issue by retrieving from a knowledge graph, where entities are connected through explicit relationships. However, GraphRAG is not a replacement for vector RAG. It is a retrieval paradigm for problems where relationships, graph structure, and provenance matter. For many applications, vector retrieval remains the fastest and most effective way to find semantically relevant information, while graph retrieval adds value when the answer depends on connected facts. But graph retrieval isn't a single technique; it requires deciding what to retrieve, how to retrieve it, and whether retrieval should happen in one pass or multiple stages. If these decisions are made correctly, GraphRAG can answer questions that vector search cannot; otherwise, it may result in a system that's slower and no more accurate than a well-designed vector RAG. This article focuses on retrieval. Graph construction is a separate issue, and schema quality affects everything. Here, we assume a well-built graph. The examples are based on a small software supply-chain graph. It's fictional, but the shape is from a real incident. In 2018, the npm package event-stream was compromised after a malicious dependency, flatmap-stream, was introduced into its dependency chain. The challenge is discovering the path shopping-app -> analytics-js -> event-stream -> flatmap-stream and connecting it to CVE-2024-1. No single text chunk contains this chain, and none of those package names indicate "compromised package." Scanners such as npm audit answer this question easily because they're built for this structure. GraphRAG can answer broader relationship questions around the same graph structure, especially when the answer requires combining multiple sources. Decision 1: Granularity Graph retrieval returns one of four units, from finest to coarsest. Node (for example, flatmap-stream and its attributes). Triplet (for example, flatmap-stream -[HAS_VULNERABILITY]-> CVE-2024-1). Path: like the red route. Paths are good for multi-hop and provenance questions. Subgraph: a connected region, like the payment component. Finer units are precise, but coarser units have more context, and more noise. Choose a unit that matches the question. Each mechanism, described in the next section, produces some units more naturally than others. Decision 2: The Six Mechanisms 1. Similarity Embed graph elements, retrieve the ones nearest the question vector. Most systems use this to find starting points. It answers "find things like this" questions on its own. Cypher CALL db.index.vector.queryNodes('pkgEmbeddings', 5, $questionVector) YIELD node, score RETURN node.name, score A simple query like "Which of our packages resemble this known-bad one?" is all that's needed. However, similarity over plain text embeddings has its limitations; it's blind to structure. So it only finds nodes that match the query and misses connected facts that don't resemble it. This method is useful for locating entry points and serving semantic lookups, but for anything that requires multiple hops, it's best to hand off to a structural mechanism. 2. Structural Traversal To get around the limitations of similarity searches, walk outward from the entry points, using techniques like neighbor expansion, breadth-first or depth-first search, and pathfinding. The granularity of what you collect depends on the approach; collecting neighbors gives you a local subgraph, while tracing routes between two entities gives you paths. Cypher // Entity-centric: what does the app build on? (a neighborhood) MATCH (:Package {name: 'shopping-app'})-[:DEPENDS_ON*1..2]->(dep:Package) RETURN DISTINCT dep // Connection question: how does the app reach vulnerable packages? MATCH p = (:Package {name: 'shopping-app'})-[:DEPENDS_ON*..5]->(bad:Package) WHERE (bad)-[:HAS_VULNERABILITY]->(:CVE) RETURN p For instance, the second query might return shopping-app -> analytics-js -> event-stream -> flatmap-stream, along with any other route to a vulnerable package. When the question is about risk, you want every possible path, not just the shortest one, because a second route is a second exposure. For large graphs, it is often more efficient to start from known vulnerable nodes and traverse backward, or constrain the search from the application side, depending on the query. Traversal is the cheapest, fastest, and easiest mechanism to explain, since you can read the route. But it has a weakness: fan-out. The practical depth limit depends on how constrained the walk is; unconstrained neighbor expansion grows rapidly, so shallow expansion is often preferred. A typed, direction-constrained path search, like the one mentioned earlier, prunes most of that growth and remains tractable deeper, which is why a four-hop dependency chase basically works in this case, but a generic four-hop expand-everything doesn't. Use traversal when the question is anchored on specific entities, and it's the best approach. 3. Graph Algorithms Two things are needed for traversal: a starting point, and a rule for edge selection. But what if you're missing one or both? There are two graph algorithms that can help. Personalized PageRank is useful when you have a starting point but no rule. It assigns a score to the mentioned entities, lets the score spread across the edges, and ranks nodes by the score they receive. Nodes that are highly reachable from the seeds through many strong paths receive higher scores. This helps find relevant nodes even if they're far away and don't share any words with the question. HippoRAG uses this for retrieval. Cypher CALL gds.pageRank.stream('supplyChain', { sourceNodes: $seedEntities, dampingFactor: 0.85 }) YIELD nodeId, score RETURN gds.util.asNode(nodeId).name AS entity, score ORDER BY score DESC LIMIT 10 For example, on our graph, if we seed analytics-js and CVE-2024-1, flatmap-stream receives a high score because it is strongly connected to both seed regions. Nothing in the query named it, though. Furthermore, directionality matters. For dependency graphs, reverse traversal or an appropriate projection is often required because vulnerabilities may be sink nodes. Community detection is especially useful for broad exploratory questions where no specific entity is known. A question like "what are the main risk areas across our dependencies?" is about the whole graph. The approach is to cluster the graph into communities, have an LLM summarize each community, and answer broad questions from those summaries. This is a key component of the Microsoft GraphRAG approach. On our graph, clustering gives us three communities: a payment-and-media stack, a web-framework stack, and the analytics subsystem with the compromised package. Cost is the main difference between the two. PageRank is expensive at query time because its scores depend on the seeds and can't be precomputed. On a large graph, you need to bound the projection to a region found by a similarity pass first. Community detection is expensive at index time because the LLM summaries are costly to compute and need to be redone when the graph changes. But queries after that are cheap. So pick PageRank when you have entities but no target. Pick communities when there's no starting entity. 4. Declarative Query With exact structural constraints in the question, it's best to query the graph directly. The model translates the question into a query language, like Neo4j's Text2Cypher. Schema grounding and validation significantly improve generated queries. Cypher // "Which volunteer-maintained packages also have a known CVE?" MATCH (pkg:Package)-[:MAINTAINED_BY]->(:Maintainer {kind: 'volunteer'}) MATCH (pkg)-[:HAS_VULNERABILITY]->(cve:CVE) RETURN pkg.name, cve.id This returns results like flatmap-stream / CVE-2024-1, combining a maintainer condition and a vulnerability condition that fuzzy mechanisms can only approximate, resulting in an exact and auditable outcome. This risk lies in the generation step, with models often inventing relationship types that sound right but don't exist in the schema; a well-formed query over imaginary edges returns zero rows without error. To mitigate this, ground the generation in the actual schema, validate the query before running it, and treat an empty result as a signal to fall back to another mechanism. Use this approach for questions that reduce to filters, counts, or joins across relationship types. 5. Generative Retrieval This mechanism works in two stages: first generating a retrieval plan that defines the relationship pattern to follow, and then translating that plan into a graph query. Reasoning on Graphs (RoG) works this way, with an LLM generating planning paths and the system retrieving the concrete paths that satisfy them. Cypher plan ← LLM("what relation path answers this?", schema) → "DEPENDS_ON* , then HAS_VULNERABILITY" paths ← graph.match(seed='shopping-app', pattern=plan) The model infers the shape of the traversal, and the graph supplies the instances. This approach fits questions where the right pattern isn't known in advance, and you don't want to hand-write a template for it. It relies on the model understanding the schema, and a one-shot plan can't correct itself unless you make the retrieval iterative. 6. Learned Retrieval A model can be trained to do the selecting, with a graph neural network scoring nodes for relevance to the question. Approaches such as G-Retriever formulate subgraph selection as an optimization problem, including variants inspired by Steiner tree formulations. The retriever is a trained component; it embeds the question, scores candidate nodes and edges, and returns the highest-value connected subgraph as evidence. This approach has demonstrated strong accuracy on hard multi-hop benchmarks, but it comes with training, serving infrastructure, and transparency costs. They're suitable for accuracy-critical question answering over a stable schema, but they're rarely the first build. Decision 3: The Paradigm You've still got to decide how many times to go to the graph. This affects both latency and accuracy. A simple approach is one retrieval, gathering everything in a single pass, which keeps latency low and is suitable for real-time answers. Iterative retrieval is another option, involving multiple passes that build on each other, useful when one pass isn't enough. It comes in two versions: fixed-rounds and adaptive. The adaptive version stops once the model has gathered sufficient information. Most production systems opt for a multi-stage approach, chaining different mechanisms together. A common pattern is using similarity retrieval to find entry points, structural traversal to expand context, and reranking to select the final evidence set. In practice, mechanisms usually combine in specific ways. Combining vector retrieval with graph retrieval is often described as HybridRAG. Letting a large language model plan the stages at query time is referred to as agentic retrieval, which is a composition pattern rather than a new mechanism. Matching Questions to Mechanisms A production system doesn't pick a mechanism per question at runtime. During the planning phase, you assess the questions, implement two or three mechanisms that cover them, and route between those in production. This table supports that assessment. The question is about…Reach forUsual granularityWhere the cost isThings semantically like X, or finding entry pointsSimilarityNode/tripletCheap, at query timeA specific entity or the routes between twoStructural traversalNode/path/subgraphCheap, at query timeMulti-hop relevance with an unknown targetPageRankRanked nodesCompute-heavy, query timeA broad theme across the whole graphCommunity detectionSubgraph + summaryExpensive, at index timeExplicit constraints: filters, counts, joinsDeclarative queryWhatever it projectsCheap, at query timeA pattern that must be inferred from the questionGenerativePath/subgraphModerate, one LLM callAccuracy-critical hard multi-hop QALearned (GNN)SubgraphExpensive, training Start with the basics. A similarity pass for entry points and structural traversal to expand. Add mechanisms as needed. PageRank or generative planning for harder questions, declarative queries for exact constraints, community summaries for thematic breadth, and learned retrieval when accuracy justifies it. Conclusion Graph retrieval involves making three key decisions. First, you need to choose the right granularity; this could be a node, triplet, path, or subgraph, depending on the answer you're looking for. The mechanism is also crucial: it's about selecting the right approach, such as similarity, traversal, graph algorithms, declarative queries, generative planning, or learned retrieval. Then there's the model: whether to use a single-pass, iterative, or multi-stage approach. By making these decisions with your system's specific questions in mind, you can design a tailored retrieval architecture rather than relying on trial and error.

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

Artificial intelligence is changing software engineering, impacting automation, user interaction, data analysis, and application development. Developers are evaluating how their technology stacks fit with these changes. For Java developers in enterprise settings, a main question is whether the Java enterprise ecosystem is prepared for AI. The short answer is yes. You do not need to abandon Java or wait for a new platform to build AI-enabled applications. Java already provides a mature ecosystem of AI libraries, model providers, APIs, and integration patterns. Jakarta EE offers the capabilities required to deploy these technologies in production-grade enterprise systems today. The ecosystem is evolving, with new initiatives exploring perfect integration of AI concepts within Jakarta EE APIs and programming models. This article reviews existing capabilities, Jakarta EE’s role within modern AI architectures, and potential future developments. AI and Software Engineering When applying artificial intelligence in software engineering, it is important to distinguish the different ways AI can be used throughout the development lifecycle. AI can assist with documentation, testing, code reviews, architecture exploration, and code generation. Architecturally, these uses fall into two categories: using AI to develop software and integrating AI within the software itself. The first category, AI-assisted software development, is currently the most common. Developers use AI tools to generate, explain, refactor, or test code. While these tools can boost productivity, they also introduce risks if not used with proper engineering discipline. Insufficient context, unreviewed code, or tools lacking architectural constraints can cause defects, security issues, complexity, or inconsistent design. AI does not replace the engineering team; it remains their responsibility to use it effectively. New methodologies are emerging to structure this interaction. Approaches like vibe coding focus on rapid development through conversational AI, while Spec-Driven Development offers explicit requirements, constraints, and context before code generation. Agent-based workflows increasingly use repositories with instructions, specifications, and Markdown files to give coding agents the required context. These approaches do not require abandoning Java; Java projects can already employ these techniques. The second category entails integrating AI within the application itself, making AI part of the application's runtime behavior rather than just assisting developers. Applications may use a large language model (LLM) to classify information, generate content, extract structured data, retrieve knowledge, execute tools, or make decisions within business workflows. This combination delivers a fundamental architectural change. Traditional enterprise applications are predominantly deterministic: developers define process flow using methods, conditions, rules, workflows, and state changes. With the same inputs and state, the execution path is predictable. In contrast, AI-enabled applications can present a dynamic execution model, where some behavior is determined at runtime via the LLM. However, not every AI-enabled application should surrender control to the model. In practice, AI architectures exist on a spectrum of autonomy. At one end, the model functions within a tightly controlled deterministic workflow. As autonomy increases, the model can select tools, plan steps, evaluate results, and coordinate more complex actions. This evolution is reflected in the Core Autonomy Patterns, which start with deterministic directed acyclic graph (DAG) workflows and progress toward more autonomous approaches such as retrieval-augmented generation (RAG), reflection, planning, ReAct, multi-agent systems, and Model Context Protocol (MCP) integrations. As flexibility increases, so does the architectural responsibility for observability, security, testing, governance, failure handling, and control. Recognizing this distinction is essential when evaluating Jakarta EE’s readiness for AI. The first category already integrates naturally with Java development tools. The second stresses the importance of the enterprise platform: AI applications still require dependency injection, configuration, REST APIs, persistence, messaging, transactions, security, observability, asynchronous execution, and integration with external systems. These are the capabilities Jakarta EE was designed to provide. Jakarta EE and AI Now Java and Jakarta EE are ready for the AI era. Integrating AI does not require leaving the enterprise Java ecosystem or waiting for new specifications. Jakarta EE applications can already use large language models (LLMs), embed AI in business workflows, and employ these capabilities within the wider enterprise platform. This is evident inside real-world applications. For example, Skillwell Simulate, a Jakarta EE-based platform, integrates with AWS services and uses Amazon Bedrock for AI features. This shows that Jakarta EE applications can adopt modern AI services while retaining the benefits of established enterprise architecture. At the lowest abstraction level, applications can integrate directly with AI providers such as OpenAI, Anthropic, Google, and Amazon Bedrock using their APIs or Java SDKs. This approach delivers full access to provider-specific features but increases coupling. Each provider uses different API models, configurations, formats, authentication, and features. Supporting multiple providers can add boilerplate and increase complexity. Enterprise developers are familiar with this challenge. Different vendors and technologies offer different capabilities, so abstractions provide a unified programming model. AI integration is now adopting a similar approach. OmniHai is a lightweight Java AI library for Jakarta EE and MicroProfile applications. Instead of requiring each vendor's SDK, OmniHai provides a consistent AIService abstraction and communicates directly with provider REST APIs. It currently supports OpenAI, Anthropic, Google AI, xAI, Mistral, Meta AI, Azure OpenAI, OpenRouter, Hugging Face, Ollama, and custom providers. With CDI, an AI provider can be injected directly into a Jakarta EE component: Java @Inject @AI(provider = AIProvider.ANTHROPIC,apiKey = "your-anthropic-api-key") private AIService claude; The application interacts with AIService instead of provider-specific APIs. This enables chat interactions to use a consistent programming model across providers: Java String response = claude.chat( "Explain microservices", ChatOptions.newBuilder() .systemPrompt("You are a helpful software architect.") .temperature(0.5) .maxTokens(500) .build() ); OmniHai also supports asynchronous and streaming operations through the same abstraction. Conceptually, this approach is similar to abstractions like EntityManager in Jakarta Persistence: the application uses a common API while implementation details remain hidden. Although not a perfect comparison, it illustrates OmniHai’s role in managing multiple AI providers. LangChain4j CDI offers a higher-level programming model. Instead of working directly with an AIService object, developers define an AI service as a Java interface. LangChain4j CDI detects interfaces annotated with @RegisterAIService and supplies their implementations as CDI beans. For example: Java @RegisterAIService public interface AssistantService { @SystemMessage("You are a helpful assistant.") String chat(String userMessage); } Developers do not write implementation classes. The infrastructure generates the implementation and connects the interface to the configured language model. The resulting service can be injected as any other CDI bean: Java @Path("/assistant") public class AssistantResource { @Inject AssistantService assistant; @GET @Path("/chat") public String chat(@QueryParam("message") String message) { return assistant.chat(message); } } This programming model will be familiar to Jakarta EE developers. It is similar to the repository abstraction in Jakarta Data, where developers define the contract through an interface and the infrastructure supplies the implementation. Although the technologies address different needs, this model reduces the amount of infrastructure code developers must write. LangChain4j goes beyond basic model invocation. It offers unified APIs for over 20 LLM providers and includes abstractions for tools, Retrieval-Augmented Generation (RAG), chat memory, structured outputs, agents, embedding stores, and other AI features. Supported integrations include Amazon Bedrock, Anthropic, Azure OpenAI, Google AI Gemini, OpenAI, Mistral, OCI Generative AI, among others. These options represent different levels of abstraction: OmniHai serves as a lightweight template-style abstraction, allowing the application to invoke operations through a common AIService. LangChain4j CDI advances this by supplying a declarative interface-based model, where developers describe the AI service and the infrastructure provides its implementation. Both approaches ensure the application stays a Jakarta EE application. Once an AI capability is available as a CDI bean, it integrates perfectly with the platform. REST endpoints can expose it, Jakarta Persistence or Jakarta NoSQL can supply data, Jakarta Security can protect its operations, Jakarta Messaging can trigger asynchronous workflows, and other Jakarta EE APIs continue their roles. The question is no longer whether Jakarta EE can integrate with AI; it already does. The key architectural decision is now the required level of abstraction: direct provider integration for maximum control, a lightweight common API like OmniHai, or a richer AI programming model such as LangChain4j CDI. Jakarta EE and Future Jakarta EE already supports AI integration, and the platform continues to evolve. Jakarta EE 12 focuses on improving the data layer, with updates to Jakarta Data, Jakarta Persistence, Jakarta NoSQL, and the new Jakarta Query specification. These improvements are especially important for AI applications that rely on enterprise data, persistence, retrieval, and contextual content. The primary AI-focused initiative is Jakarta Agentic AI, which has released its first milestone. Its purpose is not to replace LangChain4j or provider SDKs, but to offer a standard programming model for building AI agents with Jakarta EE. The specification defines a small set of concepts to structure agent workflows based on annotations, thus making the developer's life way easier: APIPurpose @Agent Declares an agent class @Trigger Defines the workflow entry point @Decision Determines whether and how the workflow proceeds @Action Defines a step in the workflow @Outcome Marks the end of the workflow @HandleException Handles exceptions inside the workflow @WorkflowScoped Provides one CDI context per workflow execution LargeLanguageModel Injectable facade for interacting with an LLM Result Represents the result of a decision This example presents a simplified fraud-detection agent and illustrates how Jakarta Agentic AI integrates with the Jakarta EE programming model. The agent uses the LargeLanguageModel facade for AI interaction and leverages Jakarta Persistence and Jakarta NoSQL to access enterprise data. As a result, AI capabilities are incorporated as part of the application, not as a separate programming environment. Java @Agent public class FraudDetectionAgent { @Inject LargeLanguageModel model; @Inject EntityManager entityManager; @Inject Template template; @Trigger private void handleTransaction( @Valid BankTransaction transaction) { } @Decision private Result checkFraud(BankTransaction transaction) { CustomerHistory history = template .find(CustomerHistory.class, transaction.customerId()) .orElse(null); String output = model.query( """ Analyze this transaction for potential fraud using the transaction and customer history. """, transaction, history); return new Result(isFraud(output), null); } @Action private void handleFraud( Fraud fraud, BankTransaction transaction) { if (fraud.isSerious()) { alertBankSecurity(fraud); } } @Outcome private void markTransaction( BankTransaction transaction) { BankTransaction managed = entityManager.merge(transaction); managed.markAsSuspect(); } } Conclusion Enterprise Java is prepared for AI today, with Jakarta EE already supporting this integration. Developers can add AI using provider SDKs, OmniHai, or LangChain4j CDI, while continuing to leverage Jakarta EE features for persistence, security, messaging, transactions, REST APIs, and enterprise data. AI enhances the existing platform as an integrated capability, rather than requiring replacement. The ecosystem continues to advance. Jakarta EE 12 enhances the data foundation, and Jakarta Agentic AI is introducing a structured programming model for building agents that integrate seamlessly with the platform. Jakarta EE is ready for AI now, and its capabilities will keep improving as the platform evolves.

By Otavio Santana CORE

Large language models have evolved from simple chat interfaces into autonomous systems capable of planning, reasoning, and interacting with external tools. The next stage of this evolution is multi-agent software engineering, where specialized AI agents collaborate to solve complex business workflows instead of relying on a single monolithic model. A planner may decompose work, researcher agents retrieve enterprise knowledge, coding agents generate implementations, reviewer agents validate outputs, and execution agents perform approved actions. Although this architecture appears attractive, production deployments reveal that coordinating multiple agents resembles building a distributed system far more than writing prompt chains. The primary challenge is not model intelligence but system reliability. Every additional agent introduces another opportunity for hallucinations, context loss, latency, retries, and cascading failures. A workflow containing five agents with individually high accuracy can still produce inconsistent outcomes because each handoff becomes another source of uncertainty. The engineering challenge therefore shifts from prompt engineering toward orchestration, state management, resilience, and observability. Most successful enterprise implementations begin with a planner-worker architecture. Instead of allowing every agent to communicate freely, a planner receives the business objective, decomposes it into smaller tasks, distributes work to specialized agents, and aggregates the responses into a final result. This pattern simplifies coordination, enables centralized policy enforcement, and provides a single location for monitoring execution. Java AgentPlan plan = planner.createPlan(request); List<CompletableFuture<AgentResult>> workers = plan.tasks().stream() .map(task -> CompletableFuture.supplyAsync( () -> worker.execute(task))) .toList(); List<AgentResult> results = workers.stream() .map(CompletableFuture::join) .toList(); return aggregator.combine(results); Bottlenecks Arise As the number of agents increases, direct synchronous communication quickly becomes a bottleneck. Event-driven messaging provides better scalability by allowing each agent to publish completed work while downstream agents subscribe only to events they understand. Kafka is particularly effective because partitions naturally distribute workloads across worker instances while preserving message ordering for individual workflows. The orchestration layer no longer manages worker availability directly and instead publishes work to topics, allowing consumer groups to handle scaling and recovery. A durable workflow engine becomes equally important. Stateless orchestration fails whenever a process crashes, a deployment occurs, or an agent exceeds execution time. Platforms such as Temporal persist workflow history so execution resumes from the last successful checkpoint rather than restarting an expensive reasoning process. This separation between orchestration and agent execution prevents duplicated work while making long-running AI workflows operationally reliable. Addressing Context Management Context management presents another significant engineering problem. Passing the complete conversation between every agent rapidly increases token consumption while reducing response quality. Instead, enterprise systems maintain workflow state separately from prompts. Business context is stored in persistent databases, semantic knowledge resides in vector stores, and external capabilities are exposed through Model Context Protocol (MCP) servers. Each agent retrieves only the information required for its current task instead of inheriting the entire execution history. Java workflowRepository.save( WorkflowState.builder() .workflowId(id) .currentAgent("SecurityReviewer") .status(Status.RUNNING) .context(serializedContext) .build() ); Standardizing communication between agents also improves maintainability. Rather than exchanging natural language, production systems often define structured contracts that include workflow identifiers, task types, priorities, and correlation identifiers. JSON { "workflowId": "WF-2041", "source": "Planner", "target": "CodeReviewer", "task": "Validate generated API", "traceId": "9bdc-421" } Structured messaging enables retries, auditing, replay, and interoperability across heterogeneous agents developed by different teams. It also aligns naturally with emerging protocols designed for agent interoperability. Reliability patterns from distributed systems remain equally valuable in AI applications. Agent failures should never stall an entire workflow. Timeouts, retries, circuit breakers, and dead-letter queues prevent individual components from consuming unlimited resources while protecting downstream services from cascading failures. Java try { AgentResponse response = future.get(20, TimeUnit.SECONDS); } catch (TimeoutException ex) { retryQueue.publish(task); circuitBreaker.recordFailure(); } Additional Issues to Consider Unlike conventional microservices, however, AI systems introduce another category of failure called reasoning loops. An agent may repeatedly invoke different tools while attempting to improve its answer without ever reaching completion. Runtime safeguards therefore extend beyond traditional retry limits to include maximum reasoning depth, token budgets, and execution deadlines. These controls prevent runaway costs while ensuring workflows terminate predictably. Production systems require complete visibility into every agent interaction. Traditional application logs reveal infrastructure failures but rarely explain why an AI workflow produced an incorrect decision. Distributed tracing with OpenTelemetry allows each planner, worker, and tool invocation to emit correlated telemetry containing workflow identifiers, agent names, execution latency, token usage, and tool calls. A single trace can reconstruct the entire reasoning path, making failures reproducible instead of mysterious. Java Span span = tracer.spanBuilder("agent-execution").startSpan(); span.setAttribute("workflow.id", workflowId); span.setAttribute("agent.name", "SecurityReviewer"); span.setAttribute("tokens.input", 1350); span.setAttribute("tokens.output", 512); worker.execute(task); span.end(); Observability should extend beyond infrastructure metrics. Enterprises benefit from tracking reasoning iterations, tool invocation frequency, retrieval latency, hallucination rates, retry counts, and token consumption. These operational metrics quickly reveal inefficient prompts, unreliable tools, or expensive reasoning loops before they impact production workloads. Testing also changes significantly. Traditional unit tests validate deterministic functions, whereas AI agents produce probabilistic outputs. Instead of asserting exact responses, enterprise pipelines evaluate workflows against acceptance criteria such as schema validation, factual correctness, safety policies, latency budgets, and execution cost. Regression suites should replay representative business workflows after every prompt, model, or orchestration change to ensure quality remains stable despite model updates. Security becomes increasingly important as agents gain permission to execute external actions. Every tool invocation should follow least-privilege principles, while generated code executes only inside isolated containers or sandboxes. Human approval remains essential for high-impact operations such as financial transactions, infrastructure changes, or customer-facing decisions. Durable workflow engines make this straightforward by pausing execution until approval arrives rather than blocking application threads. Standardizing Integrations The emergence of Model Context Protocol (MCP) further standardizes enterprise integrations. Instead of creating custom connectors for every application, MCP exposes databases, repositories, APIs, and enterprise tools through a consistent interface that any compliant agent can consume. Combined with Kafka-based messaging and workflow engines such as Temporal, MCP enables independently developed agents to cooperate without tightly coupling business logic to individual AI models. Despite growing enthusiasm, multi-agent architectures should not become the default solution. Many business problems remain better served by a single agent with carefully selected tools. Every additional agent increases latency, infrastructure complexity, operational cost, and potential failure points. Multi-agent systems become valuable only when tasks naturally decompose into specialized responsibilities requiring parallel execution, independent security boundaries, or domain-specific reasoning. Successful production deployments therefore resemble distributed systems more than prompt engineering experiments. Planner-worker orchestration, durable workflow persistence, event-driven communication, standardized protocols, resilient execution, comprehensive observability, and continuous evaluation collectively determine whether an AI system scales beyond demonstrations. A Final Word Multi-agent software engineering represents an important architectural evolution rather than simply a larger collection of language models. Organizations that approach agent collaboration with the same engineering discipline applied to microservices, distributed messaging, and cloud-native platforms will build systems capable of remaining reliable under production workloads. Those that treat agent orchestration as little more than chained prompts will likely encounter escalating costs, inconsistent behavior, and operational instability long before realizing the expected productivity gains.

Most protocol diagrams put Model Context Protocol (MCP), Agent2Agent (A2A) protocol, and Agent Communication Protocol (ACP) in three equal columns. I think that framing causes half the confusion. They aren't three interchangeable ways for agents to chat. MCP solves a capability-access problem. A2A solves a delegation problem. ACP explored a REST-first version of agent communication. Once those boundaries are separated, the architecture becomes much easier to reason about. An AI application may need to read a document, query a database, or trigger a build. Those are tool and context operations. The same application may also ask a security agent to assess a release, wait while that agent works, answer a clarification question, and collect a report. That is a different interaction, even if both flows happen inside the same user request. Practical rule of thumb: Use MCP when the caller needs a capability. Use A2A when the caller is handing responsibility for an outcome to another agent. Treat ACP as migration context, not a new third choice. At a Glance Protocol Boundary Discovery Work Unit Use It For MCP AI host/client to an MCP server Server capability negotiation and lists Tool call, resource read, prompt, or protocol request Tools, data, and reusable context A2A Client agent to remote agent Agent Card, registry, or private configuration Message or stateful Task with Artifacts Delegation and cross-agent collaboration ACP Application/agent to remote agent over REST Agent Manifest Run, message, session, and await flow Existing ACP integrations and migration 1. MCP: Give an Agent Access to Capabilities I find the Model Context Protocol (MCP) easiest to understand when I temporarily ignore the word agent. MCP is an integration contract between an AI host and external capability providers. The host might be a chat application, an IDE, or a larger agent runtime. It creates an MCP client for each MCP server it connects to. The server advertises what it offers: tools that perform actions, resources that provide context, and prompts that package reusable interaction patterns. Figure 1. MCP standardizes the connection between an AI host and external capabilities. A basic request is straightforward: The host decides that outside data or an action is needed.Its MCP client selects a discovered capability and sends a JSON-RPC request.The server validates the arguments, executes the operation, and returns structured content.The host gives that result back to the model or workflow so it can continue. MCP uses JSON-RPC 2.0 as its message protocol and supports both local communication over standard input/output (stdio) and remote communication over Streamable HTTP. It also covers initialization, capability negotiation, progress, cancellation, and errors. In other words, it is more than a convenient wrapper around function calling. A database query is not another agent. Neither is a file read, a ticket update, or a call to an internal pricing API. Modeling every capability as an agent adds identity, state, and orchestration overhead where a direct tool contract would be clearer. A practical MCP design check: Use synchronous MCP tool calls for fast, interactive, user-facing operations. For workflows that are long-running, require approvals, or need to continue independently of the client connection, use an asynchronous job or agent-based workflow instead. 2. A2A: Hand Work to Another Agent With Agent2Agent (A2A), the remote participant is an independent agent. It may use a different model, framework, programming language, cloud, memory system, or tool stack. The caller should not need access to those internals. It needs a stable contract for discovering the agent, authenticating, sending work, and receiving results. Figure 2. A2A adds discovery, delegation, task state, and structured deliverables. Discovery starts with an Agent Card. A public deployment can expose it at the well-known URI below; a private enterprise deployment may use a registry or direct configuration instead. JSON /.well-known/agent-card.json The card tells a client which skills the agent advertises, where its interfaces are, which protocol version they support, and how authentication works. In A2A 1.0, the core bindings are JSON-RPC, HTTP+JSON, and gRPC. The interaction can return a Message immediately, but the more interesting object is a Task. A Task has an ID, status, history, and output Artifacts. It may be working, waiting for more input, completed, failed, canceled, or rejected. That input-required state is what makes the protocol useful for real delegation: the remote agent can pause, ask a question, and resume without pretending the entire job was one function call. A2A also gives the caller choices for result delivery. It can wait, poll, stream ordered task events, or register a push-notification endpoint for longer work. If I expect a remote agent to exceed a five-to-ten-second interactive budget, I'd use a Task and expose progress rather than holding one opaque HTTP request open. Again, that timing is a design preference, not a protocol rule. 3. ACP: Important History, but Not a New Default The Agent Communication Protocol (ACP) took a REST-first approach. Agents published an Agent Manifest, clients submitted Runs, and the protocol supported synchronous responses, asynchronous processing, streaming, sessions, multimodal messages, and an await mechanism for missing input. It was appealing because ordinary HTTP tooling could inspect and operate the interface. In August 2025, ACP officially merged with A2A under the Linux Foundation. The ACP team announced that active development would wind down and that migration support would move users toward A2A. The REST-first idea did not disappear: A2A 1.0 includes an HTTP+JSON binding alongside JSON-RPC and gRPC. I wouldn't start a new ACP integration in 2026 unless an existing platform or partner requires it. I'd maintain a working ACP path, put a migration boundary around it, and make new agent contracts A2A-compatible. A Production Pattern Here is a small but realistic starting point: one coordinator agent, two specialist agents, and four MCP servers. The coordinator delegates test analysis to one specialist and security analysis to another. Each specialist has only the tools it needs. Figure 3. A2A coordinates independent agents; MCP gives agent-controlled access to tools and data. Example request path: Step Protocol Example Control 1 A2A Coordinator sends a release-readiness Task to the test agent Task ID, 30-second synchronous window, then stream or poll 2 MCP Test agent reads pipeline status and open defects Read-only credentials; two-second per-call starting timeout 3 A2A Security agent requests missing release scope Task moves to input-required; coordinator supplies context 4 MCP Security agent reads scanner findings and policy documents Separate scopes for scanner and document store 5 A2A Both agents return Artifacts to the coordinator Persist final artifact, status transitions, and trace ID Those numbers are not standards. They are the kind of explicit starting points that stop a prototype from turning into a chain of requests with no timeout, no owner, and no observable state. What the Protocols Do Not Solve for You Interoperability is useful. It isn't an operating model. Identity: Pass the user or workload identity across the boundary. Do not let every MCP call run as one all-powerful agent service account. Authorization: Authorize the exact tool, resource, skill, and tenant. A trusted agent should not automatically inherit access to every downstream system. Retries: Retry reads and explicitly idempotent writes. A blind retry of a payment, deletion, or ticket creation can duplicate real-world actions. Observability: Log the trace ID, calling agent, remote agent, tool name, task ID, status transition, duration, and final error category. Without those fields, a multi-agent failure becomes guesswork. Human approval: Require confirmation for destructive changes, external communication, privileged access, or decisions with material business impact. So Which One Should You Use? For tool and context access: MCP.For responsibility handoff between independent agents: A2A.For an existing ACP estate: keep it stable, then plan the move to A2A. Most serious agent platforms will use both MCP and A2A. The coordinator talks to specialist agents through A2A. Each specialist reaches its approved systems through MCP. That split is not as visually symmetrical as three protocols in parallel columns, but it matches the real engineering boundaries much better.

The Uncomfortable Truth You’ve spent days prompt-engineering your LLM. You’ve benchmarked Claude against GPT. You’ve debated whether to use Mixtral. But your RAG pipeline is still returning garbage answers, and you’re blaming the wrong component. The LLM is only as good as the context it receives. Context quality is entirely determined by retrieval. Retrieval quality is entirely determined by your embedding model. Fix the bottom, and the top fixes itself. I ran the same RAG pipeline across four embedding models on a 10,000-document legal corpus Q and A task. Same LLM (Claude Sonnet 4.6), same chunking strategy, same vector store (pgvector), same top-k=5. Only the embedding model changed. Model Retrieval P@5 Faithfulness Dims Cost/1M text-embedding-3-large 0.91 0.88 3072 $0.13 BGE-M3 (local) 0.88 0.85 1024 Free text-embedding-3-small 0.74 0.69 1536 $0.02 all-MiniLM-L6-v2 0.61 0.55 384 Free The gap between all-MiniLM-L6-v2 and text-embedding-3-large is 30 precision points. That’s not a minor tweak. That’s the difference between a product people trust and one they abandon. Your LLM had nothing to do with it. Why Embedding Models Differ So Dramatically An embedding model maps text into a high-dimensional vector space. Two chunks are “similar” if their vectors are close, measured by cosine similarity. The problem: not all models learn the same notion of similarity. A general-purpose model trained on web data will cluster “bank” near both “river” and “finance.” A domain-aware model trained on legal or financial corpora knows context. This distinction cascades into every retrieval decision your system makes. What Embedding Models Actually Learn During training, embedding models are optimized to pull semantically similar sentences closer in vector space and push dissimilar ones apart. The training data, loss function, and model architecture determine what “similar” means. Contrastive learning (SBERT, BGE): Learns from positive/negative sentence pairsMatryoshka Representation Learning (MRL, OpenAI): Encodes quality at multiple scalesLate interaction models (ColBERT): Compares token-level representations at query timeSparse + dense hybrids (BGE-M3): Combines lexical and semantic signals Key Insight: Embedding models encode your domain assumptions. If your model doesn’t understand your domain, no amount of LLM tuning will compensate for what it retrieves. Benchmarking Embedding Models on Your Own Data Don’t trust vendor benchmarks on MTEB. MTEB tests general English retrieval. Your use case is specific. Run this evaluation harness against your own corpus before committing to any embedding model: Python from sentence_transformers import SentenceTransformer from openai import OpenAI import numpy as np from sklearn.metrics.pairwise import cosine_similarity # Ground-truth query -> relevant chunk pairs from YOUR data eval_pairs = [ ("What is the penalty for breach of contract?", "Section 12.3 outlines liquidated damages of 5%..."), ("When does the agreement terminate?", "This agreement expires on December 31st 2026..."), ] def precision_at_k(embed_fn, corpus, queries, relevant_ids, k=5): corpus_embs = embed_fn(corpus) hits = 0 for i, query in enumerate(queries): q_emb = embed_fn([query]) sims = cosine_similarity(q_emb, corpus_embs)[0] top_k = np.argsort(sims)[::-1][:k] if relevant_ids[i] in top_k: hits += 1 return hits / len(queries) # Wrap OpenAI embeddings client = OpenAI() def openai_embed(texts, model="text-embedding-3-large"): resp = client.embeddings.create(input=texts, model=model) return np.array([d.embedding for d in resp.data]) # Wrap local BGE-M3 st_model = SentenceTransformer("BAAI/bge-m3") def bge_embed(texts): return st_model.encode(texts, normalize_embeddings=True) models = { "text-embedding-3-large": openai_embed, "BGE-M3 (local)": bge_embed, } for name, fn in models.items(): score = precision_at_k(fn, corpus, queries, relevant_ids) print(f"{name}: precision@5 = {score:.3f}") Run this before you commit to any embedding model. Twenty minutes of benchmarking here saves weeks of LLM debugging later. Build the evaluation dataset from your domain expert’s known query-answer pairs — even 50 pairs gives a strong signal. Matryoshka Embeddings: Large-Model Quality at Small-Model Cost OpenAI’s text-embedding-3 models support Matryoshka Representation Learning (MRL). The model is trained so that any prefix of the full embedding vector retains useful semantic structure. This means you can truncate a 3072-dimensional vector to 512 dimensions and still retain ~94% of its retrieval quality, at a fraction of the storage cost. Python from openai import OpenAI import numpy as np client = OpenAI() def embed_with_matryoshka(texts: list[str], dimensions: int = 512): """ text-embedding-3-large supports 256 -> 3072 dims. 512 dims = ~83% storage reduction, ~94% benchmark quality retained. """ response = client.embeddings.create( input=texts, model="text-embedding-3-large", dimensions=dimensions ) return np.array([item.embedding for item in response.data]) # Insert into pgvector import psycopg2 conn = psycopg2.connect(DATABASE_URL) cur = conn.cursor() for chunk_id, chunk_text in chunks: emb = embed_with_matryoshka([chunk_text], dimensions=512)[0] cur.execute( "INSERT INTO documents (id, content, embedding) VALUES (%s, %s, %s)", (chunk_id, chunk_text, emb.tolist()) ) conn.commit() With 512 dimensions, you get ~94% of the full model’s retrieval quality at ~17% of the storage and index cost. This is the default I now recommend for most production RAG pipelines. Only go to 3072 if you’re in a domain with extremely dense technical vocabulary. Going Local With BGE-M3 and Hybrid Retrieval API-based embeddings have three costs: latency, money, and privacy. If you’re embedding sensitive documents — legal contracts, medical records, internal financials — sending them to an external API is a risk your legal team will veto. BGE-M3 from BAAI solves all three problems. It runs locally, it’s multilingual (100+ languages), and it uniquely supports three retrieval modes from a single model: dense, sparse (BM25-style), and ColBERT late interaction. The hybrid of dense + sparse consistently outperforms pure dense retrieval by 5-12 points on technical corpora. Python from FlagEmbedding import BGEM3FlagModel import numpy as np # Load once, reuse across requests model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True) def hybrid_embed(texts: list[str]) -> tuple: """ Returns dense vectors + sparse lexical weights. Combine both in your vector store with Reciprocal Rank Fusion. """ output = model.encode( texts, return_dense=True, return_sparse=True, return_colbert_vecs=False ) return output["dense_vecs"], output["lexical_weights"] def reciprocal_rank_fusion(dense_ranks, sparse_ranks, k=60): """Combine dense and sparse retrieval results with RRF.""" scores = {} for rank, doc_id in enumerate(dense_ranks): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) for rank, doc_id in enumerate(sparse_ranks): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) return sorted(scores, key=scores.get, reverse=True) # Usage dense_vecs, sparse_weights = hybrid_embed(document_chunks) # Store dense_vecs in pgvector / Qdrant / Weaviate # Store sparse_weights in Elasticsearch / OpenSearch BM25 index # At query time, retrieve from both and fuse with RRF Chunking Strategy Interacts With Your Embedding Model Your embedding model and your chunking strategy are not independent decisions. A model trained on sentence pairs (like SBERT) performs best when chunks are coherent semantic units — not arbitrary 512-token windows. If you’re using semantic chunking, your choice of embedding model even affects how you split the document. Semantic Chunking With Sentence Similarity Python from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer("BAAI/bge-m3") def semantic_chunk(text: str, threshold: float = 0.75) -> list[str]: """ Split text where consecutive sentence similarity drops below threshold. Preserves semantic coherence per chunk. """ sentences = text.split('. ') embeddings = model.encode(sentences, normalize_embeddings=True) chunks, current = [], [sentences[0]] for i in range(1, len(sentences)): sim = np.dot(embeddings[i-1], embeddings[i]) if sim < threshold: chunks.append('. '.join(current)) current = [sentences[i]] else: current.append(sentences[i]) chunks.append('. '.join(current)) return chunks chunks = semantic_chunk(document_text, threshold=0.72) print(f"Created {len(chunks)} semantic chunks") # vs naive: 512-token windows often cut mid-sentence Using semantic chunking with BGE-M3 on the same legal document corpus improved my precision@5 by an additional 6 points over fixed-size chunking with the same model. The embedding model and chunking strategy compound each other. Production Architecture: Putting It Together Here is the full production-grade embedding pipeline I use, combining everything covered above: Matryoshka embeddings for cost efficiency, hybrid retrieval for precision, and a fallback to local BGE-M3 for sensitive data: Python import os from enum import Enum from dataclasses import dataclass from openai import OpenAI from FlagEmbedding import BGEM3FlagModel import numpy as np class EmbedMode(Enum): OPENAI_MRL = 'openai_mrl' # API, cost-efficient via Matryoshka BGE_HYBRID = 'bge_hybrid' # Local, privacy-safe, hybrid retrieval @dataclass class EmbedConfig: mode: EmbedMode dimensions: int = 512 # For OPENAI_MRL fp16: bool = True # For BGE_HYBRID class ProductionEmbedder: def __init__(self, config: EmbedConfig): self.config = config if config.mode == EmbedMode.OPENAI_MRL: self.client = OpenAI() else: self.model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=config.fp16) def embed(self, texts: list[str]) -> dict: if self.config.mode == EmbedMode.OPENAI_MRL: resp = self.client.embeddings.create( input=texts, model="text-embedding-3-large", dimensions=self.config.dimensions ) return {'dense': np.array([d.embedding for d in resp.data])} else: out = self.model.encode( texts, return_dense=True, return_sparse=True ) return {"dense": out["dense_vecs"], "sparse": out["lexical_weights"]} # For most teams: embedder = ProductionEmbedder(EmbedConfig(mode=EmbedMode.OPENAI_MRL, dimensions=512)) # For regulated industries (healthcare, legal, finance): # embedder = ProductionEmbedder(EmbedConfig(mode=EmbedMode.BGE_HYBRID)) The Decision Framework Here is how I choose an embedding model for any new project. Run through these questions in order: Situation Recommended model Why General English, tight budget text-embedding-3-small Best cost/quality ratio for standard RAG Best API quality, flexible cost text-embedding-3-large @ 512d (MRL) 94% quality at 17% storage cost Privacy / regulated industry BGE-M3 local + hybrid On-prem, no data leaves your infra Multilingual corpus BGE-M3 local 100+ languages, best non-English retrieval Edge / mobile / <50ms latency all-MiniLM-L6-v2 Tiny model, still useful for simple domains Unknown domain Benchmark first Run precision@5 eval before committing Common Mistakes to Avoid Using the same embedding model for indexing and a different one at query time: Vectors from different models are not comparable. Always use identical model + dimension settings for both.Not normalizing embeddings before cosine similarity: Always set normalize_embeddings= True or call np.linalg.norm(v) yourself.Choosing dimensionality based on benchmark scores, not your actual index size: At 10M+ documents, the storage cost of 3072d vs 512d is the difference between a $200/month and a $1,200/month vector DB bill.Ignoring embedding model updates: Text-embedding-3 is not the same as text-embedding-ada-002. Re-embed your entire corpus when you upgrade models. Do not mix vectors from different model versions in the same index.Skipping evaluation on your own data: MTEB leaderboard rankings do not predict performance on your specific domain. A model ranked 5th globally may outperform a rank 1 on your data. Conclusion Your LLM is not your bottleneck. Switching from GPT-4 to Claude Sonnet gives you maybe a 5% quality lift on a well-constructed RAG pipeline. Switching from all-MiniLM-L6-v2 to text-embedding-3-large gave me 30 precision points on the same pipeline, with the same LLM and the same prompt. Benchmark your embeddings first. Pick your chunking strategy second. Obsess about your LLM last. The teams winning with production AI are not the ones with the best model subscription. They are the ones who built retrieval pipelines that actually surface the right context, and it starts with the embedding model. As a rule of thumb, spend 20% of your AI engineering time on embedding evaluation. It will return 80% of your RAG quality gains.

Vibe coding is not a programming technique. It's an organizational event dressed up as one. When GitHub Copilot launched in 2021, the narrative was "AI as pair programmer." When ChatGPT arrived, it shifted to "AI as junior developer." By the time Cursor and Windsurf and Devin entered the picture, the goalposts had moved so far that we stopped noticing where they used to be. The term itself — coined by Andrej Karpathy in early 2025 — describes writing software by describing what you want and iterating on AI output until it looks right. No architecture upfront. No deep understanding of the internals. Just prompt, review, adjust, ship. The name is almost deliberately casual. Vibe. As if the whole thing is low-stakes. It is not low-stakes. Not even close. What's actually happening is that the cost of producing a working prototype has collapsed to near zero. A product manager, a designer, a technically literate founder, or a business analyst with a ChatGPT subscription can now produce something that looks — and often behaves — like what an engineering team would have spent two sprints building. The artifact exists. It runs. It answers to a curl command. And that is exactly what makes the next part so uncomfortable. What "Engineering as a Service" Actually Means on the Ground The phrase Engineering as a Service — EaaS — has been floating around enterprise architecture circles for a few years. It originally described platform teams offering standardized, self-service infrastructure to internal product teams. Sensible concept. Reasonable organizational model. That is not what I mean when I use the term now. What I'm watching happen — across teams I've worked with, consulted for, and frankly in my own org — is a quieter version: engineers becoming a validation and production-hardening layer that sits after the AI has already done the creative work. The product manager vibe-codes a proof of concept. The engineering team is handed it and asked to "make it production ready." Requirements arrive pre-defined. The architecture decision has already been made, implicitly, by whatever structure the AI generated. The engineer's job, in this model, is to clean up after the vibes. "The most dangerous moment isn't when AI writes bad code. It's when the organization stops asking engineers to think before the code exists." That shift is subtle. Gradual. And it doesn't announce itself in a restructuring memo. It shows up in how sprint planning conversations change. In the questions that stop being asked. In the job descriptions that quietly remove "system design" and add "AI code review." In the fact that your most technically sophisticated colleagues are increasingly valued for their ability to spot what the model got wrong — not for their ability to envision what should be built in the first place. It's hollowing. And the hollow feels comfortable for a surprisingly long time. The 80% Problem Nobody Is Talking About Honestly The statistic getting passed around is that AI can write 80% of the code. Maybe 90%. Some teams will tell you it's higher. They're probably right, for a certain definition of "code." What nobody says in the same breath is what that 20% contains. It contains the decision to use a distributed lock instead of an optimistic concurrency strategy, because you know your write contention pattern at 3 AM on the first of the month. It contains the choice to put that third-party API call behind a circuit breaker, because you were the one paged at midnight when their service went down for six hours two years ago. It contains the knowledge that your payment processor has a 30-second timeout that doesn't appear anywhere in their documentation, and that the retry logic the AI generated will double-charge customers under exactly the conditions that will occur in production. That 20% is not filler. It's the residue of lived system knowledge. And it cannot be prompted for, because it lives in people, not in documentation. The Skill Inversion Nobody Budgeted For Here's the part that makes people genuinely uncomfortable when you say it in a room: the engineers who are worst positioned for this transition are often the best coders. Think about it. If you built your professional identity around the craft of writing clean, efficient, elegant code — if that's the thing you're proud of, the thing you've spent ten years sharpening — you are now in a profession where that particular skill is the one being automated away fastest. The engineers who wrote beautiful Ruby. The ones who could implement a red-black tree from memory. The ones whose pull request diffs were a pleasure to read. Those skills are not worthless. But they are no longer the differentiator. The differentiator is now something harder to teach, harder to credential, and much harder to interview for: the ability to look at an AI-generated system and know — without running it — which assumptions it made, which failure modes it ignored, and which organizational constraints it has no way of knowing about. That requires something I'd describe as systems intuition. It's not algorithmic. It's not certifiable. It's the thing you develop after you've been on-call for two years, after you've traced a cascade failure through six services at 2 AM, after you've had to explain to a CFO why a "working" deployment is losing the company $4,000 an hour. You can't vibe-code your way to it. What Gets Built When Engineering Is a Service Let's say the EaaS model wins. Let's say your organization fully embraces the idea that non-engineers will prototype, AI will build, and engineers will review and harden. What does the resulting software actually look like? I know, because I've seen early versions of it. Across four teams that went deep on vibe-coding workflows in 2025, there are common artifacts starting to emerge. The code is structurally fine. Readable, even. Comments are excellent — AI comments well. Test coverage looks good on paper. But the systems have a specific flavor of wrongness that takes a while to name. They're built for the happy path with unusual thoroughness. And they fail in ways that are not in any test suite, because the failure modes weren't imagined in the prompts that generated the code. One team I spoke with — a Series B SaaS company, roughly 40 engineers — went vibe-coding-first on a new data pipeline in late 2024. Shipping velocity tripled. Incident rate was flat for two months. Then they hit Black Friday. The pipeline had no backpressure mechanism. The AI had generated clean, efficient queue processing code that assumed queue depth was bounded. Under real peak load, it consumed memory until the service OOMed, cascading into three downstream consumers. Recovery took nine hours. The postmortem finding: no one had asked the AI "what happens when the queue grows faster than we can consume it?" Because the PM who wrote the initial prompt didn't know to ask. And the engineers who reviewed the output were reviewing it for correctness, not for production failure modes they hadn't witnessed yet. So Who Actually Survives This? If the question is "what does the engineering career look like in a world where AI generates 80% of the code," the answer isn't just "learn to prompt better." That framing is too small. It optimizes for the wrong thing. The survivors are the engineers who never let their professional identity live entirely in the code. They're the ones who were always curious about why a system needed to exist, not just how to build it. The ones who sat in product strategy meetings when they didn't have to. The ones who wrote design docs before anyone asked and kept them updated after nobody read them. They're also the engineers who carry operational scar tissue. Production incidents are an education that no prompt can replicate. Every major outage you've lived through deposits something into your mental model of systems — a new category of "things that go wrong under conditions that weren't in the spec." That library of failure is, right now, one of the most underappreciated professional assets in engineering. The survivors will be engineers who can sit across from an AI-generated system and run it through a mental gauntlet: what happens when the third-party API goes down? What happens when this queue backs up for six hours? What happens when someone sends a payload that's technically valid but semantically adversarial? What happens when this runs in the EU and GDPR applies to this field? Not because they're pessimistic — but because they've seen all of those things happen. The Uncomfortable Truth About Fighting Back I want to be careful here, because the easy response to all of this is: good engineers will always be needed. And that's technically true in the same way that good writers are always needed in the age of generative text. It doesn't tell you much about the market. It doesn't tell you which specific kind of good engineering will be compensated. The uncomfortable advice, the kind I give to engineers who ask me directly: stop being the person who writes the most code, and start being the person who knows the most about what the code needs to survive in the real world. Those are different identities. They require different habits. And the transition is not comfortable, especially if you built your self-image around your coding ability. The engineers who will own the next decade are the ones who can walk into a room where an AI has already generated a candidate architecture and say — clearly, specifically, with evidence — why that architecture will fail, what it will cost, and what needs to change before anyone touches a production database. Not because they can write better code than the AI. But because they've seen this movie before, in a dozen variations, and they know how it ends. That's not a skill you can automate. Not yet. Maybe not ever.

By Dinesh Elumalai CORE

Most teams are still building AI agents like chatbots. That is fine for demos. It is not fine for production. A chatbot answers a question. An enterprise AI agent executes work. That difference sounds small, but it changes the entire architecture. Consider a customer support agent investigating a complex technical escalation. The agent may need to analyze diagnostic logs, search product documentation, find similar historical incidents, consult multiple specialized agents, wait for a support engineer to review a recommendation, and then generate a remediation plan. That workflow may take minutes, hours, or even longer. Now ask the uncomfortable engineering questions: What happens if the user closes the browser?What happens if the API request times out?What happens if one downstream system is unavailable?What happens if the model call is throttled?What happens if human approval arrives six hours later?What happens if the process restarts halfway through execution? If the answer is "we will handle that in the agent code," the architecture is already in trouble. The biggest mistake many teams make is treating the LLM as the application. In production systems, the workflow is the application. The LLM is one component inside a larger execution graph. Enterprise AI agents are not chatbots. They are distributed systems. And distributed systems need durable runtimes. The Chatbot Architecture Breaks Quickly Most early AI applications start with a simple request-response model: user request → agent API → LLM → response. This works well for Q&A, summarization, search, content generation, and basic tool calling. But enterprise workflows rarely stay that simple. A customer support agent might instead follow a flow like this: analyze the logs, search the knowledge base, find similar historical cases, run diagnostic reasoning, check severity and escalation policy, wait for human review, and only then generate a final recommendation. This is not a chat interaction. It is a long-running business process with AI inside it. The moment the agent becomes responsible for completing work across systems, the architecture needs capabilities that most chatbot implementations do not provide: Durable stateRetry policiesProgress trackingCorrelation IDsHuman approval checkpointsPartial failure handlingEvent-driven resumptionAuditabilityWorkflow versioning These are workflow orchestration concerns, not prompt engineering concerns. The Real Problem Is Execution, Not Reasoning The AI industry talks a lot about reasoning. But many production failures are not reasoning failures. They are execution failures. The model may correctly identify the next step, and the system still fails because: The workflow state was stored only in memory.The frontend session disappeared.The backend request exceeded a timeout.A transient API failure caused the entire workflow to restart.A human approval step was handled outside the agent workflow.There was no way to resume from the last completed step.The agent retried a non-idempotent action and created duplicate work. In other words, the model worked. The runtime failed. None of these failure modes are new. Durable-execution runtimes solved persist-and-resume for workflows years ago, checkpointing is older than that, and idempotency keys are payments-industry bedrock. What has changed is who is building these systems: the teams shipping agents today largely did not live through the workflow-engine era, so the discipline is being relearned. Agents also add one failure mode the classical systems never had. A workflow engine handed ambiguous state fails loudly. A language model handed ambiguous state re-reasons from scratch — it will confidently re-derive a plan, redo completed work, and re-request data it already has, and it will do so in fluent prose that looks like progress. That is a failure mode you have to design against explicitly, because it does not announce itself. This is why enterprise agent architecture needs to borrow more from distributed systems, workflow engines, and cloud orchestration than from chatbot demos. A serious AI agent platform needs to answer: How is workflow state persisted?How are long-running tasks resumed?How are retries controlled?How are external events handled?How are human decisions represented?How is progress exposed to the user?How are multiple agents coordinated?How are failures isolated? If those questions are not part of the architecture, the system is not production-ready. The Better Mental Model: Workflow First, Model Second The most useful mental model is this: The workflow is the application. The model is one activity inside it. That shift changes how systems are designed. Instead of building a giant agent that does everything, design a durable workflow that coordinates specialized capabilities. For a customer support scenario, the system might use multiple specialized agents: Diagnostic Agent: analyzes logs, symptoms, and telemetry.Knowledge Search Agent: searches product documentation and known issues.Historical Case Agent: finds similar resolved incidents.Policy Agent: checks escalation, compliance, or risk rules.Resolution Agent: synthesizes the final recommendation. Each agent has a focused responsibility. The orchestration layer coordinates execution, and it should own workflow progression, agent sequencing, parallel execution, state persistence, retry behavior, failure handling, human review, and final aggregation. This keeps the AI layer focused on reasoning and the workflow layer focused on execution. Reference Architecture: Durable Runtime for Long-Running Agents A production-oriented architecture looks more like this. A durable orchestration layer owns state, coordination, retries, and the human-in-the-loop wait; specialized agents own only their domain. Because the orchestrator checkpoints to durable state after every step, the workflow survives restarts, deploys, and days-long approval waits. The important part is not the specific cloud service. The important part is the architectural separation. The user interface starts the workflow. The durable orchestrator coordinates execution. Specialized agents perform bounded work. The workflow stores progress, handles retries, waits for human input, and resumes reliably. Azure Durable Functions is one practical implementation of this pattern because it provides stateful orchestrations, activity functions, checkpointing, retry policies, and long-running workflow support on a serverless runtime.¹ The same architectural idea can be implemented with other workflow engines. The point is not "use one specific product." The point is "do not build long-running agent execution as a stateless API." Fan-Out/Fan-In Is a Natural Pattern for Multi-Agent Systems Many enterprise AI workflows contain independent tasks. A customer support investigation can often run its diagnostic, knowledge, historical, and policy analyses in parallel — the fan-out/fan-in shape in the architecture above. The workflow fans out to multiple specialized agents. Each agent performs independent analysis. The workflow then fans in the results and synthesizes a recommendation. This maps directly onto the fan-out/fan-in pattern documented for durable orchestrations, which runs multiple functions in parallel and aggregates the results afterward. ² A simplified C# orchestration can look like this. The examples use .NET Durable Functions; the same patterns exist in the Python and JavaScript bindings, and in runtimes like Temporal. C# [Function(nameof(CustomerSupportAgentOrchestrator))] public static async Task<SupportCaseResolution> RunAsync( [OrchestrationTrigger] TaskOrchestrationContext context) { var request = context.GetInput<SupportCaseRequest>() ?? throw new InvalidOperationException("Support case request is required."); context.SetCustomStatus("Launching specialized agents"); var diagnosticTask = context.CallActivityAsync<AgentFinding>( nameof(RunDiagnosticAnalysisAgent), request); var knowledgeTask = context.CallActivityAsync<AgentFinding>( nameof(RunKnowledgeSearchAgent), request); var historicalTask = context.CallActivityAsync<AgentFinding>( nameof(RunHistoricalCaseAgent), request); var policyTask = context.CallActivityAsync<AgentFinding>( nameof(RunPolicyAgent), request); var findings = await Task.WhenAll( diagnosticTask, knowledgeTask, historicalTask, policyTask); context.SetCustomStatus("Aggregating agent findings"); var resolution = await context.CallActivityAsync<SupportCaseResolution>( nameof(GenerateDraftResolution), findings); return resolution; } This is more maintainable than building one large prompt that tries to do everything. It also gives the platform better control over which agents ran, which agents failed, which outputs were used, how long each step took, and what evidence supported the final answer. That matters in enterprise systems. Human-in-the-Loop Is Not an Edge Case Many enterprise AI systems quietly assume that agents will produce immediate answers. Real workflows often require human decisions — when confidence is low, when customer impact is high, when the recommendation involves risk, when the action changes system state, when the workflow touches regulated data, or when the escalation is sensitive. The timeline usually looks nothing like a chat exchange. Drawn to scale: the model is not the bottleneck. A typical investigation spends two minutes on AI analysis and six hours waiting for a human to approve. The slowest step is not always the LLM. It is often the human approval, dependency response, or operational handoff. This is where durable orchestration becomes essential. The workflow needs to pause without losing state. It should not keep a web request open. It should not rely on memory. It should not require a custom polling database plus a manual recovery script. Durable orchestration can model this directly: C# context.SetCustomStatus("Waiting for human review"); var reviewDecision = await context.WaitForExternalEvent<HumanReviewDecision>( "HumanReviewCompleted"); var finalResolution = await context.CallActivityAsync<SupportCaseResolution>( nameof(GenerateFinalResolution), new FinalResolutionRequest { ReviewDecision = reviewDecision }); return finalResolution; External events let a running orchestration receive signals from outside — human approvals, webhook callbacks, or other systems — without holding compute open while it waits.³ That matters because human approval should not be a side process. It should be part of the workflow. The Crash That Costs Money Durable runtimes give you at-least-once execution. After a crash, an activity may run again. For reads, that is free. For writes, it is the most dangerous window in the architecture, and it is worth being precise about where it opens. A workflow issues a customer refund. The money moves. In the instant before the runtime checkpoints that the activity completed, the process dies. On recovery, the runtime replays the activity — behaving exactly as designed — and issues the refund a second time. The orchestrator cannot prevent this, because from its point of view the activity never completed. The fix has to live in the side-effecting operation itself: every consequential write carries an idempotency key, and an operation that sees a key it has already processed returns the original result instead of acting twice. C# var refund = await context.CallActivityAsync<RefundResult>( nameof(IssueRefund), new RefundCommand( CaseId: request.CaseId, Amount: approvedAmount, IdempotencyKey: $"{request.CaseId}:goodwill-refund")); At-least-once execution guarantees a replay will eventually land in the gap between a side effect and its checkpoint. Without an idempotency key, the replay issues a second refund. With one, the operation recognizes the key and returns the original result — two calls, one refund. Resumability and idempotent writes are the same requirement seen from two sides. You cannot safely resume a workflow whose writes are not safe to replay. The Orchestrator Should Coordinate, Not Think A common mistake is putting too much logic inside the agent or the orchestrator. A better separation is simple to state: the orchestrator decides what happens next — calling activities, waiting for events, tracking status, applying retry policy, coordinating results. Activities do the work — calling models, searching systems, querying databases, invoking tools, performing side effects. For example, an activity that calls a knowledge search agent might look like this: C# public sealed class RunKnowledgeSearchAgent { private readonly IAgentExecutionClient _agentClient; public RunKnowledgeSearchAgent(IAgentExecutionClient agentClient) { _agentClient = agentClient; } [Function(nameof(RunKnowledgeSearchAgent))] public async Task<AgentFinding> RunAsync( [ActivityTrigger] SupportCaseRequest request) { var response = await _agentClient.RunAsync(new AgentExecutionRequest { AgentName = "KnowledgeSearchAgent", Prompt = $""" Search for relevant troubleshooting guidance. Case: {request.CaseId} User question: {request.UserQuestion} Product area: {request.ProductArea} Return concise findings with supporting evidence. """ }); return new AgentFinding { AgentName = "Knowledge Search Agent", Summary = response.Summary, ConfidenceScore = response.ConfidenceScore, Evidence = response.Citations, RequiresHumanReview = response.ConfidenceScore < 0.75 }; } } This keeps model calls, retrieval, tool execution, and external I/O outside the orchestration logic. That separation improves testability, recovery, and observability. Design for Partial Success Enterprise workflows should not be all-or-nothing by default. If four specialized agents run and one fails, should the entire investigation fail? Sometimes yes. Often no. A better design is to treat agent results as structured outcomes: C# public sealed record AgentExecutionResult { public required string AgentName { get; init; } public bool Succeeded { get; init; } public AgentFinding? Finding { get; init; } public string? FailureReason { get; init; } } Now the aggregation layer can reason about partial results. If the diagnostic, knowledge, and policy agents succeed and the historical-case agent fails, the system can still produce a recommendation — with an explicit caveat that historical case comparison was unavailable. This is how resilient systems behave. They degrade gracefully instead of collapsing completely. AI agents need the same discipline. Observability Is a Product Feature Users do not just want the final answer. They want to know what the system is doing. A long-running agent should expose meaningful progress — started investigation, analyzing diagnostics, searching knowledge base, finding similar cases, aggregating findings, waiting for human review, generating final recommendation, completed. This is not cosmetic. Progress visibility builds trust. From an operational perspective, the platform should track the workflow instance ID, correlation ID, case ID, current stage, agent execution duration, retry count, failure reason, human review latency, final outcome, and evidence references. If a support engineer asks, "Why did the agent recommend this?" the system should have an answer. If an operator asks, "Where are workflows getting stuck?" telemetry should show it. If a governance reviewer asks, "Which model and prompt version produced this recommendation?" that should be traceable. This is why observability belongs in the architecture, not in a dashboard added at the end. Retry Policy Is Part of the Design Long-running agents depend on external systems, and those systems will fail. They will throttle. They will time out. They will return transient errors. They will behave differently under load. Retry behavior should be explicit. C# var retryPolicy = new RetryPolicy( maxNumberOfAttempts: 3, firstRetryInterval: TimeSpan.FromSeconds(10)) { BackoffCoefficient = 2.0, MaxRetryInterval = TimeSpan.FromMinutes(2) }; var taskOptions = new TaskOptions(retryPolicy); var finding = await context.CallActivityAsync<AgentFinding>( nameof(RunKnowledgeSearchAgent), request, taskOptions); Retries should be applied carefully. Retry transient failures: HTTP 429, HTTP 5xx, temporary network failures, search service timeouts, model endpoint throttling. Do not blindly retry invalid input, authorization failures, policy violations, business rule failures, or — as the previous section argued — any non-idempotent side effect. A durable runtime gives teams a place to encode this behavior consistently. Without it, retry logic gets scattered across controllers, services, queues, and agents. Governance Matters More When Agents Act Governance becomes more important when agents stop answering questions and start influencing operational decisions. At minimum, production agent workflows should track the workflow version, agent version, prompt version, model deployment, input data sources, evidence references, reviewer decisions, final recommendation, and correlation ID. This is not bureaucracy. It is operational safety. If an agent provides a recommendation on a support case, teams need to know what information was used, which agents participated, whether a human approved the result, and how the final recommendation was generated. A durable workflow makes that lineage easier to capture, because the workflow already represents the execution path. The Test That Tells You Whether Any of This Works Architecture diagrams do not prove durability. The only trustworthy verification I have found is destructive. Kill the running workflow at an arbitrary point — not at a clean boundary, at an awkward one. Discard all in-memory and in-context state. Bring the system back up and watch what the resumed execution does. A sound design picks up exactly where the work stood, and each distinct way of failing points at a specific gap: The resumed workflow...You are missingre-derives its plan from scratchpersisted state the agent layer actually readsredoes completed stepscheckpointing at the right granularityreloads its entire history to get orienteda scoped working set per resumere-fires a side effectidempotency keys A durable runtime passes the orchestration half of this test by construction. That is what you are buying. What it does not guarantee is the agent half: whether your agents' working context, retrieved evidence, and plans are reconstructed from durable state, or were quietly living in a context window that no longer exists. Run the test end to end, including the model-facing layers. That is where it fails in practice, and it is far better to learn that on a Tuesday afternoon than during an incident. Five Lessons From Building Long-Running Agent Workflows 1. The workflow matters more than the prompt. Prompt quality matters, but it does not solve execution reliability. A great prompt inside a brittle runtime still produces a brittle system. 2. Human latency dominates model latency. Many workflows wait longer for people than for models. Design for hours, not seconds. 3. Multi-agent systems need coordination, not chaos. Adding agents is easy. Coordinating agents is hard. Without orchestration, multi-agent systems become difficult to reason about, debug, and govern. 4. Partial success is better than total failure. Enterprise systems should degrade gracefully. If one agent fails, the platform should decide whether the workflow can continue with caveats. 5. Observability is part of the user experience. A long-running agent without progress visibility feels broken. A long-running agent with clear status feels reliable. Conclusion The next phase of enterprise AI will not be won only by better prompts or larger models. It will be won by better execution architectures. Long-running agents need to coordinate multiple systems, preserve state, recover from failures, wait for human approvals, expose progress, and produce auditable outcomes. That is not chatbot architecture. That is distributed systems architecture. The model is important, but it is not the whole application. In production-grade enterprise AI systems, the workflow is the application, and durable orchestration gives that workflow a runtime. If your AI agent needs to do real work across real systems, stop building it like a chatbot. Build it like a distributed system. References Microsoft Learn, "Durable Functions overview" — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overviewMicrosoft Learn, "Fan-out/fan-in pattern scenarios in Durable Functions" — https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-fan-in-fan-outMicrosoft Learn, "Handling external events in Durable Functions" — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-external-eventsMicrosoft Learn, "Durable Functions best practices and diagnostic tools" (idempotent activities, at-least-once execution) — https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-best-practice-reference