DZone Spotlight

I spent the first six months of a project convinced we had a model quality problem. Our anomaly detection system for manufacturing telemetry was missing obvious defects; things a human operator would catch in seconds. We tried bigger models, better embeddings, more training data. Nothing moved the needle. Then one afternoon, while tracing a specific false negative, I noticed the timestamp. The sensor reading that would have triggered a correct alert had arrived 47 seconds after the decision window closed. The model never saw it. Not because the model was bad. Because the pipeline delivered the data too late for the model to act on it. That's when I stopped thinking about model architecture and started obsessing over data delivery. And honestly, everything I've built since has been shaped by a simple realization: in industrial AI, the pipeline IS the product. The model is just the last mile. How Generative AI Changed the Conversation (But Not the Bottleneck) Everyone's building AI assistants, intelligent search, predictive analytics, autonomous workflows. The conversation focuses on foundation models, prompt engineering, inference optimization. Makes sense; that's the exciting part. But in industrial environments (semiconductor fabs, energy plants, discrete manufacturing), the bottleneck isn't model capability. It's whether the right data reaches the model at the right time, in the right shape, with the right lineage attached. I've watched teams spend months fine-tuning a model that was getting stale sensor readings. Months. The model was perfectly capable. It was just blind. This is why I've come to believe that industrial AI success is a data architecture problem first and a model problem second. The reason is not because models don't matter. Instead, it is because a brilliant model on bad plumbing produces confidently wrong answers, which is worse than no answer at all. What Semiconductor Fabs Taught Me About "Real-Time" Here's where my background in semiconductor manufacturing gives me a perspective most streaming architects don't have. In a modern fab (say, a 300mm facility running at 5nm or 3nm process nodes), a single wafer passes through 500+ process steps. Each step generates telemetry: gas flow rates, chamber pressure, plasma power, temperature profiles, film thickness measurements, overlay alignment data. Multiply that by 50 wafers per lot, dozens of lots per day, and you're looking at billions of data points daily. The fab doesn't batch-process this data overnight. It can't. A wafer worth $10,000+ is moving through the line continuously. If a process parameter drifts out of spec and you don't catch it until the nightly ETL job runs, you've potentially scrapped an entire lot. That's half a million dollars gone because your pipeline was "fast enough for batch." Fabs solved this decades ago with a discipline called Fault Detection and Classification (FDC). Every equipment run is analyzed in real-time (within milliseconds of completion). Statistical models compare current sensor traces against known-good profiles. If something looks off, the system raises an alarm before the next wafer enters the chamber. This isn't some exotic research concept. It's running in every leading-edge fab on the planet right now. And the architecture behind it looks remarkably like what we're trying to build in enterprise streaming: FAB FDC ARCHITECTUREenterprise streaming equivalent Equipment sensor streams (SECS/GEM protocol) Apache Kafka / Apache Flink event streams Real-time trace comparison Stream processing with windowed aggregations SPC control charts with Western Electric rules Anomaly detection on feature pipelines Recipe parameter adjustment (APC) Automated model retraining triggers Lot genealogy / WIP tracking Data lineage and event provenance The patterns are the same. The fab version just had higher stakes, forcing better discipline earlier. Why Streaming Isn't "Faster Batch." It's a Different Mental Model. This distinction tripped me up for a while. I kept thinking of streaming as "batch that runs every second instead of every hour." That's wrong, and it leads to bad architecture. Batch assumes data is static until the next scheduled update. You collect, then process, then serve. Streaming assumes data is continuously evolving. Events flow through the platform as they occur. Applications subscribe and react while the underlying process is still unfolding. The practical difference is enormous: Batch thinking: "We'll retrain the model on last night's snapshot." Result: the model is always 8-24 hours behind reality. In a manufacturing context, that's thousands of wafers processed with stale parameters. Streaming thinking: "The feature pipeline receives fresh observations as events arrive." Result: the model's context is minutes old, not hours. Decisions happen while outcomes can still be influenced. Apache Kafka, Apache Flink, and event-driven frameworks like Apache Pulsar make this architecturally possible today. The tooling has matured. The question isn't whether streaming works; it's whether your organization has made the mental shift from "collect then analyze" to "analyze as it flows." The Hidden Engineering Nobody Wants to Talk About Building industrial AI involves way more plumbing than anyone admits during the planning phase. Behind every successful deployment lies a data platform responsible for ingesting, validating, enriching, governing, and distributing information from dozens of independent systems. In manufacturing environments specifically: Equipment comes from multiple vendors (Applied Materials, Lam Research, Tokyo Electron; each with different telemetry formats).Sampling frequencies vary wildly (100ms for some sensors, 1Hz for others, event-based for yet others).Some systems generate structured events while others produce semi-structured logs.Data quality fluctuates depending on operating conditions (a chamber during maintenance produces garbage telemetry that looks like anomalies to a naive model). Before AI can analyze any of this, the platform must reconcile these inconsistencies into a unified representation. Schema registry (Confluent Schema Registry, Apicurio), data quality frameworks (Great Expectations, dbt tests), and format standardization (Apache Avro, Protocol Buffers) do this work. It's unglamorous. Nobody writes blog posts about schema reconciliation. But I've seen more AI projects die from bad plumbing than from bad models. The ratio isn't even close. Why Data Governance Isn't Compliance Anymore. It's Model Quality. This shift snuck up on me. I used to think of governance as something the compliance team worried about: data classification, retention policies, access controls. Important, but not my problem as an architect. Then I watched a machine learning model produce wildly inconsistent predictions because it was consuming two different versions of the same feature; one from the real-time pipeline (current) and one from a batch backfill (stale). No governance framework flagged this because nobody had defined "which version should the model use?" as a governance question. In industrial AI, governance questions become engineering questions: Where did this data originate? (Lineage: Apache Atlas, OpenLineage)Has it been validated? (Quality gates in the pipeline itself.)Which version should the model use? (Catalog: Apache Iceberg's time-travel, Delta Lake's versioning.)Can this information cross regional boundaries? (Compliance-as-code in the streaming layer.) The strongest architectures I've seen integrate governance directly into the event pipeline. Metadata travels with data. Access policies apply at the stream level. Lineage is preserved through every transformation. Not as a separate process; as part of the infrastructure itself. Why RAG Quality Is a Pipeline Problem (Not a Prompt Problem) Retrieval-augmented generation has become the default architecture for enterprise GenAI. Makes sense; you ground the language model in your proprietary knowledge rather than relying solely on its training data. But here's what I keep seeing: teams spend weeks optimizing prompts and chunking strategies while their knowledge base quietly goes stale. Documents update, but embeddings don't re-index. Permissions change, but the retrieval layer doesn't reflect them. Metadata drifts from reality. The language model still generates fluent responses. They're just increasingly grounded in yesterday's (or last month's) context. RAG quality, in my experience, depends more on the freshness and accuracy of the retrieval pipeline than on the generation model sitting on top. A well-maintained knowledge pipeline with a mid-tier model outperforms a frontier model drinking from a stale index. This means treating your RAG pipeline like a streaming system: continuous ingestion, continuous re-indexing, continuous validation. Not a one-time "load the docs and forget." Building for Scale Without Burning Money Industrial AI platforms process enormous event volumes. Millions of messages per minute. Thousands of assets generating telemetry simultaneously. Multiple AI services consuming overlapping datasets. Scaling this naively (just add more brokers, more compute, more storage) gets expensive fast. What I've found works better: Process at the edge when possible. In semiconductor manufacturing, FDC analysis often runs on edge compute at the equipment level (15ms response time vs 800ms round-trip to a centralized system). The same principle applies to any industrial streaming architecture: if the decision can be made locally, don't pay the latency and cost of a centralized round-trip. Tiered storage with hot/warm/cold patterns. Real-time features stay in low-latency stores (Redis, Apache Druid). Recent history lives in columnar formats (Apache Parquet on object storage). Deep history moves to cold archives. Apache Iceberg handles this elegantly with its metadata layer. Backpressure instead of over-provisioning. Rather than provisioning for peak load 24/7, build systems that gracefully handle bursts through buffering and backpressure mechanisms. Kafka's consumer group model does this naturally when configured properly. Observability across the entire pipeline. Not just the model; the pipeline itself. OpenTelemetry for tracing, Prometheus for metrics, distributed tracing that follows an event from sensor to prediction. When something goes wrong (and it will), you need to know where the failure point is in seconds, not hours. What I'd Tell Myself Two Years Ago If I could go back to the start of that project where we spent six months blaming the model: 1. Instrument the pipeline first. Before deploying any model, measure data freshness at every stage. Know exactly how old your model's context is at inference time. If it's stale, the model doesn't matter yet. 2. Treat streaming as a prerequisite, not an optimization. For industrial AI that needs to influence real-time outcomes, batch architectures aren't "good enough for now." They're architecturally incompatible with the goal. 3. Invest in schema discipline early. It's painful and boring. It pays for itself within months. Every team I've talked to that skipped this step regretted it when they tried to add a second or third data source. 4. Governance is architecture, not documentation. If governance policies don't enforce themselves automatically in the pipeline, they don't exist in practice. They're just PDFs nobody reads. 5. The pipeline IS the AI product. The model is important but replaceable. The data infrastructure that feeds it is the durable competitive advantage. Invest accordingly. Industrial AI is maturing quickly. The teams shipping reliable systems aren't the ones with the best models. They're the ones with the best plumbing. And honestly, that's encouraging because plumbing is engineering, and engineering is what we do. I'd love to hear what's worked (or spectacularly failed) in your streaming architectures for AI. The patterns are still emerging, and I think the best ideas are coming from practitioners who've felt the pain firsthand.

Most enterprises have all three pieces. A process automation layer. A data integration strategy. An AI initiative. Yet critical decisions still fail, agents still hallucinate, and workflows still run on yesterday's data. The investments exist. The convergence does not. The problem is not a lack of technology, but a lack of architectural thinking. Process intelligence, event-driven integration, and trusted agentic AI are being built in isolation, by different teams, with different goals, on different timelines. The result is an architecture that looks complete on a slide and breaks in production. This post argues that these three capabilities form a Trinity. They only deliver their full value when they are designed to work together. Three Capabilities, One Architectural Commitment Process intelligence, event-driven integration, and trusted agentic AI each solve a real problem. Each one also creates new risks when it operates alone. The following architecture shows how the three layers connect into a single, converged system. Process Intelligence: The Layer That Gives Agentic AI Its Boundaries Process intelligence is the evolution of classic Business Process Management (BPM) into something adaptive, event-aware, and AI-ready. It is the layer where technology maps directly to business value. Every workflow connects to a concrete business outcome: a loan approved, a shipment rerouted, a fraud case resolved. Process mining observes how business processes actually run, identifies where decisions fail, and surfaces where automation would deliver the most value. Vendors like Celonis have built entire platforms around this capability. Process orchestration executes workflows, enforces business rules, and produces the audit trails that compliance teams depend on. Camunda is a leading example. Agentic process orchestration goes one step further: it allows AI agents to participate directly in workflow execution, taking autonomous actions within defined boundaries while the process layer maintains control. Automation is the business driver. Organizations adopt process intelligence to automate more, faster, with less manual intervention, while keeping humans in control of the decisions that matter. But agentic automation only works safely when the process layer defines the operational envelope: what the agent can decide alone, what requires human approval, and what must be escalated regardless of what the model recommends. This is where guardrails live in practice. Not as theoretical constraints inside a model, but as concrete workflow gates that stop, route, or escalate before an action is executed. Process intelligence is what makes automation trustworthy at scale. Event-Driven Integration: From Scheduled Batches to Live Events Event-driven integration is the architectural principle that connects operational systems continuously, based on what happens rather than when a scheduler runs. An event from a payment system, a sensor, a CRM update, or a logistics platform travels in real time to whatever system needs to act on it. Apache Kafka has become the de facto standard for event-driven integration at enterprise scale. Other options exist, including cloud-native messaging services and specialized event brokers, but Kafka is where the ecosystem has converged. What matters in any case is the commitment to events as the primary integration primitive to ensure true decoupling, scalability, and data consistency across real-time and batch systems. The market reflects this shift. Process orchestration engines have rearchitected their core runtimes to be event-driven from the ground up, built for real-time throughput and horizontal scale. Camunda's Zeebe is a leading example. Zeebe is itself an event-driven engine, which means organizations can implement event-driven workflows and lightweight integration patterns without Kafka as a prerequisite. For broader enterprise integration at scale, Apache Kafka complements the process orchestration layer, connecting the full landscape of operational systems, SaaS platforms, and data infrastructure into a single event-driven backbone. Core business applications and SaaS platforms followed. SAP S/4HANA, Salesforce CRM, and ServiceNow have all added eventing interfaces and Change Data Capture (CDC) capabilities alongside their traditional API-based request-response integrations. The direction is clear: even systems that were designed around synchronous HTTP are moving toward event-driven models. Process engines receive live state. Agentic AI systems receive current context. Decisions are made on what is actually happening, not on what happened last night. Trusted Agentic AI: Safety Is an Architecture, Not a Setting Trusted agentic AI is an architectural property, not a product feature. Agentic AI systems do not just generate responses. They take actions, trigger workflows, and interact with operational systems. That autonomy is what makes trust and safety an architectural concern rather than a model configuration. It operates at two levels. The first is the model itself. Vendors like Anthropic and Mistral build alignment, constitutional constraints, and refusal behaviors directly into their models. This provides a baseline. The second level is the process intelligence layer. A well-aligned model can still be manipulated through prompt injection or adversarial inputs. It can still hallucinate when the surrounding data is stale or incomplete. Model-level safety defines how the agent behaves within a given context. Process-level safety defines the operational envelope: what the agent is allowed to do, which decisions require human approval, and what the fallback is when the agent is wrong. Both levels are necessary. Neither is sufficient alone. When the Trinity Splits: Three Agentic AI Failure Scenarios Three short failure scenarios make this concrete. Process intelligence without event-driven integration. A workflow engine automates a credit decision. The data feeding it comes from a nightly batch export. The process runs correctly. The decision is based on a customer's financial state from 18 hours ago. The automation worked. The outcome was wrong. Event-driven integration without process intelligence. Transaction data flows in real time across systems. An agentic AI system detects an anomaly and flags a potential fraud case. But there is no process intelligence layer defining what happens next. There is no approval gate, no escalation path, no audit trail. The agent acts, or it does not, and nobody can explain which or why. Trusted agentic AI without the other two. The agent is aligned, tested, and governed at the model level. But it receives context from a batch pipeline, so its reasoning is grounded in outdated information. And no process intelligence layer enforces boundaries on what it can do next. The agent behaves well in the lab. It causes problems in production. The Trinity in Action: Process Intelligence and Agentic AI Across Three Industries The following three scenarios show this architectural model working across industries. Each one is different. The pattern is the same: an event triggers a process, an agentic AI system acts within it, and process intelligence defines the boundary between automation and human control. Financial services. A transaction event triggers an agentic AI fraud risk assessment in real time. The risk score flows into a case management workflow. Below a defined threshold, the process is automated. Above it, the process intelligence layer routes the case to a human analyst before any account action is taken. The guardrail is not inside the model. It is inside the process. Healthcare. A patient monitoring system emits a deterioration signal. The event reaches a care pathway engine, which initiates the appropriate clinical workflow. An agentic AI system recommends an intervention. The process intelligence layer requires clinician confirmation before that recommendation becomes an order. The agent informs. The human decides. The process enforces that boundary every time. Supply chain. A supplier sends a disruption signal. The event reaches the process engine before the procurement team opens their inbox. An agentic AI system analyzes inventory, evaluates alternative suppliers, and proposes rerouting options. The process intelligence layer defines which decisions the agent can execute autonomously and which require sign-off. Speed comes from the event-driven layer. Governance comes from process intelligence. Trust comes from both working together. Build the Trinity, Not the Parts This Trinity is not a new product category. It is a way of thinking about a converged architecture that most enterprises have not yet adopted. Event-driven integration ensures that every process and every agentic AI system works on current reality. Process intelligence ensures that automation stays within governed, auditable boundaries. Trusted agentic AI ensures that agents behave reliably within the context they are given, and that the process intelligence layer catches what the agent cannot. The following architecture maps the complete picture across all three layers: Organizations that invest in all three separately will keep getting the results they are getting today. Organizations that design them to converge will build something qualitatively different: infrastructure that moves fast, governs well, and earns the trust of the business. The technology exists. The architectural commitment is what is missing.

By Kai Wähner CORE

Just a few months back, I observed a test suite with a self-healing feature “fixing” a failed selector three different times during the same sprint cycle. In each instance, the fix performed its function perfectly well; however, it didn’t address the real problem of a shipped UI regression, as its sole focus was on keeping the test green. No one on the team became aware of the situation until one of the customers discovered it. That was the point when my perception of AI in Quality Assurance changed dramatically more than any keynote or LinkedIn post. Today, if you are a QA Engineer, chances are high that you have come across similar headlines as well: manual testing is dying, autonomous agents create and fix test scripts in seconds, and your career is going to be at risk soon. The fear is justified, and I do not find it irrational as a Senior QA Manager who sees new testing tools appearing in our workflows every quarter, or even faster than we can develop any governing principles. What I see happening is quite different: not only is the position not going away, but it is becoming increasingly difficult to fake. AI excels at internalizing all the mechanical, low-context tasks that previously made up the bulk of a QA Engineer’s workload, leaving only those tasks that have never been mechanical or low-context and involve judgment, risk assessment, and determining what quality looks like for a particular product. This significantly narrows the number of people qualified for the position. 1. Stop Writing Tests. Start Auditing Them. For decades, a significant proportion of QA time was spent on the technicalities: automation scripts, manual click-through of UI workflows, and broken selectors caused by someone renaming a div tag. AI is really good at these types of jobs, and pretending otherwise is just a waste of time. You should approach AI-generated tests as you would a junior engineer’s pull request: they are useful and efficient, but require your review before implementation. Learn to feed it real context, not vibes. The difference between a useless AI-generated test and a genuinely good one almost always comes down to whether you gave it the actual acceptance criteria, edge cases, and business rules, or just a vague prompt. This is a real skill, and most QA teams haven’t invested in it yet.Get comfortable with self-healing tools, and stay suspicious of them. Self-healing automation is very valuable for handling cosmetic churn in your user interface. In addition, as shown by my story above about selectors, it can silently hide the very bugs it's supposed to detect.Your value moves from writing to verifying. That’s no downgrade. The ability to check that 100 automatically created tests are useful, as opposed to just being syntactically correct, is more difficult than having written 50 by hand. 2. Learn to Test the Thing That Doesn’t Give the Same Answer Twice Every product I am working on is trying to add AI, and none of the QA processes I have seen have been designed with the requirements of such a task in mind. In conventional software, the deterministic factor is the key component; in other words, whatever the input, the output will always be the same. But with the use of AI, there is nothing like that, since the same query asked twice yields two different answers. This opens up an actual underserved field of skills, known as AI Trust, Risk, and Security Management, and, to be honest, what you call it is less important than the brawn behind it. Areas where one could actually develop some skills: Bias and fairness testing. Learning to actually evaluate whether a model’s outputs skew unfairly across groups, not just whether the demo looks fine.Hallucination detection. Building repeatable ways to check whether an LLM’s output is grounded in real data or confidently making things up. This is genuinely hard and genuinely valuable; most teams are doing it on an ad hoc basis right now.Adversarial and prompt-injection testing. Deliberately trying to break an AI system’s guardrails before someone outside your company does it for you. I’ll be honest about the caveat here: this field is young enough that best practices are still being written in real time, including by people learning on the job. Nobody has fifteen years of AI-TRiSM experience, because it didn’t exist fifteen years ago. That’s exactly why it’s a good place to plant a flag now rather than waiting for it to mature. 3. Protect the Part of Quality AI Genuinely Can’t Do AI is just a statistical machine. It doesn’t have any firsthand knowledge of being frustrated with a difficult checkout process, any cultural knowledge to know why something that works well in one place doesn’t feel right somewhere else, and no sense of that hard-to-pin-down friction that you can’t specify. AI is responsible for functional testing; however, when it comes to the people-oriented aspect of quality, that’s where I would focus my efforts: Exploratory testing following a hunch. The best bugs I’ve ever discovered came from getting a slight feeling that something was amiss and investigating it, rather than through a written test case. The hunch does not stem from any particular model.Accessibility and usability should be top priorities rather than something ticked off a box prior to deployment. Is the product actually good to use? The algorithm will tell you whether the button meets the contrast ratio requirement. It cannot tell you whether the user flow around the button is confusing.Being there when risk is being discussed. AI will tell you whether the feature complies with the specification. AI has no way of knowing if the specification itself is incorrect for your market, your users, and the regulations. This discussion has to involve a human who knows the business, not the ticket. 4. Let Production Data Tell You Where to Look AI is based on data, and therefore, your testing approach should be too. The QA engineers who actually derive useful insights from AI do not test everything equally; instead, they let the data drive them. Analyze the real usage pattern of your application and prioritize automation accordingly; focus on testing those paths that users actually use, not those that were expected according to the initial requirements specification. Close the loop with your DevOps team about what is really breaking in production. If you find out that there are constantly recurring errors of one type or another, this information is directly relevant to the testing priorities of your AI solution, not something that you talk about separately.Understand what the data pipeline looks like, at least at a high level. A significant amount of “AI testing” in the future will involve testing the data pipeline that feeds into the AI algorithm, not just the outputs. A 90-Day Plan, If You Want One If you’d rather have a concrete starting point than a philosophy, here’s roughly how I’d sequence it: A 90 Day qa plan Days 1–30 Learn the tools Bring an AI coding assistant into your actual daily automation work, not a sandbox exercise, and pay attention to where it’s confidently wrong. Days 31–60 Expand the domain Take a real course on ML fundamentals or AI testing methodology, not just a vendor’s product training. Days 61–90+ Make it visible Propose one concrete AI-driven improvement on your current team, whether that’s AI-assisted test data generation or a pilot of self-healing UI tests with a defined review process attached. Regarding tooling: this landscape moves fast enough that my suggestions for product names will be obsolete by the end of the first year, but remember that it is the categories which are meaningful, not individual product names. Natural language test generation, self-healing test execution, visual testing, and AI security testing are just a few of the currently meaningful categories. Tools in any of those categories worth considering are those that let you see and adjust the AI's decisions. The Bottom Line Anxiety about AI in testing often arises from conflating two distinct concepts. Testing is a technical task, while QA is a mindset focused on protecting the user experience. AI excels at technical tasks but cannot replicate the QA mindset. In practice, automation is removing repetitive tasks, leaving the core responsibility of defining quality for each product and user group, and identifying issues beyond a model’s reach. This results in a more meaningful, though more demanding, role.

Enterprise AI is moving beyond isolated prompt-response calls and toward systems that observe events, preserve state, invoke tools, and publish decisions back into operational workflows. In that setting, event streaming is not simply middleware. It becomes the record of how intelligent behavior unfolds over time. Kafka is designed to read, write, store, and process streams of events across distributed systems, while Kafka Streams adds joins, aggregations, windowing, event-time processing, and exactly once support for stateful stream applications. At the same time, modern agent runtimes have shifted toward durable execution, persistence, and human-governed control flows rather than single-turn prompting alone. That convergence makes Kafka a strong coordination layer for autonomous agents that need to react continuously instead of responding once and disappearing. That architectural change also alters the role of the model. In an API-centric design, the model is often treated as a synchronous dependency behind a request. In an event-driven design, the model becomes one participant in a larger decision pipeline. Observations arrive as events, context is assembled from topics and state stores, agent steps are logged, and decisions are emitted as new events for downstream systems. Because Kafka topics can be replayed and reprocessed, the same stream can feed planners, validators, enrichment services, audit consumers, and human-review workflows without creating hard coupling between those components. The resulting system is easier to inspect, easier to recover, and easier to evolve than a chain of tightly bound remote calls. Turning Kafka Into the Coordination Layer The most important benefit is not only scale. It is the replacement of brittle request chains with an append-only coordination layer. A payment event, support ticket update, equipment alarm, or fraud signal can be published once and then consumed independently by retrieval components, compliance checks, planners, and execution agents. Kafka consumer groups divide partitions across consumers in the same group, and each partition is consumed by a single consumer within that group, which preserves ordering at the partition level while still allowing horizontal scale. For agentic systems, that detail is central. If all events for the same case, customer, or device are keyed consistently, one partition becomes the serialized timeline for that entity, and the agent no longer has to reconstruct order from racing HTTP callbacks. The event log also becomes a durable memory boundary. Kafka log compaction retains the latest value for each key, which makes compacted topics useful for task state, policy snapshots, approval status, or tool metadata that must survive restarts and recover quickly. On the runtime side, agent frameworks persist checkpoints and thread-scoped state so interrupted flows can resume from a saved step instead of starting over. Used together, those layers create a pragmatic split of responsibilities, such as Kafka preserves externally visible state transitions, and the agent runtime preserves internal execution context between steps, pauses, and failures. That is exactly the kind of separation needed when autonomous behavior must remain observable without being reduced to stateless prompt calls. Designing Agent Loops Around Events Once Kafka becomes the backbone, the agent loop changes shape. The entry point is no longer a prompt alone. It becomes a domain event that is enriched, correlated, and converted into a bounded task. Research on ReAct showed the value of interleaving reasoning and acting, and current agent frameworks translate that idea into practical workflows with durable execution, interrupts, and resumable state. The production version of an autonomous agent is therefore less like a chat session and more like a state machine that reasons, uses tools, emits intermediate facts, and pauses when a policy boundary requires approval. A concise stream processor can prepare that task before the model loop begins: Java builder.stream("order-events", Consumed.with(Serdes.String(), orderSerde)) .selectKey((key, event) -> event.customerId()) .join(customerTable, this::mergeContext) .mapValues(this::toAgentTask) .to("agent-tasks"); This pattern keeps context assembly close to the log instead of scattering it across synchronous service calls. Records are keyed by stable business identity, joined with the latest customer state, and emitted as small agent-tasks messages that the runtime can consume directly. Kafka Streams is explicitly intended for stateful processing with joins, event-time semantics, and exactly-once guarantees, so the enrichment stage remains deterministic, replayable, and independent from the model-serving layer. The execution boundary can remain equally narrow: Java @KafkaListener(topics = "agent-tasks", groupId = "claims-agent") @Transactional public void handle(AgentTask task) { AgentDecision decision = agentRuntime.run(task); kafkaTemplate.send("agent-decisions", task.taskId(), decision); } A compact runtime method can express the control flow without hiding it: Java public AgentDecision run(AgentTask task) { AgentState state = stateStore.load(task.taskId()); PlanStep step = planner.next(state, task); if (step.requiresApproval()) return AgentDecision.pause(task.taskId(), "manual-review"); ToolResult result = toolExecutor.execute(step.tool(), step.arguments()); return planner.complete(task, state, result); } This arrangement matters because the runtime receives a prepared task and emits an explicit decision event instead of mutating external systems invisibly. When transactions are enabled, Spring for Apache Kafka supports exactly-once semantics for the read-process-write sequence, and Kafka itself uses idempotent producers plus transactions so retries do not create duplicate log entries. External side effects still need idempotent design when they happen outside Kafka, but the event pipeline itself becomes much more predictable and auditable. Reliability and Control in Production Reliability in event-driven AI systems is usually lost at the edges rather than inside the model call. Kafka’s exactly-once features matter because an autonomous agent often emits decisions that trigger downstream actions, compensations, or audits. Kafka Streams supports exactly-once v2, and exactly-once flows configure consumers with read_committed isolation so aborted transactions do not leak into downstream processing. The event contract matters just as much as the delivery contract. Schema Registry centralizes schemas, validates them, and enforces compatibility modes so producers and consumers can evolve independently. In practice, a stable AgentDecision schema with explicit action type, confidence, explanation reference, and approval status is usually more valuable than a loosely structured JSON envelope because it can be consumed safely by analytics jobs, rule engines, operational systems, and auditors maintained by different teams. Operational control also has to assume malformed input, tool failure, and policy limits. Kafka Connect supports dead letter queues for records that cannot be processed successfully, and Spring Kafka supports dead-letter handling for repeated listener failures. Kafka also supports SASL-based authentication and ACL-driven authorization, which matters when planners, tool executors, and audit services must have different permissions over topics and consumer groups. Combined with interrupt-driven approval workflows from modern agent runtimes, those controls allow autonomous agents to operate inside explicit safety and governance boundaries instead of as opaque background processes. Where This Architecture Fits Best This architecture is strongest when work is asynchronous, stateful, and externally observable. Fraud triage, claims handling, supply chain exception management, field-service coordination, and security operations are better fits than chat-only assistance because the hard problem is not generating a sentence. The hard problem is reacting to a changing stream of facts, correlating them by entity and time, and making bounded decisions with replayable outcomes. Event-driven AI systems with Kafka and autonomous agents are compelling because they treat intelligence as part of an operational stream rather than as an isolated endpoint. The most effective implementations keep the log authoritative, keep schemas explicit, keep agent state durable, and keep irreversible actions observable and governable. That combination produces systems that are not only responsive, but also replayable, auditable, and resilient enough for enterprise use, which is ultimately the threshold that separates a convincing demo from a production architecture.

By Uthej Mopathi CORE

Most enterprise AI post-mortems do not blame the model. They blame the storage tier that starved the accelerators, the identity policy that over-granted access, the cost model that ignored egress, the forecast that leaked future data, or the region that failed and took a business process with it. The hard part of production AI was never intelligence. It was the engineering discipline around it. This article distills the architectural patterns that decide whether a cloud AI system is trustworthy at scale, spanning infrastructure, identity, cost, operations, the applied domains, low-code assembly, platform selection, and multi-cloud resilience. It is written for engineers who have to keep these systems running, not for a keynote. Infrastructure: The Interconnect Is the Bottleneck Distributed training is a systems problem before it is a machine learning problem. When a job spans many graphics processing units (GPUs), the fabric connecting them (e.g., NVLink within a node, InfiniBand, or a vendor fabric across nodes) frequently caps throughput more than raw compute does. Accelerators wired through an ordinary network idle while they wait to synchronize gradients. Storage is the symmetric constraint. If the file system cannot deliver data at the rate the accelerators consume it, utilization collapses. The pattern is a tiered design: Hot tier: parallel or block storage feeding active training at high input/output operations per second (IOPS).Warm tier: recent data staged for quick promotion.Durable lake: object storage providing petabyte-scale durability, partitioned and lifecycle-managed underneath. Two cost drivers hide from the pricing page: data egress (moving data across regions or out of a provider) and idle warm capacity. Optimizing only the advertised compute line item guarantees a surprise on the invoice. Identity Is the Perimeter In a service-to-service AI architecture, the network perimeter is gone; identity is the boundary. A zero-trust posture, where every request authenticates and receives least privilege, contains the blast radius when a component is compromised. Across providers, identity federation is the load-bearing pattern: a principal authenticates once and is recognized everywhere, so access is granted and revoked centrally instead of reconciled across three identity systems. Policy must travel with the workload; a rule enforced on one cloud and forgotten on another is not a policy. Model authorization is the emerging frontier. As models call tools and take actions, the question moves from who can query this model to what may this model do on a user's behalf. Least privilege applied to an autonomous agent is the boundary between useful and unbounded. Cost and Operations Are a Control Loop Cost management is not a spreadsheet; it is automation. Consistent resource tagging across every cloud is the prerequisite for attribution. On top sit budgets, alerts, and automated remediation that throttles runaway spend before it escalates. Site reliability engineering (SRE) supplies measurable targets. For AI workloads, the golden signals extend beyond latency and errors to accelerator utilization, queue depth, and prediction quality. A model can be fully available and quietly wrong, so define a service level objective (SLO) for output quality, not just uptime. Three techniques earn their complexity: Spot or preemptible capacity plus checkpointing cuts training cost sharply when jobs resume cleanly after reclamation.Predictive scaling anticipates load instead of reacting to it.LLM inference optimization becomes architectural: batch requests, cache frequent responses, route easy queries to smaller models, reserve the expensive model for queries that need it. The Applied Domains Share a Spine, Differ in Physics Vision is byte-heavy. High-resolution images and video streams make the data and network layers dominant. For real-time video, decouple frame capture from analysis and sample frames rather than processing every one. Critically, a business-rule layer, never the model alone, owns consequential decisions. Every extraction should carry a confidence score used as a routing gate: Python def route_extraction(field, threshold=0.90): if field["confidence"] >= threshold: return "auto_process" return "human_review" Language is byte-light but semantically treacherous, and because it replies directly to users, errors are visible. The defining risk of generative systems is hallucination. The strongest architectural defense is retrieval grounding, forcing answers from verified sources with citations: Python def answer(question, knowledge_base): passages = knowledge_base.search(question, top_k=3) context = "\n".join(p.text for p in passages) prompt = f"Answer using ONLY this context.\n{context}\n\nQ: {question}" return model.generate(prompt), [p.source for p in passages] Forecasting is defined by time order. You cannot shuffle a time series into random splits, and the most common failure is data leakage, using information unavailable at prediction time. Test on a fair, time-ordered holdout, and always emit a prediction interval; a point forecast that hides its uncertainty invites overconfident decisions. No-Code and Low-Code: Governed or Ungoverned No-code and low-code platforms collapse build cost from a scoped project to an afternoon, which is why adoption is exploding. The symmetric risk is sprawl: hundreds of ungoverned flows handling sensitive data, owned by no one. Govern with guardrails, not gates. Restrict which connectors and data sources are permitted, assign an owner and an SLO to every production flow, then let builders move freely inside the boundary. The goal is to make the safe path the easy path. Platform Selection Without Self-Deception Vendors all claim to be fastest, cheapest, and most reliable. Benchmark to replace claims with evidence: Latency: report percentiles (p95, p99), never averages that hide the slow tail.Quality: measure on your own representative data, not a public leaderboard.Cost: model total cost of ownership, including transfer, storage, idle capacity, operations, and migration, not the headline compute rate.Reliability: verify the platform meets your recovery time objective (RTO) and recovery point objective (RPO). Combine dimensions in a weighted scorecard whose weights are fixed before scores are seen. Adjusting weights afterward to crown a favorite converts analysis into rationalization. Multi-Cloud Resilience: Design for the Day a Cloud Fails For systems a business cannot lose, a single provider is a gamble. Multi-cloud resilience deliberately places critical workloads so no single provider failure takes the business down, applied only where the cost of failure exceeds the cost of prevention. Predict rather than react. Combine leading signals into a health score and fail over proactively: Python def health_score(latency_ms, error_rate, saturation): latency_factor = max(0, 1 - (latency_ms / 1000)) error_factor = max(0, 1 - (error_rate / 0.05)) saturation_factor = max(0, 1 - saturation) return round(0.4*latency_factor + 0.4*error_factor + 0.2*saturation_factor, 3) Kubernetes makes workloads portable; data replication (with the consistency-versus-availability trade-off decided per workload) keeps data ready on the other side; and a portable foundation of federated identity, uniform policy, and centralized monitoring makes failover routine rather than heroic. The discipline that separates real resilience from a slide deck is rehearsing failure on purpose. An untested failover path is a promise, not a capability. The Judgment Layer Across every layer, value came not from the most powerful component but from the judgment applied to it: matching effort to problem difficulty, keeping humans on consequential decisions, measuring before deciding, building governance in early, and designing for change. Tools will churn; foundation models will make today's designs look quaint. That is precisely why principles outlast product knowledge. The scarce resource in enterprise AI was never intelligence. It was judgment, and judgment does not ship from the cloud.

TL;DR: A Déjà-Vu? AI adoption seems to be scaling: 37% of respondents in McKinsey’s 2026 survey report an EBIT effect from AI, and Gartner finds that 22% of organizations have scaled it across business units. Now, Agile practitioners have seen this combination before, as AI transformations and Agile transformations rhyme. Five classic failure patterns from Agile transformation adventures are back under new names: mandates from above, licenses mistaken for training, greenfield showcases, parachuted consultants, and promised payroll savings dressed up as strategy. They share one condition: organizations make AI decisions at organizational scale without leaving inspectable evidence at the workflow level in the trenches. And for good measure, let us throw in ignoring culture and excluding most of the organization’s people in the process. History Does Not Repeat Itself, but AI Transformations and Agile Transformations Do Rhyme AI transformations in large organizations are scaling, individual productivity is up, leaders still plan to increase spending, and yet enterprise financial impact remains limited: McKinsey's 2026 State of AI survey (1,719 respondents, fieldwork May 4 to June 8, 2026) puts numbers on three of the four: 44 percent of respondents say AI is scaling across their enterprise, up from 38 percent a year earlier; 80 percent of those who use AI report improved individual productivity; and 37 percent attribute any EBIT impact to AI at all, with the "AI high performer" group flat at about 6 percent.Gartner's September 2026 survey of 1,303 respondents from organizations with at least $50 million in annual revenue supplies the spending picture: 85 percent of functional leaders plan to increase AI spending in 2026, 22 percent of organizations have scaled AI across multiple business units or adopted an AI-first approach, and 11 percent do not know what their function spent on AI in 2025. Something is happening, and something is also not translating. Agile practitioners have seen that combination before. "History does not repeat itself, but it rhymes," a line widely attributed to Mark Twain despite no evidence that he said it; the attribution to Twain dates back to 1970. The attribution is shaky; nevertheless, the observation holds. I wrote the Scrum Anti-Patterns Guide about what organizations do to Agile when they adopt it from the top down. The same organizations are now doing the same things to AI, with a new generation of leaders who consider the Agile years ancient history, and five rhymes stand out. The Five Rhymes of AI Transformations Rhyme 1: The Mandate From Above IBM's June 2026 study of 2,000 C-level technology executives found that 80% reported CEO-driven AI transformation mandates, and 77% said adoption is already outpacing their governance capabilities. The most public example is Shopify. In a late-March 2025 memo that he later posted on X, CEO Tobi Lütke told the company that "reflexive AI usage is now a baseline expectation at Shopify" and that, before asking for more headcount, teams "must demonstrate why they cannot get what they want done using AI," as Tom's Hardware and TechCrunch reported. Whether that works at Shopify, I cannot judge from the outside. What I can judge is the predictable risk when that kind of memo lands in an organization where governance is already falling behind: visible compliance and invisible workarounds. Agile practitioners remember the memo announcing "we are now an agile organization" and the Sprint Reviews that followed, which were ignored by everyone who could change a decision. Rhyme 2: The Belief That This Time Training Is Optional The Agile version bought a two-day certification class and called it a transformation. Often, AI transformations skip even that: buy Copilot or ChatGPT Enterprise licenses, send an email, done. The tool is "intuitive," so the reasoning goes; it is sold as the classic example of learning by applying. Lütke's own memo contradicts this, noting that "using AI well is a skill that needs to be carefully learned." The McKinsey gap between 80% reporting individual productivity gains and 37% reporting any EBIT impact shows why individual productivity is a poor proxy for organizational change. Individuals may get more productive, whatever that means in this context, which does not imply that the organization has changed at the same time. The same survey shows where the difference lies: nearly three-quarters of high performers report fundamentally redesigning workflows because of AI, against one-quarter of everyone else. Deloitte's June 2026 pulse check of nearly 3,700 professionals found that 48% were adding AI without redesigning workflows or roles, and only 12% were redesigning workflows or roles at scale. The divide runs between organizations that change the nature of work and those that bolt AI onto whatever structure they have. Rhyme 3: The Greenfield Showcase Every transformation needs a success story for the board, so a team with no dependencies on the legacy systems, no regulatory exposure, and no operational duty builds something impressive. The Agile version was the "pilot team" in the innovation lab with the fancy toys. The AI version is the internal chatbot that answers HR policy questions and was presented at the town hall as evidence of AI's great potential. Exploration detached from production constraints is useful. The anti-pattern is mistaking evidence that something can be built for evidence that the organization has created value. BCG's 2025 survey of 1,250 senior executives found that 70% of AI's potential value sits in core business functions such as sales and marketing, manufacturing, supply chain, and pricing, which is where the showcase usually never goes, due to the "unsexiness" of the use cases. Rhyme 4: The Consultancy That Sets It Up for You In come the slide decks, the "AI transformation office," and the currently fashionable forward-deployed engineers. The role name dates back to Palantir in the early 2010s; the practice is far older. Thomas Otter, who spent years at SAP, notes that "early chunks of SAP R/1 were built at ICI and John Deere," decades before anyone called the practice forward deployment. I do not consider the practice an anti-pattern. Engineers who join the organization, learn its culture, and stay long enough to hand over applications built on understanding are legitimate. The anti-pattern is the parachute version: the engineers arrive, do the tactical technical work, and leave, and the organization is now running systems it cannot explain. Agile had the consultancy-staffed transformation office that left when the budget line ended. Rhyme 5: The Cost Story Ask most leadership teams why the organization adopts AI, and you get a story about new business, better products, and faster learning. Ask what the business case they signed off actually contains, and you find payroll. Consultancies, in my observation, sell AI as they sold offshoring: a way to remove people who do repetitive work. Cost reduction, as such, is not the anti-pattern; however, turning it into the transformation objective is. About 80% of McKinsey's high performers, and everyone else, pursue efficiency, but most high performers also pursue growth or innovation, thereby distinguishing the two approaches. Klarna ran the other experiment in public. After claiming its AI assistant did the work of 700 customer service agents, CEO Sebastian Siemiatkowski told Bloomberg in May 2025, as CX Dive reported, that "cost unfortunately seems to have been a too predominant evaluation factor when organizing this; what you end up having is lower quality," and started hiring humans again. McKinsey's respondents have noticed which story their leadership actually believes: 39% now expect AI-related job cuts, up from 32% a year earlier. Agile had the same split. The board heard "faster and cheaper"; the teams heard "better products"; and when the two stories collided, the teams lost. What the Five Rhymes of AI Transformations Share Each AI transformation rhyme has a visibility problem. Leadership can see the headcount numbers perfectly well and still optimize them; a consultancy dependency is a capability-transfer problem, while a mandate is an authority and incentive problem. What the five have in common sits one level down. In each case, the organization makes its AI decisions at organizational scale (a mandate, a license contract, a showcase budget, a vendor engagement, a business case) without leaving inspectable evidence at the workflow scale. Too often, nobody can show, for a specific workflow, who decided that AI would do this work, on what terms, under what cost constraints, who checked it, and with what result. Visibility is the symptom, and missing evidence is the condition. Scrum already had low-tech answers to similar problems: an ordered Product Backlog, an explicit Definition of Done, and a recurring Retrospective. None of them needed a platform, and none of them made leadership act. What they did was let a team generate evidence about the system it worked inside. The A3 Delegation System borrows that design principle: make consequential decisions visible before buying another layer of tooling to manage them. Six stages (Decide, Route, Hand Over, Define Done, Inspect, Roll Up), seven artifacts, and no software beyond the AI the team already uses. It is an operating discipline for AI delegation, one workflow at a time, and the evidence is a byproduct of doing the work. Where Each Rhyme Meets a Countermeasure Let us come back to the five "rhymes" and how the A3 Delegation System can mitigate these AI transformation issues: The AI Workflow Inventory makes the license fallacy and the showcase inspectable: Before anything else, the team lists the workflows it already hands to AI, each with an owner. It takes an hour, and the assumption that "people will figure it out" collapses once the list shows what they figured out. You may discover personal AI habits that were never treated as organizational workflows at all, including some touching sensitive data. The inventory also refuses the greenfield showcase by construction. Only existing workflows with a named owner enter it. The A3 Framework decision and the Routing Policy put a countermeasure against the mandate: For each inventory entry, the team decides Assist (AI drafts, you decide), Automate (delegate execution, not responsibility), or Avoid (the cost of failure is trust). Then it routes the work to a model tier by stakes and cost. Leadership can set the boundaries: approved tools, prohibited data, risk limits, or spending constraints. It cannot make the workflow-specific delegation decision from a company-wide memo; the people who know the work can do so in minutes per entry, and the decision is then on paper for leadership to read. Routing is also where the token bill becomes a decision, and precision matters here: while the price per token keeps falling, the cost of operating AI keeps rising, because cheaper tokens invite longer, more autonomous workflows that consume far more of them. Gartner predicted in August 2026 that inference costs per agentic workflow will rise more than fivefold through 2028; its analyst, Will Sommer, said, "Product leaders cannot rely on more efficient token economics to rationalize AI costs." That is the economic problem the Routing Policy addresses at the workflow level: expensive intelligence is a deliberate choice, never a default. The A3 Handoff Canvas and the AI Definition of Done make the parachute inspectable: Six fields (task split, inputs, outputs, validation, failure response, records) and a one-page quality standard per task class. Here is the ownership test for anything a consultancy or a forward-deployed engineer built: can the team fill in these two documents for the system without calling the vendor? If yes, the team owns the delegation, whoever set it up. If no, the organization is renting understanding, and the rent comes due when the engineers leave. The Delegation Audit asks one question, and it is not the cost question: Monthly or every other Sprint, 45 to 60 minutes, four checks: output and source drift, model fit, reversibility, and category creep (Assist work that quietly became unreviewed Automate). Each finding gets an owner and a decision: change the A3 category, change the tier, update the AI Definition of Done, fix the stop rule, or retire the delegation. The Audit asks whether this delegation is still sound. It does not ask what the freed capacity produced. Roll Up, the last stage, compiles what the audits show for those who ask. Whether what they show is worth paying for is a leadership decision, and it sits outside the A3 Delegation System. The system produces evidence for the value conversation, but it does not own the value decision. The AI Working Agreement wraps the other six: It records the team's rules on data, disclosure, responsibility, and review, and it is the document the team hands upward when leadership asks what "AI adoption" looks like here. It is a page that beats a slide on every occasion. Where the A3 Delegation System Stops A skeptical reader, and my readers have watched frameworks overclaim for twenty years, will ask the obvious question: am I criticizing consultancies for selling transformation frameworks and then selling my own? A3 is not an AI-transformation methodology. It is an evidence-generating delegation discipline. It cannot make leadership respond to the evidence. It can make ignoring the evidence harder. It does not determine why your organization adopts AI, nor does it replace a strategy, a portfolio decision, or the conversation about what happens to the people whose repetitive work disappears. What it does is make the absence of those decisions visible within weeks, team by team, in writing, for the price of a few hours. There is a second limit: A3 can tell you whether AI should do a piece of work, which model, what it needs, what acceptable output means, whether the delegation has drifted, who owns it, and what it is allowed to cost. It does not tell you whether the workflow should exist. Suppose the system reduces a weekly reporting workflow from 4 hours to 40 minutes, with excellent output and impeccable governance. The question that remains is why the organization produces that report at all. The A3 Delegation system can prevent undisciplined delegation. It cannot prevent an organization from competently automating work that should have disappeared. The likely next development step of the A3 system is a single field on the AI Workflow Inventory, not another canvas: what changes if this workflow works? I have not added it yet. The team should expect the visibility A3 produces to be unwelcome. A team that runs the inventory, the decisions, and the Audit inside a mandate-driven transformation produces evidence the organization may refuse to absorb. I have watched organizations refuse the evidence their Retrospectives produced for years, and the refusal told the teams more about the transformation than any all-hands did. If your leadership will not read a one-page working agreement and a monthly audit log, you have learned what the AI transformation is for. Conclusion AI transformations may repeat many of the mistakes of Agile transformations. The A3 Delegation System does not prevent organizations from making them. However, it gives teams a simple way to make some of them visible before they become expensive: Count how many of the five rhymes are playing in your organization right now. Respect yourself and be honest while aggregating those. Then put a document against one of them.

By Stefan Wolpers CORE

Over the last two years, the industry has moved from simple chatbots and retrieval-augmented generation pipelines to something fundamentally more powerful: agentic systems. These systems do not just answer questions; they plan, act, observe the consequences of their actions, and keep iterating until a goal is achieved. Whether it is Cursor writing and debugging code, Perplexity performing multi-step research, Manus executing complex tasks through code, or Gemini Deep Research producing long-form investigative reports, the underlying architecture is agentic. At the heart of every agentic system lie 2 critical design decisions. The first is the overall topology: should the system be a single agent that owns the entire problem, or a multi-agent system where specialized agents collaborate under an orchestrator? The second is the choice of internal design patterns that govern how the agent reasons, selects tools, handles errors, and improves its own output. Get these decisions right, and the system becomes reliable, scalable, and genuinely useful. Get them wrong, and you end up with brittle loops, runaway costs, or agents that hallucinate tool calls. This article provides a clear, production-oriented map of both layers. Let us examine single-agent versus multi-agent architectures, then walk through the six design patterns that dominate real-world systems today. Each pattern is illustrated with a diagram and explained with concrete examples drawn from products already in production. 1. Agentic Systems Topology Every agentic system falls into one of two broad categories. Single Agent System In a single-agent architecture, one agent owns the complete loop. It receives the user query, maintains both short-term context and long-term memory, decides which tools or MCP servers to call, observes the results, and eventually produces the final output. This design is simple to implement, easy to debug, and has lower latency for focused tasks. It is the natural starting point for most teams. The main limitations appear when the problem becomes long-horizon or requires genuinely different skills. Context windows fill up, specialized knowledge is hard to isolate, and a single failure mode can bring the entire system down. Multi-Agent System A multi-agent system introduces a meta-agent (or orchestrator) that decomposes the high-level goal and delegates work to specialized agents, for example, a data-retrieval agent, a search agent, a coding agent, or a critic. Each specialist may have its own tools and memory. Results flow back to an aggregator LLM that synthesizes the final answer. Shared or private memory and MCP servers support the collaboration. The advantages are specialization, parallelism, and higher reliability through division of labor. The costs are coordination overhead, higher token consumption, and more complex failure modes (handoff errors, inconsistent state, cascading retries). Most production systems begin as single-agent and only move to multi-agent when the benefits clearly outweigh the complexity. 2. Core Design Patterns Topology decides how agents are organized. Design patterns decide how each agent thinks and acts. The six patterns below appear, often in combination, in virtually every serious agentic product shipping today. ReACT Agent (Reason + Act) The foundational pattern used by the majority of tool-using agents. The agent interleaves three explicit steps in a tight loop: Thought: verbalized reasoning about the current state, what is known, what is still missing, and which single action would close the biggest gap.Action: a concrete tool call with specific arguments.Observation: the tool result is injected back into the context so the next thought is grounded in reality. This loop continues until the agent decides the task is complete. Pure chain-of-thought reasoning is blind to the external world; pure tool calling is planless and reactive. ReAct fuses the two and remains the default architecture in LangGraph, LlamaIndex, the OpenAI Agents SDK, Claude’s tool-use agents, and most coding assistants. The main engineering concerns are infinite loops (always set a hard iteration limit) and context growth (every thought and observation consumes tokens). CodeAct Agent Used by Manus, OpenHands, and an increasing number of advanced coding and automation systems. Instead of emitting one discrete tool call per turn, the agent writes executable code (usually Python) as its primary action space. That code can contain control flow, multiple tool invocations, data transformations, filtering, and error handling — all in a single execution step. The sandbox runs the code and returns stdout, stderr, or artifacts. If something fails, the agent can observe the error and revise the code, achieving a form of self-debugging. CodeAct collapses what would have been many ReAct steps into one more expressive action. It also leverages the full power of existing software libraries. The trade-off is the need for a secure, well-instrumented execution environment. Self-Reflection Critical for reliability and high-stakes outputs. After generating a first draft (a plan, a piece of code, or an answer), the agent or a dedicated critic LLM evaluates the output against explicit criteria: correctness, completeness, safety, style, and grounding. If the critique fails, the agent revises using the feedback. The loop continues until the critique passes. Lessons can optionally be written into long-term memory, so the same mistake is less likely in the future. Self-reflection is the agentic analog of System-2 deliberative thinking. Andrew Ng has repeatedly listed it as one of the four fundamental building blocks of agentic systems (alongside planning, tool use, and multi-agent collaboration). It is especially valuable when the cost of a wrong answer is high. Tool Use and Agentic RAG Tool use is the substrate of nearly every modern agent. Cursor is a canonical example: the agent is given a rich set of tools (file system, terminal, search, browser, cloud APIs, MCP servers) and decides at each step which tools to call and with what arguments. The quality of the tool schemas and the system prompt that teaches the model when and how to use them is often as important as the underlying model itself. Agentic RAG elevates classic retrieval-augmented generation. In a traditional RAG pipeline, the retrieval step is fixed: embed the query, fetch top-k chunks, generate. In agentic RAG, retrieval itself becomes a tool the agent can invoke repeatedly. The agent can rewrite queries, decide which sources to consult (vector database, web search, structured APIs), evaluate intermediate results, and continue retrieving until it judges the context sufficient. Only then does it generate the final grounded answer, usually with citations. Perplexity’s more advanced research modes and modern enterprise agentic RAG frameworks follow this pattern. Multi-Agent Workflow A planner or meta-agent receives the high-level goal and decomposes it into sub-tasks. Specialized agents execute those sub-tasks, often in parallel, reading from and writing to shared memory and using tools as needed. An aggregator or synthesizer LLM collects the partial results and produces the final coherent output. Optional critique or reflection loops can be added on top. This pattern shines on long-horizon, multi-skill problems such as deep research, complex software engineering workflows, or multi-document analysis. Gemini Deep Research is a prominent production example. 3. Choosing and Combining Patterns PatternBest suited forLatency / CostReliabilityComplexityReActAdaptive multi-step tool useMediumHighLow–MedCodeActComplex logic & data pipelinesLower (fewer turns)HighMediumSelf-ReflectionHigh-stakes accuracyHigherVery HighMediumTool UseAny grounded actionLowBaselineLowAgentic RAGKnowledge-intensive questionsMedium–HighHighMediumMulti-AgentSpecialized + parallel long-horizon workHigherHighHigh In practice, these patterns are almost never used in isolation. Cursor combines heavy tool use with a ReAct-style loop and test-driven reflection. Manus leans on CodeAct. Deep research systems blend multi-agent orchestration, agentic RAG, and reflection. The patterns are composable building blocks rather than mutually exclusive choices. 4. Engineering Principles That Matter Most Regardless of which patterns you choose, several engineering practices determine whether the system is production-ready: Explicit termination conditions and hard iteration limits prevent runaway loops and cost explosions.High-quality tool schemas and descriptions are often more important than the choice of model. Clear parameter types, realistic examples, and guidance on when to use each tool dramatically improve reliability.Careful memory management decides what stays in the context window versus what is externalized to a vector store or key-value memory.Full observability: Every thought, action, and observation must be logged and inspectable. Without traces, debugging agentic systems is nearly impossible.Cost, safety, and permission guardrails must be enforced at the harness level, not left to the model.End-to-end evaluation that measures real task success (not just next-token accuracy) is essential for continuous improvement. Closing Thoughts Agentic systems represent a genuine architectural shift. We are moving from models that merely generate text to systems that can plan, act in the world, observe the consequences, and improve their own behavior. The difference between a fragile demo and a reliable product usually comes down to the quality of the topology and the design patterns chosen. We have discussed in this article, the following: Single-agent versus multi-agent structure, ReAct as the workhorse reasoning-and-acting loop, CodeAct for expressive actions, self-reflection for reliability, tool use as the universal substrate, agentic RAG for dynamic knowledge retrieval, and multi-agent workflows for complex collaboration give practitioners a practical decision framework. Start simple. Begin with a single agent using the ReAct pattern and solid tools. Add self-reflection when accuracy matters. Introduce CodeAct when the logic becomes complex. Move to multi-agent architectures only when specialization and parallelism clearly pay off. Instrument everything. Measure real task success. Iterate. The systems that follow these principles are already delivering meaningful value in coding assistants, research tools, enterprise automation, and personal AI agents. The patterns themselves will continue to evolve, but the underlying ideas — interleaving reasoning with action, grounding decisions in observation, and composing specialized capabilities — are likely to remain foundational for years to come.

By Ram Ghadiyaram CORE

Tell me about that one document that no one has read. Your company's governance policy for AI systems. Forty-something pages, buried in a wiki or Confluence somewhere. The document that legal and compliance teams spent months writing, reviewing it and referencing the National Institute of Standards and Technology Artificial Intelligence Risk Management Framework and the European Union Artificial Intelligence Act, and no doubt a handful of other standards you can't quite recall. Meanwhile, every single model your team has deployed in the last year hasn't consulted that document before deployment. As I've learned in my own experience building enterprise-grade AI solutions, the space between "we have a policy" and "the policy actually prevents something undesirable from happening" is where the headaches for compliance engineers and auditors begin and where the fines from regulators are born. Thus, your governance policy should not be a document. It should be code. Documents Cannot Block a Deployment I understand why companies document governance policies. You need those policies to exist for auditing and legal clarity. But a PDF cannot stop a CI/CD pipeline from deploying code any more than a procedure printed on paper can stop a nuclear reactor from melting down. ISO 42001, NIST’s AI Risk Management Framework, and Annex IV of the EU AI Act all define what actions are necessary, but not where in your stack those actions need to happen. Frequently, the result is that the compliance team drafts a policy, the engineering team gets a slide deck during an all-hands meeting, and everyone returns to their desks confident that they just need to remember to do things differently. Let me give you an example. Say a recommendation model was deployed into production after being trained on a dataset containing personally identifiable information (PII) that was supposed to be restricted to internal analytics. The dataset had been reclassified two months earlier, but no one had updated the pipeline configuration. It took eleven weeks after deployment for a data subject access request to uncover the issue. By then, six million users had received recommendations based on data that no one was permitted to access. It took three people-weeks of work to retrain and redeploy the model, audit all downstream consumers, and report the violation to the relevant regulatory agencies. Yes, this is a theoretical scenario, but it is hardly an unusual one. Readers working on frontier AI models can probably relate, because situations like this happen at large technology companies more often than you might think. That makes it essential to adopt the following techniques to avoid problems like this in the first place. Every Governance Rule Becomes a Gate If you’ve worked in infrastructure engineering, this is not a novel problem. Policy-as-code tools such as Open Policy Agent have existed for years, and today we have Kubernetes clusters in production that simply cannot admit pods that fail to meet specific security requirements. No one manually checks every time a pod is deployed to see whether it has the right security context, because the system itself rejects insecure pods. That is how policy-as-code should work in AI systems: as a zero-trust model. Every assertion in your governance policy should translate into a machine-enforced check applied to some part of the system, returning a non-zero exit code when the check fails. Your policy becomes a set of gates that the system must pass to continue operating and must fail fast when a requirement is not met. There are three ways this becomes useful, so let’s look at each in turn. Before Training Starts: Verify What Data the Model Can See One of the most common requirements in AI governance frameworks is to classify datasets by sensitivity and restrict model training to data with a classification level equal to or below what has been approved for the model. Here’s what that might look like: Python # policy/data_classification.py from governance_engine import PolicyCheck, DataSensitivityTier class DataClassificationGate(PolicyCheck): """ Blocks model training if the dataset's sensitivity level exceeds what this model is approved to access. Maps to: NIST AI RMF GOVERN 1.2, ISO 42001 Section 6.1 """ def evaluate(self, pipeline_context): dataset = pipeline_context.dataset model_approval = pipeline_context.model.data_access_tier # Get the actual classification, not the cached one current_sensitivity = dataset.get_live_classification() if current_sensitivity.tier > model_approval.max_tier: return self.fail( reason=( f"Dataset '{dataset.name}' is classified " f"{current_sensitivity.tier.name} but model " f"'{pipeline_context.model.name}' is only approved " f"for {model_approval.max_tier.name} data. " f"To request reclassification: internal-wiki/data-tier-request" ), remediation_link="internal-wiki/data-tier-request", escalation="data-governance-oncall" ) # Everything checks out, log it for the audit trail self.record_evidence( check="data_classification_pre_training", result="PASS", model_id=pipeline_context.model.id, dataset_id=dataset.id, classification=current_sensitivity.tier.name, timestamp=self.now() ) return self.passed() A few things are worth noting about this example. First, the classification is dynamic. We are not relying on whatever value was assigned to the dataset when it was ingested, because that could easily be out of date. Datasets change, columns are added, and sensitivity is periodically reevaluated. If a dataset’s classification is raised, any model trained on it should have its data access tier raised as well. Second, notice that the failed assertion includes a remediation link. We are giving the person who triggered the failure a direct pointer to what they should do next. This is critically important. One reason companies struggle so much with governance policy is that engineers understandably resist controls that slow down their work. Productivity will suffer if they are blocked without a clear path forward. If your policy is a wall they have to climb over, they will find ways to avoid climbing it. It is better to turn that wall into a door that opens only when the proper permission is presented. At Serving Time: The Gateway That Cannot Be Skipped If you're at all familiar with the announcement of Meta's Llama Guard model as part of the Purple Llama research series in late 2023, you might recognize this next section. To quote the announcement, "Unlike traditional approaches, Llama Guard does not utilize rule lists or keyword matching; instead, it leverages a separate model for enhanced security, trained to discern whether the input or output of Llama models violates any given safety policy." In other words, the Llama Guard model is policy-as-code: all safety policies are encoded directly in the system prompt of the model, allowing it to classify any given prompt or response as either safe or unsafe. This is a tremendously powerful technique, and while you might not be in a position to train dozens of guard models for every system you want to protect, you can most likely implement a similar pattern at the serving gateway layer for your models. YAML # policy/serving_gateway.yaml # This config lives WITH the model definition. # When the model deploys, its governance deploys with it. model_policies: - applies_to: "credit-risk-scoring/v2" enforcement_mode: BLOCK # vs AUDIT_ONLY for rollout pre_inference: - check: caller_authentication require_tier: 3 # Only Tier 3+ services can call this deny_message: "Credit models require Tier 3 service auth. See internal-wiki/model-auth" - check: input_data_restricted_fields blocked_fields: ["social_security_number", "date_of_birth", "ethnicity", "address"] action: STRIP_AND_LOG # Remove the field, log the attempt post_inference: - check: output_contains_pii scan_for: ["ssn_pattern", "phone_pattern", "email_pattern"] action: REDACT - check: fairness_monitor protected_attributes: ["gender", "race", "age_group"] max_disparity_ratio: 1.25 action: ALERT # Don't block, but fire an alert to the fairness oncall always: - check: audit_log_all fields: ["caller_id", "input_hash", "output_hash", "latency_ms", "timestamp"] retention_days: 2555 # 7 years, regulatory requirement This pattern encodes several important governance rules. First, notice that the policies are attached directly to the model itself. When the model is deployed, its policies are deployed with it. You cannot deploy the model without also deploying the policies that govern how it may be used. That means the policies are not stored separately in a governance database or knowledge management system. Instead, they live directly in the model’s configuration. This has several advantages, including making the policies much easier to discover. There is no way to deploy the model without reviewing its policies, which helps ensure those policies are considered and followed. After Deployment: Compliance Drifts Without Continuous Checks So you've deployed your model, and it passed all the policy checks during deployment. Great. Now, assume several months later the model starts serving requests that violate policy. The world changes over time; new regulations come into law; models' concepts drift; users' behaviors evolve. This is entirely realistic, and not hypothetical - the EU AI Act's implementing acts are currently still being drafted. You must perform both continuous checks and deployment-time checks in order to account for changes in the model after deployment: Python # policy/continuous_compliance.py from governance_engine import ContinuousMonitor, AlertSeverity monitor = ContinuousMonitor( schedule="*/5 * * * *", # Every five minutes alert_channel="ai-compliance-oncall" ) @monitor.regulation("eu_ai_act.article_14") def check_human_oversight(system_context): """ EU AI Act Article 14 requires that high-risk AI systems are designed to allow effective human oversight. We interpret this as: there must be a working override endpoint, it must respond fast enough to be meaningful, and a human must have actually reviewed outputs recently. """ if system_context.risk_classification != "HIGH": return # Only applies to high-risk systems # Can a human actually intervene? override = system_context.get_override_endpoint() if not override.is_healthy(): monitor.alert( AlertSeverity.P1, f"Human override endpoint DOWN for {system_context.model_id}. " f"High-risk system operating without oversight capability.", runbook="confluence/internal/ai-override-recovery" ) # Is the override fast enough to matter? if override.p99_latency_ms > 500: monitor.alert( AlertSeverity.P2, f"Override latency {override.p99_latency_ms}ms, too slow " f"for meaningful human intervention.", runbook="confluence/internal/ai-latency-optimization" ) # Has anyone actually looked at this thing recently? days_since_review = system_context.days_since_last_human_review() if days_since_review > 30: monitor.alert( AlertSeverity.P3, f"No human review of {system_context.model_id} in " f"{days_since_review} days. Scheduling mandatory review for the compliance team.", auto_action="schedule_review" ) The value of this approach is that it ties directly back to a specific regulation. If an auditor asks, “How do you know you’re complying with the EU AI Act?” you can show exactly which alerts the check has triggered and when it last ran successfully. That is critical for demonstrating that your systems are actually doing what they are supposed to do. It is much easier to say, “Here is a dashboard showing the results of this check over the last 180 days,” than to explain in natural language all the steps you take to maintain ongoing compliance with a regulation. Mistakes That Cost Us Months (and Probably Millions) Don’t Ship Fifty Policies on Day One Leadership will ask why it is taking so long. Even so, don’t try to deploy data classification, access control, drift monitoring, and fairness audits all at once. Engineers will start filing bypass requests faster than your governance team can process them. Instead, pick one policy that would have prevented your last major incident if it had been in place. Deploy that policy. Let engineers see it, push against it, and learn how it works and how it can help catch real production issues before they become public relations nightmares. Once they understand it, iterate from there. Make Failures Helpful, Not Hostile If a policy fails, make sure it tells the user what they did wrong, why it is a problem, and how to fix it. A policy that simply says “DENIED” will have much lower adoption than one that says, “This dataset contains Tier 3 data, but your model is only approved for Tier 2 datasets. See the link below if you want to formally request a change to your model’s approval tier.” Version Your Policies in Source Control Use proper version control for your policies—not a governance database or a wiki, but real source code version control that enforces pull requests and code review. That way, when a regulation changes, you can update the policy, review the changes, test it, and deploy it to production just like any other code. It also makes it much easier to roll back a change if something goes wrong. The Audit Trail Is the Product Every time a policy runs, capture the outcome along with information about the model, dataset, and any other contextual details that could be useful to an auditor. If someone asks, “How are you preventing the use of sensitive data in model training?” you should be able to show them a dashboard containing the results of every relevant data classification check run over the last 90 days. The Regulatory Reality The regulatory landscape around AI is expanding quickly. The EU AI Act is now in force, the NIST AI Risk Management Framework is becoming an important reference point, state legislatures are passing new laws, and standards such as ISO/IEC 42001 are increasingly becoming de facto requirements. None of these developments are going away, and companies cannot expect a PDF policy reviewed once a year to be enough for compliance. The companies that handle this environment well will treat compliance as a force multiplier rather than overhead. Policy-as-code at deployment time gives you gates, while continuous runtime monitoring gives you guardrails. Together, they give engineers enough context to reason about their systems while adding friction only where it matters most. The result is safer products, more productive engineers, and a compliance posture you can actually demonstrate to an auditor.

Most vehicle tracking systems ask one database to do everything. For example, store the road network, query it with recursive CTEs, write live positions to the same database, run analytics on the same tables, and so on. It works until the graph queries slow down, high-frequency writes start competing with reads, and analytics queries time out. This article shows a different approach: three databases, each doing what it's genuinely good at. Neo4j Aura for the road network graph, Databricks Lakebase for live vehicle positions, and Databricks Lakehouse for historical analytics. Ten simulated vehicles move around a real city following real road connections loaded from OpenStreetMap. Two Streamlit dashboards show live positions and analytics. The whole system is driven by a single YAML configuration file, so switching from London to San Francisco or Singapore means changing a single file and rerunning five notebooks. The full source code is available on GitHub. The Three-System Architecture The architecture has three distinct layers: Neo4j Aura holds the road network — intersections, road segments, zone topology, and shortest paths. It answers graph questions that a relational database may handle awkwardly.Databricks Lakebase holds the live operational data — vehicle positions written every two seconds by a simulator, vehicle statuses, and trip records. It's a fully managed Postgres database inside Databricks, handling OLTP workloads with standard psycopg2 connectivity.Databricks Lakehouse holds the analytical history — position data synced from Lakebase into a Delta table, aggregated by zone and road segment. None of these systems knows about the others. The intelligence sits in the application layer — the simulator, the Streamlit dashboards and the analytics notebook — which orchestrates queries across all three and combines the results. The Road Network in Aura We'll use OSMnx to download the drivable road network for the London Borough of Merton from OpenStreetMap and load it into Aura. The graph model is straightforward: Cypher (:Intersection {node_id, lat, lon, street_count, location}) -[:ROAD {osmid, name, highway, maxspeed, oneway, length_m}]-> (:Intersection) Merton's road network produces thousands of intersection nodes and thousands of directed road relationships. A POINT INDEX on the location property enables fast nearest-neighbor lookups -- finding the intersection closest to any GPS coordinate runs in milliseconds. The reason for Aura is simple: the road network is a graph and graph queries are where Aura excels. Finding the shortest path between two zones is a single Cypher function call: Cypher MATCH path = shortestPath((start)-[:ROAD*..300]->(end)) RETURN length(path) AS hops The equivalent in SQL requires a recursive CTE that grows in complexity with every additional hop. For zone reachability queries — "which zones can a vehicle reach within two hops?" — the difference is even more pronounced. We also define five logical zones as bounding boxes within the borough and store them as Zone nodes with ADJACENT_TO relationships. This gives us a zone adjacency graph that the simulator uses for routing decisions. The Simulator The simulator loads the entire road graph from Aura into memory at startup — one query, one dictionary, no further Aura calls during the simulation loop. It then places ten vehicles at their home intersections and moves each one along Breadth-First Search (BFS)-computed routes. Vehicles don't route randomly. They have a home zone and a 70% chance of staying in or near it. The remaining 30% of the time, they cross into any zone in the borough, producing occasional longer cross-city runs. Every two seconds, each vehicle writes its current coordinates to Lakebase: Python cursor.execute(""" INSERT INTO vehicle_positions (vehicle_id, lat, lon, speed_kmh, current_zone) VALUES (%s, %s, %s, %s, %s) """, (vehicle_id, lat, lon, speed_kmh, current_zone)) The simulator runs as a background subprocess launched from a Jupyter notebook, continuing independently while the Streamlit dashboards are open. The Live Vehicle Tracker The vehicle tracker (app.py) refreshes every three seconds and shows three pydeck layers on a CARTO basemap: Vehicle icons – one car icon per vehicle at its current GPS positionTrail lines – each vehicle's last 20 positions, colored by home zoneShortest path – a black line showing the road-network shortest path between any two selected zones, computed on demand from Aura Figure 1 shows vehicles moving on the Merton map with trail lines and a shortest path highlighted between two zones. Figure 1. Streamlit Vehicle Tracker. The sidebar shows a bar chart of zone activity over the last 10 minutes and a nearest-driver lookup -- given a zone, which vehicle is currently closest to it? The haversine distance calculation runs against the latest position of every vehicle, using zone center coordinates that map to real road intersections. The Analytics Dashboard The analytics dashboard (analytics_app.py) connects to all three systems simultaneously. Every 30 seconds, it syncs new position records from Lakebase into a Lakehouse Delta table and runs two analytical queries. Figure 2 shows an analytics dashboard with zone activity over time across all five zones. Figure 2. Analytics Dashboard. The chart on the left-hand side shows position update counts per zone per minute over the last hour — a live view of which parts of the city are busiest: SQL SELECT current_zone AS zone, DATE_TRUNC('minute', recorded_at) AS minute, COUNT(*) AS updates FROM vehicle_positions_delta WHERE current_zone IS NOT NULL GROUP BY current_zone, DATE_TRUNC('minute', recorded_at) ORDER BY minute, zone The chart on the right-hand side is the architectural highlight: a cross-system join that answers "which named roads carry the most vehicle traffic?" Lakebase has the position records (latitude, longitude, per vehicle per tick). Aura has the road names (what named road each intersection belongs to). Neither system alone can answer the question. The join runs in Python using pandas. Road names and coordinates are loaded from Aura once at startup and cached. Position coordinates come from Lakebase via the Lakehouse Delta table on each refresh. Coordinates are rounded to three decimal places (~100m precision) and joined: Python joined = pos_df.merge( road_df[["road_name", "highway", "lat_r", "lon_r"]], on=["lat_r", "lon_r"], how="inner" ) Primary roads dominate because BFS routing naturally follows main roads when finding shortest paths. The YAML Configuration System Every city-specific value lives in a single config.yaml file which contains zone definitions, vehicle assignments, map coordinates, and the OpenStreetMap place name. YAML city: name: "London Borough of Merton" osmnx_place: "London Borough of Merton, UK" network_type: "drive" map_lat: 51.410 map_lon: -0.188 map_zoom: 12 Switching cities means copying a different config file and re-running five notebooks. Three example config files are included: Merton (London), San Francisco, and Singapore. For cities where OSMnx's place name geocoding doesn't produce a usable polygon boundary, a pyrosm-based approach clips a Geofabrik regional file to a bounding box instead. The pre-clipped files for San Francisco and Singapore are included in the GitHub repo, so you can run those configs without any additional data preparation. A companion config_validator.py validates the file on load and raises clear errors if anything is missing or malformed. Why Three Systems? The answer is that each system does something the others can't do efficiently. Neo4j Aura handles graph traversals — shortest paths, multi-hop reachability, nearest-node spatial lookups. These are awkward in SQL and natural in Cypher. Databricks Lakebase handles high-frequency OLTP writes — hundreds of inserts per minute, sustained, with foreign key constraints and BIGSERIAL auto-increment. Databricks Lakehouse handles analytical aggregations over historical data — counting position records by zone and minute, joining across large datasets. Columnar storage and parallel execution make this fast. The three-system architecture isn't complexity for its own sake. Each system earns its place by doing something the others would handle poorly. The Free Online Book The full system — all notebooks, both Streamlit apps, the YAML config system and seven chapters of detailed explanation — is available as a free online book. The book covers the road network loading and data cleaning, zone and adjacency graph setup, Lakebase table design, the BFS simulator, both Streamlit dashboards, the analytics notebook, and all the gotchas and lessons learned. The code is on GitHub under Apache 2.0. The pre-clipped OSM data files are available under the Open Database License (ODbL). Summary We've built a real-time fleet operations dashboard using three database systems, each doing what it does best: Neo4j Aura for road network graph queries and shortest path computation, Databricks Lakebase for high-frequency vehicle position writes, and Databricks Lakehouse for historical analytics over Delta tables. The interesting engineering is in the joins that cross system boundaries — finding the nearest driver uses Aura's spatial index, routing vehicles uses BFS over an in-memory graph loaded from Aura, and identifying the busiest named roads joins position data with road names via pandas. A YAML configuration file drives the entire system, making it straightforward to point the same codebase at a different city. The architecture demonstrates that a multi-database approach isn't inherently complex — it becomes simpler when each system has a clear, non-overlapping role. The full source code is available on GitHub.

By Akmal Chaudhri CORE

The Index Can Be Fast and Still Be Wrong Vector search teams usually define performance with query latency, recall, and throughput. Those measures matter, but they can all look healthy while the system returns a stale version of a document that changed minutes ago. The index is fast. The answer is still wrong. This failure is easy to miss because a vector index is normally downstream from the source of truth. Between a database write and a searchable embedding sit event capture, transport, chunking, model inference, index mutation, and cache invalidation. Freshness is the end-to-end property produced by that entire chain. Freshness Needs a Contract Saying that updates are processed quickly is not a contract. A useful freshness SLO states which source version must be searchable, how long the pipeline may lag, and what the query path should do when that guarantee cannot be met. For example, a service might require 99 percent of committed updates to become searchable within 60 seconds, while deletes must disappear within 10 seconds. The distinction matters because showing old text is inconvenient, but returning deleted or access-revoked content can become a security incident. Capture the Write Without a Dual-Write Gap The first failure appears when application code writes the business row and publishes an indexing event as two separate operations. If the database commit succeeds and the publish fails, the source changes without any durable instruction to update the index. Retrying the request does not reliably repair that gap. A transactional outbox avoids the split. The application updates the entity and inserts an outbox record in the same database transaction. A change-data-capture process then publishes the outbox record asynchronously. SQL BEGIN; UPDATE documents SET body = :body, version = version + 1 WHERE id = :id; INSERT INTO embedding_outbox (event_id, entity_id, source_version, operation) SELECT :event_id, id, version, 'UPSERT' FROM documents WHERE id = :id; COMMIT; Make Every Event Versioned and Idempotent Delivery systems retry. Partitions rebalance, workers crash after writing but before acknowledging, and older events can arrive after newer ones. The index consumer must therefore treat duplicate and out-of-order delivery as normal behavior, not an edge case. Each event should carry an immutable event identifier, entity identifier, monotonic source version, operation, and payload reference or hash. The consumer applies a mutation only when the incoming version is newer than the indexed version. That compare-and-set must be atomic in the index or in a strongly consistent metadata store beside it. Python def apply(event, index, embed): current = index.metadata(event.entity_id) if current and current.source_version >= event.source_version: return "already_applied" if event.operation == "DELETE": index.delete_if_newer(event.entity_id, event.source_version) return "deleted" vector = embed(event.content) index.upsert_if_newer( id=event.entity_id, vector=vector, metadata={"source_version": event.source_version} ) return "updated" Deletes Are First-Class Data Upserts get most of the design attention because they create embeddings. Deletes are more dangerous because there is no new content to process. A delete event must survive the same durable path and carry a version that prevents an older upsert from resurrecting the record later. Keep tombstones long enough to cover the maximum replay and recovery window. If a full rebuild reads a snapshot taken before a delete, the rebuild process must also consume the change stream from the snapshot position forward. Otherwise, the old record can quietly return when the new index is promoted. Model Versions Belong in the Index Schema Fresh source data can still be semantically stale when query and document vectors were created by different embedding models. Store the embedding model identifier, chunking configuration version, and normalization settings with every indexed item. Treat those fields as part of the index schema. A model upgrade should normally create a new physical or logical index generation. Dual-write new updates, backfill historical content, validate retrieval quality, then switch query traffic. Mixing vectors from incompatible spaces in one collection creates a failure that looks like weak relevance but cannot be tuned away. Use Watermarks to Measure What the Pipeline Has Proven A queue-depth metric shows workload, not freshness. The more useful signal is a source-position watermark: the highest committed database position or entity version that the searchable index has fully applied. Compare that watermark with the source head to measure version lag and event-time lag. Parallel consumers complicate this because one partition can race ahead while another is stuck. The global searchable watermark is bounded by the slowest required partition. Reporting the fastest worker hides exactly the stale slice users are likely to hit. Guard Queries When Freshness Matters Some requests know the minimum version they require. A write API can return the committed source version, and a later search request can send that version as a read-your-writes token. The query layer then checks whether the relevant index watermark has caught up. The fallback depends on the product. The service can wait briefly, route to a fresher generation, perform a source-of-truth lookup, or return a clear retryable status. Serving an older result without saying so should not be the default. Python def search(query, minimum_version=None): if minimum_version is not None: if index_watermark() < minimum_version: raise RetryableFreshnessError( "search index has not reached the required version" ) return vector_index.search(query) Rebuild Without Creating a Freshness Blackout Large indexes eventually need rebuilding because schemas, models, or partition layouts change. A safe rebuild uses a snapshot plus a change-stream handoff. Record the snapshot position, bulk-load the snapshot into a new generation, replay every later event, and promote only after its watermark reaches the live index. The promotion itself should be an atomic alias or routing change. Keep the previous generation available for rollback until both correctness and latency checks pass. A rebuild is not complete when bulk loading ends. It is complete when the new generation proves that no committed change was skipped. Operate Freshness Like Availability The dashboard should track source-to-index lag percentiles, oldest unapplied event age, consumer retry rate, dead-letter volume, version conflicts, delete lag, model-version distribution, and watermark gaps by partition. Alerts should be tied to the freshness SLO rather than to queue depth alone. Periodic reconciliation closes the final gap. Sample source entities, compare their versions and hashes with indexed metadata, and repair mismatches through the normal event path. The goal is not to pretend delivery is perfect. The goal is to make drift observable, bounded, and repairable. Test Failure Modes Before Launch Also test partial degradation. If one embedding worker pool is unavailable, confirm that lag remains visible and query guards behave as designed. If the dead-letter path fills, verify that alerts fire before the SLO is exhausted. These exercises turn recovery assumptions into executable evidence and reveal whether the pipeline can repair itself without manual database edits. Freshness behavior deserves fault-injection tests, not only happy-path integration tests. Pause one consumer partition, duplicate a batch, deliver versions out of order, fail an embedding call after the index write, and replay a snapshot across a recent delete. Each test should assert the final indexed version, not merely that the worker returned success. The Missing SLO Production vector search is a replicated data system with expensive transformation in the middle. Once that is clear, familiar distributed-systems rules apply: capture changes durably, version every mutation, make consumers idempotent, preserve deletes, expose watermarks, and rebuild from a known log position. Latency tells you how quickly the index answered. Freshness tells you whether it answered from the world that exists now. A production search system needs both guarantees, because a fast answer from yesterday is still a failure. References Debezium Outbox Event RouterPostgreSQL Logical Decoding ConceptsApache Flink Timely Stream Processing and Watermarks