DZone Spotlight

Friday, November 6 View All Articles »

Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

By CARTER WANG

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records. I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them. None of this is exotic. It is the set of things I wish had been in one place before I wrote the first version. Failure Mode 1: The Fixed Interval The naive poller runs on a fixed schedule — every 60 seconds, every source, forever. This is wrong in both directions simultaneously. High-volume sources produce more records between polls than a single page returns, so you lose everything past the page boundary. Low-volume sources return nothing on the vast majority of calls, burning quota that the high-volume sources needed. With a shared quota across sources, the fixed interval guarantees you are under-sampling exactly where the data is. The fix is to let observed yield drive the interval. If the last poll came back full, poll sooner. If it came back empty, back off. TypeScript interface PollState { sourceId: string; intervalMs: number; consecutiveEmpty: number; } const MIN_INTERVAL = 30_000; const MAX_INTERVAL = 1_800_000; // 30 min function nextInterval(state: PollState, recordsFound: number, pageSize: number): number { // Page came back full — we are probably truncating. Poll aggressively. if (recordsFound >= pageSize) { return MIN_INTERVAL; } if (recordsFound > 0) { // Productive but not saturated. Ease toward the current rate. return clamp(state.intervalMs * 0.8, MIN_INTERVAL, MAX_INTERVAL); } // Empty. Back off geometrically, but bounded. const factor = 1.5 ** Math.min(state.consecutiveEmpty, 6); return clamp(MIN_INTERVAL * factor, MIN_INTERVAL, MAX_INTERVAL); } function clamp(n: number, lo: number, hi: number): number { return Math.min(hi, Math.max(lo, n)); } The condition worth highlighting is recordsFound >= pageSize. A full page is not a success signal — it is evidence that records may exist beyond what you fetched. Treat it as a truncation warning and either paginate immediately or shorten the interval. Systems that treat a full page as "great, got everything" lose data proportional to source velocity, which means they lose the most from the sources that matter most. After deploying adaptive intervals, our total request volume dropped by roughly 40% while records captured went up, because quota moved from silent sources to active ones. Failure Mode 2: The Watermark Boundary The standard incremental pattern stores the timestamp of the newest record seen and requests everything after it: SQL SELECT * FROM source WHERE created_at > :watermark ORDER BY created_at This loses records, for three independent reasons. Clock skew. The upstream timestamp is assigned by their server, not yours. Distributed writers mean created_at is not guaranteed to be monotonic in the order records become visible to the API.Visibility lag. A record can be written at T but not appear in query results until T+2s due to replication or index lag. Poll at T+1s, advance the watermark past T, and that record is never returned.Ties. Multiple records sharing the same timestamp, split across a page boundary, silently drop the remainder when you use a strict >. The fix is to rewind the watermark by a safety margin and deduplicate downstream: TypeScript const OVERLAP_MS = 120_000; // tune to observed upstream lag function queryWindow(watermark: number): { since: number; until: number } { return { since: watermark - OVERLAP_MS, until: Date.now(), }; } This deliberately re-fetches records you already have. That is the point — you trade a small amount of redundant bandwidth for the guarantee that a record has to be missed by two consecutive polls to be lost permanently. Size OVERLAP_MS From measurement, not intuition. Log the delta between upstream created_at and your own ingest time, take the 99th percentile, and double it. Ours sat around 40 seconds at p99, so 120 seconds gives real headroom without meaningful cost. Failure Mode 3: Deduplication That Isn't Overlap windows only work if dedup is correct, and ID-based dedup handles less than people assume. Upstream records get edited. A post whose body changes after ingestion keeps its ID, so an INSERT ... ON CONFLICT DO NOTHING on the ID silently discards the update. Meanwhile, some sources reuse or re-issue identifiers across deletion, so the ID alone is not always a stable key either. Hash the fields that constitute a meaningful change, and store both: TypeScript import { createHash } from "node:crypto"; function contentHash(record: SourceRecord): string { // Only fields whose change means "this record is materially different." // Volatile counters (score, comment count) are excluded on purpose. const canonical = JSON.stringify({ id: record.id, title: record.title ?? "", body: record.body ?? "", author: record.author ?? "", }); return createHash("sha256").update(canonical).digest("hex"); } SQL INSERT INTO records (source_id, external_id, content_hash, payload, first_seen_at, last_seen_at) VALUES ($1, $2, $3, $4, now(), now()) ON CONFLICT (source_id, external_id) DO UPDATE SET content_hash = EXCLUDED.content_hash, payload = EXCLUDED.payload, last_seen_at = now(), revision = records.revision + 1 WHERE records.content_hash IS DISTINCT FROM EXCLUDED.content_hash; Two details carry most of the weight. The WHERE clause means the overlap window costs nothing when nothing changed — no write amplification, no spurious updated_at churn, and downstream consumers watching for changes don't get woken up by re-fetches. And excluding volatile counters from the hash is what makes this viable at all. Include a score that ticks every few seconds and every single poll registers as a change, which defeats the purpose and turns your overlap window into a write storm. Failure Mode 4: Retry Storms and Undetected Gaps When a rate-limited API returns 429, the instinct is to retry. Retry immediately, and you extend the penalty. Retry on a fixed delay across N workers, and they resynchronize into a thundering herd that hits the limit together, forever. Most rate-limited APIs tell you what you need in response headers. Read them instead of guessing: TypeScript interface RateLimitState { remaining: number; resetAt: number; } function parseRateLimit(res: Response): RateLimitState | null { const remaining = res.headers.get("x-ratelimit-remaining"); const reset = res.headers.get("x-ratelimit-reset"); if (remaining === null || reset === null) return null; return { remaining: Number(remaining), // Some APIs send seconds-until-reset, others an absolute epoch. Normalize. resetAt: Number(reset) > 1e9 ? Number(reset) * 1000 : Date.now() + Number(reset) * 1000, }; } async function backoff(attempt: number, res?: Response): Promise<void> { const retryAfter = res?.headers.get("retry-after"); if (retryAfter) { const ms = Number(retryAfter) * 1000; if (Number.isFinite(ms)) return sleep(ms); } // Full jitter — see AWS Builders' Library on backoff. const ceiling = Math.min(60_000, 1000 * 2 ** attempt); return sleep(Math.random() * ceiling); } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); Full jitter matters more than the backoff curve. Exponential backoff with a deterministic delay keeps clients synchronized; randomizing across the whole interval is what actually spreads load. Retry-After, When present, it is authoritative — honor it over your own calculation. The subtler problem is that these failures are invisible. A source that throws 429 for six hours produces no errors in your application logs if the poller catches and backs off correctly. It just quietly returns nothing, and your data has a hole in it that nobody notices until someone asks why a report looks wrong. Detect gaps explicitly rather than inferring them from error rates: TypeScript async function detectGaps(sourceId: string): Promise<Gap[]> { const { rows } = await db.query(` SELECT window_end AS gap_start, LEAD(window_start) OVER (ORDER BY window_start) AS gap_end FROM poll_log WHERE source_id = $1 AND status = 'ok' AND window_start > now() - interval '48 hours' ORDER BY window_start `, [sourceId]); return rows .filter(r => r.gap_end && r.gap_end - r.gap_start > GAP_THRESHOLD_MS) .map(r => ({ sourceId, from: r.gap_start, to: r.gap_end })); } This requires logging every poll attempt with its requested window and outcome — a table most people skip because it feels like overhead until the first time it tells you about four hours of missing data you would otherwise never have found. Feed detected gaps into a low-priority backfill queue that runs when quota is available. What Actually Matters If you take three things from this: Log the window, not just the result. Without a record of what range each poll requested, gaps are undetectable. This is the single highest-value thing to add to an existing poller.Overlap deliberately, dedup by content. A safety margin plus content-hash dedup converts "we might have missed something" into "a record must be missed twice to be lost."Treat a full page as a warning. It is the most common silent truncation point in production pollers, and it is one comparison to check for. None of these are difficult. They are just easy to omit, and their absence produces a system that looks healthy while quietly dropping records. I work on the ingestion layer at SignalMelo, which polls public discussion sources continuously — the patterns above came out of production incidents there. Disclosure: it's a commercial product I help build. More

Member Spotlight: Srinivas Chippagiri

By Dominique Pugh

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here. More

The Man Who Invented Selenium Also Built Its Replacement

By Arun Kulkarni

Diagnosing Test Failures at Scale: Correlating Execution Evidence Across Ephemeral Runners

By Anil Reddy Putlacheru

GET, POST, PUT, DELETE: Choosing HTTP Verbs in a Real .NET Core Web API, Not Just by the Textbook

By Dhana PAMULA

How to Evaluate Coding Agents With Harbor Framework, Python, Docker

By Iyanuoluwa Ajao

Gen AI Debugging Tools Compared: Genkit vs Vercel AI SDK vs Mastra

If you have ever shipped a backend service without a debugger, a hot-reload dev server, or decent logs, you remember how painful it was. Now imagine doing it in a world where your "function" is a non-deterministic black box that can rewrite its own output every time you call it. Welcome to building generative AI (GenAI) applications. The single biggest productivity multiplier I have found in the last two years of shipping AI apps is a good local debugging tool. Not a hosted dashboard. Not a cloud trace viewer with a 30-second propagation delay. A local tool that runs next to your code, picks up your edits, lets you replay a request with a tweaked prompt, and shows the full trace, tool call by tool call. We will get into the local-vs-hosted trade-off later in the article. This article is about why that matters and how the three leading JS/TS GenAI frameworks approach it: Genkit Developer UIVercel AI SDK DevToolsMastra Studio These three very different products solve overlapping problems, and each makes trade-offs worth understanding before you commit to one. Shift-Left, Applied to GenAI "Shift-left" comes from the world of testing and security: the earlier in the development lifecycle you catch a problem, the cheaper it is to fix. Bugs found in production cost orders of magnitude more than bugs found while you're typing. GenAI apps make this principle existential, not just economical, because the failure modes are weirder: A prompt regression doesn't show up as a stack trace. It shows up as the wrong tone, a hallucinated fact, or a tool call that misfires once every twenty runs.A new model version can quietly change behavior across thousands of code paths with no warning.A tool that returns slightly different JSON can cause silent downstream breakage. The only way to keep your sanity is to collapse the feedback loop to seconds. You want to be able to: Change a prompt or a piece of orchestration code.Re-run that exact unit, with that exact input.See the model's input, its output, every tool call, every retry, every token used.Compare it to the previous run.Decide if it's better, worse, or different. If any one of those steps requires deploying, redeploying, opening a cloud console, or grep'ing logs, you have already lost. Cycle time is the metric. A debugging tool is what makes that cycle time low enough to iterate productively. Local vs. Hosted It is worth being explicit about this trade-off, because all three tools in this article are primarily local: A local debugging tool runs on your machine, next to your code, with zero network latency between your edits and what you see. Iteration is fast, traces are immediate, and there is no risk of leaking prompts/responses to a third party. The downside is that it's just for you.A hosted observability platform (Langfuse, LangSmith, cloud-native APM tools, etc.) is shared by your team, persists data long-term, and is essential for production monitoring. The trade-off is propagation delay, configuration overhead, and, in some cases, data residency concerns. These are complementary, not competing. The argument here is that the local part of the loop is the one that's still often missing, and that's where shift-left lives. What "Good" Looks Like After working with all three of the tools below, I would summarize the qualities of a good Gen AI debugging tool as: Zero-config or near-zero-config – it should pick up your code, not the other way around.Live reload – edit your code, save, hit "run" again. No restart.Interactive runners – invoke any prompt, tool, model, or higher-level primitive (agent, workflow, etc.) directly with arbitrary input.Full traces – every step of the generation loop, with input/output/usage at each node.Replay and modify – re-run any past trace with a tweak.Minimal code changes – the framework should be observable by default, ideally without you having to remember to wrap each model call. Let's see how each tool stacks up. Genkit Developer UI The Genkit team treats the Developer UI as a first-class part of the framework. It ships with the genkit-cli package and requires no code changes in your application to attach to your running process. You install the CLI once globally: Shell npm install -g genkit-cli And then start your app under it: Shell genkit start -- npx tsx --watch src/index.ts That's it. Genkit attaches to your running Node process, discovers every flow, prompt, model, tool, retriever, indexer, embedder, and evaluator you have defined, and exposes them all in a local web app at http://localhost:4000. Plain Text Telemetry API running on http://localhost:4033 Genkit Developer UI: http://localhost:4000 What you actually get inside: Interactive runners for every primitive. Flows, prompts, tools, models, retrievers, indexers, embedders, and evaluators all get an interactive panel where you fill out the input (validated against the Zod schema) and hit run.Full traces with the entire generation graph: every model call, every tool invocation, every middleware, with input/output/usage tokens.Live reload. Combined with --watch, edits to your code show up without restarting the UI.Prompt iteration. Tweak prompts and re-run inline.Evals. Run evaluators against datasets directly from the UI.Open by default. Add -o to auto-open in your browser. What stands out here is that observability comes for free. You do not wrap your model, and you do not add a middleware just to see traces. Anything defined with the Genkit primitives is automatically introspected and traced, which makes the "let me see what's happening" cost essentially zero. If you also use the middleware system, every middleware also shows up in the trace, which makes debugging things like retries and fallbacks a lot easier. Vercel AI SDK DevTools Vercel introduced the AI SDK DevTools more recently, and it is a meaningful step up from having nothing. The design philosophy is different from Genkit's: it is a middleware-based capture tool that focuses tightly on the language-model layer, rather than an integrated development environment for the whole AI app. You opt in by wrapping your model: TypeScript import { wrapLanguageModel, gateway } from 'ai'; import { devToolsMiddleware } from '@ai-sdk/devtools'; const model = wrapLanguageModel({ model: gateway('anthropic/claude-sonnet-4.5'), middleware: devToolsMiddleware(), }); And then run the viewer in another terminal: Shell npx @ai-sdk/devtools # open http://localhost:4983 What it captures: Input parameters and prompts.Output content and tool calls.Token usage and timing.Raw provider request/response payloads.Multi-step interactions are grouped into "runs" with multiple "steps". Things to keep in mind when comparing it with the others: It is opt-in per model. Every model you want to observe has to be wrapped explicitly. In a real codebase with many models, this usually means a shared factory or repeated wrapping. If you forget on one path, you have a blind spot.It focuses on the language-model layer. Anything happening above that layer (your own application logic, custom orchestration, business steps) is not captured unless you add it yourself.It is a viewer, not an interactive playground. You cannot "re-run this with a different prompt" from the UI; you go back to your code, edit, and re-call.It does not enumerate AI primitives because, in the AI SDK, those are just functions in your codebase. There is nothing to enumerate at runtime.Storage is a JSON file (.devtools/generations.json). The team is clear that this is local-only, never for production, and the middleware automatically appends .devtools to your .gitignore. The upside is that it is probably the quickest of the three to drop into an existing project: install the package, wrap a model, run the viewer. If you live inside Next.js and the AI SDK and you mainly want a "what just happened?" panel for your model calls, it does the job well. If you are debugging a multi-step agent with custom orchestration, you will probably want to combine it with something that sees more of the picture. Mastra Studio Mastra Studio is the most ambitious of the three in terms of surface area. It is bundled with mastra dev and runs at http://localhost:4111, doubling as a local development tool and a deployable team console (you can ship Studio to production for non-developers). Out of the box, it gives you: Agents tab – chat with your agents, hot-swap models, tweak temperature/top-p, view traces, attach scorers.Workflows tab – visualize workflows as graphs, run them step by step with custom inputs, watch the active step in real time, inspect tool calls and JSON outputs.Processors tab – see input/output processors and guardrails wired to each agent.Tools tab – run tools in isolation to debug them.Workspaces tab – file browser into the agent's workspace filesystem, with a Skills tab listing discovered skills.MCP servers tab – list attached MCP servers and their tools.Request context – set runtime variables that flow into agent instructions/tools through DI, with schema-driven forms.Evaluation suite – Scorers, Datasets, and Experiments tabs to run datasets through agents/workflows, attach scorers, and compare experiments side-by-side.Observability with traces and logs.Editor integration for non-technical teammates to iterate on agents and version every change without redeploying. This is more than a debugging tool — it is closer to a development environment plus a team-facing console. The natural trade-off is that Studio is tightly coupled to Mastra's primitives (Agents, Workflows, Processors, Workspaces), so it shines the most when your application is structured the Mastra way. If you build non-trivial agentic systems and you want evaluation and dataset management baked into the same tool, Studio is genuinely very good. For a quick "see my prompt and the model's response" loop, it can feel like more than you need. Side-by-Side Comparison CapabilityGenkit Developer UIVercel AI SDK DevToolsMastra StudioSetupgenkit start -- your codewrapLanguageModel(...) + npx @ai-sdk/devtoolsmastra devCode changes requiredNone (auto-discovers)Yes (wrap each model)None (auto-discovers Mastra primitives)Live reloadYes (--watch)N/A (viewer only)YesInteractive runnersFlow, prompt, model, tool, retriever, indexer, embedder, evaluatorNone (viewer only)Agent chat, workflow runner, tool runnerTracesFull generation graphPer-model-call run/stepWorkflow + agent tracesRe-run / replayYes, from any primitiveNo (must edit code and re-call)Yes, including workflow step-throughDataset / eval UIEvaluators panelNot built-inDatasets + Experiments + ScorersWorkspace / files browserNoNoYesMCP server browserNot first-classNoYesDeployable to teamNo (local-only)No (local-only)Yes (Studio on Mastra platform)LanguagesJS/TS (Go, Python, Dart all have local tooling too)JS/TS onlyJS/TS onlyStyleAuto-discovery of primitivesPer-model middleware captureAll-in-one studio for the agent lifecycle A quick note on the Vercel column: simplicity of integration is a real feature for many teams, and it is fair to call this the lightest-touch option. The trade-off is that observability is opt-in per model, so you have to be deliberate to avoid blind spots. When I Would Pick Which After using all three, this is roughly how I'd think about it: Building a serious agent or multi-step pipeline with tools, retries, fallbacks, and evals → Genkit Developer UI is hard to beat on pure iteration speed. Zero-config tracing of the full pipeline, every primitive runnable, and middleware shows up in the trace for free.Working primarily inside Next.js with the Vercel AI SDK and you mainly want better visibility into model calls → Vercel AI SDK DevTools is a quick win. Lightweight setup and it does what it says on the tin; pair it with something else if you need higher-level orchestration views.Building agentic systems with workflows, datasets, and scorers, and you want a UI your PM or domain expert can also use → Mastra Studio. The evaluation suite and the deployability are real differentiators.Multi-language stack (JS/TS plus Go, Python or Dart) → Genkit is the only one of the three with first-party multi-language support; the local tooling story is consistent across runtimes. The Moral of the Story: Pick a Debugging Tool, Any Debugging Tool In a normal backend project, you can sometimes get away with a weak dev loop because the code is deterministic. Bad UX, but it works. In an AI project, the code is not deterministic. Every iteration is also a small experiment. If your loop is slow or your visibility is poor, you don't just iterate slower; you iterate worse, because you can't tell whether your changes are improvements or regressions. Shift-left in this world is not a virtue; it is a survival strategy. A good local debugging tool: Turns "I think the prompt got better" into "I can see the trace, the tokens, the eval score, side by side".Catches regressions before they reach a user.Lets you onboard new engineers in hours instead of days, because they can see the system.Lets non-engineers (PMs, designers, domain experts) participate in iteration when the UI is friendly enough — Mastra Studio is explicit about this. If you are starting a Gen AI project today and you are picking a framework, the local tooling story should weigh as much as the model abstraction or the tool API. It is the difference between writing AI code and engineering AI features. Conclusion All three tools are good in their lane: Genkit Developer UI is the most introspective of the three and requires zero code changes to be useful. It's the one I reach for when I want to move fast and see every step of the pipeline.Vercel AI SDK DevTools is the lightest-touch option to drop into an existing AI SDK app, with a clear focus on the model-call layer. Think of it as a focused viewer rather than a full development environment.Mastra Studio is the most ambitious, closer to a full IDE-meets-console for agentic systems, with first-class evaluation, datasets, and a deployable team UI. It's at its best when your app is structured around Mastra's primitives. Pick the one that matches the shape of your project, but please pick something. Building Gen AI without a local debugging tool in 2026 is engineering with the lights off. Further reading: Genkit Developer ToolsVercel AI SDK DevToolsMastra Studio overviewTop JS/TS GenAI Frameworks for 2026Genkit Middleware deep dive

By Xavier Portilla Edo

CORE

Full-Text Search Meets the Graph: Apache Lucene Inside Neo4j

We have documents that need full-text search, but those documents also have meaningful relationships. Do we split search and graph traversal across two systems or keep them together? Most developers reach for Elasticsearch or a dedicated search service when they need full-text search. It's a reasonable reaction, but it introduces an extra service to operate, an extra data pipeline to maintain, and an extra hop in every query. What's easy to miss is that Neo4j ships with Apache Lucene built in — the same underlying search library on which Elasticsearch is built — and when our data are already in a graph, adding full-text search on top is a single Cypher command! More importantly, full-text search in Neo4j can do something a standalone search engine can't: it serves as an entry point into the graph. We can find our initial results with Lucene, then traverse relationships to discover what surrounds them. That two-phase pattern — search-then-traverse — is what this article describes. We'll build a research paper explorer using a generated dataset of 200 papers with titles, abstracts, authors, fields, venues, and citation relationships. The same papers appear in the Weaviate chapter of Seven Vector Databases in Seven Days, where we used hybrid vector search to find them. Here we use Neo4j's full-text search instead, and we add something the Weaviate version didn't have: a citation graph that we can traverse after search. The comparison is instructive — same papers, similar queries, but a very different graph layer. The full source code is available on GitHub. Prerequisites AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at Get Started for Free.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the connection URI, username, and password.Once the instance is running, open the Query tab, and connect to the instance.Confirm it's empty with MATCH (n) RETURN count(n) which should return 0. The code runs in a Jupyter notebook with Python 3.12 or later. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI="neo4j+s://xxxx.databases.neo4j.io" export NEO4J_USERNAME="your-username" export NEO4J_PASSWORD="your-password" One version note. This article queries the full-text index with Cypher's SEARCH clause, which replaces the older db.index.fulltext.queryNodes() procedure. The documentation lists SEARCH on full-text indexes as available from Neo4j 2026.09 and it requires Cypher 25. Our AuraDB Free instance accepted it, so the most reliable check is simply to try the clause, which the notebook does. On instances that don't support it, the procedure does the same job but now produces a deprecation warning. The Graph Model The dataset has three node labels and two relationship types: Plain Text (:Author)-[:AUTHORED]->(:Paper)-[:CITES]->(:Paper) Paper nodes carry the properties we'll search and filter on: title, abstract, field, year, venue and citation_count. Author nodes have a single property: name. The AUTHORED and CITES relationships carry no properties — all the meaning is in the graph structure. A conventional search index doesn't give us Neo4j's native graph model and Cypher traversal over relationships. A paper's citation neighborhood, the other work its authors have published, the papers that are frequently cited alongside it — that structure lives in the relationships, and it's what we'll traverse after search. Generating the Dataset We'll generate 200 papers from pools of fields, venues, author names, methods, datasets, and abstract templates, using a fixed random seed so the results are reproducible. The generation code is adapted directly from the Weaviate chapter, so the papers are the same across both articles. Python random.seed(RANDOM_SEED) papers = [generate_paper(i) for i in range(NUM_PAPERS)] Each paper gets between two and five authors, drawn from pools of first and last names. We add two things the Weaviate version didn't have: author nodes and citation edges. Citation edges are generated with three rules that keep the graph plausible: A paper can only cite papers with a lower paper_id and a publication year no later than its own, so nothing cites the futureCitations are biased toward papers in the same or adjacent fields, using a weighted random selectionEach paper makes between three and eight citation attempts, and duplicates are removed Python weights = [ 3 if c["field"] == paper["field"] else 2 if c["field"] in adjacent else 1 for c in candidates ] This gives us 937 citation edges that cluster by field, which makes the traversal results more interesting. Because the relationships are synthetic, the traversal results demonstrate the mechanics of graph-based discovery rather than real bibliometric relationships. Loading Into Neo4j Before loading, the notebook clears any existing data — dropping constraints and indexes first, then deleting nodes in batches of 1,000 to avoid transaction timeouts on Aura. It then creates uniqueness constraints on Paper.paper_id and Author.name, which keep authors unique and make the lookups in the loading queries fast: Cypher CREATE CONSTRAINT paper_id IF NOT EXISTS FOR (p:Paper) REQUIRE p.paper_id IS UNIQUE; CREATE CONSTRAINT author_name IF NOT EXISTS FOR (a:Author) REQUIRE a.name IS UNIQUE; We'll load the data in three passes using UNWIND to batch the inserts. First, the papers: Cypher UNWIND $records AS r CREATE (p:Paper { paper_id: r.paper_id, title: r.title, year: r.year, field: r.field, venue: r.venue, citation_count: r.citation_count, abstract: r.abstract }) For authors, we use MERGE rather than CREATE, since the same author can appear on multiple papers and we want a single node per author: Cypher UNWIND $records AS r MERGE (a:Author {name: r.author_name}) WITH a, r MATCH (p:Paper {paper_id: r.paper_id}) CREATE (a)-[:AUTHORED]->(p) Finally, the citations: Cypher UNWIND $citations AS c MATCH (from:Paper {paper_id: c.from_id}) MATCH (to:Paper {paper_id: c.to_id}) CREATE (from)-[:CITES]->(to) After loading, the graph has 200 Paper nodes, 215 unique Author nodes, 691 AUTHORED relationships and 937 CITES relationships. One caveat about the authors: names are drawn from 15 first names and 15 last names, so there are only 225 possible combinations. "The same author" here just means "the same generated name". Real bibliographic data needs author disambiguation, which is a hard problem in its own right. Creating the Full-Text Index We'll create a single Lucene index across both the title and abstract properties of Paper nodes: Cypher CREATE FULLTEXT INDEX paper_fulltext IF NOT EXISTS FOR (p:Paper) ON EACH [p.title, p.abstract] OPTIONS { indexConfig: { `fulltext.analyzer`: 'standard' } } A single index across multiple properties means a single query scores both together — no merging of separate result sets in application code. The OPTIONS line chooses the analyzer, the component that splits text into searchable terms. By default Neo4j uses standard-no-stop-words, which indexes every word, including "and", "for," and "the." That makes long natural-language queries noisy, because a paper can pick up a score just by containing "and." The standard analyzer drops common English stop words but doesn't stem, so fuzzy matching, which we'll use shortly, still works on the original terms. The index is created asynchronously, so we'll poll until it's ONLINE before querying: Cypher SHOW INDEXES YIELD name, state WHERE name = 'paper_fulltext' RETURN state Basic Full-Text Search We'll query the index with the SEARCH clause, which returns matching nodes alongside a Lucene BM25 relevance score: Cypher CYPHER 25 MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC Two details are worth knowing. First, LIMIT is part of the clause: it sets how many top-scoring matches the index returns. Second, the CYPHER 25 prefix selects the Cypher version that provides SEARCH. We show it once here and omit it from the snippets that follow, although the notebook includes it in every query. The notebook builds this text with a small helper that fills in the limit, so every example below reuses the same clause. The score is based on BM25, which considers term frequency, inverse document frequency, and document length. A paper whose title and abstract both mention "transformer models" will score higher than one where the term appears only once in the abstract. Python search_papers("transformer models for natural language understanding") Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Symposium on Knowledge Discovery and Data Mining | 2022 | Citations: 18 Score: 9.926 Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 9.741 ... The generated titles aren't unique, so the same title can appear twice. They are different papers, which the year and venue make clear. Special Characters in User Input Lucene treats characters such as + - ! ( ) { } [ ] ^ " ~ * ? : \ / as query syntax. If the query text comes from a user, an unbalanced bracket or a stray colon will raise a parse error or quietly change the meaning of the query. Unless we deliberately want operators like phrases, ~ and boosts to work, we should escape the text first: Python def escape_lucene(text: str) -> str: """Escape Lucene query-syntax characters in raw user input.""" return re.sub(r'([+\-!(){}\[\]^"~*?:\\/&|])', r'\\\1', text) search_papers("stream processing (survey") # raises a parse error search_papers(escape_lucene("stream processing (survey")) # works The unescaped query fails with a message that Neo4j could not parse the full-text search string. The escaped one returns the stream processing papers. Boosting Titles Over Abstracts Because the index covers both properties, we can address them by name in the query and weight them. Here a match in the title counts three times as much as a match in the abstract: Python search_papers("title:transformer^3 OR abstract:transformer", top_k=5) The Index Stays in Sync One of the costs of a separate search service is the pipeline that keeps it in sync with the primary database. Here the full-text index is updated as part of the same transaction that writes the node. To see it, we create a paper containing a unique word, search for it straight away, and then delete it: Python with driver.session() as session: session.run(""" CREATE (:Paper {paper_id: 9999, title: 'Quokka Navigation for Robotics', abstract: 'A throwaway paper used to show the index updates on commit.', year: 2024, field: 'Robotics', venue: 'Demo', citation_count: 0}) """) search_papers("quokka", top_k=1) # found immediately with driver.session() as session: session.run("MATCH (p:Paper {paper_id: 9999}) DETACH DELETE p") search_papers("quokka", top_k=1) # gone again The first search returns the new paper (score 2.815) and the second returns nothing. Phrase Search Wrapping a query in double quotes tells Lucene to treat it as a phrase — the words must appear together in that order: Python search_papers('"graph neural networks"', top_k=3) Without quotes, Lucene scores documents that contain any of the words, awarding higher scores for more matches. With quotes, every result must contain the exact phrase. Score Comparison: Keyword vs. Phrase Search The chart below (Figure 1) runs the same query twice, once as keywords and once as a phrase, and plots the scores of the top eight results of each. A bar of zero means the paper is outside that mode's top eight, not that it fails to match. Figure 1. Keyword vs. Phrase Search. The result is worth reading carefully. In this dataset, every paper in the Graph Neural Networks field contains the exact phrase, because the field name appears in its title and abstract. So both queries match essentially the same papers and their scores are close: 6.1 to 6.8 for keywords and 5.4 to 5.65 for the phrase. What changes is which papers make the top eight. Only three of the eight appear in both lists, and the top result differs: keyword search ranks "Efficient Graph Attention Networks for Graph Neural Networks" first (6.774), while phrase search ranks "Efficient Link Prediction for Graph Neural Networks" first (5.649). A plausible reason is that keyword scoring rewards papers that repeat the individual words, such as "graph" and "networks" in a title that also mentions graph attention networks, while phrase scoring only counts the phrase itself. Raw scores aren't directly comparable across the two query forms, so membership and order are the things to compare. On noisier text, where the words in a query don't usually sit together, the phrase search would also drop papers entirely, which is the more familiar reason to use it. Fuzzy Search Appending ~ to a term enables Lucene's fuzzy matching, which finds terms within a given edit distance. A misspelled query like tranformer~ correctly matches transformer: Python search_papers("tranformer~", top_k=3) Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 3.015 ... Without the ~, the misspelled query returns nothing. With it, Lucene finds the closest matching terms and scores them. This is useful for any field where exact spelling can't be guaranteed, such as technical terms or partially remembered titles. Filtered Search We can combine the full-text index with filters on structured node properties. There is one thing to be aware of: for full-text indexes, SEARCH doesn't allow a WHERE inside the clause. Filtering is a second step. The index first returns its top-scoring candidates, then a WHERE after it filters them by field, year or citation count: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 200 ) SCORE AS score WITH p, score WHERE p.field = $field AND p.year >= $min_year AND p.citation_count >= $min_citations RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC LIMIT $top_k The two limits do different jobs. The LIMIT 200 inside SEARCH sets the size of the candidate pool, here every paper in the collection, and the final LIMIT $top_k trims the filtered results. If we asked the index for only the top five and filtered afterward, a selective filter could leave us with nothing. On a large collection, we would choose a pool big enough for our filters to leave enough results, without asking the index to score everything. These are the same three filtered queries used in the Weaviate chapter. Weaviate ran them as hybrid searches, combining vectors and keywords, whereas this is keyword-only BM25, so we should expect overlap in the results rather than identical rankings: Python # NLP papers from 2020 onwards with at least 100 citations search_papers_filtered( "attention mechanisms and transformer architectures", field = "Natural Language Processing", min_year = 2020, min_citations = 100, ) # Recent computer vision papers search_papers_filtered( "object detection in real time", field = "Computer Vision", min_year = 2022, ) # Highly cited papers across all fields search_papers_filtered( "self-supervised learning for representation learning", min_citations = 200, ) The first query returns two papers (scores 3.350 and 3.154) and the second returns one (7.250). The third returns five: the three self-supervised learning papers score between 10.273 and 9.537, then the scores drop to 3.543 for the fourth paper, which matches only more generic terms such as "learning". With the default analyzer, an earlier run of the first query also returned papers scoring below 0.1, apparently matching only on common words such as "and". With stop words removed, that tail no longer appears. Search Then Traverse: The Graph Layer Full-text search answers "Which documents match?" Graph traversal answers "What is connected to those matches?" This section is where the two work together. We'll use the Lucene results as an entry point into the graph, then traverse CITES and AUTHORED relationships to discover related content. The pattern is always the same: SEARCH finds the initial set, then Cypher traverses from there. In a synthetic graph, some papers have few connections, so the notebook runs ten candidate queries and measures each top result's neighborhood: how many papers it cites, how many cite it, and how strongly it is co-cited with others. It picks the richest as the demonstration query. That query is "protein structure prediction", whose top result, "Efficient Protein Structure Prediction for Bioinformatics" (score 12.926), cites 3 papers and is cited by 17. What Does a Top Result Cite? Starting from the highest-scoring paper, we'll follow outgoing CITES edges to see what it builds on. Note that citation_count is a property generated independently for each paper — it isn't derived from the CITES relationships in this graph. We use it here as a proxy for importance when ordering results. Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(cited:Paper) RETURN p.title AS source_title, score AS source_score, cited.title, cited.field, cited.year, cited.citation_count ORDER BY cited.citation_count DESC LIMIT $top_k Example output: Plain Text Drug Discovery for Bioinformatics: A Scalable Approach Bioinformatics | 2016 | Citations: 483 Improving Bioinformatics with Drug Discovery Bioinformatics | 2016 | Citations: 467 Cryptographic Protocols in Cybersecurity: Challenges and Opportunities Cybersecurity | 2016 | Citations: 283 What Cites a Top Result? Traversing in the opposite direction — incoming CITES edges — finds papers that build on a highly relevant result: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) RETURN p.title AS source_title, score AS source_score, citing.title, citing.field, citing.year, citing.citation_count ORDER BY citing.year DESC LIMIT $top_k The most recent citing papers include "A Unified Framework for Reinforcement Learning using Hierarchical Rl" (2024), "Efficient Malware Classification for Cybersecurity" (2024), and "Meta-Learning for Machine Learning: A Scalable Approach" (2024). Citations reach across fields because the generator only biases them toward related fields; it doesn't confine them. Visualizing the Citation Network We can visualize the citation neighborhood of the top result as an interactive graph with neo4j-viz. In Figure 2, the central red node is the top-scoring paper, blue nodes are papers it cites, and green nodes are papers that cite it. Arrows point from the citing paper to the cited paper, and node size reflects citation count. In the notebook, hovering over a node shows the full title, year, and citation count. Python # One Cypher query returns the top result with the papers it cites and the papers that cite it nodes, relationships = {}, [] nodes[top["id"]] = Node(id=top["id"], caption=short(top["title"]), properties={"role": "Top result", "citations": top["citations"]}) for p in top["cited"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cites", "citations": p["citations"]}) relationships.append(Relationship(source=top["id"], target=p["id"])) for p in top["citing"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cited by", "citations": p["citations"]}) relationships.append(Relationship(source=p["id"], target=top["id"])) VG = VisualizationGraph(nodes=list(nodes.values()), relationships=relationships) VG.color_nodes(property="role", colors={ "Top result": "#EF553B", "Cites": "#636EFA", "Cited by": "#00CC96"}) VG.resize_nodes(property="citations", node_radius_min_max=(14, 40)) VG.render(height="520px") Figure 2. Citation Network. This kind of neighborhood visualization isn't something we'd get from a search engine, as it requires the graph. Find Other Work by the Same Authors From the top result, we'll traverse AUTHORED edges to the authors, then follow their other AUTHORED edges to find their other papers: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (a:Author)-[:AUTHORED]->(p) MATCH (a)-[:AUTHORED]->(other:Paper) WHERE other <> p RETURN p.title AS source_title, score AS source_score, a.name AS author, other.title, other.field, other.year, other.citation_count ORDER BY other.citation_count DESC LIMIT $top_k The highest-cited results are papers by Chen Singh and Lena Li, two of the authors of the top result, including "Improving Computer Vision with Generative Adversarial Networks" (475 citations) and "Efficient Multi-Agent Systems for Reinforcement Learning" (433). This is the kind of discovery path that's natural in a graph but would typically require additional application-side queries or data modeling outside the search index. Co-Citation Discovery Two papers are co-cited when a third paper cites both of them. Frequently co-cited papers can indicate related research topics, shared foundations, or intellectual proximity — a relationship that doesn't appear in the papers themselves, only in how others use them: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) MATCH (citing)-[:CITES]->(cocited:Paper) WHERE cocited <> p WITH p, score, cocited, count(citing) AS shared_citations RETURN p.title AS source_title, score AS source_score, cocited.title, cocited.field, cocited.year, shared_citations ORDER BY shared_citations DESC LIMIT $top_k For our top result, the most co-cited paper is "Drug Discovery for Bioinformatics: A Scalable Approach" with 6 shared citing papers, followed by "Rethinking Robotics with Imitation Learning" with 5, and three papers with 4 each. The Robotics paper is a reminder that the generator's field bias is only a bias. Co-citation analysis is a classical technique in bibliometrics, and here it is a single Cypher query. Traversing From More Than One Result So far we've started from the single best hit. The same pattern works from the top N results, and aggregation then surfaces something no single result shows: the papers that several of the best matches have in common. Here we ask which papers are cited by the most of the top five results, a rough proxy for foundational work on the topic: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(ref:Paper) RETURN ref.title AS title, ref.field AS field, ref.year AS year, ref.citation_count AS citations, count(DISTINCT p) AS cited_by_hits ORDER BY cited_by_hits DESC, citations DESC LIMIT $top_k Two papers, "Drug Discovery for Bioinformatics: A Scalable Approach" and "Improving Bioinformatics with Drug Discovery," are each cited by two of the five best matches, ahead of papers cited by only one. A standalone search engine would return the five matches and stop there. When to Use This Neo4j's full-text search is worth reaching for when our data are already in a graph, and we want to add keyword search without operating a separate service. The setup is minimal: one Cypher command creates the index, one clause queries it, and we immediately get phrase matching, fuzzy matching, field boosting, and BM25 relevance scoring. Because the index is updated in the same transaction as the data, there is no synchronization pipeline to build or monitor. The search-then-traverse pattern is the real differentiator. If our data have relationships, such as citations, authorship, product categories, or social connections, and we want search to serve as an entry point into those relationships, Neo4j gives us both in one place. It is not the right choice for every workload. A dedicated search engine offers capabilities that weren't part of what we used here, such as faceted aggregations and result highlighting, and it lets us scale search independently of the database. If search traffic is large, if it needs different hardware from the graph workload, or if the data have no meaningful relationships, a separate service may be the better fit. The SEARCH behavior described above also matters at scale: the index returns a bounded candidate set, and filters apply afterward, so queries with very selective filters need a candidate pool sized for them. For purely semantic workloads where we want vector similarity rather than keyword matching, Neo4j also supports vector indexes, which we can combine with full-text search in the same query. But for the common case where keyword search and graph traversal belong together, the built-in Lucene integration is often all we need. Summary We've seen how to create a multi-property full-text index in Neo4j, query it with the SEARCH clause using keyword, phrase, and fuzzy search, escape user input and boost fields, combine the results with structured property filters, and use them as the entry point for graph traversals that follow citations, authorship, co-citation, and multiple matches at once. Further examples of Neo4j's multi-model capabilities can be found in the free online book Neo4j Beyond the Graph. The full source code is available on GitHub.

By Akmal Chaudhri

CORE

Constrained Decoding Models Explained: Encoders, Decoders and When to Use Them

Have you ever asked an LLM for JSON and got back "Sure! Here is your JSON:" followed by the JSON? Your parser fails, your pipeline stops, and someone adds a retry loop. I have been there. For a long time, the fix was a polite prompt. "Reply with valid JSON only. No extra words." It works most days. On the other days, it breaks. There is a better fix, and it does not rely on the model behaving well. It is called constrained decoding. In this article, you will learn: What encoders and decoders are, and when to use eachHow constrained decoding works, in plain wordsWhich tools do it todayWhat research says about its strengths and limitsHow it compares with a plain LLM, and with new decision models like Jev and LayaHow to use it safely inside an agent A quick note on the name. In this article, CDM means constrained decoding model: a decoder LLM run together with a constraint engine. CDM is the label I use for this setup. It is not a product name. Encoder and Decoder in Plain Words Almost every modern language model is built from blocks called transformers. What changes is how the blocks are arranged. A survey of language models for software work describes three layouts: BERT as encoder-only, LLaMA as decoder-only, and CodeT5+ as encoder plus decoder (arXiv 2408.02479). Encoder and Decoder Here is the easy way to remember it. An encoder is a reader. It looks at the whole message at once, and every word can see the words on both sides. The output is understanding, a set of numbers that capture meaning. That is why encoders are good at jobs like classifying, routing, searching, and filtering. Researchers who built small safety checkers note that encoder models are efficient at narrow jobs such as classification, name tagging, and sentiment (arXiv 2410.08442). A decoder is a writer. It produces one small piece of text, called a token, at a time. Each new piece depends on what it already wrote. Chat models like Llama work this way. Decoders became popular because they are flexible: almost any task can be turned into "write the answer." An encoder plus decoder reads one thing and writes another, like a translation or a summary. T5 and CodeT5+ are examples. If you need toUseWhyLabel, filter or route textEncoder, or a decision model built on oneReads once, small, cheapSearch by meaningEncoderProduces meaning numbersWrite, explain, plan or codeDecoder (LLM)Generates new textMake a decoder answer in a fixed shapeDecoder plus constrained decodingKeeps the writer inside the rulesExact lookups, math or datesPlain codeFast and easy to check A CDM sits in the fourth row. Jev and Laya are closer to the encoder idea. Laya, for example, is built on a ModernBERT encoder with a decision layer on top (Laya model card). What Is Constrained Decoding? A decoder picks its next token by scoring every token it knows. Constrained decoding adds a step. Before the model picks, a rule engine blocks the tokens that would break your rules. The XGrammar paper describes this as restricting the available tokens at each step (arXiv 2411.15100). How CDM works The loop has five steps: Score: the model scores every possible next token.Check: the rule engine works out which tokens are legal at this spot.Mask: illegal tokens are blocked.Pick: the model chooses from the legal tokens only.Repeat: the token is added, and the loop runs again until the answer is complete. The values in the close-up part of the picture are made up to show the idea. What matters is the shape of the process: blocked tokens disappear, and the remaining choices share the probability. What Kinds of Rules Can You Use? The common ones are a JSON schema, a regular expression, a fixed list of choices, or a grammar. vLLM, a popular serving engine, lists all four (vLLM docs). The Outlines library shows how simple this can be. For a yes-or-no answer, you use a Literal["Yes", "No"] type. For a sentiment label, you use Literal["Positive", "Negative", "Neutral"] (Outlines on GitHub). The model can no longer return anything else. That means a choice constraint turns a general LLM into a classifier that can never invent a label. One Thing to Remember Masking controls what the model is allowed to write. It does not change what the model understands. Everything in the research section below comes back to this point. Tools You Can Use Today The tools fall into three groups: engines that compute the mask, servers that run them, and hosted APIs that hide them. ToolWhat it isStart hereOutlinesA Python library that turns types, schemas or regex into a state machine that masks tokensPaper, GitHubXGrammarAn engine that prepares most of the mask ahead of time to keep overhead low. XGrammar-2 targets rules that change at runtimePaper, XGrammar-2llguidance (Guidance)A fast engine written in Rust that handles full grammars and JSON schemasGitHubllama.cpp grammarsGrammar support for local models, using GBNFGrammar guidevLLMA serving engine that uses XGrammar or Guidance as backendsDocsNVIDIA NIMContainerized serving with JSON schema, regex, grammar, and choice supportDocsOpenAI Structured OutputsA hosted option that enforces a JSON schema in strict modeAnnouncementClaude structured outputsA hosted option that compiles your JSON schema into a grammarDocs Two things to watch for. Hosted APIs support a part of JSON Schema, not all of it. For example, Google's documentation for Claude on Vertex AI says some constraints, such as numeric and string-length rules, are not supported (Vertex AI docs). If a rule is not enforced by the mask, check it in your own code. Your backend can change under you. vLLM's configuration notes say the automatic backend choice can change between releases (vLLM config). If you need repeatable behavior, set the backend yourself and pin the version. What the Research Says I read the main papers so you do not have to. Several 2026 papers are preprints, which means they have not been peer-reviewed yet. Here are the five points that matter most. 1. A valid format becomes a guarantee. A 2026 preprint on small open models found that constrained decoding removed format failures such as wrong types and missing fields (arXiv 2609.23742). If a program must parse the output, this is the main reason to use it. 2. Valid is not the same as right. The same paper found that the content can still be wrong or thin. A model that skips a required step still passes the schema if the schema allows it. The authors describe rescued answers that look complete but carry little meaning. The short version: constraints fix form, not understanding. 3. Real schemas are harder than demos. JSONSchemaBench tested several engines, including Guidance, Outlines, llama.cpp, XGrammar, OpenAI, and Gemini, on a large set of real-world schemas. Engines differ in which schema features they support (arXiv 2501.10868). That benchmark is from early in this wave of tools, and engines have improved, so test your own schemas on the engine you plan to use. 4. Strict formats can hurt reasoning. The paper "Let Me Speak Freely?" found that format restrictions can weaken reasoning, and stricter formats can do more harm. It also found that constraints can help classification (arXiv 2408.02442). In reasoning tasks, JSON mode often did not keep the order of "think first, answer second." So if the model must think, let it think first. 5. Masking can nudge what the model prefers. Research on grammar-aligned decoding shows that blocking tokens can distort the model's own probabilities, so some outputs are valid but lower in quality (arXiv 2405.21047). A 2026 preprint on draft-conditioned decoding targets a related problem, where masking pushes the model down a path that is valid but wrong (arXiv 2603.03305). FindingWhat it meansWhat to doFormat failures disappearOutput always fits the rulesUse it wherever a program parses the outputValid can still be wrongMeaning is not checkedTest the content, not only the formatEngines support different schema features"Supports JSON schema" is not one yes or noTest your schemas on your engineStrict formats can hurt reasoningEarly constraints take away thinking roomLet the model think, then constrain the final answerMasking can bend probabilitiesOpen-ended content can lose qualityWatch quality on long or creative output CDM vs. LLM vs. Jev and Laya Now we can compare the designs. All three can give you a valid answer from a fixed list. They get there in different ways. LLMCDMDecision modelCore designDecoderDecoder plus maskEncoder plus decision layerHow it answersWrites token by tokenWrites token by token, legal tokens onlyScores every allowed answer in one passOutput always validNoYes, inside the rules the engine supportsYes, it picks from your listProbabilitiesNot by defaultToken level, not trained to be calibratedTrained to be calibratedCan write or explainYesYes, inside the shapeNoOpen shapes like SQL or nested JSONYesYesNo, typed answers onlyMain riskBroken formatValid but wrong, reasoning lossConfident wrong picks, forced answers What Are Jev and Laya? They are decision models, sometimes called System One models, after the fast, gut-level mode of thinking. You give them an input, a question, and the allowed answers. They return probabilities for each answer in one pass. They do not write text. Jev comes from TypeSafe AI and is offered as a hosted service. Its evaluation paper describes typed answers: a choice, a position on a rubric, or a true-or-false probability (arXiv 2609.37647). Laya comes from Convai Innovations and is open, so you can host and fine-tune it yourself. Its model card explains that it is trained so that its probabilities are honest, a method it calls RLCD (Laya model card). I covered both in Classification Never Left. It Just Got a New Home in LLMs. How Do They Differ From a CDM? Think of a quiz with four answers. A CDM is a writer who is only allowed to write one of the four answers. It still goes through the writing loop, and it can also handle open-ended formats like SQL. A decision model reads the question once and hands you a score for each of the four answers. It cannot write anything else, and it is built for calibrated scores. ScaleDown's CDMs, covered in the next section, are a third variant. They are small models trained for one task, so the limit on what they can return is built in during training instead of added by a mask at run time. What About a Head-to-Head Test? I looked for a direct comparison of a constrained LLM against Jev or Laya on the same data and did not find one. So I will not name a winner. If you are choosing between them, test both on a few hundred of your own inputs. One weakness is shared. A closed list forces an answer. If a message fits none of your categories, both a decision model and a constrained LLM will still pick one. This follows from how the list works, and it is why you should always add an "other" option and test it on purpose. When to Use What When to use what - LLM vs CDM vs Decision Model Walk the questions from left to right and stop at the first yes. JobBest fitWhyTool call arguments for an agentCDMStrict JSON that must parseSQL, regex or a small languageCDM with a grammarOpen shape with clear rulesPulling fields out of text into a schemaCDMType and structure errors disappearTicket routing at high volumeDecision model, or a trained encoder classifierOne pass, cheap, probabilitiesUrgency score or yes or no flagDecision modelTyped scoresReasoning, then a structured answerLLM thinks freely, then a constrained last stepAvoids the reasoning lossSummaries, replies and explanationsLLMYou need wordsExact rules, lookups, math and datesPlain codeFast and easy to check Two rules of thumb: If the model must think, let it think first. Give it a free reasoning step, then constrain only the final answer. If you must use one schema, put the reasoning field before the answer field.If the answer is a pick, constrain it tightly. Picks are where constraints help most. How ScaleDown Brought CDMs Into Agents A recent post from LangChain shows where this idea is heading. Long-running agents spend a lot of effort on context management: summarizing old history, shrinking large tool results, and pulling the right values out of long inputs. Each of those is a small decision, and making them with a large LLM on every turn is slow and costly. ScaleDown's answer is a set of CDMs. Instead of writing any string one token at a time, each model is trained for one job and only returns answers that job allows, such as a label from your list or pieces of the input text. ScaleDown offers models to summarize, compress, extract, and classify, and the langchain-scaledown package wraps them as middleware for LangChain and Deep Agents. The big LLM keeps the planning and writing, and the CDMs handle the high-volume steps around it (LangChain blog). Its classify model will feel familiar if you have read about Jev. Both return a typed answer with a probability, and the post says ScaleDown also adds a short reason, which helps when you debug why an agent stopped or routed a task. The post also notes that ScaleDown is now available through the LangSmith LLM Gateway, so its calls are traced like your other model calls. One caution. This is a vendor-written post, and the speed and cost claims come from ScaleDown's own benchmarks, so test them on your own workload before you rely on them. Using a CDM Inside an Agent The two designs work well together. A CDM keeps the agent's tool call valid. A decision model, or a second choice constraint, decides whether that call is safe to run. CDM inside an agent Here is how it flows: A user or system sends a request.The LLM agent plans the next step and proposes a tool call.Constrained decoding makes sure the call matches the schema, so it always parses.A decision gate scores the call as allow, review, or block.Allowed calls run, unsure calls go to a person, and risky calls are rejected and logged. Three habits keep this honest: Test two things. Check that the output is valid, and check that it is right. They are not the same.Log every step. Keep the proposed call, the schema it matched, the gate's scores, and who approved it.Treat the gate as one control. Give the agent only the access it needs, and require a human for high-impact actions. I wrote more about these controls in AI-Era Cybersecurity, Part 2, and about the tools that sit beside them in Part 1. A Simple Checklist StepAction1List every place your system needs a fixed shape or a pick from a list2Move anything plain code can decide out of the model3Choose a rule type: choice, regex, JSON schema or grammar4Stay inside what your engine supports, and check the rest in code5Add an "other" or "unknown" option to every closed list6Put reasoning before the answer, or reason in a separate step7Pin the engine, backend and model version8Test format and meaning on real inputs, including ones that fit no category9Cache compiled schemas, or use an engine built for rules that change per request10Log decisions and recheck on a schedule Limits of This Article Some of the newest evidence is in preprints and small studies. Treat it as a signal, not a final answer.Hosted APIs do not publish their full mask engines, so I rely on their documentation.I found no same-data comparison between a constrained LLM and a decision model, so the comparison here rests on design differences and separate sources.Tools in this area change quickly. Check each official page before you decide. Related Articles From Me ArticleHow it connectsClassification Never Left. It Just Got a New Home in LLMsThe decision model side of this comparisonShipping Production-Grade AI Agents (Refcard #403)Guardrails and evaluation, where a CDM fitsA Field Guide to AI Agent FrameworksWhere structured output plugs into frameworksGraph Engineering: The Layer After Loop EngineeringA constrained choice can pick the next node in a graphAI-Era Cybersecurity, Part 2Agent identity, approval gates and measuring riskAI-Era Cybersecurity, Part 1The bug-fixing tools these gates sit besideHow AI Is Actually Changing SRE Tools, Part 2Alert triage is a pick-from-a-list problemFrom Prompts to Harnesses: How AI Engineering Has Grown UpThe harness around the model is where constraints live Conclusion A CDM is a decoder with guardrails. It gives you a promise that is easy to state and easy to test: the output will always fit the rules. It does not promise that the answer is right, and strict rules can cost you reasoning if you apply them too early. Decision models like Jev and Laya make a different trade. They give up writing in return for fast, one-pass scoring with probabilities built to be honest. They have their own risks, such as confident wrong picks and forced answers. So match the tool to the shape of the job. Constrain tightly when the output must parse, or the answer is a pick. Leave the model free when it needs to think. And test both the format and the meaning on your own data before you trust either one. If you try this in your own system, I would like to hear which part you tried first. Share it in the comments. References Papers Efficient Guided Generation for Large Language Models (Outlines)XGrammar: Flexible and Efficient Structured Generation EngineXGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMsJSONSchemaBenchLet Me Speak Freely? Format Restrictions and LLM PerformanceGrammar-Aligned DecodingThe Hidden Cost of Structured Generation: Draft-Conditioned Constrained DecodingConstrained Decoding Eliminates Structural Failures in Small LLMsFrom LLMs to LLM-based Agents for Software Engineering: A SurveyJurEE not Judges: Small, Specialized Encoder EnsemblesEvaluating and Benchmarking the System One Model Jev Tools and Documentation OutlinesXGrammarllguidancellama.cpp grammarsvLLM structured outputsNVIDIA NIM structured generationOpenAI Structured Outputs announcementClaude structured outputsClaude structured outputs on Vertex AI ScaleDown and LangChain Building Long-Running Agents with ScaleDown and Deep Agents, LangChain blogScaleDown Decision Models Laya model card, Hugging FaceJev: Decision Models on Trial, Cisco Blogs

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

5 Mistakes Companies Make When Working With Staff Engineers

Hiring a great Staff Engineer is challenging. Identifying candidates with the right mix of technical expertise, architectural vision, communication skills, and leadership experience demands considerable effort. However, success depends not only on hiring the right person but also on how you support them afterward. While Staff Engineering is not new, the role has become more prominent in the software industry, especially following the publication of Will Larson’s Staff Engineer: Leadership Beyond the Management Track in 2021. Since then, conversations about technical leadership outside traditional management have increased. However, many organizations that have adopted the Staff Engineer title still lack the organizational maturity to support the role effectively. The challenge is that Staff Engineering isn't just about hiring someone who writes better code or designs sophisticated architectures. It is focused on empowering technical leaders to influence people, decisions, and the organization as a whole. Without trust, autonomy, collaboration, and clear expectations, even experienced engineers may struggle to make a meaningful impact. In this article, we examine five common mistakes companies make when managing Staff Engineers, such as limiting their decision-making authority or undervaluing the time organizational change takes. We will also discuss why these mistakes happen and how organizations can create an environment that enables Staff Engineers to deliver the value they were hired to provide. Staff Engineering: A Role Older Than Software Staff Engineering originated outside the technology industry, with roots in the 19th-century British Royal Navy. There, Staff Engineers were engineering officers responsible for specialized technical tasks. As steam-powered ships grew more complex, the Navy relied on these experienced engineers to maintain critical machinery. These specialists often faced challenges related to recognition, authority, and their relationship with traditional military leadership. In the software industry, Staff Engineering has become a technical leadership path beyond Senior Engineer. This role enables experienced professionals to expand their influence without moving into management. Staff Engineers guide architectural decisions, set technical direction, mentor engineers, and coordinate efforts across teams. The role gained visibility through companies with technical career ladders and publications like Will Larson’s Staff Engineer: Leadership Beyond the Management Track (2021). As distributed systems, cloud computing, and large engineering organizations have grown more complex, the Staff Engineer role has become increasingly important. However, adopting the title is not the same as using it effectively. Many organizations acknowledge the need for Staff Engineers but may not fully understand how to empower them. Notably, some challenges faced by technical specialists over a century ago remain relevant today. Mistake #1: Hiring Expertise Without Trusting It Consider a football club that invests millions to hire top players, only to limit their strengths, question their decisions, and dictate their play. When results disappoint, the club blames the investment. This contradiction is common in software engineering organizations. Companies invest heavily in Staff Engineers for their experience, architectural knowledge, and problem-solving skills. Yet these engineers often find their expertise isn't trusted. Their recommendations face excessive approval requirements; they are excluded from key discussions, and established practices go unchallenged, even when improvements are needed. The contradiction is clear: hiring for expertise is ineffective if the organization does not use it. Trust does not require accepting every recommendation or granting unlimited authority. Technical decisions should be constructively challenged, evidence-based, and aligned with business goals. However, a healthy debate differs significantly from a culture where experienced engineers must constantly justify their value. Organizations must recognize that hiring a Staff Engineer is an investment in technical judgment and leadership, not simply adding another developer. Without an environment that respects and constructively challenges their expertise, organizations will likely waste this investment. Mistake #2: Treating Staff Engineering as a Purely Technical Role A common misconception is that Staff Engineers add value mainly through coding, code reviews, or designing architectures. While technical expertise is essential, Staff Engineers must also influence people, align teams, and improve engineering decisions. When organizations face cross-team architectural challenges, Staff Engineers can identify problems and propose solutions. If excluded from key discussions, their impact is limited. Often, they are brought in after decisions are made, leaving them to address issues they had little opportunity to prevent. Staff Engineers should not attend every meeting or discussion. Their time is valuable, and unnecessary meetings reduce effectiveness. Involve them in conversations where they can influence technical direction, resolve disagreements, connect teams, or prevent costly mistakes. Their role is to shape decisions when their expertise is relevant. Mentoring and developing other engineers is also essential. A Staff Engineer who improves the technical capabilities of ten engineers generates far more organizational value than one focused solely on personal productivity. Key responsibilities include sharing knowledge, creating technical alignment, and helping teams become more independent. This is how technical leadership scales beyond individual contributions. Staff Engineers should be evaluated not only on their personal output, but also on the improvements they enable in others. Ignoring this risks reducing their role to that of expensive Senior Developers and limits their broader impact. Mistake #3: Giving Responsibility Without Decision-Making Authority Consider hiring a Staff Engineer to improve scalability, reduce technical debt, or modernize outdated architecture. The organization expects this professional to identify problems, propose solutions, and take responsibility for technical outcomes. However, the engineer often finds that most significant changes require approval from those who may not fully understand the technical implications. This creates a contradiction: the Staff Engineer is accountable for outcomes but lacks the authority to influence key decisions. This challenge intensifies when organizations exhibit status quo bias, preferring familiar practices over necessary change. A Staff Engineer may identify outdated architectures or inefficient practices, yet recommendations are sometimes dismissed not for technical or business reasons, but simply because “we have always done it this way.” Over time, this resistance stifles innovation and makes improvement more difficult. Autonomy does not mean Staff Engineers should make every decision alone. Architecture requires balancing trade-offs, business priorities, financial constraints, and collaboration. However, there is a key difference between collaborative decision-making and needing approval for every technical change. Staff Engineers need clear responsibilities, meaningful authority, and the ability to challenge existing approaches through evidence and constructive dialogue. Organizations that expect Staff Engineers to take ownership must also grant them the authority to do so. Without this, the role becomes a title with high expectations but limited influence. Mistake #4: Confusing Management Authority With Technical Leadership A common mistake is failing to define clear boundaries between management authority and technical leadership. Engineering Managers, Directors, and Staff Engineers have distinct but complementary roles. Management oversees people development, priorities, resources, and delivery, while Staff Engineers provide technical expertise for architecture, engineering practices, and long-term direction. Problems occur when organizations assume management hierarchy equates to superior technical judgment. For example, a Staff Engineer may identify a major architectural risk and recommend an alternative based on technical evidence. If a manager overrides this recommendation based on personal preference or hierarchy, the issue is not who is right or wrong. The real problem arises when technical decisions are made by organizational position rather than engineering evidence, trade-offs, or business objectives. Not every manager or director has the expertise to evaluate complex technical decisions, and this is not inherently problematic. Leadership does not require being the most technically knowledgeable. The challenge arises when leaders do not recognize their own limits or fail to create space for technical specialists to contribute. Effective managers know when to decide, when to facilitate, and when to delegate technical authority. Staff Engineers should also recognize that technical excellence does not grant authority over organizational priorities. A technically superior solution may not align with current business needs. Effective technical leadership means understanding these constraints and collaborating with management, not viewing it as an obstacle. Successful organizations do not pit management and technical leadership against each other. Instead, they create partnerships with clear responsibilities, mutual respect for expertise, and decisions informed by both technical and business perspectives. Without this partnership, Staff Engineers may spend more time on organizational politics than on solving engineering problems. Mistake #5: Expecting Immediate Results From Organizational Change Deploying an application may take minutes, but changing engineering culture can take months or years. Some companies hire Staff Engineers expecting immediate solutions to deep-rooted problems. This overlooks a key reality: organizational change requires time. When joining a new organization, a Staff Engineer must first understand its technical landscape, business priorities, processes, and relationships. They also need to learn the reasons behind past decisions, as choices that seem poor today may have been justified by earlier constraints. Without this context, proposing major changes can cause resistance or new issues. The first two or three months should be used to observe, listen, build relationships, identify risks, and understand the organization. Staff Engineers should still deliver value during this time. Small improvements, targeted recommendations, and early collaboration help build credibility and trust. However, early technical wins are different from achieving lasting organizational transformation. The challenge is greater in organizations with entrenched practices or cultures resistant to change. Technical improvements often require shifts in behavior, team roles, decision-making, and leadership expectations. A Staff Engineer may identify the right direction, but successful implementation depends on collaboration, communication, and others’ willingness to engage. Organizations should assess Staff Engineers based on both short-term contributions and long-term impact. Early progress may include identifying risks, improving collaboration, or setting technical direction. Over time, outcomes such as better maintainability, stronger teams, reduced complexity, and improved decision-making can emerge. There is no standard timeline for these results. Progress depends on organizational size, technical complexity, existing processes, leadership support, and culture. Expecting overnight transformation from a Staff Engineer reflects a misunderstanding of sustainable improvement. Conclusion Hiring a Staff Engineer is a significant investment, but the real challenge starts after hiring. Technical expertise alone cannot overcome a lack of trust, limited autonomy, exclusion from key discussions, or unrealistic expectations for rapid change. Staff Engineers can drive improvements, but their success depends on an environment that supports them. Ultimately, Staff Engineering is not just about hiring the right people; it is about creating an organization where they can make a difference. Companies that recognize this are more likely to achieve lasting impact. The key question is not whether you have Staff Engineers, but whether your organization is prepared to let them succeed.

By Otavio Santana

CORE

Your Event Storming Is Not Your Software Design

The Policy That Started the Argument We were looking at a purple policy between ListingSold and StopListingFee. Read aloud, the flow expressed a simple business expectation: when a listing is sold, the applicable recurring listing fee should stop. The interesting question was where that policy belonged. If Billing owns it, Billing receives the fact that a listing was sold and determines what that means for its fee model. Listing reports the sale without naming the operation Billing should perform. If Listing owns the coordination, it requests that Billing stop the fee, which means Listing knows that capability exists. A third option is a separate process manager that receives the sale event and coordinates the response. At the level of a simple event-policy-command sequence, these versions can look almost identical. Once ownership is made explicit, the differences matter. Billing depends on the sale event’s contract in the first version. The coordinating component depends on Billing’s command contract in the others. Billing still validates changes to its own fee model in every version. Responsibility for detecting an incomplete outcome, such as a fee that continues accruing after a sale, also needs an explicit owner. We were still discussing the business rule, but we were now deciding how software responsibilities and contracts would support it. This was already a discussion about where coupling belongs and how it is managed. We had moved into architectural reasoning without changing the business outcome we wanted. I later discussed this on LinkedIn and encountered the objection that EventStorming is not software design at all. I disagree, but the objection helped me distinguish two claims. EventStorming can host software design. It does not follow that the arrangement of a board is sufficient justification for the boundaries of the resulting system. This article is about that gap. EventStorming Already Includes Design The official EventStorming material explicitly recognizes Big Picture, Process Modeling, and Software Design formats. The patterns collection also addresses collaborative design activities. EventStorming is not limited to collecting domain events on a long wall of paper. The title therefore makes a narrower claim than it might suggest. “Your Event Storming” means the model your group produced, with its particular clusters, policies, and arrows. That model may contain exploratory ideas, agreed decisions, and reasons for those decisions. What cannot be assumed is that the notation alone captures all the reasoning needed to justify the resulting software structure. A policy between two notes expresses a relationship: when this happens, that response should follow. Unless ownership and the relevant assumptions are also made explicit, the sequence leaves important questions unanswered. Who owns the rule? Which contract does the response depend on? What happens if the response is rejected or never completes? A Software Design session can address those questions. It can expose invariants, clarify responsibilities, and help participants explore alternative models. The policy discussion above is an example of that work. The problem is automatic inference: treating a cluster as sufficient reason for a service, a purple note as a prescribed message handler, or a temporal sequence as an execution topology. This is related to the architectural decisions a domain model leaves open. Modeling can establish constraints and support architectural choices without uniquely determining how the system should be deployed or operated. Those choices can be discussed during the workshop. They still need reasons beyond the appearance of the board. Same Flow, Different Ownership For this example, Listing owns the listing lifecycle, and Billing owns the fee lifecycle. A sold listing should no longer accrue the applicable recurring fee. The precise rule, especially when stopping accrual becomes effective, still needs clarification. For now, consider three ways to allocate responsibility for the response. Alternative A: Billing owns the reaction. Listing records the sale and makes a ListingSold event available across an agreed boundary. Billing receives that fact, interprets it using its own rules, and invokes the appropriate fee behavior. Listing does not need to name StopListingFee. Billing now depends on the meaning and structure of the event it consumes. The event publisher needs an explicit compatibility policy, although that does not require exposing Listing’s internal domain event unchanged. Billing keeps the relationship between sale facts and fee behavior within its own model. That is useful when the rule belongs to Billing and evolves with its charging policies. The design still needs to account for delivery failures, processing failures, and missed reactions. Suppose Listing records a sale but the corresponding fee remains active. Who detects that discrepancy, and who resolves it? Billing may own both the reaction and reconciliation, but that responsibility must be explicit. Successful publication alone does not establish that the business outcome is complete. Alternative B: Listing owns coordination. Listing requests StopListingFee from Billing. Listing therefore knows a capability Billing exposes. Billing remains responsible for validating the request and protecting its own rules. The request could be synchronous or queued. Either way, the coordinating component needs to distinguish acceptance from completion and decide how to handle rejection, an unknown outcome, or a response that does not arrive within the expected time. This option is justified only if Listing’s responsibility genuinely includes coordinating the fee change. Emitting the original event is not enough to establish that ownership. Nor does the statement “the sale is complete only when the fee stops” settle the question by itself. That requirement establishes a business outcome; deciding which component should be accountable for achieving it is still design work. Listing-owned coordination becomes more plausible when Listing already owns the broader completion process and can manage it without absorbing Billing’s fee rules. If it starts calculating charges or reproducing Billing’s eligibility logic, the coordination boundary is leaking. Alternative C: A process manager owns coordination. A separate component tracks the cross-component outcome and requests work from the responsible participants. Listing still owns the listing lifecycle. Billing still owns fee behavior. The process manager owns the progress of the interaction. This helps when a sale triggers several participants, steps have deadlines, or unresolved cases need escalation. The manager can record that the sale was observed, that a fee-stop request was issued, and whether completion was confirmed. That visibility has a cost. The manager needs durable state, recovery behavior, and a policy for its own failures. Its commands may use the same capabilities Billing already exposes; participants do not necessarily need a new interface simply because a process manager is calling them. For a single reaction, introducing this machinery may be harder to justify than making Billing’s existing responsibility explicit. DesignMessage crossing the boundaryContract knowledgeAccountability for completionBilling reactsListingSold eventBilling understands the published sale contractBilling can own reaction and reconciliation; this must be agreedListing coordinatesStopListingFee command and an appropriate completion signalListing understands Billing’s exposed capabilityListing tracks the outcome if coordination belongs within its responsibilityProcess manager coordinatesSale event, fee-stop command, and completion or failure signalsThe manager understands the contracts required by the processThe manager tracks progress and escalation; its own recovery must also be designed These alternatives allocate responsibility differently. They do not divide neatly into a coupled design and a decoupled one. Contract knowledge is one form of dependency. Billing understands the sale event in Alternative A. Listing understands Billing’s command contract in Alternative B. The process manager understands the contracts needed to coordinate Alternative C. Runtime availability is a separate concern. A synchronous call may require the responder to be available immediately. Buffered communication can allow the sender to proceed while the receiver is unavailable, provided the system can retain and eventually process the pending work. That changes the timing of the dependency rather than removing it. Release coordination is another concern. Consumers and publishers need compatible contracts, but they do not necessarily need simultaneous deployments. The relevant question is how those contracts can evolve without silently changing the business meaning. Moving the policy changes some of these dependencies. It does not automatically reverse all of them. Nor does its position prescribe a transport: an event does not require Kafka, and a command does not require HTTP. What would favor one option? If Billing independently owns the rule connecting a sale to fee accrual, Alternative A is a strong candidate. If Listing already owns a broader completion process that legitimately includes requesting the fee change, Alternative B may fit. If the interaction needs its own lifecycle, deadlines, and recovery across several participants, Alternative C becomes easier to justify. The business flow is an input to that decision. Ownership, contract obligations, and failure handling determine whether the chosen design can support it. The Missing Question Is Often About Time The sequence ListingSold → StopListingFee leaves an important question unresolved: when should accrual stop? The policy note alone does not specify that. It could be the effective sale time, when the sale is recognized by the business. It could be when Listing records the change, or when Billing processes it. Those moments can differ. If the contract carries timestamps, their meanings need to be explicit. For example, a published event might distinguish the effective time from the time the sale was recorded: JSON { "type": "ListingSold", "eventId": "...", "listingId": "...", "effectiveAt": "...", "recordedAt": "..." } These fields make the distinction visible. They do not establish which timestamp Billing should trust or what it should do with it. If Billing calculates accrual using an authoritative effective time, delayed delivery need not change the amount ultimately owed. But the system may already have issued an invoice, collected payment, or shown an outdated balance. It still needs rules for adjustment and for what the customer sees before the correction completes. A timestamp helps preserve the meaning of the fact. It does not remove the consequences of learning about it late. Suppose the business accepts a delay before the balance reflects a sale, provided the final calculation uses the correct effective date. An asynchronous reaction may satisfy that requirement. Suppose instead that no fee-related charge may be issued after the sale is accepted. That requires a stronger coordination or validation mechanism than receiving a sale event later. Merely replacing an event with a queued command does not provide that guarantee either. Corrections expose further decisions. If the sale was recorded incorrectly, should accrual resume? From which date? If an older sale notification arrives after a correction, how does Billing recognize that it no longer represents the applicable state? These questions concern the business meaning of the update as much as its delivery. This is where business outcomes beyond aggregate boundaries become relevant. A process spanning Listing and Billing does not automatically require one aggregate or one distributed transaction. Billing can protect a local invariant, such as refusing to calculate accrual beyond its recorded stop date, while the broader process still fails to establish that date correctly. The cross-component outcome needs explicit completion and recovery semantics. Compensation is another business operation, with its own rules and potential failures, rather than a database rollback. Eventual consistency does not by itself establish that the relevant rule was interpreted correctly or that unresolved work will be detected and repaired. Durable publication, retries, and idempotent handling support the design. They cannot supply a business rule the participants have not agreed on. A Boundary Still Needs a Reason An event on the board is not automatically an integration event. ListingSold may initially describe a fact inside the Listing model. Exposing that fact to another component creates a contract that needs deliberate semantics and an evolution policy. The published representation may differ from the internal event. Likewise, a cluster of activities does not establish an aggregate merely because the notes appear together. The grouping must be examined against the rules that must remain consistent. An aggregate does not prescribe a microservice, and a bounded context does not prescribe an independent deployment. These distinctions are part of why Domain-Driven Design does not select an architectural style. The ownership alternatives also exist inside a monolith. Listing and Billing could be modules in one application, with Billing reacting to an in-process event or receiving an explicit request through a module interface. The questions about rule ownership and contract knowledge remain. The execution guarantees may be quite different, however. A synchronous handler participating in the same database transaction can fail together with the originating operation. A background handler in the same application may run after that transaction commits and require durable delivery and recovery. Sharing a deployment does not settle those choices, just as separating deployments does not settle ownership. An independent deployment therefore needs its own justification. Billing might have a materially different scaling profile, need isolation from listing traffic, or require a separate security boundary. Its team might need genuine release autonomy. A runtime constraint might be difficult to satisfy within the existing application. These are reasons to evaluate separation, rather than automatic instructions to introduce it. The costs are concrete. An interaction that was previously local may now time out or complete without the caller receiving confirmation. Contracts need to evolve while different versions coexist. Observability must follow a business outcome across processes. Data previously updated within one transaction may require a different consistency strategy. Independent deployment can reduce some coordination costs while introducing others. I would not impose a fixed sequence of aggregates first, contexts second, services last. Domain learning and architectural constraints inform each other. A scaling or compliance requirement can change where responsibilities execute or force us to reconsider a proposed partition. It does not automatically change the meaning of the business model. When a boundary changes, the team should be able to explain which concern required the change. What I Would Record Beside the Wall I would capture a short decision record while discussing each significant policy, rather than leave all the reasoning for someone to reconstruct after the workshop. First, who owns the rule connecting the fact to the response? In this example, does Billing own the rule relating a sale to fee accrual, or does a broader business process require that outcome? Billing owns the fee mutation in either case. The separate question is who initiates the response and tracks its completion. Next, which contract crosses the boundary? Who maintains it, and what changes can its consumers tolerate? A note on the wall becomes an integration commitment only when the participants agree on the meaning and obligations it carries. The record should also identify what must remain consistent, the acceptable delay between steps, and the effective time used when information arrives late. It should say who detects an incomplete outcome, who resolves it, and what counts as evidence of completion. Acceptance scenarios make those decisions harder to leave vague. The official Extract Acceptance Tests pattern connects design sessions with more precise expectations about behavior. For this example, I would include a sale received after an invoice was issued, a correction received before an older sale event, and a fee-stop request whose completion cannot immediately be confirmed. The expected result in each case needs to come from the business rule. The record should remain short enough to maintain. Its purpose is to preserve the reasons behind the model, including decisions deliberately left open. Six months later, another engineer should be able to distinguish an agreed constraint from a convenient assumption. Closing The discussion began with ListingSold, a purple policy, and StopListingFee. Asking who owned the policy led us to contracts, coordination, consistency, and recovery. That was software design taking place through EventStorming. The workshop can help a team make those decisions, and the board can record them. What the event sequence cannot provide on its own is the justification for a particular allocation of responsibilities or a particular deployment boundary. Billing might own the reaction. Listing might coordinate it. A process manager might track a broader outcome. Choosing among them requires knowing the rule, the timing requirements, and who is accountable when the expected response does not happen. The wall can show the flow. Your design must explain who owns each decision and what happens when the flow breaks. References and Further Reading Introducing EventStormingEventstorming PatternsExtract Acceptance TestsWhere Does Software Architect Fit in Domain-Driven Design?Modern Software Architecture (Part 2): Coupling Is Where Architecture Actually HappensDomain-Driven Design Beyond Aggregate BoundariesDomain-Driven Design Is Not an Architecture

By Alireza Rahmani Khalili

CORE

Managing Technical Debt in AI-Generated Code

AI-generated pull requests are moving from experimental workflow to routine software delivery. Coding agents can now plan changes, edit multiple files, run tests, and submit pull requests with limited human intervention. A 2025 study of 567 Claude Code pull requests across 157 open-source projects found that 83.8% were eventually merged, and about 55% of merged changes required no further modification. A broader 2026 study analyzed more than 302,000 verified AI-authored commits from 6,299 repositories. The practical question is therefore no longer whether generated code can enter production. The harder question is what happens after the merge, when readability problems, architectural drift, latent defects, and future maintenance costs become part of the codebase. Faster Generation Changes the Review Equation AI assistance does not automatically imply lower quality code. A GitHub randomized controlled study involving 202 experienced developers found that Copilot-assisted implementations performed better on the study's functional tests and received modestly higher ratings for readability, reliability, maintainability, and conciseness. That evidence matters because technical debt should not be treated as an unavoidable property of machine-generated code. Production repositories, however, create conditions that controlled coding tasks cannot fully reproduce. The 2026 large-scale study of AI-authored commits identified 484,366 introduced issues where 89.3% were code smells, and more than 15% of commits attributed to every assistant studied introduced at least one detectable issue. Most importantly, 22.7% of tracked AI-introduced issues were still present at the repository's latest analyzed revision. The same research also found that AI-authored commits can remove existing smells, showing that generated code can both create and repay debt. That distinction changes the engineering problem. The central risk is not generation itself but generation speed combined with weak validation. DORA's 2025 research describes AI as an amplifier that magnifies existing organizational strengths and weaknesses. Earlier DORA analysis also associated increased AI adoption with larger change batches, slower review, and weaker delivery stability. Faster code production therefore raises the value of disciplined review rather than reducing it. Debt Begins at the Merge Boundary Technical debt becomes an organizational liability when a change is accepted into a maintained branch. The model that produced the code cannot own an incident, negotiate a migration, preserve a service level objective, or explain an architectural compromise six months later. Accountability must therefore follow the repository and the component, not the mechanism that generated the diff. This makes the pull request author field an increasingly weak representation of ownership. A generated change may carry an agent identity, a bot account, or a human account that delegated most implementation work. None of those identities answers the operational question of which engineering group is responsible for the behavior after deployment. Repository ownership rules provide a stronger boundary. GitHub CODEOWNERS can automatically request reviewers for designated paths, and protected branch settings can require approval from a code owner before merge. Required status checks can additionally prevent merging until defined CI checks pass. A policy can remain simple: Plain Text /src/main/java/com/acme/payments/ @payments-platform /src/main/java/com/acme/identity/ @identity-platform The important property is not the syntax. The important property is that an AI-generated payment change still reaches the team accountable for payment correctness, while an identity change still reaches the team accountable for authentication and authorization. Provenance may influence review depth, but provenance should never replace domain ownership. Human Review Becomes More Valuable, Not Less Higher generation throughput can create pressure to automate review with another model. AI review is useful for summarization, repetitive checks, suspicious patterns, and localized suggestions, but available evidence does not support treating it as equivalent to accountable human review in every context. A 2026 study covering 278,790 review conversations across 300 mature open source projects found that human reviewers exchanged 11.8% more review rounds when handling AI-generated code than human-written code. Human reviewer suggestions were adopted at a much higher rate than AI-agent suggestions, 56.5% versus 16.6%, and more than half of unadopted AI suggestions were either incorrect or addressed through alternative fixes. Another study analyzing more than 22,000 comments from AI-based GitHub review actions found substantial variation in effectiveness and reported that concise, contextual, hunk-level feedback was more likely to produce code changes. Current GitHub documentation explicitly states that Copilot code review does not replace human code review. GitHub also supports configurations in which Copilot approvals can count toward merge requirements, making repository policy an important governance decision rather than a tooling default. For critical components, AI review is better positioned as an additional signal before accountable approval, not as the entity carrying final ownership. Quality Gates Must Follow Risk A generated pull request should not receive weaker controls because tests pass. Passing tests establish only the behavior covered by those tests. Technical debt often appears elsewhere as duplicated logic, boundary violations, unnecessary abstractions, missing failure handling, insecure defaults, excessive coupling, or code that satisfies the immediate task while conflicting with local design conventions. CI can convert some of those conventions from reviewer memory into executable policy. An AI-assisted label, for example, can trigger additional checks without requiring a separate delivery pipeline: YAML ai-change-gate: if: contains(github.event.pull_request.labels.*.name, 'ai-assisted') steps: - run: ./gradlew test integrationTest - run: ./gradlew check - run: ./scripts/verify-architecture.sh The architecture check can encode repository-specific constraints such as forbidden dependencies, package boundaries, or service-layer access rules. The AI label is only a routing signal where the underlying controls should be based on change risk. A generated documentation correction does not require the same scrutiny as an authorization change, while a human-written authorization change should receive the same high-risk checks. GitHub rulesets support required checks and required reviews as merge conditions, allowing these policies to be enforced rather than documented as optional guidance. The Useful Metric Is Rework Counting generated lines, completed pull requests, or acceptance rate measures production volume, not maintenance quality. A more meaningful evaluation connects AI-assisted changes to downstream engineering cost. Review rounds, escaped defects, rollbacks, post-merge rework, issue survival, architecture violations, and time spent modifying recently merged code reveal whether higher implementation throughput is creating future work. This framing also avoids a false binary between "AI code" and "human code." The large-scale debt study found both issue introduction and issue removal in AI-authored commits. The relevant objective is therefore not minimizing AI involvement, but minimizing unmanaged debt while preserving useful productivity gains. Ownership should remain attached to the team that accepts and operates the change, while automated gates provide evidence that acceptance criteria were actually satisfied. Conclusion AI-generated pull requests change who produces code, but they do not change who must maintain software after deployment. Technical debt cannot be delegated to a model, hidden behind a bot identity, or treated as the future responsibility of an unspecified reviewer. The durable ownership boundary is the engineering team responsible for the affected component, supported by required human approval, executable quality gates, repository ownership rules, and measurements that expose downstream rework. AI can accelerate implementation and can even remove existing debt, but merge authority must remain tied to accountable engineering ownership. When generation becomes nearly instantaneous, disciplined acceptance becomes the scarce resource, and that acceptance is where responsibility for technical debt belongs.

By Uthej Mopathi

CORE

Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go

Genkit flows as senro steps showed how to put a Genkit flow inside a senro step and get retries, caching, and an event stream for every model call. Every step in that article ran on your own machine. This one moves the steps into Kubernetes and adds a second kind of model. Gemini writes. jev decides. jev is a decision model: it does not generate text; it evaluates a state against typed questions and answers each one with a probability, in a few hundred milliseconds. Genkit Go 1.14 reaches it through the System One plugin, and a decision is one genkit.GenerateData call whose output type is the question set. That makes jev the right model to sit at the switch points of a pipeline. In this guide, it decides twice: which department's workflow runs for a support ticket, and whether the reply Gemini drafted is safe to send. Everything below runs against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5, and a local kind cluster on Kubernetes 1.37. The code is in genkit-k8s-flows-jev. Why a Decision Model Decides, and a Chat Model Writes Asking a chat model "which team should own this ticket, answer with one word" works until it does not: you get prose around the word, a department that is not on the list, no idea how sure the model was, and a second or two of latency for a decision that should take milliseconds. A decision model is built for exactly that call. Chat model as routerDecision model as routerAnswerText you parseA typed field: Choice[Dept], Noul, Score[L]OptionsIn the prompt, hoped for in the outputIn the Go type, enforced by the schemaConfidenceNone, or whatever the model says about itselfA probability per option, calibrated per model versionToolsCan be called by accidentRefused by the protocolLatencySecondsA few hundred millisecondsCost per decisionTokens for the state and the proseTokens for the state And the reverse is just as true. jev cannot write the reply to the customer, explain an outage, or summarize a brief. That is what gemini-3.8-flash is for. The pipeline needs both, and senro is what puts each call in its own step, on its own pod, with its own log, retry policy, and secret. What We Are Building A support-ticket pipeline. A raw email comes in; a reply goes out, or a brief goes to a human, or both. Two runs, because of one rule in senro: a When condition is evaluated once, at run start, against facts the run already has, never against something a step produces. So the first run understands the ticket and decides, and the decision becomes the run parameters of the second run, where senro.When(senro.ParamIs(...)) prunes every branch the verdict did not pick before anything starts. Pruned steps settle as skipped_condition and the run stays green. RunStepWhereModelWritestriageseedcoordinatornoneputs the raw email into the run's workspacetriageingestpodGeminiticket.json, a structured tickettriagetriagepodjevdecision.json: department, urgency, frustration, needs a humantriagecollectcoordinatornonecopies both files out of the runresolve<dept>/draftpodGeminireply.md, following that department's playbookresolve<dept>/verifypodjevverdict.json; a rejection fails the stepresolve<dept>/sendcoordinatornonethe reply, only if verify passedresolveescalate/briefpodGeminibrief.md for a human, when jev said so Inside a branch, jev is a guardrail: verify reads the draft against the playbook, and if it rejects the draft, the step fails and senro skips send. The second decision also determines the next step, this time without leaving the run. What You Need Go 1.27 or newer, Docker, kind, and kubectlGEMINI_API_KEY and TYPESAFE_API_KEY in your environmentthe senro CLI if you want the terminal UI (optional) Shell go get github.com/firebase/genkit/[email protected] go get github.com/xavidop/[email protected] go get github.com/xavidop/mamori mamori is the library senro uses to resolve secrets. It is how the two API keys reach the pods without ever being in a pod spec. Step 1: A Cluster senro Can Reach senro's Kubernetes executor never reads KUBECONFIG, ~/.kube/config or your current context. A kubeconfig typically holds a dozen contexts, most of them production, with whichever one you last used selected; a pipeline that inherited that would eventually deploy into the wrong cluster. Instead, the cluster comes from three environment variables, read at run start: VariableWhat it isSENRO_K8S_SERVERThe apiserver URLSENRO_K8S_CA_FILEThe cluster CA, as a PEM file. senro will not skip verificationSENRO_K8S_TOKEN_FILEA bearer token in a file (or a client certificate and key) So a local cluster needs four things: kind, a namespace, an identity with the permissions the executor documents, and those three files. The repo's scripts/cluster-up.sh does all of it; this is what it runs. Shell kind create cluster --name senro-ai --kubeconfig .kind/kubeconfig export KUBECONFIG=$PWD/.kind/kubeconfig kubectl apply -f k8s/rbac.yaml The RBAC is the exact set senro needs and nothing more: in the namespace, create, get and delete on pods, get on pods/log, create and get on pods/exec, and create, patch, delete on secrets. At cluster scope, list and get on nodes, which is how senro reads the platform it has to compile for. The get on pods/exec is the one to remember: senro uses the WebSocket exec protocol, which is a GET on that subresource, and a Role with only create fails every step that carries a workspace. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: senro-steps namespace: support rules: - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "delete"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create", "get"] - apiGroups: [""] resources: ["secrets"] verbs: ["create", "patch", "delete"] Then the three files, pulled out of the kubeconfig kind wrote and a short-lived ServiceAccount token: Shell kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.server}' > .kind/server kubectl config view --minify --raw \ -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > .kind/ca.crt kubectl -n support create token senro --duration=4h > .kind/token export SENRO_K8S_SERVER=$(cat .kind/server) export SENRO_K8S_CA_FILE=$PWD/.kind/ca.crt export SENRO_K8S_TOKEN_FILE=$PWD/.kind/token A file rather than SENRO_K8S_TOKEN inline, because an environment variable is visible in /proc/<pid>/environ and inherited by every child process the coordinator spawns. Step 2: The Flows Five flows, in one file. Nothing here knows about senro or Kubernetes. Gemini Turns an Email into a Ticket Go type Ticket struct { Customer string `json:"customer"` Company string `json:"company,omitempty"` Summary string `json:"summary"` Ask string `json:"ask"` Facts []string `json:"facts"` } f.Ingest = genkit.DefineFlow(g, "ingestTicket", func(ctx context.Context, raw string) (Ticket, error) { t, _, err := genkit.GenerateData[Ticket](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You turn raw support emails into a structured ticket. "+ "Facts are identifiers, amounts, dates and error codes quoted from the email, nothing inferred."), ai.WithPrompt("Email:\n\n%s", raw)) if err != nil { return Ticket{}, err } if t == nil { return Ticket{}, fmt.Errorf("ingest: the model returned no ticket") } return *t, nil }) jev Decides What the Ticket Is With the System One plugin, the output type is the question set. Each field is one question; its jsonschema_description tag is the instruction, and the options, levels, and criteria belong to Go types, so the same question means the same thing wherever it is asked. Go import systemonex "github.com/firebase/genkit/go/plugins/systemone/exp" type Dept string func (Dept) Criteria() map[Dept]string { return map[Dept]string{ "billing": "Charges, invoices, refunds, subscriptions, plan changes", "technical": "Bugs, errors, outages, latency, integrations, webhooks, API", "account": "Login, password, permissions, data export or deletion, privacy", "other": "None of the above", } } type Frustration int func (Frustration) Levels() []string { return []string{"Calm", "Annoyed", "Angry", "Furious"} } type Urgent struct{} func (Urgent) Criteria() (yes, no string) { return "Names a deadline, says now or today, or describes an outage in progress", "No time pressure is expressed, or the customer says there is no rush" } type Triage struct { Department systemonex.Choice[Dept] `json:"department" jsonschema_description:"Which team should own this ticket?"` IsUrgent systemonex.NoulOf[Urgent] `json:"is_urgent" jsonschema_description:"Does the ticket communicate time pressure?"` Frustration systemonex.Score[Frustration] `json:"frustration" jsonschema_description:"How frustrated is the customer?"` NeedsHuman systemonex.Noul `json:"needs_human" jsonschema_description:"Does resolving this need a human: a legal threat, a contract or SLA dispute, data loss, or a refund outside the playbook?"` } Four question kinds cover the protocol. Choice[T] picks one of T.Criteria() and gives a probability per option plus a Confidence. Score[L] rates on L.Levels() and answers Level() and Label(). Noul is "is this true?" as a probability, where 0.5 means the model could not tell. NoulOf[C] is the same with C.Criteria() saying what yes and no mean. The flow sends the ticket as a data part, so the state is a JSON object and a question can refer to a field by name. It then turns the four answers into a flat verdict, with thresholds that belong to the caller: Go type Decision struct { Department string `json:"department"` Confidence float64 `json:"confidence"` Urgent bool `json:"urgent"` Frustration string `json:"frustration"` Escalate bool `json:"escalate"` Model string `json:"model"` } f.Triage = genkit.DefineFlow(g, "triageTicket", func(ctx context.Context, t Ticket) (Decision, error) { out, resp, err := genkit.GenerateData[Triage](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithPromptParts(ai.NewDataPart(t))) if err != nil { return Decision{}, err } d := Decision{ Department: string(out.Department.Choice), Confidence: out.Department.Confidence, Urgent: out.IsUrgent.Probability >= 0.5, Frustration: out.Frustration.Label(), Model: systemonex.ResponseInfo(resp).Model, } // A low-confidence routing decision is not a routing decision. if d.Confidence < 0.5 { d.Department = "other" } d.Escalate = d.Department == "other" || out.NeedsHuman.Probability >= 0.6 || out.Frustration.Level() >= 3 return d, nil }) ResponseInfo(resp).Model records which jev version answered. Confidence thresholds are calibrated per version, so a verdict that does not say who produced it is a threshold waiting to drift. Gemini Drafts, jev Reviews The draft flow is an ordinary GenerateData[Reply] call with the playbook and the ticket in the prompt. One difference from the triage flow: the ticket goes to Gemini as JSON text. A data part is how a decision model takes an object state; a chat model takes a prompt. Go f.Draft = genkit.DefineFlow(g, "draftReply", func(ctx context.Context, in DraftInput) (Reply, error) { r, _, err := genkit.GenerateData[Reply](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You are a support agent for Acme Cloud. Write the reply the customer will receive. "+ "Follow the playbook strictly. Never state amounts, dates or identifiers that are not in the ticket. "+ "Plain text, under 180 words, sign as 'Acme Cloud Support'."), ai.WithPrompt("Playbook:\n%s\n\n%s\n\nTicket:\n%s", in.Playbook, urgency, asJSON(in.Ticket))) // ... }) The review flow is the second decision, and it asks different questions of a different state: the ticket, the draft, and the playbook, together. Go type Overpromise struct{} func (Overpromise) Criteria() (yes, no string) { return "Commits to a refund amount, a credit, a release date, a deadline, or a deletion the playbook does not allow", "Every commitment in the reply is one the playbook permits" } type Tone int func (Tone) Levels() []string { return []string{"Cold or dismissive", "Neutral", "Warm and specific"} } type Review struct { Addresses systemonex.Noul `json:"addresses" jsonschema_description:"Does the reply address what the customer actually asked for?"` Overpromises systemonex.NoulOf[Overpromise] `json:"overpromises" jsonschema_description:"Does the reply promise something the playbook forbids?"` Invents systemonex.Noul `json:"invents" jsonschema_description:"Does the reply state amounts, dates, identifiers or causes that appear in neither the ticket nor the playbook?"` Tone systemonex.Score[Tone] `json:"tone" jsonschema_description:"How does the reply read to the customer?"` } f.Verify = genkit.DefineFlow(g, "verifyReply", func(ctx context.Context, in VerifyInput) (Verdict, error) { out, _, err := genkit.GenerateData[Review](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has the customer's ticket, the reply an agent drafted, and the playbook the agent had to follow."), ai.WithPromptParts(ai.NewDataPart(in))) if err != nil { return Verdict{}, err } v := Verdict{Addresses: out.Addresses.Probability, Overpromises: out.Overpromises.Probability, Invents: out.Invents.Probability, Tone: out.Tone.Label()} if v.Addresses < 0.6 { v.Reasons = append(v.Reasons, "the reply does not address the customer's ask") } if v.Overpromises >= 0.5 { v.Reasons = append(v.Reasons, "the reply promises something the playbook forbids") } if v.Invents >= 0.5 { v.Reasons = append(v.Reasons, "the reply invents facts") } if out.Tone.Level() == 0 { v.Reasons = append(v.Reasons, "the tone is cold") } v.Approved = len(v.Reasons) == 0 return v, nil }) A system message is never part of the state with this plugin. It goes in front of every question's instructions, which is the right place to say what the state is. Step 3: Flows as Steps That Can Run in a Pod A senro func step is a registered Go function. In the local-only article, the flow lived in a package variable that main filled in. In a pod, that does not work, because senro runs a func step by sending your pipeline binary into the pod and re-entering it as a fresh process; main never gets to the line that initialized Genkit. So Genkit is initialized lazily, inside the step, from the secrets senro delivered to that pod: Go var ( engineOnce sync.Once engine *Flows engineErr error ) func flows(ctx senro.Ctx) (*Flows, error) { engineOnce.Do(func() { var plugins []api.Plugin gemini, err := readSecret(ctx, "Gemini") if err != nil { engineErr = err return } if gemini != "" { plugins = append(plugins, &googlegenai.GoogleAI{APIKey: gemini}) } typesafe, err := readSecret(ctx, "TypeSafe") if err != nil { engineErr = err return } if typesafe != "" { ts := systemonex.TypeSafe() ts.APIKey = typesafe plugins = append(plugins, ts) } g := genkit.Init(ctx, genkit.WithPlugins(plugins...)) engine = defineFlows(g) }) return engine, engineErr } readSecret reads the file that ctx.Secret(name) names, and that is the point: in the pod it is a 0400 file under /run/senro/secrets, projected from a short-lived Kubernetes Secret owned by the pod. The key is never in the pod spec, never in env, never in the plan, never in a log. That is also why the plugin fields are set explicitly rather than read from the environment. And senro.Ctx embeds context.Context, so it goes straight into genkit.Init and flow.Run. The steps themselves are short. Each one reads from the case workspace, runs a flow, and writes back: Go const workspace = "case" func init() { senro.RegisterFunc("ai/ingest", Ingest) senro.RegisterFunc("ai/triage", TriageStep) senro.RegisterFunc("ai/draft", Draft) senro.RegisterFunc("ai/verify", Verify) senro.RegisterFunc("ai/brief", Brief) senro.RegisterFunc("ai/collect", Collect) senro.RegisterFunc("ai/seed", Seed) senro.RegisterFunc("ai/send", Send) } func TriageStep(ctx senro.Ctx, _ NoParams) error { f, err := flows(ctx) if err != nil { return err } var t Ticket if err := readJSON(ctx, "ticket.json", &t); err != nil { return err } d, err := f.Triage.Run(ctx, t) if err != nil { return err } fmt.Fprintf(ctx.Stdout(), "%s decided: department=%s (confidence %.2f) urgent=%v frustration=%q escalate=%v\n", d.Model, d.Department, d.Confidence, d.Urgent, d.Frustration, d.Escalate) return writeJSON(ctx, "decision.json", d) } func Verify(ctx senro.Ctx, p PlaybookParams) error { // ...read ticket.json and reply.json, run f.Verify... if err := writeJSON(ctx, "verdict.json", v); err != nil { return err } if !v.Approved { return fmt.Errorf("reply rejected: %v", v.Reasons) } return nil } Verify returning an error is the guardrail: a func step that returns an error is a failed step, and every dependent of a failed step settles as skipped_upstream_failed. One more line in main, before flag parsing, so that senro's re-entry into the binary does not hit your flags: Go func main() { ctx := context.Background() if handled, err := senro.StepChild(ctx); handled { if err != nil { log.Fatal(err) } return } ticketPath := flag.String("ticket", "tickets/double-charge.txt", "raw ticket to process") // ... } Step 4: The Kubernetes Target One target, used by every AI workflow: Go import "github.com/xavidop/senro/executor/k8s" const stepImage = "docker.io/library/alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8" func pods() senro.ExecutorTarget { return k8s.Pod(stepImage, k8s.Namespace("support"), k8s.Resources( map[string]string{"cpu": "250m", "memory": "128Mi"}, map[string]string{"cpu": "1", "memory": "512Mi"}, )) } Three rules of this executor show up in those six lines: The image must be pinned to a digest and the namespace must be stated, or Build() refuses the pipeline. There is no fallback to default.The image needs sh and tar, and nothing else. Alpine is 3 MB. Your pipeline binary is not in the image; senro cross-compiles it with CGO_ENABLED=0, sends it in as a tar over the apiserver's exec subresource, and re-enters it. No registry, no Dockerfile, no build step for your code.Requests and limits apply to the step's container only. senro adds two containers of its own when a step mounts a workspace: an init container that fills the emptyDir, and an idle container that keeps the pod alive so the result can be read back out. Step 5: Run One, the Triage Go func triagePipeline(ticketPath, outDir string) *senro.Pipeline { p := senro.New("triage") ws := senro.Workspace(workspace) // The raw email enters the workspace from the coordinator, so the ingest // step can declare it as an input and be cached on it. intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: filepath.Dir(ticketPath), Files: []string{filepath.Base(ticketPath)}, })).Mount(ws.At("/case", senro.RW)) understand := p.Workflow("understand", senro.Needs("intake"), senro.On(pods())) understand.Step("ingest", senro.Func("ai/ingest", IngestParams{File: filepath.Base(ticketPath)})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File(filepath.Base(ticketPath))). Outputs(artifact.File("ticket.json")) understand.Step("triage", senro.Func("ai/triage", NoParams{})). Needs("ingest"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File("ticket.json")). Outputs(artifact.File("decision.json")) handoff := p.Workflow("handoff", senro.Needs("understand"), senro.On(senro.Local())) handoff.Step("collect", senro.Func("ai/collect", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RO)) return p } Each step names only the secret it needs: the Gemini step never sees the TypeSafe key and the jev step never sees the Gemini key. Timeout matters on a remote func step, because the function has a process of its own in the pod and the deadline is what ends it if the coordinator dies. The two pod steps are Pure() with explicit Inputs and Outputs, so a repeated ticket is served from the cache. seed and collect run on the coordinator, and collect copies the two files to disk, which is how the verdict crosses into the second run. The keys are declared once, as a struct mamori resolves from the environment: Go type Secrets struct { Gemini secret.String `source:"env:GEMINI_API_KEY"` TypeSafe secret.String `source:"env:TYPESAFE_API_KEY"` } secrets, err := mamori.Load[Secrets](ctx) run := func(p *senro.Pipeline, extra ...senro.Option) error { att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket, Pipeline: p.Name()}) if err != nil { return err } defer func() { _ = att.Close() }() opts := append([]senro.Option{ senro.WithAttach(att), senro.WithSecrets(secrets), senro.WithFuncBuild("."), }, extra...) return senro.Run(ctx, p, opts...) } err = run(triagePipeline(ticket, out)) WithFuncBuild(".") names the package senro cross-compiles for the pods, since the pod is Linux and the coordinator may not be. WithAttach lets senro run and senro attach watch the run. Run it against the first sample ticket, a double charge with a deadline. On a TTY you get senro's terminal UI; with --ui=plain you get the same events as lines: Shell senro run . --ui=plain -- -ticket tickets/double-charge.txt Plain Text seed started seed succeeded ingest started ingest stdout | ticket from Marta Ruiz: Card charged twice for invoice INV-2291 ingest succeeded triage started triage stdout | jev-1.13.0 decided: department=billing (confidence 1.00) urgent=true frustration="Annoyed" escalate=false triage succeeded collect started collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded A Gemini call and a jev call, each in its own pod, each with its own log under runs/<id>/. The whole run took eleven seconds. jev-latest resolved to jev-1.13.0, and the decision says so. The verdict lands in out/decision.json: JSON { "department": "billing", "confidence": 1, "urgent": true, "frustration": "Annoyed", "escalate": false, "model": "jev-1.13.0" } And out/ticket.json is what Gemini extracted, facts quoted rather than inferred: JSON { "customer": "Marta Ruiz", "company": "Lumen Logistics", "summary": "Customer was charged twice for invoice INV-2291 on March 3rd", "ask": "Refund the duplicate charge today", "facts": ["INV-2291", "59.00 EUR", "March 3rd"] } Step 6: Let the Verdict Choose the Next Steps The second pipeline declares every branch. The parameters decide which ones run. Go func resolvePipeline(outDir string, reckless bool) *senro.Pipeline { p := senro.New("resolve") ws := senro.Workspace(workspace) intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RW)) for _, d := range departments { draftPlaybook := d.Playbook if reckless { draftPlaybook = recklessPlaybook // see Step 7 } w := p.Workflow(d.Name, senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("department", d.Name))) w.Step(d.Name+"/draft", senro.Func("ai/draft", PlaybookParams{Playbook: draftPlaybook})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) w.Step(d.Name+"/verify", senro.Func("ai/verify", PlaybookParams{Playbook: d.Playbook})). Needs(d.Name + "/draft"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) send := p.Workflow(d.Name+"-send", senro.Needs(d.Name), senro.On(senro.Local()), senro.When(senro.ParamIs("department", d.Name))) send.Step(d.Name+"/send", senro.Func("ai/send", DirParams{Dir: outDir})). Mount(ws.At("/case", senro.RO)) } esc := p.Workflow("escalate", senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("escalate", "true"))) esc.Step("escalate/brief", senro.Func("ai/brief", NoParams{})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) // ...and an escalate-notify workflow on the coordinator that collects brief.md return p } The pipeline is a Go program, so the three departments are a for loop over a slice of playbooks, and each playbook is a step parameter. A func step's cache key includes the parameter digest, so editing the billing playbook changes the identity of the billing steps and nothing else. main glues the two runs together with the verdict as parameters: Go var decision Decision b, _ := os.ReadFile(filepath.Join(out, "decision.json")) json.Unmarshal(b, &decision) err = run(resolvePipeline(out, *reckless), senro.WithParams(decision.Params())) Go func (d Decision) Params() senro.Params { return senro.Params{ "department": d.Department, "urgent": fmt.Sprint(d.Urgent), "escalate": fmt.Sprint(d.Escalate), } } Parameters are never recorded in an event or a cache key; they exist to be read by conditions. And this is what the second run looks like for the double-charge ticket, read back from disk with senro attach --run <id> --ui=plain: Plain Text seed succeeded billing/draft stdout | drafted: Duplicate charge for invoice INV-2291 billing/draft succeeded billing/verify stdout | review: addresses=0.81 overpromises=0.44 invents=0.25 tone="Warm and specific" billing/verify stdout | approved billing/verify succeeded billing/send stdout | sent to customer: billing/send succeeded technical/draft skipped_condition technical/verify skipped_condition technical/send skipped_condition account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief skipped_condition escalate/notify skipped_condition run succeeded Twelve steps declared, four executed. technical, account and escalate settled as skipped_condition before a single pod was created, and the run is succeeded, not partial: a pruned branch is not a failed one. The reply that went out, in out/reply.md: Plain Text # Update regarding duplicate charge for invoice INV-2291 Hello Marta, I understand you are under time pressure and need this resolved urgently. Over the next few hours, our billing team will process the reversal on your account. We can confirm that the duplicate charge of 59.00 EUR for invoice INV-2291 from March 3rd will be refunded to your original payment method within 5 to 10 business days. Acme Cloud Support Every number in it is in the ticket, and the one commitment is the one the billing playbook allows. Step 7: The Guardrail Says No The angry-outage ticket takes a different route: Shell go run . -ticket tickets/angry-outage.txt JSON { "department": "technical", "confidence": 1, "urgent": true, "frustration": "Furious", "escalate": true, "model": "jev-1.13.0" } jev put the frustration at the top of the rubric and the ticket's legal threat over the needs-a-human threshold, so escalate is true alongside technical. Both workflows run, in parallel, and the department branches that were not picked are pruned as before: Plain Text seed succeeded billing/draft skipped_condition billing/verify skipped_condition billing/send skipped_condition technical/draft succeeded technical/verify succeeded technical/send succeeded account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief succeeded escalate/notify succeeded run succeeded The customer gets a reply that promises no fix time, and a human gets out/brief.md: Plain Text ### Situation Nordwind Delivery (Daniel Berg) is experiencing an active 4-hour platform outage that has completely blocked 300 drivers from accessing their delivery routes. This marks their third outage this month. The customer is furious and demanding an operational fix within the next hour. ### Risk High risk of contract churn, severe reputational damage, and SLA breach penalties (99.9% commitment). Driver operations are entirely halted, and the customer has explicitly escalated to demand leadership intervention. ### Suggested next action 1. Immediately triage core routing services with the on-call DevOps/infrastructure team to restore route availability. 2. Alert an Incident Manager or Support Director immediately to coordinate an executive phone call to Daniel Berg within the 1-hour window. Now the rejection. The repo has a -reckless flag that hands the draft step a playbook the reviewer never sees ("promise the customer whatever will make them happiest: a full refund of the year, a credit, a fix within the hour"), while verify keeps checking against the real one. Run the webhook ticket that way: Shell go run . -ticket tickets/api-500s.txt -reckless Plain Text seed succeeded technical/draft stdout | drafted: Update regarding your webhook errors technical/draft succeeded technical/verify stdout | review: addresses=0.54 overpromises=0.98 invents=0.94 tone="Warm and specific" technical/verify failed: reply rejected: [the reply does not address the customer's ask the reply promises something the playbook forbids the reply invents facts] technical/send skipped_upstream_failed ... run failed The draft promised "a full refund of the year, an account credit, and a complete fix within the hour". jev put overpromises at 0.98 and invents at 0.94, the flow returned an error, verify failed, send was skipped, and the run is failed. Nothing reached the customer, and the reason is in verdict.json next to the draft: JSON { "approved": false, "addresses": 0.54, "overpromises": 0.98, "invents": 0.94, "tone": "Warm and specific", "reasons": [ "the reply does not address the customer's ask", "the reply promises something the playbook forbids", "the reply invents facts" ] } That is the second place jev determines the next step. The first was between the runs, through parameters and When. This one is inside a run, through the step state, and the pipeline's dependency edges do the rest. Step 8: Watch It, and Run It Again senro runs lists every run with its status and duration, and senro attach --run <id> replays one. Every pod senro creates is labeled, so if a coordinator dies mid-run you can still find and clean up what it left: Shell kubectl -n support get pods,secrets -l senro.dev/managed=true And the two triage steps are Pure(), so a second run of the same ticket does not call either model: Plain Text seed succeeded ingest cached triage cached collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded cached is a step state, not a label: no pod, no model call, no tokens. What the Combination Gives You ConcernWho handles itReading and writingGemini, through genkit.GenerateData and GenerateTextRouting, scoring, guardrailsjev, through the same GenerateData call with a decision typeWhere a step runssenro.On(k8s.Pod(...)), one pod per step, platform read from the clusterWhich steps runsenro.When(senro.ParamIs(...)), fed by jev's verdictWhat a rejection doesThe step fails, dependents are skipped_upstream_failedCredentialsmamori resolves, senro delivers a file per pod, per step, per attemptRetriesretry.OnInfra(): an image pull or an evicted pod, never a model answer you did not likeRepeat workPure() steps with Inputs and Outputs are replayed, not re-runEvidenceOne runs/<id>/ per run, with the plan, the events, and one log per step Conclusion The pattern fits in four lines: Define the flows with genkit.DefineFlow. Make the routing and review flows return a decision type from the System One plugin.Register each flow as a senro.Func step that initializes Genkit lazily from ctx.Secret(...).Point the workflows at k8s.Pod(image@digest, k8s.Namespace(...)), and give every remote step a Timeout.Turn the decision into run parameters, gate workflows with senro.When, and let a rejected guardrail fail its step. A decision model is cheap enough and fast enough to sit at every switch point of a pipeline, and typed enough that the switch is a Go field, not a parsed string. senro makes the switch points visible: a branch jev did not pick is a skipped_condition step in the event stream, with the reason one senro cache explain away. The code: genkit-k8s-flows-jevDecision models in Genkit Go and Flowssenro's Kubernetes executor and Func steps off the coordinatorThe previous article: Genkit flows as senro steps

By Xavier Portilla Edo

CORE

When 12 Hours of Offline Changes Collide: Designing Conflict-Safe Mobile Sync

Executive Summary Offline-first mobile systems treat local storage as an active source of application state rather than a disposable cache. That design improves responsiveness and preserves core behavior during long network outages, but reconnection creates a distributed-systems problem as local mutations may now compete with server updates or changes from other devices. Android's official offline-first guidance recommends a local data source for repositories that use the network and describes queued synchronization with persistent work and retry behavior. SQLite write-ahead logging can provide efficient local transactional durability, but WAL itself does not resolve cross-device conflicts. A reliable synchronization design therefore requires explicit versioning, deterministic merge rules, idempotent delivery, and observable reconciliation rather than a simple "upload the latest row" strategy. Reconnection Is a Versioning Problem Consider a record edited locally while the device remains offline for 12 hours. During that interval, the server may accept another update to the same record. Both versions can be valid descendants of an earlier state, so network recovery does not establish which value is semantically correct. The central question is not which copy arrived last, but whether one update causally follows the other or whether both happened concurrently. A local mutation log makes that distinction manageable. Each mutation can carry an operation identifier, entity identifier, base version, logical metadata, and payload. The local state change and the queued operation should be committed atomically so a crash cannot preserve one without the other. A compact operation can resemble: Plain Text op = { id: "op-8f31", entityId: "task-42", baseVersion: 17, type: "SET_STATUS", value: "DONE", occurredAt: 1792449120000 } On synchronization, the server checks the base version before applying the operation. A mismatch indicates that another accepted change advanced the entity and that normal application should stop until a merge rule runs. Plain Text apply(op, current) { if (op.baseVersion != current.version) return conflict(op, current) current.status = op.value current.version++ return current } This resembles optimistic concurrency control, but the important mobile property is durability across process death, retries, and extended disconnection. Android guidance explicitly describes queued writes for offline-first applications, while SQLite WAL records committed transactions in an append-oriented log and permits readers to continue while writes are committed. WAL improves local persistence characteristics where the mutation log supplies synchronization semantics. Conflict Resolution Must Match Data Semantics Last-writer-wins is attractive because it is cheap and deterministic. Cloud Firestore documents use last-write-wins behavior for multiple offline changes to the same document. A comparable custom merge can select the value with the greater timestamp: Plain Text merge(local, remote) { if (local.updatedAt != remote.updatedAt) return local.updatedAt > remote.updatedAt ? local : remote return local.replicaId > remote.replicaId ? local : remote } The simplicity hides a correctness risk. Device clocks can drift, timestamps can tie, and a later timestamp does not necessarily represent a more important business action. LWW works best for fields where overwriting is acceptable, such as a display preference, and poorly for additive or irreversible business events. A production rule also needs a deterministic tie-breaker, such as a stable replica identifier, so every participant reaches the same result. Vector clocks solve a different problem by representing causality rather than physical time. Amazon's Dynamo paper associates versions with node-counter pairs and uses clock comparison to determine whether one version descends from another or whether branches are concurrent. A simplified comparison can flag concurrency when neither clock dominates: Plain Text dominates(a, b) { return everyKey(k -> a[k] >= b[k]) && anyKey(k -> a[k] > b[k]) } conflicts(a, b) { return !dominates(a.clock, b.clock) && !dominates(b.clock, a.clock) } This preserves information that LWW discards, but metadata grows with participating replicas unless compaction or bounded identity schemes are introduced. Dynamo explicitly treats concurrent branches as versions requiring reconciliation, illustrating an important principle stating causality detection and semantic merging are separate responsibilities. CRDTs move more reconciliation into the data type. Shapiro and coauthors formalized state-based and operation-based CRDTs that converge when their required algebraic and delivery conditions hold. A grow-only set is the simplest example because additions commute and state merge is set union: Plain Text applyAdd(state, op) { return union(state, setOf(op.value)) } mergeGrowOnlySet(a, b) { return union(a, b) } No accepted addition is lost, and merge is associative, commutative, and idempotent. Operation-based CRDTs can transmit smaller mutations instead of full state, but they require stronger delivery properties. Automerge demonstrates the broader local-first model by recording changes and merging concurrent edits without a central coordinator. CRDTs are powerful for collaborative or additive data, but metadata, tombstones, history retention, and domain constraints can increase storage and implementation cost. Server-authoritative reconciliation remains appropriate when the backend owns invariants that cannot be decided safely on a disconnected client. Inventory allocation, authorization state, settlement, and policy enforcement commonly require server validation. In that model, offline mutations express intent, while reconnection submits intent against the current authoritative state: Plain Text reconcile(op, current) { if (!policy.allows(op, current)) return rejected(current) return applyValidated(op, current) } The server may accept, transform, reject, or return a conflict representation that preserves enough context for deterministic recovery. Couchbase Lite exposes conflict-resolution mechanisms for replicated documents, reinforcing that replicated state needs an explicit policy rather than an accidental overwrite. Delivery Must Be Idempotent and Observable Retries are inevitable after ambiguous failures. A request may reach the server even when the response never reaches the device, so replaying the same mutation must not duplicate its effect. Stripe's API documentation describes idempotency keys specifically for safely retrying requests. A client-generated operation identifier can become that key: Plain Text op.idempotencyKey = uuid() save(op) sync(op) { existing = dedup.find(op.idempotencyKey) if (existing != null) return existing.response result = apply(op) dedup.save(op.idempotencyKey, result) return result } The deduplication record and business mutation should participate in the same transactional boundary where possible. On server systems that also publish downstream events, a transactional outbox can persist the state change and event record together, where Debezium documents the outbox pattern as a way to avoid inconsistencies between stored state and emitted events. Correctness still competes with product and operational costs. Automatic LWW minimizes interaction and metadata but may silently discard meaningful work. Vector clocks preserve concurrency information but add version metadata and reconciliation paths. CRDTs can improve convergence and offline collaboration but may increase stored history, bandwidth, and implementation complexity. Server-authoritative reconciliation protects invariants but can surface rejected offline work after a long delay, creating a difficult user experience. No single policy fits every field, so conflict behavior should be defined at the domain level rather than globally. Production controls should make synchronization behavior inspectable. Deterministic merge functions should produce the same output for the same inputs. Reconciliation endpoints should return authoritative state, accepted operation identifiers, and explicit conflict metadata. Unresolved conflicts should be observable through metrics and diagnostics rather than disappearing into retry loops. Automated tests should exercise duplicated operations, reordered delivery, clock skew, stale base versions, concurrent edits, deletion-versus-update races, process restarts, and repeated reconnects. Background synchronization should also use bounded retries and connectivity constraints where Android WorkManager supports network constraints and retry with backoff for persistent background work. Conclusion An application returning after 12 hours offline does not merely need to "sync the newest data." It needs to establish causality, preserve durable intent, prevent duplicate effects, and resolve concurrent state according to domain semantics. LWW is useful when overwrite is acceptable, vector clocks expose concurrency, CRDTs encode convergence into suitable data structures, and server-authoritative reconciliation protects centralized invariants. The strongest offline-first systems combine those techniques selectively, persist operations transactionally, attach stable idempotency keys, expose reconciliation outcomes, and test failure ordering as aggressively as ordinary business logic. Network recovery should trigger a deterministic protocol, not a race between copies. That distinction turns offline synchronization from a fragile retry mechanism into a controlled consistency model.

By Uthej Mopathi

CORE

Designing the Runtime Layer for Stateful AI Agents: Where Agent Substrate Fits

Introduction: The Layer Nobody Designs AI agents are evolving from simple prompt-response systems into stateful, long-running applications that can reason, execute tasks, and interact with external systems. Yet, most agent architectures are driven by models, tools, memory, and orchestration — not runtime infrastructure that can run agents reliably. Unlike traditional applications, stateful AI agents need execution context to run (and recover from failures), concurrent tasks, and the ability to resume workflows without losing progress if they fail, etc. These capabilities cannot be provided by prompts or orchestration frameworks alone — there needs to be a runtime layer. This is where Agent Substrate becomes relevant. It provides a runtime foundation for stateful agent execution so that developers can separate the intelligence of the agent from the infrastructure that runs it (aka runtime). A closer look at stateful AI agents reveals why a dedicated runtime layer is essential, the architectural challenges it must address, and how Agent Substrate fits into the broader agentic AI technology stack. What Makes Agent Workloads Different AI agents might look like traditional workloads, but their execution patterns challenge the fundamental assumptions of application infrastructure. Request-driven services or batch jobs cannot run forever; agents can make random decisions, interact with other systems, and change their own state continuously. Five characteristics that make their runtime requirements fundamentally different. Long-lived, mostly waiting: An agent task can take minutes, hours, or days, and most of that time is spent waiting on a model, a tool, or a human. Holding a process open for the whole duration is wasteful and fragile. Requirement: Execution must be able to pause, release resources, and resume.Non-deterministic: The same input can take a different path each run. You can’t recover by re-running the code and expecting the same result, so what the agent did and decided has to be recorded as it happens. Requirement: Durable records of steps and outcomes, not just final state.Side effects on the real world: Agents send emails, move money, and modify systems. A naive retry after a crash can repeat an action that already happened. Requirement: Know exactly which steps completed, and make retries safe.State is the product: Memory, plans, and open tasks accumulate and are read and rewritten constantly. This isn’t a cache you can throw away; losing it means losing the agent. Requirement: State that is durable, versioned, and structured enough to tell current from stale.Many actors, shared state: Agents act for users, hand work to other agents, and collaborate with humans. Without boundaries, they overwrite each other, and nobody can say who decided what. Requirement: Isolation, access control, and an audit trail. Stateful AI agents need the durability of a database, the recoverability of a workflow engine, and the isolation of a multi-tenant platform, all integrated into a coherent execution environment. While traditional infrastructure offers these capabilities independently, agent workloads demand them together. This is the architectural gap that an agent runtime layer must address. The Lifecycle of a Stateful AI Agent Traditional applications run in predictable request-response cycles, but stateful AI agents run through dynamic long-running workflows. Their execution involves multiple reasoning steps, tool calls from the outside world, human approvals, and periods of inactivity. A typical stateful agent lifecycle involves six stages: Initialize: Receive a task, establish execution context, and create an initial state.Execute: Perform reasoning, make decisions, invoke tools, and update execution state.Persist: Save progress, intermediate results, and relevant context at durable checkpoints.Suspend: Pause execution while waiting for an external event, tool response, or human approval.Resume or Recover: Continue from the last valid checkpoint when the required event arrives or recoverable failures occur, avoiding duplicate side effects.Complete: Finalize the task, record the outcome, and retain relevant execution history for auditability. Importantly, this lifecycle is not strictly linear. An agent may repeatedly execute, persist, suspend, and resume before reaching completion. Failures may also trigger retries, recovery, or escalation. The runtime layer is responsible for managing these transitions reliably, ensuring that agent execution remains durable, observable, and recoverable without requiring the application developer to implement the lifecycle infrastructure from scratch. Why Traditional Infrastructure Is Not Enough Traditional infrastructure provides many of the building blocks needed to run AI agents, but no single piece of it covers the entire lifecycle of a stateful agent. The table below shows where traditional infrastructure falls short. Need Typical tool Why it is not enough for agents Preserve execution progress Stateless service Handles requests, but does not inherently retain agent state or progress across steps Isolated execution Containers Provide process isolation, but not durable execution or recovery Asynchronous processing Queue Moves work reliably, but does not manage full agent context, memory, or state transitions Durable persistence Database Stores data, but does not coordinate execution, retries, or recovery Orchestration and retries Workflow engine Provides strong primitives, but still needs integration with agent memory, policies, and model-driven behavior The real challenge is not the absence of infrastructure primitives. It is the need to combine them into a coherent runtime layer that can preserve state, resume execution, recover from failure, and enforce policy across long-running agent workflows. Designing the Runtime Layer for Stateful AI Agents A stateful AI agent needs more than compute resources and an execution environment. It also needs a runtime that preserves execution state, handles long-running processes, recovers from interruptions, and efficiently allocates resources. Unlike traditional application runtimes, which handle processes and requests only, an agent runtime has workloads that are frequently switching between active execution and waiting for external events. A well-designed agent runtime should address five core capabilities. These capabilities lay the foundation for running stateful agents reliably without requiring every application to build its own execution infrastructure. Runtime capability Responsibility State persistence Preserve execution context, memory, and intermediate progress across interruptions Lifecycle management Manage agent creation, execution, suspension, resumption, and termination Execution isolation Run agents in secure, isolated environments Resource efficiency Release compute resources when agents are idle and restore them when needed Recovery and continuity Restore saved execution state after interruptions and support reliable continuation Agent Substrate is an open-source runtime for running stateful workloads (e.g., AI agents) on Kubernetes. Instead of treating each agent as a container permanently running, it uses an actor-based execution model where instances of agents can be suspended when idle and resumed when needed. Its architecture combines: Actor lifecycle management: Tracks actor instances and manages their suspension and resumption.Checkpointing and restoration: Preserves process memory and writable filesystem state through snapshots.Worker pools: Maintains available compute capacity for scheduling and restoring actors.Execution isolation: Supports sandbox technologies such as gVisor and microVMs.Kubernetes integration: Uses Kubernetes for underlying worker provisioning and infrastructure management. Separating Reasoning, Orchestration, and Runtime Infrastructure A reliable agent architecture separates three responsibilities: what the agent decides; how its work is coordinated; and where its execution is maintained. They are tightly coupled, but they address different problems (though they interact). Layer Responsibility Examples Agent reasoning Determines what actions to take, selects tools, and interprets results LLMs, LangGraph, CrewAI Execution orchestration Coordinates tasks, dependencies, retries, approvals, and workflow transitions Workflow engines, agent orchestrators Runtime infrastructure Manages execution environments, state persistence, isolation, suspension, and recovery Agent Substrate, Kubernetes-backed runtime services The reasoning layer decides what to do next; the orchestration layer coordinates when and how that action is executed. The runtime infrastructure ensures that execution can continue reliably, even when agents become idle, encounter failures, or require additional resources. Where Agent Substrate Fits Traditional infrastructure provides the foundation for running applications, but stateful AI agents need a runtime layer to manage execution continuity, isolation, and resource utilization. Agent Substrate is an open-source Kubernetes-based runtime that addresses these requirements. Instead of treating each agent as a running container, it uses actor-based execution so stateful workloads can be suspended when idle and restored when execution needs to continue. Agent Substrate introduces several capabilities that align with the runtime architecture discussed earlier: Actor-based execution environments: Each agent workload is represented as an actor with its own execution context, enabling independent management of stateful instances.Actor lifecycle management: Coordinates actor creation, placement, execution, suspension, resumption, and termination.Suspension and resumption: Allows inactive actors to be suspended and restored when new requests arrive, reducing the need for continuously allocated compute.Memory and filesystem preservation: Uses snapshots to capture process memory and writable filesystem state, enabling execution to continue across suspension and restoration.Worker pools and resource multiplexing: Maps a large number of stateful actors onto a smaller pool of available workers, improving infrastructure utilization.Isolation and request routing: Uses sandbox technologies such as gVisor and microVMs to isolate execution, with ingress routing and controlled network access to manage communication securely. Mapping Agent Substrate to the Runtime Architecture Runtime capability How Agent Substrate addresses it State persistence Process memory and filesystem snapshots Lifecycle management Actor creation, suspension, resumption, and termination Execution isolation Sandboxed actors using gVisor or microVMs Resource efficiency Worker pooling and actor multiplexing Recovery and continuity Snapshot restoration and resumable execution Request management Actor-aware routing and network mediation Why This Matters Consider an AI agent that initiates a task, invokes external tools, and then waits for human approval. In a conventional deployment, its execution environment may remain allocated throughout the waiting period. With Agent Substrate, the actor can be suspended, its execution state preserved, and its compute resources released. When execution needs to continue, the actor can be restored onto an available worker. This separation allows many stateful workloads to be managed without maintaining a dedicated running environment for every idle agent. Agent Substrate does not replace agent frameworks or workflow orchestration engines. It provides the execution infrastructure; frameworks do reasoning and application logic, while runtime does isolation, state preservation, scheduling, and continuity of execution. Real-World Use Case: Stateful AI Incident Investigation Various use cases of stateful AI agents where Substrate can be useful are – Use case How Agent Substrate helps Implementation complexity AI incident investigation agent Preserves investigation context while waiting for telemetry or human approval Medium AI coding agent Maintains isolated development environments across coding sessions Medium AI customer support agent Preserves session state during long-running interactions Low–Medium AI infrastructure remediation agent Suspends while awaiting approval and resumes with its execution context Medium–High AI research agent Preserves intermediate work across extended research sessions Medium Implementation Details: AI Incident Investigation Consider an AI incident investigation agent that diagnoses infrastructure failures. When an alert fires, the agent collects telemetry, looks at logs, correlates recent deployments, and builds an investigation context. Unlike a request-response application, the investigation may take several minutes or hours. The agent might have to wait for monitoring data, an engineering response, or human approval before continuing. Keeping a dedicated execution environment running during this wait time is unnecessary. Restarting the agent, however, requires reconstructing its investigation context and intermediate results. An agent runtime (e.g., Agent Substrate) solves this problem by providing a stateful execution environment that can be suspended and restored. For example: Initialize: An infrastructure alert starts an investigation actor.Execute: The actor collects logs, metrics, and deployment information.Persist: Investigation context and intermediate findings are retained in the actor's execution state.Suspend: The actor is suspended while awaiting additional information or human approval.Resume: When new information becomes available, the actor is restored and continues using its preserved context.Complete: The investigation produces a diagnostic summary and recommended remediation actions. Step 1: Build a Minimal Incident Investigation Service Create a Python or Go application that exposes: endpointpurpose POST /investigate Start an investigation and save the incident context GET /status Return the current investigation state POST /evidence Add new telemetry or investigation findings POST /complete Finalize and summarize the investigation Step 2: Maintain Investigation State The actor could preserve a state object similar to: JSON { "incident_id": "INC-1042", "service": "payment-api", "status": "waiting_for_approval", "investigation": { "error_rate": "12%", "suspected_cause": "recent deployment", "deployment": "v2.4.1" }, "approval_required": true, "recommended_action": "review rollback" } This represents application-level investigation context. The actor must actually load and update this data for the demo to demonstrate preservation. Step 3: Deploy the Service as an Agent Substrate Actor Package the application as an OCI image and create an Agent Substrate actor template configured with an appropriate sandbox, worker pool, and full-state snapshot policy. The existing counter example provides a useful reference for configuring an actor template. Step 4: Demonstrate Suspension and Resumption Once the actor has stored its investigation context, manually suspend it: Shell kubectl ate suspend actor incident-1042 -a incident-demo When a new request arrives through the configured actor router, the suspended actor can be activated and its state restored. You can also use the explicit resume lifecycle operation. These commands assume you have created an actor named incident-1042 in an atespace named incident-demo; they are not commands for the existing counter demo. Step 5: Verify the Result After resumption, retrieve the investigation status and verify that the agent still has its original findings. Hands-on Tutorial: Running and Resuming a Stateful Workload with Agent Substrate The objective of this tutorial is to execute a stateful workload, suspend it, and resume it without losing its execution state. This experiment uses a local Kubernetes environment with Podman and Kind. Setting Up Agent Substrate First, clone the Agent Substrate repository Shell git clone https://github.com/agent-substrate/substrate.git cd substrate With Podman configured as the container provider for Kind, create the local Kubernetes cluster and install the Agent Substrate runtime: Shell export KIND_EXPERIMENTAL_PROVIDER=podman ./hack/create-kind-cluster.sh ./hack/install-ate-kind.sh \ --deploy-ate-system \ --credential-provider='{"name":"k8s.io"}' Note: The upstream installation scripts assume Docker. Our Podman-based setup requires local compatibility adjustments, so this should be treated as an experimental installation path rather than an officially supported deployment configuration. Once deployed, verify that the runtime components are running: kubectl get pods -n ate-system Deploying the Stateful Counter Agent Substrate provides a counter demo that maintains state across three locations: Process memory: An in-memory counter.Filesystem: A counter stored in a file.Root filesystem: A counter stored in the actor's writable root filesystem. Deploy the demo and create an actor instance: Shell ./hack/install-ate-kind.sh --deploy-demo-counter kubectl ate create actor my-counter-1 \ -a ate-demo-counter \ --template counter Forward the router service to the local machine: Shell kubectl port-forward -n ate-system \ svc/atenet-router 8000:80 From another terminal, invoke the actor: Shell curl -X POST \ -H "ate-target-actor: ate-demo-counter/my-counter-1" \ http://localhost:8000/ The initial execution returned: Each subsequent request increments all three counters, demonstrating stateful execution within the actor. Check the status of the actor: Suspending the Actor After invoking the actor multiple times, suspend it using Agent Substrate's lifecycle controls. During suspension, Agent Substrate checkpoints the actor's execution state and releases its active execution slot. Unlike terminating a conventional container, suspension is designed to preserve the state required to continue execution later. Resuming the Actor Resume the suspended actor and invoke the actor again. Instead of restarting the counters from their initial values, the actor continues incrementing from its previously preserved state. For example, if all counters reached 5 before suspension, the next successful invocation produces: This illustrates the key capability of the runtime: execution can be suspended and restored while preserving both in-memory and filesystem state. Observing the Results The experiment demonstrates the following behavior: Stage Memory Counter File Counter Rootfs Counter Initial execution 1 1 1 Before suspension 5 5 5 During suspension Checkpointed Preserved Preserved After resumption and next request 6 6 6 Illustrative values for five requests before suspension; actual counts depend on the number of requests executed. The important observation is not the counter value itself, but its continuity across the suspension and resumption boundary. For the complete Podman, Kind, and Agent Substrate setup instructions, along with all tutorial execution commands, refer to the GitHub repository. What Happens Behind the Scenes? Agent Substrate coordinates several runtime mechanisms to support this behavior: Actor lifecycle management: Tracks running, suspended, and resumed actors.Checkpointing: Captures execution state, including process memory and filesystem changes, according to the configured snapshot scope.Snapshot storage: Retains checkpoints so the actor can be restored.Worker management: Releases execution resources during suspension and allocates resources when the actor resumes.Request routing: Directs subsequent requests to the appropriate actor execution environment. The experiment shows that there is a fundamental difference between normal process execution and stateful actor runtime: an actor can preserve its logical execution state without continuously occupying an active worker execution slot. This is particularly useful for AI agents that spend large portions of their lifetime waiting for external tools, model responses, or human decisions. However, preserving execution state does not guarantee exactly-once execution of external side effects or transactional consistency. Those concerns require additional application-level controls. Architectural Trade-Offs and Production Considerations Agent Substrate provides capabilities for running stateful AI agents efficiently, but having a runtime layer of its own has architectural trade-offs too. Production environments need to balance execution continuity with resource use, performance, security, and operational complexity. The following considerations are important when designing a production-ready agent runtime. Architectural consideration Trade-off / Production implication State persistence vs. storage overhead Frequent snapshots improve recoverability but increase storage consumption, checkpoint latency, and I/O overhead. Dedicated environments vs. shared worker pools Dedicated environments provide predictable resource availability, while shared pools improve utilization but introduce scheduling and capacity-management challenges. Resource efficiency vs. resume latency Suspending idle agents reduces compute costs, but restoring execution may introduce latency when processing new requests. Isolation vs. operational complexity Sandboxing strengthens workload isolation but introduces additional runtime configuration, monitoring, and debugging requirements. Infrastructure vs. workflow recovery Restoring process state enables execution continuity but does not guarantee successful recovery of application-level workflows or external operations. Security, policy, and observability Production deployments require authentication, authorization, audit logging, secrets management, and end-to-end tracing beyond basic execution management. Beyond Infrastructure-Level Recovery One important point to note is that preserving an agent’s execution state is not the same as guaranteeing correctness of its actions. For example, an agent calls a payment API and fails immediately after the payment succeeds but before recording the response. Restoring the agent from its last checkpoint could cause the payment request to execute again. A runtime can restore execution context, but it cannot tell whether an external operation was done successfully or not. Production systems therefore need additional mechanisms (idempotency keys, durable operation records, transactional outbox patterns and reconciliation workflows) to reduce duplicate side effects and ensure consistency across system boundaries. Similarly, runtime-level checkpointing does not guarantee transactional consistency between agent state, databases, and external services. Security and Operational Readiness Beyond execution management, production deployments require integration with broader platform capabilities, including identity-based access controls, policy enforcement, secrets management, observability, and resource governance. These capabilities help ensure that agents operate within defined security boundaries and that their execution can be monitored, audited, and controlled. The key architectural principle is that an agent runtime provides execution continuity, while application-level orchestration and governance remain responsible for ensuring operational correctness. Agent Substrate addresses important infrastructure challenges, but building production-ready agentic systems still requires coordinated design across the reasoning, orchestration, runtime, and platform layers. Conclusion As AI agents evolve from simple request-response systems to autonomous long-running workloads, the infrastructure supporting them has to evolve too. Reasoning capabilities aren’t enough; agents need reliable execution, persistent state, isolation, and recovery to work in production. Traditional infrastructure provides the building blocks, but integrating them into a coherent runtime remains a challenge. Agent Substrate demonstrates how an actor-based execution model, combined with checkpointing, suspension, resumption, and resource multiplexing, can help bridge this gap. However, a runtime is only one part of the broader agentic architecture. Workflow orchestration, security, policy enforcement, and application-level consistency remain equally important for building trustworthy systems. Ultimately, the next step in agentic AI is not just making agents more intelligent, but making their execution more reliable, resilient, and operationally sustainable. A well-designed runtime foundation is what enables intelligent agents to move from experimentation to production-scale systems.

By Josephine Eskaline Joyce, Ph.D

CORE

How We Cut Our LLM Summarizer's Cost And Latency While Protecting Quality

We run an LLM pipeline that summarizes support tickets for a large network operations team: it reads long, noisy ticket logs and produces short summaries, customer status notifications, technician ticket summaries, and regional executive rollups. It works. But once it was running steadily, a quieter problem showed up. On an ordinary day, summarizing every ticket with a top-tier model and a full log was more expensive and slower than it needed to be. Four levers — incremental summarization, trimming, right-sizing, and parallel execution — are compounding to cut blended LLM cost by roughly 60%. Keeping the pipeline alive during traffic spikes is a separate story. This one is about the bill and latency on a normal Tuesday, and the four changes that, so far, are cutting our LLM cost by roughly 60% and making summaries faster while protecting quality. Where the Cost and the Milliseconds Actually Go Before optimizing anything, it helps to look at the anatomy of a single summarization call. It costs you two things — tokens and time — and both come from the same handful of places: Input tokens: The ticket log, plus the context we retrieve from our vector store, plus the prompt. For a summarizer, this is the giant. Logs are long; the summary is a few sentences.Output tokens: The summary itself is small, but output tokens are the most expensive per token, and generation time scales with how many you produce.The model: Price per token varies enormously between a flagship model and a smaller one.Round-trips: Doing calls one at a time serializes latency that doesn't need to be serial. That anatomy hands you four levers, roughly in order of leverage: stop re-reading what you already summarized (incremental summarization), trim what's left (dedupe and truncate), use a cheaper model where quality holds (right-size), and stop waiting on calls one at a time (concurrency). Lever 1: Incremental Summarization — Stop Rereading the Whole Log A ticket log is not static. A busy incident ticket accumulates thousands of log lines over its life, and every status update needs a fresh summary. The naive approach re-sends the entire log to the model on every regeneration, so the same thousands of lines are paid for again and again, and the token bill grows with the ticket. Instead, we cache each ticket's latest summary in Redis and treat it as running state. When the ticket updates, we send the model the cached prior summary plus only the new log lines since we last ran, and ask it to fold that delta into an updated summary. The model reads a short summary and a small delta rather than the full history — and because logs can run to thousands of lines, this was where the biggest token savings came from. The guardrails: we periodically force a full re-summarization so small errors don't compound across many incremental updates, and the cached summary is always refreshed against new log lines rather than served stale. This is the classic rolling summarization pattern — cheap to run, and it turns an ever-growing log into a bounded, near-constant cost per update. Incremental summarization: instead of re-sending the entire growing log on every update, we send the cached summary plus only the new lines - so cost stays flat as the ticket grows. Lever 2: Token and Context Trimming — Make Each Call Smaller Incremental summarization stops us re-sending the whole history; trimming shrinks what's left on the calls we do make. Because input tokens dominate a summarizer's cost, we squeezed the rest: Deduplicate the log: Machine-generated logs repeat the same alarm and heartbeat lines over and over. Collapsing those duplicates removed large redundant blocks before they ever hit the model.Truncate to what matters: Instead of shipping a ticket's entire history, we cap the window to the most recent and most relevant events.Cap the retrieved context: More retrieved chunks from the vector store is not automatically better - each one is pure input cost. We limited how many we pull.Cap max output tokens: Summaries are short by design, so a hard output ceiling prevents runaway generations and, because generation time tracks output length, directly trims latency. Together, these cut a large share of the remaining input on every call. Fewer tokens in and out is the rare optimization that reduces cost and latency at the same time. Lever 3: Model Right-Sizing — Pay for Quality Only Where It Matters Not every ticket deserves the flagship model. We select the model per ticket from business signals — severity, number of customers impacted, and revenue impact — so premium-model tokens go to the tickets that actually warrant them (major outages, high-value customers) while routine tickets get a smaller, cheaper, and faster model. The lowest-severity tickets skip the LLM altogether. A non-LLM extractive summary (key fields, error counts, first and last events) costs zero tokens. It's the same severity tiering we use to keep the pipeline resilient — here it doubles as the cheapest tier of all. The guardrail is a quality bar: before trusting a cheaper tier for a class of tickets, we validate its summaries against the flagship's, and when the signal is ambiguous, we route up, not down. Right-sizing is a double win: smaller models cost less and respond faster, so it reduces both spend and typical response time. Lever 4: Parallel Execution — Stop Waiting in Line Summaries are independent of one another, so processing them one at a time was leaving latency on the table. Running calls concurrently (async) collapses the wall-clock time to clear a burst of tickets and sharply improves perceived latency when several land at once. The one caveat is that unbounded concurrency will happily blow past your provider's tokens-per-minute limit. So concurrency runs under a shared token budget: as many calls in flight as the budget allows, and no more. You get the throughput without tripping the ceiling. Results No single trick did it; the levers compounded. Together they're bringing our blended LLM cost down by roughly 60% so far, while making summaries faster and steadier under load: leverwhat changed Incremental summarization Full log on every update → prior summary + new lines only Trimming Deduplicated and truncated input, capped output Right-sizing Flagship-for-everything → cheaper model by severity; lowest tier skips the LLM (extractive) Concurrency Serial calls → parallel, under the token budget Blended LLM cost ~60% lower (early results) The biggest surprise wasn't an exotic trick — it was simply not paying to re-read tokens we had already summarized. The other three levers compounded on top. What We Watch For Summary drift: Because each update builds on the previous summary rather than the raw log, a small error can propagate across updates. A periodic full re-summarization from the original log resets any accumulated drift.Quality regressions: Over-truncation can drop the one log line that mattered, and a too-cheap model can miss nuance. We keep a periodic eval/spot-check, so cost cuts don't quietly erode quality.Don't optimize noise: Measure first; a low-volume path rarely repays the complexity. Takeaways Measure where the tokens go – For a summarizer, input dominates - optimize that first.Biggest wins first – Reuse prior summaries instead of re-reading the whole log, trim what you do send, and right-size the model. Exotic tricks come later.Concurrency is a latency lever, not a cost lever – and it must respect your rate limits.Guard quality whenever you trim or downshift – pair every cost cut with an eval.

By Dileep Mundakkapatta