Coding

I stumbled onto this pattern while building agents with Deep Agents, watching a tool registry grow past the point where sending every schema on every turn still made sense. What follows is the pattern itself, stripped down so it drops into any tool-calling agent loop today regardless of framework, backed by a benchmark against a synthetic 61-tool registry. The Problem: Tool Count Grows, Relevance Per Turn Doesn't Most agent frameworks assemble the tool list once at construction time and send the whole thing to the model on every call, regardless of what that turn is about. Fine at 5-10 tools. Once you wire up a handful of MCP servers (Slack, GitHub, Linear, a calendar, a CRM, each contributing 3-6 schemas), every turn starts carrying 40-60 tool definitions whether the user asked about a Slack message or not. That costs you twice over. Every schema (name, description, parameter spec) gets serialized into every request, so tokens spent on tools irrelevant to this turn are tokens not spent on the task. And the accuracy hit is real: more candidates in the list give the model more chances to grab a similarly named or similarly described tool instead of the right one, something the benchmark below reproduces directly. Sending the whole registry every time is the actual bug here, not the size of the context window. The Pattern The pattern boils down to three moving parts: score the registry against the current turn's intent, send only the top-K, and give the model an explicit way to ask for something it can't currently see. Scoring ranks the full registry against the latest user message, and a plain token-overlap measure (Jaccard similarity between the query's words and each tool's name plus description) turns out to be enough to separate on-topic from off-topic; no embeddings needed. Top-K plus an always-include set caps what actually gets sent to the model; a handful of tools an agent can't function without (file I/O, a task/subagent tool) sit outside the filter entirely. Then there's the piece that makes any of this safe to ship: a discover tool the model can call to search the complete, unfiltered registry when nothing in its filtered view fits. A match gets pinned into the always-visible set for the rest of the conversation, so the worst case is one extra tool call, never a tool the model silently doesn't know exists. The implementation below has no framework dependency, just the algorithm, and represents a tool as a plain {name, description} pair, a strict subset of what OpenAI function-calling, Anthropic tool-use, LangChain's BaseTool, and MCP tool listings all expose, so it drops into any of them. Python """A portable, zero-dependency tool selection pattern for tool-calling agents.""" from __future__ import annotations import re from collections.abc import Iterable from dataclasses import dataclass _TOKEN_RE = re.compile(r"[a-z0-9]+") @dataclass(frozen=True) class Tool: """Minimal tool description: what any framework's tool object reduces to.""" name: str description: str def _tokenize(text: str) -> set[str]: return {m.group(0) for m in _TOKEN_RE.finditer(text.lower())} def lexical_score(tool: Tool, query_tokens: set[str]) -> float: """Jaccard overlap between a tool's name+description tokens and the query tokens.""" if not query_tokens: return 0.0 tool_tokens = _tokenize(f"{tool.name} {tool.description}") if not tool_tokens: return 0.0 return len(tool_tokens & query_tokens) / len(tool_tokens | query_tokens) def select_tools( tools: Iterable[Tool], query: str, *, top_k: int, always_include: frozenset[str] = frozenset(), pinned: frozenset[str] = frozenset(), scorer=lexical_score, ) -> list[str]: """Return the names of the top-K tools most relevant to `query`, plus keepers.""" tools = list(tools) keep_names = always_include | pinned if len(tools) <= top_k: return [t.name for t in tools] query_tokens = _tokenize(query) candidates = [t for t in tools if t.name not in keep_names] ranked = sorted(candidates, key=lambda t: scorer(t, query_tokens), reverse=True) selected = {t.name for t in ranked[:top_k]} return [t.name for t in tools if t.name in keep_names or t.name in selected] def discover(tools: Iterable[Tool], query: str, *, scorer=lexical_score) -> Tool | None: """Search the full registry for the single best match to `query`. Returns `None` if nothing scores above zero -- callers should surface that as "no match found" rather than silently picking an arbitrary tool. """ query_tokens = _tokenize(query) tools = list(tools) if not tools: return None best = max(tools, key=lambda t: scorer(t, query_tokens)) return best if scorer(best, query_tokens) > 0 else None select_tools costs nothing below top_k; it's a no-op until the registry is actually large enough to matter. scorer is a keyword hook, so swapping lexical_score for a cosine-similarity function over an embedding model changes nothing else in the function. What the Tests Actually Check Python """Tests for the portable tool_selector module (excerpt).""" from tool_selector import Tool, discover, select_tools def _tools(*pairs: tuple[str, str]) -> list[Tool]: return [Tool(name=n, description=d) for n, d in pairs] def test_pinned_tool_survives_an_unrelated_turn() -> None: """Simulates turn 2 of a conversation where turn 1's discover() pinned a tool.""" tools = _tools( ("weather_lookup", "get the current weather forecast for a city"), ("calculator", "evaluate a basic arithmetic expression"), ) result = select_tools( tools, "what is the weather forecast today", top_k=1, pinned=frozenset({"calculator"}), ) assert set(result) == {"weather_lookup", "calculator"} def test_discover_finds_the_right_tool_by_description() -> None: tools = _tools( ("weather_lookup", "get the current weather forecast for a city"), ("calculator", "evaluate a basic arithmetic expression"), ) match = discover(tools, "evaluate an arithmetic expression") assert match is not None assert match.name == "calculator" The discover test surfaced a real limitation while I was writing it. An early draft queried discover(tools, "I need to crunch some numbers") against a calculator tool described as "evaluate a basic arithmetic expression," and it failed outright: the two strings share zero tokens. Lexical scoring has no concept of synonymy, so whatever query gets handed to discover has to share vocabulary with the target tool's description; in practice that means the model has to formulate a reasonable search term rather than forward the user's literal wording. It's a real constraint of the zero-dependency approach, and the main argument for the scorer= hook: swap in an embeddings model once tool vocabulary and user vocabulary diverge enough to bite you. Measuring It: A recall@K Benchmark Rather than mock an LLM's tool-picking behavior (which amounts to testing my own mock), I measured the one thing that doesn't need a model in the loop at all: does the correct tool survive the filtering step? If the right tool gets cut before the model ever sees the list, no amount of model capability brings it back. It's a 61-tool registry, modeled on what four or five real MCP servers actually expose (Slack, GitHub, Linear, Jira, Gmail, Calendar, Drive, Notion, a CRM, web search, weather, finance, plus a small always-include filesystem core), roughly the tool count teams report after wiring up a handful of MCP servers rather than an inflated worst case. 30 labeled queries span about 18 domains, split between direct phrasing ("send a direct message to alice on slack") and indirect phrasing ("let the team know in the channel that the deploy finished"), the same split deepagents' own tool-selection evals use. Recall@K Across the Six top_k Settings Tested top_krecalltools sentpayload (chars)reduction557%545292%1067%1090584%1570%151,35875%2077%201,81067%3080%302,71651%60100%605,4322% Unfiltered baseline: every turn sends all 61 tools, 5,523 chars, every time. Recall@K rises while payload reduction falls as top_k grows from 5 to 60, crossing between k=15 and k=20 That table breaks down into two separate questions worth pulling apart: how good is the trade at a reasonable K, and how much does pushing K higher actually buy you? At top_k=10, you get 67% recall for an 84% payload reduction. For a lexical scorer with zero setup cost, that's a genuinely good trade, and the 33% of misses aren't silent failures; they're what the discover escape hatch exists for: one extra tool call, the tool gets found, and it's pinned for the rest of the thread. Recall also climbs slowly as top_k grows: going from 10 to 30 tools sent buys only 13 more points. Past a certain point you're paying most of the unfiltered cost for a shrinking accuracy gain, and if you need recall above roughly 80% without raising top_k that far, that's the signal to swap in the scorer= embeddings hook instead of continuing to raise K. The k=10 misses look like this: Plain Text MISS query='file a bug report on the backend repo' expected='github_create_issue', got=[..., 'github_create_pr', ...] MISS query='mark this jira ticket as in progress' expected='jira_transition_issue', got=[..., 'jira_create_issue', ...] MISS query="what's 340 divided by 12" expected='calculator', got=['read_file', 'write_file', ..., 'slack_search_messages'] The misses cluster around two failure modes: tools in the same domain sharing most of their vocabulary (github_create_issue and github_create_pr both score high on "github", "create", "repo"), and short, generic queries that share almost no tokens with the target description ("what's 340 divided by 12" versus "evaluate a basic arithmetic expression"). Both are what the escape hatch is designed to catch, and both are cases where embeddings-based scoring would do meaningfully better. The full registry, query set, and benchmark script run about 150 lines, small enough to paste into any project and adapt to your own tool list. The numbers are reproducible without an API key or a specific model. Before vs. After, on an Actual Agent Run The recall@K numbers above measure the scoring step in isolation. To see the effect on a real conversation, the same 3-turn scenario was run twice through an actual create_agent graph with a checkpointer (once unfiltered, once with ToolSelectionMiddleware(top_k=1, always_include=frozenset())) against 4 domain tools: weather_lookup, stock_price, translate_text, calculator. Tools sent to the model per turn, same conversation, with and without filtering Turn 2 is the interesting one: the question, "I need to crunch some numbers but don't see a tool for that, can you check?", was deliberately worded to score zero against calculator's own description. The model genuinely can't see a calculator tool in its filtered list and has to fall back to the discover escape hatch: Plain Text TURN 2: "I need to crunch some numbers but don't see a tool for that, can you check?" model call (before discover_tools ran) -> model was sent 2 tools: ['discover_tools', 'weather_lookup'] discover_tools returned: Found tool `calculator`: Evaluate a basic arithmetic expression. It is now available for the rest of this conversation. TURN 3: "translate hello to French" (same thread -- calculator pin should persist) model call -> model was sent 3 tools: ['calculator', 'discover_tools', 'translate_text'] state['tool_selection_pinned'] on this thread: ['calculator'] calculator shows up in turn 3's tool list even though that turn is about translation: that's the pin from turn 2 persisting through the checkpointer as designed. Without the middleware, every turn sends all 4 schemas regardless of relevance. With it, turns 1 and 2 send 2 tools each and turn 3 sends 3, and the tool the model couldn't initially see gets recovered through exactly one extra call. Wiring It Into an Existing Framework The algorithm above has zero framework knowledge, on purpose. Here's how it plugs into LangChain / deepagents' middleware system, which intercepts the tool list before every model call via wrap_model_call: Python from tool_selector import Tool, select_tools class ToolSelectionMiddleware: """Sketch: adapt to your framework's actual middleware hook signature.""" def __init__(self, *, top_k: int = 15, always_include: frozenset[str] = frozenset()): self.top_k = top_k self.always_include = always_include def wrap_model_call(self, request, handler): latest_query = _latest_human_message_text(request.messages) candidate_tools = [Tool(t.name, t.description) for t in request.tools] keep = set(select_tools( candidate_tools, latest_query, top_k=self.top_k, always_include=self.always_include, )) filtered = [t for t in request.tools if t.name in keep] return handler(request.override(tools=filtered)) This is a sketch, deliberately not copy-pasteable middleware. A fuller version wires the discover_tools escape hatch as an injected tool with per-thread pin state, so pins don't leak across concurrent sessions. Treat the API shape here as illustrative rather than stable; the algorithm underneath is the part worth keeping regardless of framework. One detail worth flagging for anyone building an injected-context tool in LangChain/LangGraph: if your escape-hatch tool takes a runtime/context parameter the framework injects automatically (ToolRuntime, for instance), the module defining it must not use from __future__ import annotations. Postponed annotations turn the type hint into a string at definition time, so injection detection that inspects the live signature won't recognize it. The tool then breaks silently when invoked through the framework's actual call path, even though a direct unit test would never catch it. The Native Alternative: Claude's Tool Search Tool If you're calling the Claude API directly rather than going through an agent framework, Anthropic now ships a server-side version of this same idea: the Tool Search Tool (tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119). You declare it alongside your other tools, mark the tools you don't want sent by default with defer_loading: true, and Claude searches the deferred set and pulls in only what's relevant, as a tool_search_tool_result block. Python { "tools": [ { "type": "tool_search_tool_bm25_20251119", "name": "tool_search_tool_bm25" }, { "name": "github_create_issue", "description": "...", "defer_loading": true }, { "name": "slack_send_dm", "description": "...", "defer_loading": true } ] } This isn't a mere hosted copy of the DIY pattern. The model does the searching itself, so there's no lexical-overlap or embedding logic to maintain: Claude decides what's relevant and searches for it. Discovered schemas also get appended to the request rather than swapped in. Changing which tools are visible mid-conversation would normally invalidate the prompt cache, since the tool list renders at the start of the prefix, but tool search sidesteps that: the deferred tools stay out of the initial prefix, and appending to the end doesn't rewrite what came before. What you lose relative to the DIY version is curatorial control: there's no equivalent of always_include or an explicit per-thread pin you can inspect and log, since the whole mechanism lives server-side. If you need that visibility, or you're not on a framework/model that supports the Tool Search Tool, the portable version above is the fallback. If you're calling Claude directly and don't need fine-grained control over what's exempted from filtering, reach for the native tool first: it's less code to maintain, and it solves the caching problem for free. Takeaways Tool registries grow faster than most agent code accounts for. Two or three MCP servers is enough to cross the point where sending every schema on every turn starts costing accuracy, not just tokens. A zero-dependency lexical scorer recovers most of that benefit (67% recall at 84% payload reduction at top_k=10 on a 61-tool registry), and it's the escape hatch, not the scorer's raw accuracy, that makes shipping something this lossy safe. Test that escape hatch through the real framework call path rather than by calling the underlying function directly: injected-parameter bugs and cache-invalidation bugs both hide from unit tests that bypass the framework's actual entry point. And if you're calling the Claude API directly, check whether the native Tool Search Tool already covers your case before building any of this yourself. The full tool_selector.py, its test suite, and the benchmark script are small enough to fit in a gist; reach out if you want them as a standalone repo rather than reconstructing from the code blocks above.

By Ninaad Rao CORE

When I first started building enterprise applications with Large Language Models (LLMs), I fell into a trap that almost every developer encounters. I thought that scaling an AI system simply meant refining a single, massive prompt. I wrote complex system instructions, packed the context window with rules, and expected a single stateless API call to act as a researcher, analyst, and copywriter all at once. In production, this monolithic approach failed repeatedly. When processing dynamic data streams, the model flattened nuanced details, skipped critical execution steps, and regularly generated highly confident hallucinations. Through these failures, I realized the core problem: we are expecting a single inference step to manage an entire engineering workflow. To build predictable, production-grade software, I had to redesign my architecture. I moved away from monolithic prompts and began decoupling complex tasks into role-based, multi-agent frameworks in Python. My Breaking Point: The Competitive Intelligence Engine Failure Problem The necessity of this architectural shift became clear to me during a deployment for an enterprise technology firm. My team was tasked with building a competitive intelligence engine to track daily competitor product launches, analyze changing pricing sheets, and generate technical battlecards for our global sales team. My first iteration used a single, closed-source model wrapper. The prompt instructed the LLM to read raw HTML fragments from target URLs, extract feature updates, compare them against our internal capabilities matrix, and output a structured battlecard. During local testing with a few static URLs, it worked well. But when I went live against a shifting market, the system kept breaking without much notice: The Production Vulnerabilities I Encountered Context flattening: When parsing multiple long competitor pricing tiers, the model routinely dropped nuanced constraints, such as specific seat-count thresholds. It simply averaged out the data. Severe information loss: Instead of extracting the live web data provided in the context window, the model slipped back into its static pre-training data, hallucinating older features that the competitor had deprecated months prior. Prose without substance: Because the model had to handle data extraction, comparative reasoning, and copy editing simultaneously, it prioritized linguistic fluency over technical depth. The output looked like excellent marketing prose, but it was factually useless to our sales engineers. To fix this, I completely dismantled the monolithic prompt. I decoupled the system into three distinct programmatic agents, creating a clear engineering pipeline: Step 1: Establishing a Model-Agnostic Execution Boundary When I design multi-agent systems, my first rule is that agents must be decoupled from specific model providers. A production agent should depend on a stable, programmatic interface. This approach allows me to swap a cloud API like OpenAI for a local, open-weights model running via Ollama without changing a single line of business logic. Here is the standardized execution node I developed for this framework: Python import os from openai import OpenAI # I initialize the client container using environment boundaries client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) def execute_agent_inference(messages: list, target_model: str = "gpt-4o-mini") -> str: """ Provides a standardized execution node for all upstream agents to communicate with the designated model endpoint. """ response = client.chat.completions.create( model=target_model, messages=messages, temperature=0.1 # Low temperature enforces deterministic reasoning ) return response.choices[0].message.content Step 2: The Strategist Agent (Task Decomposition) The execution loop begins with the Strategist Agent. I isolated this node to handle a single cognitive task: ingestion and planning. Its sole job is to break down a broad user request into a chronological sequence of distinct tasks. Python def strategist_agent(user_objective: str) -> list: """ Ingests a broad objective and returns a structured execution plan. """ system_prompt = """ You are a project strategist. Your job is to break down a broad research objective into an ordered, numbered list of specific, non-overlapping data requirements. Do not summarize the topic. Output only the numbered steps. """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Objective: {user_objective}"}, ] raw_plan = execute_agent_inference(messages) # Parse the numbered rows into a clean Python list return [line.strip() for line in raw_plan.split("\n") if line.strip()] By forcing the system to map out its roadmap before running any resource-heavy tasks, I ensure the application maintains a strict operational scope. Step 3: Integrating External Tools With the Extraction Agent An agent is only as good as the data it consumes. I designed the Extraction Agent to never guess or extrapolate. Instead, I equip it with specific Python functions that fetch live, real-world data before it runs an inference cycle. Here, I define a simulated web search utility and an internal vector store look-up tool: Python def fetch_live_web_data(query: str) -> str: """ Simulates a live web lookup via external search providers like Tavily or SerpAPI. """ return f"[Live Web Match] Found current market documentation regarding: {query}" def query_internal_vector_store(query: str) -> str: """ Simulates a vector database query for internal technical specifications. """ return f"[Vector DB Match] Internal baseline spec data for: {query}" def extraction_agent(allocated_task: str, running_context: str) -> str: """ Gathers factual data using external retrieval tools before forming response notes. """ # Execute the tools first to ground the agent's context in real data web_insights = fetch_live_web_data(allocated_task) internal_insights = query_internal_vector_store(allocated_task) system_prompt = """ You are a data extraction agent. Your job is to analyze tool outputs and compile precise, evidence-dense technical notes. Strictly ground your response in the provided tool outputs. Do not extrapolate. """ user_payload = f""" Current Task: {allocated_task} Prior Context: {running_context} Tool Outputs: - {web_insights} - {internal_insights} """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_payload}, ] return execute_agent_inference(messages) Step 4: The Technical Reviewer Agent (Synthesis and Audit) The final step in my pipeline is the Technical Reviewer Agent. I do not use this agent as a passive text formatter. Instead, I design it to act as an internal critic that actively checks the gathered research for missing technical data. Python def technical_reviewer_agent(compiled_research_notes: str) -> str: """ Audits research materials and synthesizes a structured final technical report. """ system_prompt = """ You are a technical reviewer. Synthesize a clean report from the provided research notes. CRITICAL RULES: 1. Organize your output using clear Markdown headings and bullet points. 2. Do not introduce general knowledge or unverified claims. 3. If the data contains gaps, note them explicitly instead of smoothing over them. """ messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Research Notes:\n{compiled_research_notes}"} ] return execute_agent_inference(messages) Step 5: Constructing the Orchestration Loop With all my agents built, I put them together using a central orchestrator function. This loop manages the execution sequence, updates the running memory context between steps, and passes state across agent boundaries. Python def run_intelligence_engine(target_topic: str) -> str: """ Coordinates the execution sequence, updates persistent memory boundaries, and returns the finalized asset. """ print(f"[*] Initializing Strategy Phase for: {target_topic}") execution_steps = strategist_agent(target_topic) accumulated_notes = [] persistent_memory = "" for idx, step in enumerate(execution_steps, 1): print(f"[>] Executing Phase {idx}: {step[:50]}...") # Pass the running context so the agent knows what has been researched so far step_output = extraction_agent(step, persistent_memory) accumulated_notes.append(step_output) # Update the persistent memory to prevent duplicate work in later steps persistent_memory += f"\n[Completed Phase {idx} Info]: {step_output}\n" print("[*] Compiling and Reviewing Final Deliverable...") final_report = technical_reviewer_agent("\n".join(accumulated_notes)) return final_report if __name__ == "__main__": report_output = run_intelligence_engine( "Analyze competitor pricing models for cloud infrastructure shifts" ) print("\n--- Final Report Output ---\n") print(report_output) Resolving the Hidden Challenge: Information Degradation When I first launched a framework like this, I noticed a subtle engineering issue: the information handoff problem. When multiple agents pass unstructured text back and forth, the data risks losing clarity at each step. If the Strategist designs broad steps, the Extractor returns summarized notes, and the Reviewer formats them aggressively, the final output loses its technical precision. To keep your multi-agent networks at their best in production, I recommend implementing these two programmatic practices: 1. Maintain Strict Structural Memory Controls Never pass a raw conversational history across agent boundaries. Instead, require your extraction nodes to return explicit, structured technical updates (such as clear key-value maps or clean markdown bullet records). This approach preserves specific variables like precise pricing values or hardware specs, all the way to the final synthesis step. 2. Implement Automated Validation Gates Do not use an LLM to check if its own output is correct. Instead, place deterministic variable Python validation gates between agent handoffs. I write small programmatic checks to verify that the text matches a required schema, meets minimum character counts, or contains key terms extracted from the retrieval tools before letting the pipeline proceed. Measurable Production Outcomes Transitioning our enterprise tracking engines from monolithic prompt templates to this decoupled multi-agent architecture delivered immediate, verifiable improvements across our core operational metrics: Drastic reduction in hallucination rates: By isolating the extraction agent and grounding its context entirely in live tool calls, our documented hallucination rate fell from 7.2% to less than 0.2%.System traceability: When an output degrades, my team and I no longer dig through thousands of lines of a single prompt history. We simply look at the independent logs of each agent to find exactly where the data chain broke, reducing our Mean Time to Resolution (MTTR) from hours to minutes.Operational maintainability: I can update, optimize, or replace individual components, such as updating a web scraping API or refining the Reviewer's styling guide, without breaking or re-testing the rest of the application ecosystem. Conclusion The true test of an enterprise AI application is not how well it runs a basic query on a local development machine. Real success is defined by how reliably the application handles messy, dynamic data in production over time. By separating monolithic prompts into a coordinated pipeline of role-based agents, I turned unpredictable model outputs into stable, dependable software infrastructure. A perfect framework lies in distributing cognitive responsibility, creating clear interfaces, and engineering strict control boundaries around your models. Thank you for reading. Designing multi-agent AI systems for enterprise LLM workflows goes beyond calling powerful models; it requires thoughtful system design, coordination between agents, strong observability, and scalable architecture that can operate reliably in production.

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along.

A large API response becomes a client problem long before it becomes a network problem. A browser can receive hundreds of megabytes and still become unresponsive while buffering bytes, parsing one enormous JSON document, retaining duplicate object graphs, and rendering too much state on the main thread. The reliable solution is not a larger timeout. It is to stop treating the response as a synchronous document and start treating it as a durable, observable job whose data arrives in bounded pieces. Browser streams support incremental consumption and backpressure, while background workers allow long-running processing to remain independent of user-interface scripts. The Response Becomes a Job, Not a Payload The public API should acknowledge work quickly and return a stable job identifier rather than hold an HTTP connection open until every upstream page has been fetched. A 202 Accepted response establishes that contract without implying completion. The client can then subscribe to progress events, request a partial view, or retrieve a final artifact when the job reaches a terminal state. RFC 9110 defines 202 Accepted specifically for requests accepted for processing when processing has not necessarily completed. Java @PostMapping("/reports") public ResponseEntity<JobAccepted> create(@RequestBody ReportRequest request) { String jobId = UUID.randomUUID().toString(); workflowClient.start(reportWorkflow::run, jobId, request); return ResponseEntity.accepted() .header("Location", "/reports/" + jobId) .body(new JobAccepted(jobId, "QUEUED")); } This endpoint performs no large download or expensive transformation. It creates an addressable unit of work and returns immediately. The browser remains responsive because the initial response is tiny, while server capacity is protected from long-lived request threads. The job record should expose states such as queued, fetching, indexing, ready, failed, and canceled, with progress kept monotonic and coarse enough to remain trustworthy. Temporal Owns the Long-Running Control Flow Temporal fits the control plane because Workflow state survives process crashes and worker restarts, while failure-prone operations such as remote API calls belong in Activities with explicit timeouts and retry policies. Temporal documentation distinguishes deterministic Workflow logic from non-deterministic Activities and provides retry, timeout, heartbeat, and message-passing mechanisms for long-running execution. Java @WorkflowMethod public ResultRef run(String jobId, ReportRequest request) { String cursor = null; int sequence = 0; do { PageRef page = activities.fetchAndStore(jobId, cursor, sequence); activities.publishChunkReady(jobId, page); cursor = page.nextCursor(); sequence++; } while (cursor != null && !canceled); activities.buildIndex(jobId); activities.publishCompleted(jobId, sequence); return new ResultRef(jobId, sequence); } @SignalMethod public void cancel() { canceled = true; } Only references and counters should cross Workflow boundaries. Passing raw pages through Temporal causes every Activity input and result to accumulate in Event History. Temporal warns that large histories increase Workflow Task latency, documents a 50 MB or 51,200-event history limit, and recommends external storage plus Continue-As-New for large or long-running executions. The response body therefore belongs in object storage, while Temporal retains keys, checksums, cursors, and status. The fetching Activity should checkpoint often enough to support retries without restarting the transfer. Heartbeat details can carry the last committed cursor or byte range. Temporal recommends heartbeats for long-running Activities because missed heartbeats can trigger failure detection and retry. Java public PageRef fetchAndStore(String jobId, String cursor, int sequence) { UpstreamPage page = upstream.fetch(cursor); String key = storage.put(jobId + "/" + sequence, page.bytes()); Activity.getExecutionContext().heartbeat( new FetchCheckpoint(sequence, page.nextCursor()) ); return new PageRef( key, sequence, page.nextCursor(), page.sha256() ); } Kafka Carries Facts, Not Giant Documents Kafka is most effective as the event backbone, not as a substitute for object storage. Events should describe what happened and point to durable data, ChunkStored, ChunkIndexed, JobProgressed, JobCompleted, or JobFailed. Kafka enforces record-size limits at both producer and broker levels, so pushing multi-megabyte fragments into records creates brittle configuration coupling and expensive retries. Every event should use jobId as the key. Kafka partitions are ordered logs, and records sharing a key normally land in the same partition, preserving per-job sequence while allowing unrelated jobs to scale across partitions. Consumer groups distribute partitions across workers and rebalance them when membership changes. Java public void publishChunkReady(String jobId, PageRef page) { ChunkReady event = new ChunkReady( jobId, page.sequence(), page.storageKey(), page.sha256() ); kafkaTemplate.send("report-events", jobId, event); } Duplicate delivery must be assumed at every boundary. Kafka producer idempotence prevents duplicate writes caused by producer retries when compatible acknowledgment and in-flight settings are used, but downstream side effects still require idempotent consumers. An indexer can enforce uniqueness with (jobId, sequence, checksum) and commit its database transaction before acknowledging the Kafka offset. Backpressure should be expressed through bounded concurrency rather than hidden in memory. An Activity can publish one stored chunk at a time, while indexer lag indicates downstream pressure. Temporal can pause between pages when lag crosses a threshold, or consumers can scale until partition count becomes the limit. The Client Receives Progress and Bounded Content Server-sent events are sufficient when communication is primarily server-to-client. The protocol uses text/event-stream, keeps a persistent HTTP connection, and represents each notification as a small text block. A projection service can consume Kafka events, maintain the latest job state, and expose a resumable stream using application event IDs Java @GetMapping( value = "/reports/{jobId}/events", produces = MediaType.TEXT_EVENT_STREAM_VALUE ) public Flux<ServerSentEvent<JobEvent>> events( @PathVariable String jobId) { return eventProjection.stream(jobId) .map(event -> ServerSentEvent.<JobEvent>builder() .id(event.sequence().toString()) .event(event.type()) .data(event) .build()); } The client should render status changes and small previews, not append the full raw response into application state. When direct streaming is required, the Fetch API exposes the response body as a ReadableStream, allowing chunk-by-chunk processing rather than waiting for completion. Parsing should occur incrementally, with CPU-heavy decoding or transformation moved to a Web Worker, whose execution remains separate from user-interface scripts. Final delivery should usually be a paginated query API, a range-readable artifact, or a signed download URL. A giant JSON reconstruction endpoint merely recreates the original failure at the last step. RAG Turns Stored Volume Into a Useful Interface RAG becomes valuable after chunks are durably stored. Each chunk can be normalized, split along semantic boundaries, embedded, and indexed with metadata containing the job identifier, source sequence, object key, and byte range. The original RAG formulation combines parametric generation with retrieved non-parametric memory, grounding generation in selected passages rather than the entire corpus. Java @KafkaListener( topics = "report-events", groupId = "rag-indexers" ) public void onChunkReady(ChunkReady event) { if (index.exists( event.jobId(), event.sequence(), event.checksum())) { return; } byte[] payload = storage.get(event.storageKey()); chunker.split(payload).forEach(chunk -> index.upsert( event.jobId(), event.sequence(), chunk ) ); progress.markIndexed( event.jobId(), event.sequence() ); } The query path retrieves only the most relevant chunks and sends those bounded passages to the model. Raw object references remain attached so generated statements can link back to source material. RAG should not conceal incomplete ingestion; the query service must expose index coverage and reject complete-report requests until all expected chunks are indexed. Java public Answer answer(String jobId, String question) { List<Passage> context = index.search(jobId, question, 8); return generator.generate(question, context); } This layer changes the client experience from downloading everything before anything is useful to inspecting progress, searching partial results, and retrieving only relevant evidence. It also keeps model context bounded when the source response is extremely large. A Responsive System Is Built From Explicit Boundaries The essential boundary is simple: Temporal owns durable intent and recovery, Kafka distributes compact facts, object storage holds large bytes, RAG builds a searchable semantic view, and the client receives only bounded updates or explicitly requested slices. Each component solves a different failure mode, and none is forced to carry the complete response through an interface designed for small messages. The resulting architecture prevents UI freezes, survives retries and restarts, supports cancellation and replay, and makes large upstream results useful before a monolithic download could finish. Large-response handling becomes reliable when completion is modeled as a process rather than a payload.

By Uthej Mopathi CORE

A pull request arrives. A few hundred lines of Java implementing the new discount rule: tiered thresholds, a regional exception, something about loyalty tiers that nobody can quite explain. It compiles. The tests pass. An LLM wrote it in about forty seconds. Now: who reviews it? The person who owns that rule is in commercial operations. She knows exactly which customers should get the discount and why the regional exception exists, and she cannot read Java. The person who can read Java has no idea whether the thresholds are right. He will check that the code looks reasonable, because that is the only thing he is equipped to check. So the review that happens is not the review that matters. That is the problem I keep coming back to, and it has nothing to do with how good the model is. This Is Not an Argument About Whether the Model Is Good Enough Most objections to generated code are about competence. The model hallucinates an API. It gets an edge case wrong. It writes something that works on the happy path and falls over in production. I find these arguments unconvincing because they expire. Models get better. Any position resting on today's error rate is a position with a shelf life, and people who staked one out three years ago have mostly had to retreat from it. The durable question is different. It is not how well the model writes. It is what the thing it writes is permitted to say. A model that never makes a mistake, handed Java, can still emit Runtime.getRuntime().exec(...). Not because it is malicious or confused — because that sentence is available in the language it was asked to write. Competence and authority are separate axes, and improving the first does nothing to the second. "Write it in Java" Is a Much Bigger Grant Than Anyone Means Consider what you actually authorize when you ask for a discount rule in Java. You authorize file system access. Network sockets. Reflection. Thread creation. Process execution. Every class on the classpath, including the ones that talk to your database, your payment provider, and your secrets manager. You authorize the loading of new code at runtime. Nobody intends to grant any of this. It arrives free with the language, the way a house key also opens the shed. The task needed perhaps six operations — look up an order, total it, check a customer's tier, apply a discount, log the decision, approve or refuse — and the language you handed over contains everything Java contains. That gap, between the authority the task requires and the authority the language confers, is the whole of it. It exists whether or not the model is trustworthy. It exists whether or not anyone acts on it. It is just very large, and it is not visible in the pull request. The Usual Guardrails Are Denial Lists The standard responses all share a shape. Tell the model in the prompt not to touch the file system. Review the generated code. Run static analysis and flag dangerous calls. Run it in a sandbox with a restricted security policy. Every one of these asks you to enumerate what must not happen, over a space of things that can happen which is effectively unbounded. You are writing a deny-list against a general-purpose language. You have to think of exec. Then of reflection reaching exec. Then of the dependency that shells out on your behalf. Then of the next one. We learned this lesson in security a long time ago and reached a settled answer: allow-lists beat deny-lists, because the allow-list is finite and you wrote it. Somehow, when the subject is generated code, we reach for the deny-list again. Shrink the Language, Not the Model The alternative is to stop constraining a powerful language and instead supply a small one. Give the model a vocabulary that contains exactly the operations the domain has — the six from earlier, say — and nothing else. Not a restricted Java. A different, much smaller language, whose entire vocabulary is a list your team wrote in advance, in Java, on purpose. Generated business logic then looks like this: Python PROGRAM ApproveOrder(orderId INTEGER, limit DECIMAL) RETURNS BOOLEAN DECLARE purchase Order DECLARE total DECIMAL purchase = LOAD_ORDER(orderId) total = ORDER_TOTAL(purchase) IF total > limit THEN REJECT purchase, "over limit" RETURN FALSE END IF APPROVE purchase RETURN TRUE END. LOAD_ORDER, ORDER_TOTAL, REJECT and APPROVE are not part of the language. They are Java classes somebody decided to expose. Order is a Java object the program can hold and pass and never look inside — there is no purchase.customer.account.balance here, only the operations the domain chose to have. Two things change, and the second matters more than the first. The obvious one: dangerous programs are no longer forbidden, they are inexpressible. If the model emits DELETE_ALL_ORDERS, nothing rejects it on policy grounds. The name means nothing. The program does not compile, for the same reason a typo does not compile. There is no deny-list because there is nothing to deny. The less obvious one: the commercial operations manager can read the program above. She can tell you whether the threshold is right, whether the rejection reason is the one the contract requires, whether an approval should have been logged. The review moves to the person who owns the rule. That is the review that was missing at the start of this article, and no amount of static analysis over generated Java produces it. A small language buys something else, quietly. With no data structures, one global scope, no null, and a compiler that refuses to run a program that reads a variable before it is set, entire families of subtle wrongness have nowhere to live. Not caught — absent. What It Costs, and What It Does Not Buy I would not trust this argument from someone who only listed the advantages, so here are the bills. You have to design the vocabulary. Somebody sits down and decides that the domain has ORDER_TOTAL and CUSTOMER_RISK and not forty other things. That is real work, done before the first generated line, by someone who understands the domain. And if nobody on your team can write that list, this approach will not help you. It will only show you that the list does not exist. That is worth finding out, but it is not a pleasant morning. Complex algorithms stay in Java. Business rules are algorithms too, and they belong in the small language; that is the point. But route optimization, a scoring model, anything with real computational substance belongs behind a function the small language calls. The signal is usually that you want to build up a data structure, or that you want a helper you can call from three places. Both mean you have wandered out of business logic and should walk back. The boundary bounds naming, not doing. This is the limit people miss, and overstating it is how the idea gets dismissed. A function you expose can do anything Java can do. RUN_SHELL_COMMAND is a perfectly registrable operation. The vocabulary is only as narrow as the operations you chose, and choosing them badly gets you exactly the exposure you were avoiding. There are no resource limits yet. A generated program can still loop forever. This one is a gap rather than a decision: the interpreter walks the program one statement at a time, so a step budget or a deadline is a small addition rather than a redesign, and it will go in when somebody needs it. Until then, untrusted input needs the same containment any untrusted workload needs. What you get is narrower than "safe" and more useful than it sounds: the set of things a generated program can name is finite, written down, and reviewable by a human before anything is generated at all. When I Would Still Write Java If the thing is genuinely computational, write Java. If it is a one-off that will be deleted next week, use whatever is nearest — Java, Python, a shell script — and let the model write it; do not build a vocabulary for something with a life expectancy of days. If the rules change so fast that the vocabulary would be obsolete before it settled, the overhead will not pay for itself. And if your business logic is already reviewed by people who can read it, understand it, and are accountable for it being right — you may not have the problem this solves. Plenty of teams do not. But if you are about to let a model write business rules in Java, ask the question I started with, because the answer is usually uncomfortable. Somebody is going to approve that pull request. Are they the person who knows whether the rule is correct? If not, the language is too big. I have been building a small language along these lines: BUBAS, an orchestration language for subject-matter experts, embedded in Java. The example above is real BUBAS. The idea does not require my implementation, though — the argument is about the size of the language you hand over, and you can shrink yours however you like.

By Peter Verhas CORE

Parallel coding agents create a concurrency problem before they create a productivity gain. Two autonomous processes that edit the same checkout can overwrite files, invalidate assumptions, contaminate test state, or produce changes that are individually correct but jointly incompatible. A safer operating model treats each agent as an isolated contributor with a dedicated Git worktree, an explicit file-level contract, deterministic validation commands, and no authority to integrate directly into the protected branch. Git worktrees provide multiple linked working trees for one repository, while modern coding-agent platforms independently reinforce the same principle through isolated sandboxes, scoped write access, and controlled network permissions. Isolation Before Parallelism The repository should expose one branch and one working directory per agent. Git worktrees are preferable to two processes sharing a checkout because each linked worktree has its own checked-out branch and worktree metadata while remaining attached to the same repository. Git explicitly supports multiple working trees and provides lifecycle commands for adding, listing, removing, locking, and pruning them. A practical setup can start both tasks from the same known commit: Shell git fetch origin git worktree add ../agent-auth -b agent/auth origin/main git worktree add ../agent-checkout -b agent/checkout origin/main The important property is not directory convenience but isolation of mutable state. The authentication agent can compile, format, generate files, and modify its branch without changing the checkout seen by the checkout agent. This mirrors the isolation used by cloud coding agents: OpenAI describes Codex cloud tasks as isolated containers, while GitHub limits its cloud coding agent to a dedicated branch and subjects that branch to repository protections. Parallelism still requires ownership boundaries. Separate worktrees prevent filesystem collisions, but Git cannot prevent two branches from independently editing the same contract. A useful policy assigns feature-local paths to each agent and reserves cross-cutting files such as dependency manifests, database migrations, CI workflows, shared schemas, and public interfaces for an integration task. Concurrent edits to build.gradle, an OpenAPI document, or a shared DTO can create semantic conflicts even when Git reports no textual conflict. The safest default is therefore narrow write scope, not broad repository access. Contracts Turn Prompts Into Boundaries Agent instructions should be treated as executable operating contracts rather than conversational prompts. Current agent systems already support repository-level instruction files, and Codex reads AGENTS.md before work begins and supports directory-specific overrides, while GitHub Copilot repository instructions can describe how a project should be built, tested, and validated. A tool-neutral contract can make scope and completion criteria machine-checkable: YAML agent: checkout base: origin/main allowed_paths: ["src/main/java/com/acme/checkout/**", "src/test/java/com/acme/checkout/**"] forbidden_paths: ["build.gradle", ".github/**", "api/**"] validation: ["./gradlew test --tests '*Checkout*'", "./gradlew spotlessCheck"] integration: "rebase-then-review" The contract should be enforced outside the model as well. An agent stating that only checkout files changed is weaker than a gate deriving the changed-path set from Git. git diff is designed to compare trees, commits, the index, and working-tree state, so scope checks can be based on repository truth rather than agent self-reporting. A completion gate can remain deliberately small: Shell git diff --check git diff --name-only origin/main...HEAD ./gradlew clean test The changed-path output can be matched against the contract before review. A clean build matters because two long-running agents can leave generated output or caches that conceal missing dependencies. Feature-specific tests provide fast local feedback, but the final gate should run the repository’s normal clean validation path. The agent contract should also require small, coherent commits so rejected or accepted changes remain separable during integration. Integration Is a Gate, Not a Merge Integration should occur only after the branch is refreshed against the current base. Git rebase replays topic-branch commits on top of an upstream base, which makes stale assumptions visible before final validation. For a short-lived agent branch, the sequence is straightforward: Shell git fetch origin git rebase origin/main ./gradlew clean test A conflict during rebase is useful information, not merely friction. It signals overlapping ownership or an assumption that changed while the agent was running. Conflict resolution should preserve the current base contract first, then reapply the feature intent, followed by the complete validation suite. Re-running only the previously failing test is insufficient because the resolved file may sit on a wider dependency path. Merge, rebase, and cherry-pick serve different integration needs. git merge incorporates the histories of diverged branches, while rebase rewrites a topic branch by replaying its commits onto another base. git cherry-pick applies the changes introduced by selected commits and is useful when only part of an agent branch is acceptable. Cherry-picking should remain selective rather than becoming a substitute for disciplined branches, as partial adoption becomes difficult when commits mix refactoring, generated files, dependency changes, and feature logic. The most dangerous failure is a green branch that becomes red only after another agent merges. Strict required status checks reduce that risk by requiring a branch to be up to date with its base before merging, and GitHub merge queues can validate changes against the latest target branch plus queued changes. Even without a hosted merge queue, the same principle can be implemented with a temporary integration branch that combines both agent branches and runs the full build before either change reaches main. Security and Lifecycle Bound the Blast Radius Coding agents execute model-generated commands, so repository isolation should be paired with credential isolation. Network access should remain disabled unless task requirements justify it, filesystem write access should be limited to the assigned worktree, and production credentials should never be placed in repository files or general shell profiles. OpenAI’s Codex security guidance describes workspace-limited local writes, network-off defaults, and cloud secrets that are removed before the agent phase, GitHub similarly recommends minimum GITHUB_TOKEN permissions and avoiding plaintext sensitive data in workflow files. CI should enforce the same boundary. Protected branches can require successful checks and reviews before integration, and secret-scanning push protection can block recognized credentials before they enter repository history. Agent-generated workflows deserve additional scrutiny because automation with write credentials expands the blast radius beyond source edits. A default read-only token, explicit permission elevation for narrowly defined jobs, and human approval for changes to workflows or deployment configuration provide a stronger control plane. GitHub’s secure-use guidance explicitly recommends least-privilege workflow credentials. Worktrees should be disposable after integration. Git recommends git worktree remove for finished linked worktrees and provides prune for stale administrative metadata. Unclean worktrees are protected from ordinary removal unless force is requested, which makes final inspection practical before deletion. Cleanup can remain explicit: Shell git worktree remove ../agent-auth git worktree remove ../agent-checkout git worktree prune Conclusion Running two coding agents safely on one codebase is primarily a source-control and governance problem. Reliable parallelism comes from isolated worktrees, narrow path ownership, versioned agent instructions, Git-derived scope checks, clean validation, protected integration, least-privilege credentials, and deliberate cleanup. The central rule is simple: agents may work concurrently, but mutable state, authority, and acceptance must remain separated. With that boundary in place, parallel agent execution becomes an auditable engineering workflow rather than two autonomous processes racing inside the same repository.

By Uthej Mopathi CORE

Any input/output operation, be it accessing a file, handling an HTTP request, or a database connection, is based on 3 fundamental system concepts — file descriptors, kernel memory, and heap size. This article discusses how modern languages help developers handle behind-the-scenes file descriptor, kernel memory, and heap management. These three concepts are major bottlenecks for scaling. 1. File Descriptors A file descriptor is just a positive number that is used by the kernel to identify any open input/output stream or connection. It is defined by the kernel for a process. The following file descriptors are defined by default for a process: 0 – Standard Input (stdin)1 – Standard Output (stdout)2 – Standard Error (stderr) Any subsequent I/O operation gets the next available integer as file-descriptor. The file descriptor value can be adjusted by using the ulimit -n command in Linux. Each application, whether it is a web server written in Java Spring Boot, an API server written in Go using net/http and gorilla-mux, or a Python Flask app, is a single process. Each process has only 1024 file descriptors defined by default. That means each application can perform only 1024 I/O operations simultaneously. This seems like an amazing concept when we talk about scaling our application or API server. As many times as we come across this question — how can we scale our API server or web application to handle 100k or 1 million requests per second? This is where our modern languages play their role very beautifully behind the scenes to enable developers to develop the application to handle such scale. 2. Kernel Memory At a lower layer than file descriptors, when an incoming TCP connection hits the network card, the Linux kernel performs a 3 Way TCP handshake for that connection. The handshake lifecycle includes the states: SYN -> SYN-ACK -> ACK. The number of requests equal to the defined file descriptor value are processed immediately, assigned a file descriptor, and forwarded to the application for further processing. When FDs are exhausted, the Kernel maintains a queue for requests waiting for FDs to become available so your application can process them. The same thing happens when a request is processed, and the response is ready to be sent back to the client. This queue is maintained within RAM by read buffers(rmem) and write buffers(wmem). The size of buffers is defined in memory by the kernel and is dynamic, depending on network throughput, round-trip time, and memory pressure. The kernel network memory is non-paged, i.e cannot be swapped to disk. It’s a big bottleneck as it directly depends on physical memory. For example, if there are 100,000 open connections and each connection holds an average of 128KB of kernel memory, it comes to 12.8GB of physical RAM. This is clearly a kernel overhead, and it doesn’t show up in JVM heap metrics or Go runtime statistics. rmem and wmem buffers are governed by kernel parameters defined in /proc/sys/net/ipv4/ 3. Heap Size When TCP connections are assigned file descriptors and kernel memory is reserved, they enter user space, which is the memory managed by the application runtime — Java JVM, Node.js V8 Engine, Python interpreter, Go runtime, etc. Each connection stores objects in the heap within three categories: Connection metadata – Keep-alive timers, IP State, Socket Wrappers, etc.Cryptographic session context – handshake caches, cipher states, TLS/SSL keys, etc.Serialized payload buffers – response queues, JSON strings, ORM entity maps, etc. A connection that is encrypted via TLS takes a lot more space in the heap compared to a regular connection. For an encrypted connection, the application has to save symmetric keys, cipher contexts, session tickets, etc. onto the heap. A regular TCP socket object in the heap consumes 2KB to 5KB of space, whereas a TLS 1.3 socket object consumes 20KB to 100KB of heap space. If an API maintains 10,000 idle TLS connections, it will consume 200MB to 1GB of heap space. When an application runs, the runtime asks the kernel for memory space as the application creates objects. The application keeps creating objects, and the kernel keeps reserving memory for those objects; this is called the heap. The maximum heap size can be defined by different programming languages at runtime; for example, in Java, -Xmx4g reserves 4GB for the heap. The operating system promises to provide that much memory as heap space for the application, but it doesn’t reserve it all at once. As the application creates objects, the kernel continues to reserve memory. When objects are marked as done, the garbage collector removes them from the heap. When an incoming request hits our API server, the application uses heap space to convert raw bytes to the application-specific data structure. Once the application finishes processing the request and returns the response, those objects in the heap become unreachable or dead. When the garbage collector sweeps those objects to reclaim that memory, it doesn’t return the memory immediately; instead, the JVM or Go runtime keeps that freed memory in its internal pool. If a new HTTP request arrives within 1 millisecond, the runtime assigns the required memory from the free memory in the pool. Now imagine 10,000 new requests arriving at the same time, each with 2MB of raw bytes, and the runtime trying to allocate heap for the objects; the app instantaneously uses 20GB of memory. This is called GC thrashing, as the runtime rapidly creates required objects in the heap faster than the GC can clean them. The garbage collector is an application thread itself; when the heap gets 80%-90% full, the garbage collector panics and consumes 100% of CPU cores to scan millions of memory pointers to find dead objects. The runtime, like the JVM or Node.js garbage collector, may stop other code execution while it reorganizes the memory. So, how do runtimes like Go and the JVM handle GC thrashing? Go follows a simple strategy – avoid creating objects on the heap. The fastest GC collector is the one that has nothing to collect. The Go compiler compiles the application to see if variables outlive their functions. If a struct is used only inside a function, Go pushes the struct to the stack instead of the heap, and the stack pointer just drops when the function returns. The memory is reclaimed in 1 CPU cycle without even involving the garbage collector. If Go does have to clean the heap, its GC runs concurrently along with other goroutines and is broken into several micro pauses. Go provides sync.Pool to help developers to reuse heap memory while creating objects. For example to instead of creating millions of []bytes for JSON parsing for every new request, developers can use sync.Pool as follows: Go // Instead of creating a new buffer for every HTTP request: var bufferPool = sync.Pool{ New: func() any { return new(bytes.Buffer) }, } func handleRequest(w http.ResponseWriter, r *http.Request) { buf := bufferPool.Get().(*bytes.Buffer) // 1. Grab an existing buffer from pool buf.Reset() defer bufferPool.Put(buf) // 2. Put it back when done! // Parse JSON into 'buf' without allocating new heap memory } By recycling buffers via sync.Pool, high-concurrency APIs can handle 100,000 requests/sec with near-zero new heap allocations. Java takes a different approach. Because Java applications historically create millions of short-lived objects on the heap, the JVM relies on Generational Hypotheses and Generational Collectors (like G1GC, ZGC, and Shenandoah). G1GC can be used like java -XX:+UseG1GC while running Java applications. G1GC divides the Heap memory into physical regions: Young Generation (Eden & Survivor spaces) and Old Generation. It kind of sorts objects into different regions so that it doesn't have to scan the complete heap and can clean where most of the marked objects live. We can also mention -XX:MaxGCPauseMillis=200 to tell G1 to pause the application for no more than 200ms, but this is not guaranteed. Older JVM collectors like Parallel GC used to freeze the entire application to clear the heap when full, leading to multi-second latency spikes. Modern JVMs introduce ZGC (Z Garbage Collector) and Shenandoah. ZGC uses specialized CPU pointer references to track moved objects in real time. ZGC can clean, move, and compact terabytes of heap memory concurrently while your API requests are actively running. ZGC guarantees GC pause times under 1 millisecond, regardless of whether your heap is 500 MB or multi-terabytes. Conclusion Keep track of these three core concepts — file descriptors, kernel memory, and heap size to know when to scale. 1. File Descriptor Saturation Signals File descriptors represent the system's open handles. When an application hits its FD threshold, the operating system stops accepting connections. The following are example scenarios that indicate when to scale. Check Kernel-wide statistics from /proc/sys/fs/file-nr, per process fds - /proc/<pid>/fd, Prometheus exposes process_open_fds. If it consistently breaches the 80–85% threshold, it's time to scale. You have already tuned ulimit -n and LimitNOFILE up to standard safety thresholds (e.g., 65,536 or 104,857), but process FD counts continue climbing toward the max. Network interfaces show growing SYN-to-LISTEN socket counts and drops in netstat -s under the listen queue overflow metric. 2. Kernel Memory Pressure Signals Because TCP receive (rmem) and transmit (wmem) buffers are non-paged, they cannot overflow onto disk swap. When kernel network memory fills up, the OS drops packets. Below are the scenarios related to kernel memory breach. Check /proc/net/sockstat under TCP: inuse and matching /proc/sys/net/ipv4/tcp_mem thresholds. Netstat counters (netstat -s | grep -i retrans) show a sharp rise in TCP Retransmission rates (>1–2%). Latency spikes occur because the kernel is dynamically shrinking socket buffers down to tcp_rmem minimums (4 KB) to avoid running out of physical RAM, throttling TCP window sizes. 3. Heap Size & Garbage Collection (GC) Thrashing Signals When user-space heap allocations outpace the garbage collector's ability to sweep dead objects (like parsed JSON payloads or session states), application performance collapses. The runtime (JVM or Go) spends more than 15–20% of its total CPU time running GC sweeps (go_gc_cpu_fraction or JVM GC CPU utilization). In Go, metrics show the pacer triggering Mark Assist, stealing CPU time from worker goroutines to help clean up memory. You can check the runtime package /cpu/classes/gc/mark/assist:cpu-seconds metrics to see if GC is asking for more help from CPU. In Spring Boot, you can use Actuator and Micrometer to expose relevant endpoints to monitor the threshold values.

The first version of almost every API client I write looks embarrassingly simple. Send a request. Parse the JSON. Return the result. Something like this: JSON import requests def get_data(url):response = requests.get(url)response.raise_for_status()return response.json() For a quick test, that is usually enough. Then I leave it running for a while. Eventually the connection hangs, the API returns a 500, or I get a 429 because I was a little too aggressive with polling. That is usually the point where the “simple client” stops being simple. The interesting part is not making the request. It is deciding which failures are worth retrying and which ones should fail immediately. That distinction matters more than adding a generic retry loop around everything. The First Thing I Add Is a Timeout I used to treat timeouts as an optional detail. I do not anymore. A request without an explicit timeout can wait much longer than expected when the remote service is slow or unreachable. For a script that runs once, that is annoying. For a worker or monitoring process, it can gradually turn into a much larger problem. So even before thinking about retries, I normally start with: Python response = requests.get(url,params=params,timeout=10) Ten seconds is not a universal recommendation. It depends on what the API is doing. But I prefer having a number I deliberately chose over letting a network call wait indefinitely. For a lightweight market-data endpoint, ten seconds already feels generous. For a large export or a slower internal service, I might choose something else. The important part is that the timeout is intentional. Not Every Error Should Be Retried This was probably the mistake I made most often when I first started adding retry logic. The naive version looks like: Python for attempt in range(5):try:return make_request()except Exception:time.sleep(2) It feels robust because the program “keeps trying.” In reality, it can make things worse. If the server returns 401 Unauthorized, retrying the same request five times will not fix the credentials. If the endpoint returns 404, waiting two seconds and asking for the same missing resource again is usually pointless. If the request itself is invalid, a retry just repeats the same bad request. The failures I usually consider temporary are things such as: connection errors and timeouts429 Too Many Requestssome 5xx server errors Everything else deserves more careful treatment. A client should not confuse persistence with resilience. A Small Retry Function For smaller projects, I like keeping the behavior visible rather than hiding everything inside a large abstraction. A basic version might look like this: Python import randomimport timeimport requests RETRYABLE_STATUS_CODES = {429,500,502,503,504} def get_json(url, params=None, max_attempts=4):for attempt in range(max_attempts):try:response = requests.get(url,params=params,timeout=10) if response.status_code in RETRYABLE_STATUS_CODES:raise requests.HTTPError(f"Temporary HTTP error: {response.status_code}",response=response) response.raise_for_status()return response.json() except (requests.Timeout,requests.ConnectionError,requests.HTTPError) as exc: if attempt == max_attempts - 1:raise delay = (2 ** attempt) + random.uniform(0, 1) print(f"Request failed: {exc}. "f"Retrying in {delay:.2f}s") time.sleep(delay) There is nothing particularly sophisticated here. That is partly why I like it. I can read the function six months later and immediately understand what it will retry. Why I Add Jitter The random.uniform(0, 1) part looks insignificant, but it solves a real problem. Imagine several workers call the same API and all receive a temporary failure at roughly the same moment. Without jitter, they might all retry after: 1 second2 seconds4 seconds8 seconds They stay synchronized. So instead of reducing pressure on the service, they repeatedly hit it together. Adding a small random component spreads those retries out. For one local script, this barely matters. For multiple workers or scheduled jobs, it starts to matter quite a lot. It is a small example of something I see often in backend work: code that behaves perfectly with one process can behave very differently when twenty copies are running. 429 Needs a Little More Respect Rate limits are also a case where simply retrying quickly is the wrong response. If an API says “slow down,” sending the same request again immediately is not resilience. It is ignoring the server. If the response includes a Retry-After header, I would rather respect it: Python retry_after = response.headers.get("Retry-After") if retry_after:delay = float(retry_after)else:delay = (2 ** attempt) + random.uniform(0, 1) This also makes the client less dependent on my guess about how aggressive the rate limit is. When there is no explicit guidance, exponential backoff is still a reasonable fallback. Logging the Failure Is More Useful Than It Sounds One thing I underestimated for a long time was logging. When I was running scripts manually, print() felt good enough. The problem appears later, when somebody asks: “Why did this job miss data at 03:12?” If all I know is “the request eventually failed,” debugging becomes guesswork. At minimum, I want to know: When the request failedWhich endpoint failedThe HTTP status if one existedWhich retry attempt it wasHow long the client waitedWhether the final attempt failed permanently For a real service, I would use Python's logging module instead of scattered print statements. The logs do not need to be verbose. They need to answer questions later. That is a different goal. Retrying Writes Is More Dangerous GET requests are usually where retry logic feels straightforward. POST requests make me more cautious. Suppose a client submits an order or creates a resource. The server processes it successfully, but the connection drops before the client receives the response. From the client's perspective, the request “failed.” If it blindly retries, the operation could happen twice. This is where idempotency becomes important. If an API supports idempotency keys or client-generated request IDs, I use them for operations where duplicate execution would be a problem. Otherwise, I want the retry behavior for writes to be much more conservative than the behavior for reads. This is especially relevant in financial systems. A duplicate market-data request is annoying. A duplicate order is something else entirely. The Same Pattern Shows Up in Trading APIs I run into these problems a lot when looking at market-data and trading integrations. The domain makes the trade-offs easier to see because APIs are often being called continuously rather than once. A price-monitoring process may run for hours. A worker may request candles repeatedly. A trading application may depend on several remote services at the same time. BYDFi is one platform I encounter through my work, so I mention it here as a disclosed real-world example rather than an independent recommendation. The useful engineering lesson is not specific to one exchange. Whether the client talks to a trading platform, payment provider, cloud service, or internal API, the same questions keep appearing: What happens when the service is slow? Which errors are temporary? How often should I retry? Could retrying create a duplicate side effect? What information will I need when debugging this tomorrow? Those questions are much more important than the first successful API response. I Usually Keep the Client Boring There is always a temptation to turn a small HTTP wrapper into a miniature framework. I try not to. For most projects, I would rather have a client with obvious behavior than one with ten layers of abstraction. The version I want is usually boring: explicit timeout, a short list of retryable errors, limited attempts, backoff, jitter, useful logs, and special handling for operations that should not be duplicated. Nothing about that is clever. That is the point. Network failures are already unpredictable enough. I do not want the recovery logic to be unpredictable too. Final Thoughts Getting a 200 OK is the easiest part of building an API integration. The real work starts when the remote service does something you did not expect. I have found that the most reliable clients are not the ones that retry the most. They are the ones that have a clear opinion about failure. They know when to wait. They know when to try again. And, just as importantly, they know when to stop.

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text <actor-name>.<atespace>.actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

Most teams don't decide to build microservices. They get pushed into it. One app grows for a couple of years. More people push into the same codebase. Then a change to something totally unrelated breaks checkout on a Tuesday. Nobody planned that. That's usually when someone says it, half-joking, half not: maybe we should just split this thing up. And Node.js is the name that comes up. Not because anyone ran a deep framework comparison. Honestly, half the time it's already running the API layer and chewing through small request/response calls all day, so nobody has to fight for it. It's already there. Easiest sell in the room. What people get wrong going in: the win isn't "we use Node.js now." It's narrower than that. Node.js microservices earn their keep when a service actually needs to scale on its own — checkout during a flash sale, say, while the blog section sits idle. Split things up without that need, and you haven't built microservices. You've built one tightly coupled app, just now with network calls between the pieces instead of function calls. Same mess. Slower. The real work is designing the microservices architecture in Node.js properly: keeping services loosely coupled, getting them to talk without one outage taking three other services down with it, and figuring out which pieces genuinely need their own database versus which ones are fine sharing. That's what this covers. Core Architecture Components A Node.js microservices architecture usually has the same handful of pieces, even if the specifics change from one company to the next. componentpurposecommon tools API Gateway Routes requests, handles auth, rate limiting Express Gateway, Kong, NGINX Service Framework Builds individual business services Express, Moleculer Synchronous Calls Request/response between services axios, fetch, gRPC Async Messaging Event-based communication RabbitMQ, Kafka Resiliency Prevents cascading failures Opossum (circuit breaker) Containerization Isolates services and dependencies Docker Orchestration Scaling, restarts, rollouts Kubernetes Logging Centralized, searchable logs Winston, Pino Monitoring Tracks performance and health Prometheus, Grafana 1. API Gateway Clients never talk to your services directly. They hit the gateway first, and it figures out where the request needs to go. This is usually also where auth checks happen and where rate limiting lives, so one client can't flood the system with requests. 2. Individual Services Behind the gateway are the actual services, each one handling a single piece of the business: orders, users, whatever it is. Express is still the default choice for building these. Some teams are moving to Moleculer instead, since it's built specifically for microservices rather than being a general framework stretched to fit. When choosing a Node.js microservices framework, the right option depends on how much infrastructure your team wants the framework to handle. 3. Database Per Service This is the corner teams cut, and it always shows up later, usually a few months in, once nobody remembers why the shortcut got taken. If the order service and the user service are both querying the same database, you don't actually have two services. You have one database wearing two name tags. Each service needs to own its data, full stop. Need something from another service? Ask through its API, or listen for the event it fires. Don't go around the back and query its tables directly; that's the shortcut that turns into a rewrite. 4. Message Broker Not every interaction needs an answer right away. When someone places an order, the order service shouldn't sit around waiting for a confirmation email to go out; it fires off an event and moves on to the next request. Something else, usually RabbitMQ or Kafka, is listening for that event and deals with it on its own time. How to Build Microservices With Node.js The honest answer to how to build microservices with Node.js is: don't start by spinning up five repos. Start by figuring out where the actual boundaries are. Each service needs to own one business capability, its data, its logic, everything it needs to run without leaning on another service to function. A practical Node.js microservices tutorial usually comes down to a sequence like this: Define service boundaries: Figure out the independent business functions: users, orders, payments, notifications, whatever they are for you.Create a Node.js project for each service: Deployable on its own, with its own dependencies and config. Not a shared node_modules folder pretending to be independent.Choose the right framework: Express is fine for lightweight services; reach for a dedicated Node.js microservices framework such as Moleculer when you need more built-in.Expose APIs: Give each service a clean REST or gRPC interface for anything synchronous. This approach keeps building microservices with Node.js focused on business boundaries rather than simply splitting a large codebase into smaller applications. Node.js Microservices Example A simple Node.js microservices example could be an e-commerce application divided into four services: User service: Manages customer accounts and authentication.Product service: Handles product information and inventory.Order service: Creates and tracks customer orders.Notification service: Sends email or other order-related notifications. For example, when a customer places an order, the Order Service can publish an order.created event. The Notification Service listens for that event and sends the confirmation without forcing the Order Service to wait for the email process to finish. This is also a practical example of how to create microservices in Node.js: start with independent business capabilities, expose only the interfaces other services need, and use events when a response isn't required immediately. Communication Strategies Some requests need an answer right away. Others just need to notify another service that something happened, and nobody's waiting on a response. Most Node.js microservices setups use a mix of both. Synchronous Calls One service asks, waits, gets an answer back. That's really it. Most of the time plain HTTP is enough: axios or fetch, nothing fancy. gRPC only earns its keep once two services are hammering each other with requests constantly and the JSON overhead starts showing up in your latency numbers. It runs over HTTP/2, uses Protocol Buffers, and has smaller payloads. Asynchronous Messaging Different situation. Order comes in; the order service doesn't need to hang around until the confirmation email actually sends; it just says done and picks up the next request. Somebody else deals with the email later. RabbitMQ if you care about routing, sending different messages down different paths. Kafka if you're dealing with volume, logs, activity streams, stuff that never really stops flowing. Design Patterns for Resiliency Distributed systems fail in ways a single app never does. One service going down shouldn't mean the whole system goes down with it, so a few patterns exist specifically to contain that damage. Circuit Breaker Something's failing, so the instinct is to retry, and retrying just adds load to a service that's already drowning. A circuit breaker cuts that off. After enough failures in a row, it stops sending requests to that service for a stretch and lets it recover instead of burying it further. Opossum is what most people reach for in Node.js when they're setting this up. Saga Pattern You can't roll back a transaction across three different databases the way you'd roll back one. So instead of a single transaction, you get a chain; each step commits on its own in its own service. If step four fails, you don't just stop; you run backward through one, two, and three, undoing what already happened. It's not clean. It's what you're left with once one database per service is no longer optional. Idempotent Consumers Networks resend messages sometimes; that's just how it goes. If your order service can't tell a retry apart from a brand new order, you end up double-charging someone eventually. A uniqueness check on the event solves most of this; before acting on a message, the service checks whether it's already seen it. Dead Letter Queues Some messages are never going to process no matter how many times you retry them: bad data, a broken payload, whatever the cause. Rather than let one bad message jam everything behind it, it gets pulled into its own queue and dealt with separately later, instead of stalling the rest of the line. Production Deployment and Observability Getting this running locally is one thing. Running it in production with actual traffic is where most of these decisions get tested for real. Containerization Each service, along with its database and anything else it depends on, gets wrapped in its own Docker container. This keeps one service's dependencies from clashing with another's, and it means what runs on your laptop is basically the same thing that runs in production, no more "works on my machine." Orchestration By the time you've got more than two or three containers, doing this manually just doesn't hold up. Kubernetes takes that off your plate; more traffic comes in, it spins up more instances on its own. Something crashes, it gets restarted without anyone needing to notice at 3 am. Rolling out a new version doesn't mean downtime either; it shifts traffic over gradually. And on the security side, secrets management means your API keys aren't just sitting in a config file somewhere waiting to get committed to git by accident. Centralized Logging Logging to a file on each individual server doesn't work once you've got a dozen services running across different machines. Nobody's going to SSH into ten boxes trying to piece together what happened. Tools like Winston or Pino send structured logs somewhere central instead, so you can actually search across everything at once when something breaks. Metrics and Monitoring The goal is finding out something's wrong before a user emails you about it. In a Node.js system specifically, event loop lag is the one to watch closely; a blocked event loop doesn't throw an error, it just quietly slows everything down until someone notices things feel off. Memory usage and response times matter too, obviously. Prometheus is usually what's pulling these numbers together, and Grafana is where you'd actually go look at them. Wrapping Up None of this is complicated on its own: gateway, services, a message broker, some way to keep failures from spreading. What makes it hard is doing all of it at once, correctly, while the system is already handling real traffic and you don't get a do-over if you get the database boundaries wrong on day one. Node.js fits well here mostly because it doesn't get in the way. It's lightweight, it handles the kind of request volume microservices tend to generate, and the ecosystem around it — Express, gRPC libraries, message broker clients — is mature enough that you're not building plumbing from scratch. Whether you're pulling a monolith apart piece by piece or starting fresh, the patterns covered here (separate databases, circuit breakers, idempotent consumers, proper observability) are the parts that actually determine whether the system holds up once it's under load, not just when it's running clean on your laptop.