DZone Spotlight

Saturday, September 5 View All Articles »

Designing Safe Agent Permissions: Why Least Privilege Must Exist Outside the Model

By Igboanugo David Ugochukwu

CORE

Nine seconds. That's how long it took an AI coding agent to delete a production database and its backups at PocketOS, a car-rental software vendor, in April 2026. By founder Jer Crane's account, an AI coding agent hit a credential mismatch during a routine staging task, searched the codebase, and found an API token in an unrelated file. That token carried blanket permissions across the Railway infrastructure API. The agent used the token. Database and backups were gone before anyone noticed. Nobody attacked PocketOS. No credentials were stolen, no prompt injection ran, no malware executed. The agent pursued a goal, hit an obstacle, and used the authority it had been handed to clear it. The problem isn't that the agent could delete a production database. The problem is that nothing in the authorization architecture prevented it from doing so — for a task that never should have touched production at all. That's the distinction this entire discipline turns on, and it's why the fix has to live in the architecture, not in a system prompt telling the agent to be careful. Permission Is Authority Over an Operation, Not Access to a System Most agent failures trace back to the same habit: authorizing at the integration level rather than at the operation level. "This agent can call the Railway API." "This agent has a Salesforce connection." Those describe access to a system, not authority over specific actions inside it — and that gap is exactly where PocketOS lost its database. A token scoped for "the Railway API" turned out to include the ability to delete production volumes, a capability nobody making the staging-fix decision actually wanted the agent to have, but nobody had explicitly excluded either. The fix is to define permission as a tuple: agent → action → resource, evaluated against live context by a policy engine at call time. A billing agent doesn't get billing:write; it gets something closer to billing.refund.issue, gated by conditions like amount <= $1,000 AND customer.region == agent.allowedRegions. That check has to happen when the agent tries to act, against an external policy engine — OpenFGA, Cedar, Open Policy Agent — not get baked into a prompt where the model is trusted to enforce its own boundaries. Layered onto that tuple is a risk hierarchy, because not all operations carry equal weight. Observe (read-only queries) can generally run autonomously, subject to row/column filtering. Retrieve/Modify (reversible writes — updating a record, sending a templated email) needs short-lived scoped tokens and before/after logging. Act (external side effects — calling a partner API, triggering a workflow) warrants just-in-time authorization and rate limits. Escalate (irreversible or sensitive — wire transfers, deleting production data, changing security config) requires explicit human approval, full stop. PocketOS had legitimate Tier 2 authority to modify staging configuration. What it actually exercised was Tier 4 authority to destroy production infrastructure, because the credential it stumbled into didn't distinguish between the two. Give Agents Identities, Not Borrowed Authority The corollary is that every agent needs its own identity — not a shared service account, not a borrowed human token. Shared credentials mean a compromised or overreaching agent inherits every privilege that account carries, and per-agent auditability disappears entirely. The mechanics for fixing this already exist in enterprise IAM; they just need to extend to non-human identities. Provision each agent as a distinct workload identity — an OIDC client or service principal — and manage its lifecycle through SCIM, so a retired workflow doesn't leave a live credential behind for someone to stumble across later. Use OAuth 2.0 client credentials or token exchange to obtain short-lived, task-scoped access tokens rather than static keys sitting in a config file. A credential that expires in minutes and is scoped to one task can't become the thing an agent finds three months later while fixing an unrelated bug — which is precisely the mechanism that sank PocketOS. Least Privilege Applies to Data as Much as to Actions Data exposure doesn't require a destructive action at all — just a retrieval system that doesn't enforce the boundaries a human user would respect. The reference case is Microsoft Copilot's "EchoLeak" vulnerability, tracked as CVE-2025-32711 and disclosed in mid-2025: a malicious email reached Copilot through normal retrieval, and the agent autonomously pulled internal Word documents, PowerPoint files, and Outlook content and transmitted them to an attacker-controlled server — zero-click, no user interaction required, evading Microsoft's own cross-prompt injection classifier in the process. Simon Willison, who coined the term "prompt injection" in 2022, named the underlying structural risk in June 2025: the "lethal trifecta." Any agent that simultaneously has access to private data, processes untrusted content, and can communicate externally is set up so that, in his words, "an attacker can easily trick it into accessing your private data and sending it to that attacker." Remove any one leg and the exploit collapses. The practical answer is filtering at the query layer — a WHERE tenant_id = :agentTenant clause, a database view that already excludes restricted columns — not a hope that the model will voluntarily ignore fields it technically received. Once sensitive data reaches an LLM's context window, downstream controls cannot reliably undo that exposure. A useful discipline is treating what an agent actually sees as the intersection of two permission sets — what the agent identity may touch, and what the human it's acting for may touch — rather than either alone, since agents routinely have broader technical reach than the specific person they're currently helping. Delegation Must Reduce Authority, Never Inherit It Multi-agent architectures introduce the same failure one layer up. When a primary agent spins up a specialized sub-agent, the instinct is to let it inherit whatever the parent was authorized to do — the shared-service-account mistake, moved up a layer of abstraction. Security architects have a decades-old name for this: the confused deputy problem. A sub-agent summarizing public documentation doesn't need write access to the ticketing system just because its orchestrator has it. There's a standard mechanism for avoiding it: OAuth2 token exchange (RFC 8693) lets a parent hand a sub-agent a narrower, short-lived on-behalf-of token — scoped to the sub-task, carrying the original human requester's identity as context rather than the parent's identity as a substitute. Every hop should independently hit the policy engine again; nobody skips the check because an upstream step already passed one. This is also the sharpest lens for reading Anthropic's November 2025 disclosure of GTG-1002, a Chinese state-linked group that manipulated Claude Code into functioning as a largely autonomous intrusion framework against roughly thirty organizations. Anthropic says the operators didn't write novel exploits — they convinced the agent it was performing a legitimate, authorized security assessment, breaking the operation into innocuous-looking steps, and let it run reconnaissance and credential harvesting with 80–90% of tactical work executed without a human in the loop. Independent researchers have challenged parts of Anthropic's framing, but the architectural lesson remains: an agent granted broad authority can turn seemingly benign subtasks into a larger chain of actions unless each delegation is independently authorized. Audit Trails Have to Answer More Than "What Did the Model Say" For every meaningful action, the record needs to capture who initiated the request, which agent identity acted, which tool or API was invoked and with what parameters, what operation was requested against what resource, which policy allowed or denied it, whether a human approved it, and a correlation ID tying the action to the broader workflow. For reversible writes, log the before/after diff, not just the fact a write occurred. And the logs need to be tamper-evident — append-only or cryptographically signed — because a log an agent (or an attacker) can edit isn't an audit trail; it's a suggestion. This is the layer that's usually missing. One research effort that reviewed 7,246 publicly reported AI incidents from September 2023 through May 2026 verified 344 as enterprise-relevant, and found that in 188 of them, an autonomous system caused harm directly in production with no attacker anywhere in the chain. Separately, a Cloud Security Alliance survey of 418 security and IT professionals, published in April 2026, found that 65% of respondents reported at least one AI-agent-related incident in the preceding twelve months. Neither number requires an adversary to make the risk alarming. The first shows that autonomous systems can cause production harm without an attacker; the second shows how widespread agent-related incidents have already become. A Worked Example Take a common agentic task: cancel a customer's order and issue the refund. Done properly, this is a sequence of independently checked steps, not one blanket-authorized action. The agent authenticates via OIDC and receives a JWT carrying its own agent ID, with the customer's request attached as context. Before touching the order, it asks the policy engine: can this agent cancel this order, given it hasn't shipped? If yes, it gets a token scoped to exactly that action. Only after cancellation succeeds does it request a second, separate authorization for the refund. Under a set threshold — say $50 — it's granted a fresh, narrowly scoped token for the payment call; above it, the same check routes to a human approval queue and the agent waits. Every step, allowed or denied, is logged with the agent ID, the policy version, and a correlation ID linking cancel-then-refund as one traceable workflow. None of this requires exotic infrastructure — it's OAuth2, a policy engine, and the discipline to check twice instead of once. Before deploying an agent, ask: Does it have its own identity, distinct from any shared account or human token?Is every tool permission scoped to a specific operation and resource, not a whole system?Is sensitive data filtered before it reaches the model, not trusted to the model afterward?Does every delegation to a sub-agent reduce authority rather than pass it along whole?Can you reconstruct every consequential action — and who authorized it — from the audit trail alone? Architecture, Not Prompt Engineering None of the incidents above happened because a model wasn't smart enough. They happened when a credential, role, or inherited permission gave the agent more authority than the task required — or when the surrounding architecture failed to constrain what that authority could reach — and the agent, reasoning correctly within the authority it had, found and used the edges of it. Least privilege for agents is an architectural boundary enforced outside the model, at the point where the agent actually touches a tool, a record, or another agent's output. Treat permissions as a system-identity problem, and you can reason about, log, and revoke them. Treat them as an instruction, and you get PocketOS: nine seconds, no attacker required. Sources "AI Agent Destroys Production Database in 9 Seconds," Zenity, April 2026 — https://zenity.io/blog/ai-agent-database-deletion-pocketos"Claude-Powered Cursor AI Agent Deletes an Entire Company Database in 9 Seconds," CX Today, April 29, 2026 — https://www.cxtoday.com/security-privacy-compliance/claude-powered-cursor-ai-agent-deletes-an-entire-company-database-in-9-seconds-is-your-customer-data-secure/Halamish, E. and Tokarev, V., "Agent-Inflicted Damage: Inside the Real-World Failures of Enterprise AI Systems," Cyera Research, May 28, 2026 — https://www.cyera.com/research/agent-inflicted-damage-inside-the-real-world-failures-of-enterprise-ai-systems"Autonomous but Not Controlled: AI Agent Incidents Now Common in Enterprises," Cloud Security Alliance / Token Security, April 2026 — https://cloudsecurityalliance.org/artifacts/autonomous-but-not-controlled-ai-agent-incidents-now-common-in-enterprisesWillison, S., "The lethal trifecta for AI agents," June 16, 2025 — https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/Anthropic, "Disrupting the first reported AI-orchestrated cyber espionage campaign," November 13, 2025 — https://www.anthropic.com/news/disrupting-AI-espionage More

Enterprises Should Assume AI Agents Will Delete Their Production Base

By Meir Wahnon

Traditional identity mechanisms weren’t built to account for the autonomous and unpredictable nature of AI agents. So enterprises shouldn’t be surprised when they make unpredictable – or even destructive – decisions, like deleting a company’s entire production base. Without the right guardrails in place, these new actors will continue to wreak havoc across organizations. One doesn’t have to look far to find examples of this playing out in real time. PocketOS recently shared that a Cursor agent deleted its production database and backups while working on a routine task in what the company thought was a secure staging environment. The entire scenario unfolded in just 9 seconds, but took days to resolve. When the company asked the agent why it took that action, it admitted that it had “guessed instead of verifying.” This incident serves as an important cautionary tale: even seasoned developers working within the parameters of a staging environment aren’t immune to agents going rogue. Agents represent an entirely new paradigm. What makes them valuable (i.e., the autonomy to make decisions and take actions) is also what makes them risky. If a rogue agent can cause this much chaos in a sandbox, one can only imagine how harmful targeted, malicious prompt injection could be in production systems. Organizations need to strike a balance between giving agents enough freedom to provide value, but not so much that they accidentally blow up their operations. Simply prompting agents not to take harmful actions isn’t enough – you need infrastructure-level mitigations. If an action is possible, you can bet an agent will take it. Enterprises should expect agents to uncover exposed credentials or other loopholes they didn’t even know existed – it’s built into their design. Here are 5 ways every organization should be strengthening its agentic identity strategy: 1. Give Agents Their Own First-Class Identities Agents aren’t humans, machine identities, or your other typical NHIs. They’re an entirely new class of entities that warrant their own first-class identity. Agents need identities that make them traceable, observable, and bound to specific users so organizations can continuously monitor their actions, scoped permissions, and who they’re acting on behalf of. This way, if an agent makes a mistake, you can pull the plug quickly without impacting other agents and systems. 2. Reinforce Least-Privileged Access Enterprises need a robust approach to enforcing least-privileged access. Static credentials were suited to the machine identities of the past — not the highly dynamic, agentic identities running amok in code bases today. Ephemeral credentials that limit what agents can do and for how long are crucial. Agent credentials should be measured in minutes — not months. Additionally, progressive scoping ensures that agents can access only the minimum scope required to complete a task. Should they need additional permissions as the task progresses, they can ask for them, but a human must vet that request. 3. Authorize Read-Only Access for Sensitive Data Not all data should be treated equally, and agent permissions need to reflect that. Read-only access is lower risk than enabling write access, so it should be the standard for agents interacting with sensitive information. That way, if an agent veers off course, it won’t be able to alter systems or records that house sensitive data. 4. Set Up Sandbox Environments Sandbox environments are critical for letting agents learn, experiment, and fail, without impacting production systems. The PocketOS agent mishap illustrated just how crucial it is that these environments are truly secure. When configured correctly, sandboxes can help organizations identify unexpected behaviors, permission issues, or security risks before they have the chance to impact customers and/or operations. 5. Strong UX Makes for Better Human-in-the-Loop (HITL) Checkpoints HITL controls are essential for keeping agents in check. But many people experience consent fatigue after trying to decipher and approve dozens (or hundreds) of agent requests every day. UX matters here. Agent requests need to be transparent and human-readable, so users quickly know exactly what permissions they’re granting and why. Poor UX makes it harder for humans to identify sensitive actions and can lead to over-permissioning. Until enterprises modernize their approach to agentic identity, they can expect agents to occasionally take wildly unpredictable actions — like deleting their entire codebase. We’ll undoubtedly see more stories like this emerge as organizations rush to deploy agents without first understanding the new set of rules and implications surrounding their identity. It’s vital to establish these guardrails now — before your agents discover the ones you forgot to build. More

Ampere PMU Profiler: A Guide to Microarchitecture Profiling

By Bhakti Hinduja

Refcard #267

Getting Started With DevSecOps

By Akanksha Pathak

CORE

Refcard #291

Code Review Core Practices

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

Video and Audio as Knowledge Sources: Content Understanding in Microsoft Foundry IQ

Your largest corpus is the one nobody indexed. Here's how to turn recordings into grounding data an agent can cite. The Corpus You Already Have and Never Indexed Every organization is sitting on years of recorded meetings, support calls, training sessions, conference talks, and screen recordings. Almost none of it is retrievable. When someone asks "what did we decide about the vendor migration," the answer exists — in a 47-minute recording nobody will ever scrub through. The reflex fix is to bolt a transcription service onto your RAG pipeline: run Whisper, dump the text into a blob container, index it. It works, sort of, and then you discover what you lost. The transcript has no speakers, no timestamps you can link back to, no slide content, no distinction between the presenter reading a bullet and someone in the room disagreeing with it. Your agent can now quote the meeting but can't tell you when it happened or who said it. Azure Content Understanding is Microsoft's answer to that gap: it ingests documents, audio, images, and video and extracts the most critical information to power well-grounded generative and agentic solutions, combining Document Intelligence's traditional AI with LLM-based content reasoning. And as of Build 2026, it is integrated with Foundry IQ standard mode for built-in content extraction inside Microsoft's retrieval and agent workflows. That integration is the subject of this article. Specifically, the part everyone gets wrong: there are two ways to get media into a knowledge base; they are not equivalent, and the one people assume exists is the one you should verify before you plan a sprint around it. This is a companion to my earlier piece on connecting a Foundry IQ knowledge base to LangGraph over MCP. That one covered retrieval. This one covers what you feed it. Architecture Figure 1 — Two paths into the same knowledge base. The native path is a flag on the knowledge source; the explicit path runs analyzers yourself and lands the output as text. The Two Paths, Stated Plainly Path A — native extraction inside the ingestion pipeline. Setting the contentExtractionMode property to standard on file-based indexed knowledge sources (Azure Blob, SharePoint, OneLake) enables Content Understanding functionality within the ingestion pipeline. One property. No orchestration code. This shipped in the Foundry IQ 2026-05-01-preview release, which focused on richer Content Understanding extraction and image serving for multimodal agentic retrieval. Path B — explicit analysis, then index the output. You run Content Understanding yourself, write the resulting Markdown and fields to blob storage, and point a normal knowledge source at that. More moving parts, complete control. Here is the honest caveat, and I would rather you hear it from me than discover it in week three: Microsoft's Foundry IQ extraction announcements emphasize document understanding — layout, tables, figures, and document-embedded images — rather than audio and video ingestion. Content Understanding standard mode itself is documented for documents, images, audio, and video. Whether your blob knowledge source will accept an .mp4 today, in your region, at your API version, is a question you should answer with a five-minute test rather than an assumption. So: test Path A first, build on Path B if it doesn't cover your media types. Path B is what this tutorial walks through in detail, because it works regardless, and because understanding it makes Path A trivial to adopt when it covers you. Prerequisites A Microsoft Foundry resource and a Content Understanding resource.An LLM deployment for analyzers. Analyzers are powered by LLM and embedding models you deploy in Foundry, and GPT-5.2 improves custom field extraction enough to avoid prompt-engineering gymnastics on mixed layouts, domain-specific language, and multilingual content. Analyzers built on GPT-4.1 continue to run unchanged.An Azure AI Search service for the knowledge base. Shell pip install 'markitdown[az-content-understanding]' azure-search-documents azure-identity Pin Your API Version Before Writing a Line of Code Content Understanding's GA API is 2025-11-01. The preview versions 2024-12-01-preview and 2025-05-01-preview were slated for retirement on July 15, 2026 — a date that has now passed, so if you inherited a codebase targeting either, it is already broken or about to be. Note the asymmetry with Foundry IQ, which is on 2026-05-01-preview for the content-extraction features. You will be running a GA content service against a preview retrieval service. Plan your support expectations accordingly. Step 1: Choose the Analyzer, and Know What Each Modality Costs You Content Understanding ships prebuilt analyzers per modality, and MarkItDown auto-selects among them: documents route to prebuilt-documentSearch, video to prebuilt-videoSearch, and audio to prebuilt-audioSearch. They are not interchangeable, and the differences determine what your agent can cite: DocumentsAudioVideoPrebuilt analyzerprebuilt-documentSearchprebuilt-audioSearchprebuilt-videoSearchPrimary outputLayout-aware MarkdownTranscript with structureTranscript plus visual contextStructure preservedHeadings, tables, figure descriptionsUtterance boundariesScene and segment boundariesNatural citation anchorPage and figure IDTimestampTimestamp and frameGrounding you gainTable cells stay in their tableWho spoke, and whenWhat was on screen, not just saidWhat you still loseNothing much, this is the mature pathVisual aids referenced verballyFine detail in dense slidesCustom field schemaYesYesYesBest forContracts, reports, formsSupport calls, interviewsMeetings, demos, training The row that matters most is natural citation anchor. A document chunk cites a page; an audio chunk cites a timestamp. If you flatten audio into plain text before indexing, you throw away the only anchor that makes a recording navigable — and no amount of clever chunking downstream will recover it. Step 2: The Fastest Path From a Recording to Indexable Text MarkItDown with the Content Understanding backend is the shortest route, and it is genuinely a few lines. Zero configuration auto-selects the analyzer per file type: Python from markitdown import MarkItDown md = MarkItDown(cu_endpoint="<content_understanding_endpoint>") doc = md.convert("report.pdf") # → prebuilt-documentSearch video = md.convert("meeting.mp4") # → prebuilt-videoSearch audio = md.convert("call.wav") # → prebuilt-audioSearch print(video.markdown) The output is Markdown with headings, tables, and figure descriptions inline — exactly the shape downstream chunkers and embedding models prefer. That last clause is the whole argument for this approach: you are not inventing a format; you are producing the format the rest of the stack already wants. With a custom analyzer, the output carries extracted fields as YAML front matter above the body: Python md = MarkItDown( cu_endpoint="<content_understanding_endpoint>", cu_analyzer_id="my-meeting-analyzer", ) result = md.convert("standup-2026-08-14.mp4") --- contentType: video fields: MeetingTitle: Vendor migration review Decision: Proceed with phased cutover Owner: A. Rivera ---  ... That front matter is not decoration. It is your metadata filter, your citation payload, and the difference between "the agent found a relevant meeting" and "the agent told me we decided to proceed, and here is the timestamp." Step 3: Design a Field Schema Worth Extracting Prebuilt analyzers give you good transcripts. Custom analyzers give you answers to questions you ask repeatedly — and those are what make retrieval feel like it understands your business rather than your file formats. Two constraints to plan around: Custom analyzers are built in Content Understanding Studio, not the Foundry portal. The Foundry portal surfaces prebuilt analyzers and a playground, with a deep link into CU Studio that preserves your project context, but custom analyzer creation lives in Studio. Expect to move between two tools. Standard mode is per-file. Standard mode handles single files with straightforward field extraction — documents, images, audio, or video without cross-file analysis or complex reasoning. Pro mode is for multi-step reasoning and cross-file analysis. Since Foundry IQ's integration is with standard mode, per-file extraction is your ingestion contract. Questions like "which meetings contradicted the Q2 plan" are retrieval-time work for the agentic retrieval engine, not ingestion-time work for the analyzer. A schema that earns its keep on meeting recordings: FieldWhy it earns its placeDecisionThe single most-asked question of any meeting corpusOwnerTurns retrieval into accountabilityDueDateEnables recency and deadline filtersSystemsMentionedLets an agent scope to "anything about the billing service"UnresolvedQuestionsSurfaces what a summary would smooth over Resist the urge to extract a summary field. The retrieval engine synthesizes at query time against the actual question; a pre-baked summary just adds a lossy paraphrase your agent might cite instead of the source. One operational note from Microsoft that applies whenever you change models: run side-by-side against your existing eval set before flipping production traffic, since confidence scores, latency, and output accuracy can all shift with a new model. Step 4: Land the Output Where a Knowledge Source Can Reach It Write the Markdown to blob storage, one file per recording, with the front matter intact: Python import pathlib from azure.identity import DefaultAzureCredential from azure.storage.blob import BlobServiceClient from markitdown import MarkItDown md = MarkItDown(cu_endpoint=CU_ENDPOINT, cu_analyzer_id="my-meeting-analyzer") blobs = BlobServiceClient( account_url="https://stfoundryiqprod.blob.core.windows.net", credential=DefaultAzureCredential(), ).get_container_client("meeting-transcripts") def ingest(path: str) -> str: """Analyze one recording and land the result as indexable Markdown.""" result = md.convert(path) name = pathlib.Path(path).stem + ".md" blobs.upload_blob(name, result.markdown.encode("utf-8"), overwrite=True) return name Keep the source recording and the derived Markdown in separate containers. You want the knowledge source pointed at text only, and you want the original media addressable for playback when a citation resolves. Mixing them means either indexing binaries you can't use or losing the link back to the thing a user actually wants to watch. Step 5: Create the Knowledge Source and the Knowledge Base Now it is an ordinary Foundry IQ ingestion. Foundry IQ automates document chunking, vector embedding generation, and metadata extraction for indexed knowledge sources, and schedules recurring indexer runs for incremental refresh. Python from azure.identity import DefaultAzureCredential from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( AzureBlobKnowledgeSource, AzureBlobKnowledgeSourceParameters, KnowledgeBase, KnowledgeSourceReference, ) client = SearchIndexClient(endpoint=SEARCH_ENDPOINT, credential=DefaultAzureCredential()) source = AzureBlobKnowledgeSource( name="meeting-recordings-ks", description=( "Transcribed and structured meeting recordings: decisions, owners, " "due dates and unresolved questions, with timestamps. " "Use for questions about what was decided, by whom, and when." ), azure_blob_parameters=AzureBlobKnowledgeSourceParameters( connection_string=BLOB_CONNECTION, container_name="meeting-transcripts", # Path A: enable Content Understanding inside the ingestion pipeline. # Test this against your actual media types before relying on it. content_extraction_mode="standard", ), ) client.create_or_update_knowledge_source(knowledge_source=source) client.create_or_update_knowledge_base(knowledge_base=KnowledgeBase( name="meetings-kb", knowledge_sources=[KnowledgeSourceReference(name="meeting-recordings-ks")], retrieval_instructions=( "Prefer the most recent meeting when decisions conflict. " "Always surface the timestamp and speaker with any quoted decision." ), )) Two things to be deliberate about. The description is load-bearing. The agentic retrieval engine uses it to plan queries and select sources, so write it the way you would brief a new colleague: what is in here, and what kinds of questions it answers. "Meeting transcripts" is a wasted field. The content_extraction_mode="standard" flag is your Path A test. Point it at a container holding one .mp4 and one .md, run the indexer, and look at what got indexed. If the media file produced chunks, you can skip Steps 2 through 4 for that file type. If it didn't, your pipeline is already correct and you have lost five minutes. What Happens at Query Time Figure 2 — Ingestion runs once per recording; retrieval runs per question. The timestamp survives both, which is the point. Nothing about retrieval changes because the source was a video. That is the entire payoff: the agentic retrieval engine plans queries, selects sources, runs parallel searches, and aggregates results, returning extractive data with citations so agents can reason over raw content and trace answers to source documents. Your recordings are now just documents that happen to have timestamps. The 2026-05-01-preview release also added image serving — surfacing document-embedded images during agentic retrieval. For a slide-heavy deck or a demo recording, that is the difference between "the presenter showed an architecture diagram" and actually returning the frame. Combined with figure extraction from Office files, where each figure is retrievable by ID via GET /contentunderstanding/analyzerResults/{operationId}/files/figures/{figureId}, you can build citations that resolve to a picture rather than a paraphrase of one. Making Citations Survive the Pipeline This is where most media-RAG implementations quietly fail. The transcript is indexed, retrieval works, the answer is correct — and the citation says "meeting-2026-08-14.md," which is useless to someone who wants the 12 seconds where the decision was made. Three rules: Never strip the front matter during chunking. If your chunker treats YAML as noise, the extracted fields never reach the index and your metadata filters silently match nothing.Keep timestamp markers inside the chunk text, not only in metadata. Extractive retrieval returns content; if the timestamp lives only in a sidecar field, the model has nothing to quote.Store a deterministic link back to the media. A blob URL plus a timestamp fragment costs you one field at ingestion and turns every citation into a playable link. The test: ask your agent a question, take the citation it returns, and try to get to the exact moment in the recording. If you can't do it in one click, the pipeline isn't finished, however good the answer text looks. Troubleshooting SymptomLikely causeWhat to doMedia files in the container produce no chunksThe knowledge source didn't ingest that file type nativelyFall back to Path B: analyze with CU, index the Markdown output404 or version errors from the CU endpointCode still targeting a retired preview APIMove to the GA API version 2025-11-01Custom analyzer missing in the Foundry portalCustom analyzers are created in CU StudioUse the deep link from Foundry; project context is preservedExtraction quality dropped after a model changeConfidence scores, latency, and accuracy shift between modelsRe-run your eval set side by side before shifting production trafficFields extracted but never filterableFront matter stripped during chunkingPreserve YAML through the chunker; verify fields landed in the indexCitations resolve to a file, not a momentTimestamps kept only in metadataKeep markers in chunk text and store a media URL per chunkCross-file questions return thin answersStandard mode is per-file by designLet the agentic retrieval engine handle cross-document reasoning at query timeOffice figures not retrievableFigures need fetching by IDUse the figures/{figureId} endpoint referenced from the Markdown Where to Go Next A few things landed or were slated for July 2026 that change the calculus here, and are worth confirming against current docs before you design around them: a synchronous API for Read and Layout, an agentic understanding mode for complex documents, data zone and global zone processing for residency, improved custom analyzer training from your own examples, and labeled training data no longer being stored in CU so training inputs stay in your own storage. That last one matters most if compliance review is what's blocking you. The natural follow-on build is a routing layer: send short call recordings straight through prebuilt-audioSearch, send slide-heavy sessions through prebuilt-videoSearch so you keep the visual channel, and reserve custom analyzers for the recording types you query weekly. Extraction quality and cost both track how well that routing matches your corpus. References What is Foundry IQ? — knowledge sources, indexing automation, agentic retrieval and ACL enforcement.Content Understanding standard and pro modes — per-file vs. cross-file analysis, and the preview API retirement notice.Content Understanding models and deploymentsFoundry vs. Content Understanding Studio — which tool does what.Supported document formats and input file limitsBuild a RAG solution with Content UnderstandingContent Understanding in LangChainWhat's new in Azure Content Understanding at Build 2026 — the Foundry IQ standard-mode integration, MarkItDown backend, GPT-5.2 analyzers, and the July roadmap.Improved data processing features in Foundry IQ: richer content extraction and data enrichment — contentExtractionMode, SharePoint indexing expansion, and image serving.Foundry IQ: improve recall by up to 54% with knowledge basesBuild 2026 Session BRK242 — "Turn your agents into action" — agentic understanding mode and the Foundry IQ integration, demoed end to end.MarkItDown on GitHub — the CU-backed converter used throughout this article.Microsoft Agent Framework — for registering CU as an agent tool instead of a pipeline stage.

By Jubin Soni, FBCS

CORE

The Startup Time Trick Hiding Inside Your Docker Build

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along.

By Garima Agarwal

The Bottleneck of Scaling

Any input/output operation, be it accessing a file, handling an HTTP request, or a database connection, is based on 3 fundamental system concepts — file descriptors, kernel memory, and heap size. This article discusses how modern languages help developers handle behind-the-scenes file descriptor, kernel memory, and heap management. These three concepts are major bottlenecks for scaling. 1. File Descriptors A file descriptor is just a positive number that is used by the kernel to identify any open input/output stream or connection. It is defined by the kernel for a process. The following file descriptors are defined by default for a process: 0 – Standard Input (stdin)1 – Standard Output (stdout)2 – Standard Error (stderr) Any subsequent I/O operation gets the next available integer as file-descriptor. The file descriptor value can be adjusted by using the ulimit -n command in Linux. Each application, whether it is a web server written in Java Spring Boot, an API server written in Go using net/http and gorilla-mux, or a Python Flask app, is a single process. Each process has only 1024 file descriptors defined by default. That means each application can perform only 1024 I/O operations simultaneously. This seems like an amazing concept when we talk about scaling our application or API server. As many times as we come across this question — how can we scale our API server or web application to handle 100k or 1 million requests per second? This is where our modern languages play their role very beautifully behind the scenes to enable developers to develop the application to handle such scale. 2. Kernel Memory At a lower layer than file descriptors, when an incoming TCP connection hits the network card, the Linux kernel performs a 3 Way TCP handshake for that connection. The handshake lifecycle includes the states: SYN -> SYN-ACK -> ACK. The number of requests equal to the defined file descriptor value are processed immediately, assigned a file descriptor, and forwarded to the application for further processing. When FDs are exhausted, the Kernel maintains a queue for requests waiting for FDs to become available so your application can process them. The same thing happens when a request is processed, and the response is ready to be sent back to the client. This queue is maintained within RAM by read buffers(rmem) and write buffers(wmem). The size of buffers is defined in memory by the kernel and is dynamic, depending on network throughput, round-trip time, and memory pressure. The kernel network memory is non-paged, i.e cannot be swapped to disk. It’s a big bottleneck as it directly depends on physical memory. For example, if there are 100,000 open connections and each connection holds an average of 128KB of kernel memory, it comes to 12.8GB of physical RAM. This is clearly a kernel overhead, and it doesn’t show up in JVM heap metrics or Go runtime statistics. rmem and wmem buffers are governed by kernel parameters defined in /proc/sys/net/ipv4/ 3. Heap Size When TCP connections are assigned file descriptors and kernel memory is reserved, they enter user space, which is the memory managed by the application runtime — Java JVM, Node.js V8 Engine, Python interpreter, Go runtime, etc. Each connection stores objects in the heap within three categories: Connection metadata – Keep-alive timers, IP State, Socket Wrappers, etc.Cryptographic session context – handshake caches, cipher states, TLS/SSL keys, etc.Serialized payload buffers – response queues, JSON strings, ORM entity maps, etc. A connection that is encrypted via TLS takes a lot more space in the heap compared to a regular connection. For an encrypted connection, the application has to save symmetric keys, cipher contexts, session tickets, etc. onto the heap. A regular TCP socket object in the heap consumes 2KB to 5KB of space, whereas a TLS 1.3 socket object consumes 20KB to 100KB of heap space. If an API maintains 10,000 idle TLS connections, it will consume 200MB to 1GB of heap space. When an application runs, the runtime asks the kernel for memory space as the application creates objects. The application keeps creating objects, and the kernel keeps reserving memory for those objects; this is called the heap. The maximum heap size can be defined by different programming languages at runtime; for example, in Java, -Xmx4g reserves 4GB for the heap. The operating system promises to provide that much memory as heap space for the application, but it doesn’t reserve it all at once. As the application creates objects, the kernel continues to reserve memory. When objects are marked as done, the garbage collector removes them from the heap. When an incoming request hits our API server, the application uses heap space to convert raw bytes to the application-specific data structure. Once the application finishes processing the request and returns the response, those objects in the heap become unreachable or dead. When the garbage collector sweeps those objects to reclaim that memory, it doesn’t return the memory immediately; instead, the JVM or Go runtime keeps that freed memory in its internal pool. If a new HTTP request arrives within 1 millisecond, the runtime assigns the required memory from the free memory in the pool. Now imagine 10,000 new requests arriving at the same time, each with 2MB of raw bytes, and the runtime trying to allocate heap for the objects; the app instantaneously uses 20GB of memory. This is called GC thrashing, as the runtime rapidly creates required objects in the heap faster than the GC can clean them. The garbage collector is an application thread itself; when the heap gets 80%-90% full, the garbage collector panics and consumes 100% of CPU cores to scan millions of memory pointers to find dead objects. The runtime, like the JVM or Node.js garbage collector, may stop other code execution while it reorganizes the memory. So, how do runtimes like Go and the JVM handle GC thrashing? Go follows a simple strategy – avoid creating objects on the heap. The fastest GC collector is the one that has nothing to collect. The Go compiler compiles the application to see if variables outlive their functions. If a struct is used only inside a function, Go pushes the struct to the stack instead of the heap, and the stack pointer just drops when the function returns. The memory is reclaimed in 1 CPU cycle without even involving the garbage collector. If Go does have to clean the heap, its GC runs concurrently along with other goroutines and is broken into several micro pauses. Go provides sync.Pool to help developers to reuse heap memory while creating objects. For example to instead of creating millions of []bytes for JSON parsing for every new request, developers can use sync.Pool as follows: Go // Instead of creating a new buffer for every HTTP request: var bufferPool = sync.Pool{ New: func() any { return new(bytes.Buffer) }, } func handleRequest(w http.ResponseWriter, r *http.Request) { buf := bufferPool.Get().(*bytes.Buffer) // 1. Grab an existing buffer from pool buf.Reset() defer bufferPool.Put(buf) // 2. Put it back when done! // Parse JSON into 'buf' without allocating new heap memory } By recycling buffers via sync.Pool, high-concurrency APIs can handle 100,000 requests/sec with near-zero new heap allocations. Java takes a different approach. Because Java applications historically create millions of short-lived objects on the heap, the JVM relies on Generational Hypotheses and Generational Collectors (like G1GC, ZGC, and Shenandoah). G1GC can be used like java -XX:+UseG1GC while running Java applications. G1GC divides the Heap memory into physical regions: Young Generation (Eden & Survivor spaces) and Old Generation. It kind of sorts objects into different regions so that it doesn't have to scan the complete heap and can clean where most of the marked objects live. We can also mention -XX:MaxGCPauseMillis=200 to tell G1 to pause the application for no more than 200ms, but this is not guaranteed. Older JVM collectors like Parallel GC used to freeze the entire application to clear the heap when full, leading to multi-second latency spikes. Modern JVMs introduce ZGC (Z Garbage Collector) and Shenandoah. ZGC uses specialized CPU pointer references to track moved objects in real time. ZGC can clean, move, and compact terabytes of heap memory concurrently while your API requests are actively running. ZGC guarantees GC pause times under 1 millisecond, regardless of whether your heap is 500 MB or multi-terabytes. Conclusion Keep track of these three core concepts — file descriptors, kernel memory, and heap size to know when to scale. 1. File Descriptor Saturation Signals File descriptors represent the system's open handles. When an application hits its FD threshold, the operating system stops accepting connections. The following are example scenarios that indicate when to scale. Check Kernel-wide statistics from /proc/sys/fs/file-nr, per process fds - /proc/<pid>/fd, Prometheus exposes process_open_fds. If it consistently breaches the 80–85% threshold, it's time to scale. You have already tuned ulimit -n and LimitNOFILE up to standard safety thresholds (e.g., 65,536 or 104,857), but process FD counts continue climbing toward the max. Network interfaces show growing SYN-to-LISTEN socket counts and drops in netstat -s under the listen queue overflow metric. 2. Kernel Memory Pressure Signals Because TCP receive (rmem) and transmit (wmem) buffers are non-paged, they cannot overflow onto disk swap. When kernel network memory fills up, the OS drops packets. Below are the scenarios related to kernel memory breach. Check /proc/net/sockstat under TCP: inuse and matching /proc/sys/net/ipv4/tcp_mem thresholds. Netstat counters (netstat -s | grep -i retrans) show a sharp rise in TCP Retransmission rates (>1–2%). Latency spikes occur because the kernel is dynamically shrinking socket buffers down to tcp_rmem minimums (4 KB) to avoid running out of physical RAM, throttling TCP window sizes. 3. Heap Size & Garbage Collection (GC) Thrashing Signals When user-space heap allocations outpace the garbage collector's ability to sweep dead objects (like parsed JSON payloads or session states), application performance collapses. The runtime (JVM or Go) spends more than 15–20% of its total CPU time running GC sweeps (go_gc_cpu_fraction or JVM GC CPU utilization). In Go, metrics show the pacer triggering Mark Assist, stealing CPU time from worker goroutines to help clean up memory. You can check the runtime package /cpu/classes/gc/mark/assist:cpu-seconds metrics to see if GC is asking for more help from CPU. In Spring Boot, you can use Actuator and Micrometer to expose relevant endpoints to monitor the threshold values.

By Vishal Bhatia

How I Run Two AI Coding Agents on One Codebase

Parallel coding agents create a concurrency problem before they create a productivity gain. Two autonomous processes that edit the same checkout can overwrite files, invalidate assumptions, contaminate test state, or produce changes that are individually correct but jointly incompatible. A safer operating model treats each agent as an isolated contributor with a dedicated Git worktree, an explicit file-level contract, deterministic validation commands, and no authority to integrate directly into the protected branch. Git worktrees provide multiple linked working trees for one repository, while modern coding-agent platforms independently reinforce the same principle through isolated sandboxes, scoped write access, and controlled network permissions. Isolation Before Parallelism The repository should expose one branch and one working directory per agent. Git worktrees are preferable to two processes sharing a checkout because each linked worktree has its own checked-out branch and worktree metadata while remaining attached to the same repository. Git explicitly supports multiple working trees and provides lifecycle commands for adding, listing, removing, locking, and pruning them. A practical setup can start both tasks from the same known commit: Shell git fetch origin git worktree add ../agent-auth -b agent/auth origin/main git worktree add ../agent-checkout -b agent/checkout origin/main The important property is not directory convenience but isolation of mutable state. The authentication agent can compile, format, generate files, and modify its branch without changing the checkout seen by the checkout agent. This mirrors the isolation used by cloud coding agents: OpenAI describes Codex cloud tasks as isolated containers, while GitHub limits its cloud coding agent to a dedicated branch and subjects that branch to repository protections. Parallelism still requires ownership boundaries. Separate worktrees prevent filesystem collisions, but Git cannot prevent two branches from independently editing the same contract. A useful policy assigns feature-local paths to each agent and reserves cross-cutting files such as dependency manifests, database migrations, CI workflows, shared schemas, and public interfaces for an integration task. Concurrent edits to build.gradle, an OpenAPI document, or a shared DTO can create semantic conflicts even when Git reports no textual conflict. The safest default is therefore narrow write scope, not broad repository access. Contracts Turn Prompts Into Boundaries Agent instructions should be treated as executable operating contracts rather than conversational prompts. Current agent systems already support repository-level instruction files, and Codex reads AGENTS.md before work begins and supports directory-specific overrides, while GitHub Copilot repository instructions can describe how a project should be built, tested, and validated. A tool-neutral contract can make scope and completion criteria machine-checkable: YAML agent: checkout base: origin/main allowed_paths: ["src/main/java/com/acme/checkout/**", "src/test/java/com/acme/checkout/**"] forbidden_paths: ["build.gradle", ".github/**", "api/**"] validation: ["./gradlew test --tests '*Checkout*'", "./gradlew spotlessCheck"] integration: "rebase-then-review" The contract should be enforced outside the model as well. An agent stating that only checkout files changed is weaker than a gate deriving the changed-path set from Git. git diff is designed to compare trees, commits, the index, and working-tree state, so scope checks can be based on repository truth rather than agent self-reporting. A completion gate can remain deliberately small: Shell git diff --check git diff --name-only origin/main...HEAD ./gradlew clean test The changed-path output can be matched against the contract before review. A clean build matters because two long-running agents can leave generated output or caches that conceal missing dependencies. Feature-specific tests provide fast local feedback, but the final gate should run the repository’s normal clean validation path. The agent contract should also require small, coherent commits so rejected or accepted changes remain separable during integration. Integration Is a Gate, Not a Merge Integration should occur only after the branch is refreshed against the current base. Git rebase replays topic-branch commits on top of an upstream base, which makes stale assumptions visible before final validation. For a short-lived agent branch, the sequence is straightforward: Shell git fetch origin git rebase origin/main ./gradlew clean test A conflict during rebase is useful information, not merely friction. It signals overlapping ownership or an assumption that changed while the agent was running. Conflict resolution should preserve the current base contract first, then reapply the feature intent, followed by the complete validation suite. Re-running only the previously failing test is insufficient because the resolved file may sit on a wider dependency path. Merge, rebase, and cherry-pick serve different integration needs. git merge incorporates the histories of diverged branches, while rebase rewrites a topic branch by replaying its commits onto another base. git cherry-pick applies the changes introduced by selected commits and is useful when only part of an agent branch is acceptable. Cherry-picking should remain selective rather than becoming a substitute for disciplined branches, as partial adoption becomes difficult when commits mix refactoring, generated files, dependency changes, and feature logic. The most dangerous failure is a green branch that becomes red only after another agent merges. Strict required status checks reduce that risk by requiring a branch to be up to date with its base before merging, and GitHub merge queues can validate changes against the latest target branch plus queued changes. Even without a hosted merge queue, the same principle can be implemented with a temporary integration branch that combines both agent branches and runs the full build before either change reaches main. Security and Lifecycle Bound the Blast Radius Coding agents execute model-generated commands, so repository isolation should be paired with credential isolation. Network access should remain disabled unless task requirements justify it, filesystem write access should be limited to the assigned worktree, and production credentials should never be placed in repository files or general shell profiles. OpenAI’s Codex security guidance describes workspace-limited local writes, network-off defaults, and cloud secrets that are removed before the agent phase, GitHub similarly recommends minimum GITHUB_TOKEN permissions and avoiding plaintext sensitive data in workflow files. CI should enforce the same boundary. Protected branches can require successful checks and reviews before integration, and secret-scanning push protection can block recognized credentials before they enter repository history. Agent-generated workflows deserve additional scrutiny because automation with write credentials expands the blast radius beyond source edits. A default read-only token, explicit permission elevation for narrowly defined jobs, and human approval for changes to workflows or deployment configuration provide a stronger control plane. GitHub’s secure-use guidance explicitly recommends least-privilege workflow credentials. Worktrees should be disposable after integration. Git recommends git worktree remove for finished linked worktrees and provides prune for stale administrative metadata. Unclean worktrees are protected from ordinary removal unless force is requested, which makes final inspection practical before deletion. Cleanup can remain explicit: Shell git worktree remove ../agent-auth git worktree remove ../agent-checkout git worktree prune Conclusion Running two coding agents safely on one codebase is primarily a source-control and governance problem. Reliable parallelism comes from isolated worktrees, narrow path ownership, versioned agent instructions, Git-derived scope checks, clean validation, protected integration, least-privilege credentials, and deliberate cleanup. The central rule is simple: agents may work concurrently, but mutable state, authority, and acceptance must remain separated. With that boundary in place, parallel agent execution becomes an auditable engineering workflow rather than two autonomous processes racing inside the same repository.

By Uthej Mopathi

Your Spark Job Isn't Slow Because of Bad Code. It's Slow Because of the Wrong Join

I learned this lesson the hard way. We had a critical data pipeline running for over 3 hours every single day. The logic was perfectly clean. The overarching schema was explicitly right. There were absolutely no obvious memory leaks, and absolutely nothing looked fundamentally broken in the raw PySpark transformations. Then I finally checked the physical query plan. Under the hood, Apache Spark was quietly executing a massive Sort-Merge Join to merge a multi-terabyte fact table with a dimensional lookup table that was barely 50MB in total size. One single line of code changed—wrapping that exact tiny lookup table cleanly in a broadcast() hint—and the exact same analytic job plummeted from 3 hours to just 18 minutes. That was it. One word saved us hours of daily compute and massive underlying cloud FinOps costs. Wrong join types are financially devastating directly because they are completely silent. Spark will not throw an aggressive exception. Your pipeline will not explicitly fail. It will simply execute your logic confidently 10× slower than it architecturally ever needs to. Here is the exact mental model I exclusively use now every single time I write a distributed join in Apache Spark. TL;DR: Silent shuffle bottlenecks kill massive Spark performance. Always explicitly broadcast small tables (< 200MB), default natively to Sort-Merge for massive dual-sided joins, and actively, aggressively leverage AQE skew joins to fundamentally prevent heavy task skew. Check your physical plans! 1. The Small Table: Always Broadcast When actively joining a massive fact table heavily against a tiny dimension table (like cleanly mapping a primitive status_id logically to a status_name), globally shuffling the multi-terabyte fact table wildly across the distributed cluster is architectural suicide. The rule: If a table is reliably under 200MB, rigorously physically force a Broadcast Hash Join.The mechanism: Spark intelligently bypasses the massive network shuffle entirely. It simply naturally copies the tiny 50MB table directly into the RAM of every single native worker node, allowing them to map data logically and locally. The Implementation Python from pyspark.sql.functions import broadcast # Wrapping the small lookup table strictly natively in a broadcast hint enriched_df = massive_fact_df.join( broadcast(small_lookup_df), "customer_id", "left" ) 2. Both Sides Massive: Default to Sort-Merge If you are systematically actively joining two massive, multi-terabyte tables accurately together (e.g., dynamically merging historical transactions cleanly with historical web_sessions), you physically cannot organically broadcast data without instantly dynamically triggering brutal Out-Of-Memory (OOM) driver exceptions. The Rule: Default heavily unconditionally to the Sort-Merge Join.The Mechanism: This is Spark's absolute most robust, incredibly stable joining algorithm physically built for massive scale. Spark heavily and organically shuffles the massive data wildly across the cluster so that precisely matching keys uniquely land physically on the exact same nodes, fundamentally and strictly sort them, and actively, efficiently, and accurately merge them natively. It is technically slower than a pure broadcast, but it is incredibly beautifully resilient inherently at petabyte scale. The Implementation Python # No explicit hints structurally required. Spark will cleanly natively default seamlessly to Sort-Merge for massive large datasets. final_df = massive_transactions_df.join( massive_sessions_df, "user_id", "inner" ) 3. Highly Skewed Data: Enable AQE Skew Join In heavy enterprise datasets, physical data is rarely organically distributed perfectly evenly. Imagine an active e-commerce platform where the default "Guest Customer" cleanly accounts for physically 60% of all universal platform transactions. If you intelligently execute a naive Sort-Merge Join broadly on customer_id, one single isolated Spark executor will physically be forced systematically to exclusively process the entire massive 60% "Guest" chunk. The other 199 regular executors will efficiently and cleanly finish in seconds and sit completely idle while that one node globally grinds to a halt. The rule: Actively, safely leverage Adaptive Query Execution (AQE) dynamically to natively, beautifully split heavily skewed partitions dynamically.The mechanism: AQE actively, dynamically, and securely detects massively skewed partitions directly mid-flight, accurately splitting them cleanly into optimally smaller, incredibly uniform, reliable sub-partitions so they can be effectively and seamlessly processed rapidly and cleanly in parallel. The Implementation Python # Ensuring AQE and Skew Join optimization are physically aggressively cleanly enabled locally in the specific cluster config spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") The Silent Hero: Adaptive Query Execution (AQE) The part most data engineers fundamentally miss: AQE has actually been turned ON by default since Apache Spark 3.2. This means Spark is reading actual statistical data mid-job. If you execute a Sort-Merge Join on a massive table that unexpectedly shrinks to 40MB after an aggressive .filter() clause, AQE will actively intercept the job mid-flight and organically auto-switch the execution directly into a lightning-fast Broadcast Join. You technically don't have to code anything for this to happen. It organically just happens. But you do have to verify it. You must explicitly verify that spark.sql.adaptive.enabled is active in your environment. You must actively understand exactly what it is doing—because when AQE occasionally guesses wrong (usually due to stale table statistics), you need to know precisely how to aggressively override it with manual hints. Conclusion Performance tuning in distributed compute engines fundamentally comes down to actively understanding the physical network shuffle. Check your explicit joins. Aggressively read your physical query plans (using .explain()). And never blindly trust default configurations at the enterprise level. What is the absolute worst join performance bottleneck you have ever hit in production? Let me know in the comments below!

By Syed Siraj Mehmood

Building a Python API Client That Doesn’t Fall Apart When the API Misbehaves

The first version of almost every API client I write looks embarrassingly simple. Send a request. Parse the JSON. Return the result. Something like this: JSON import requests def get_data(url):response = requests.get(url)response.raise_for_status()return response.json() For a quick test, that is usually enough. Then I leave it running for a while. Eventually the connection hangs, the API returns a 500, or I get a 429 because I was a little too aggressive with polling. That is usually the point where the “simple client” stops being simple. The interesting part is not making the request. It is deciding which failures are worth retrying and which ones should fail immediately. That distinction matters more than adding a generic retry loop around everything. The First Thing I Add Is a Timeout I used to treat timeouts as an optional detail. I do not anymore. A request without an explicit timeout can wait much longer than expected when the remote service is slow or unreachable. For a script that runs once, that is annoying. For a worker or monitoring process, it can gradually turn into a much larger problem. So even before thinking about retries, I normally start with: Python response = requests.get(url,params=params,timeout=10) Ten seconds is not a universal recommendation. It depends on what the API is doing. But I prefer having a number I deliberately chose over letting a network call wait indefinitely. For a lightweight market-data endpoint, ten seconds already feels generous. For a large export or a slower internal service, I might choose something else. The important part is that the timeout is intentional. Not Every Error Should Be Retried This was probably the mistake I made most often when I first started adding retry logic. The naive version looks like: Python for attempt in range(5):try:return make_request()except Exception:time.sleep(2) It feels robust because the program “keeps trying.” In reality, it can make things worse. If the server returns 401 Unauthorized, retrying the same request five times will not fix the credentials. If the endpoint returns 404, waiting two seconds and asking for the same missing resource again is usually pointless. If the request itself is invalid, a retry just repeats the same bad request. The failures I usually consider temporary are things such as: connection errors and timeouts429 Too Many Requestssome 5xx server errors Everything else deserves more careful treatment. A client should not confuse persistence with resilience. A Small Retry Function For smaller projects, I like keeping the behavior visible rather than hiding everything inside a large abstraction. A basic version might look like this: Python import randomimport timeimport requests RETRYABLE_STATUS_CODES = {429,500,502,503,504} def get_json(url, params=None, max_attempts=4):for attempt in range(max_attempts):try:response = requests.get(url,params=params,timeout=10) if response.status_code in RETRYABLE_STATUS_CODES:raise requests.HTTPError(f"Temporary HTTP error: {response.status_code}",response=response) response.raise_for_status()return response.json() except (requests.Timeout,requests.ConnectionError,requests.HTTPError) as exc: if attempt == max_attempts - 1:raise delay = (2 ** attempt) + random.uniform(0, 1) print(f"Request failed: {exc}. "f"Retrying in {delay:.2f}s") time.sleep(delay) There is nothing particularly sophisticated here. That is partly why I like it. I can read the function six months later and immediately understand what it will retry. Why I Add Jitter The random.uniform(0, 1) part looks insignificant, but it solves a real problem. Imagine several workers call the same API and all receive a temporary failure at roughly the same moment. Without jitter, they might all retry after: 1 second2 seconds4 seconds8 seconds They stay synchronized. So instead of reducing pressure on the service, they repeatedly hit it together. Adding a small random component spreads those retries out. For one local script, this barely matters. For multiple workers or scheduled jobs, it starts to matter quite a lot. It is a small example of something I see often in backend work: code that behaves perfectly with one process can behave very differently when twenty copies are running. 429 Needs a Little More Respect Rate limits are also a case where simply retrying quickly is the wrong response. If an API says “slow down,” sending the same request again immediately is not resilience. It is ignoring the server. If the response includes a Retry-After header, I would rather respect it: Python retry_after = response.headers.get("Retry-After") if retry_after:delay = float(retry_after)else:delay = (2 ** attempt) + random.uniform(0, 1) This also makes the client less dependent on my guess about how aggressive the rate limit is. When there is no explicit guidance, exponential backoff is still a reasonable fallback. Logging the Failure Is More Useful Than It Sounds One thing I underestimated for a long time was logging. When I was running scripts manually, print() felt good enough. The problem appears later, when somebody asks: “Why did this job miss data at 03:12?” If all I know is “the request eventually failed,” debugging becomes guesswork. At minimum, I want to know: When the request failedWhich endpoint failedThe HTTP status if one existedWhich retry attempt it wasHow long the client waitedWhether the final attempt failed permanently For a real service, I would use Python's logging module instead of scattered print statements. The logs do not need to be verbose. They need to answer questions later. That is a different goal. Retrying Writes Is More Dangerous GET requests are usually where retry logic feels straightforward. POST requests make me more cautious. Suppose a client submits an order or creates a resource. The server processes it successfully, but the connection drops before the client receives the response. From the client's perspective, the request “failed.” If it blindly retries, the operation could happen twice. This is where idempotency becomes important. If an API supports idempotency keys or client-generated request IDs, I use them for operations where duplicate execution would be a problem. Otherwise, I want the retry behavior for writes to be much more conservative than the behavior for reads. This is especially relevant in financial systems. A duplicate market-data request is annoying. A duplicate order is something else entirely. The Same Pattern Shows Up in Trading APIs I run into these problems a lot when looking at market-data and trading integrations. The domain makes the trade-offs easier to see because APIs are often being called continuously rather than once. A price-monitoring process may run for hours. A worker may request candles repeatedly. A trading application may depend on several remote services at the same time. BYDFi is one platform I encounter through my work, so I mention it here as a disclosed real-world example rather than an independent recommendation. The useful engineering lesson is not specific to one exchange. Whether the client talks to a trading platform, payment provider, cloud service, or internal API, the same questions keep appearing: What happens when the service is slow? Which errors are temporary? How often should I retry? Could retrying create a duplicate side effect? What information will I need when debugging this tomorrow? Those questions are much more important than the first successful API response. I Usually Keep the Client Boring There is always a temptation to turn a small HTTP wrapper into a miniature framework. I try not to. For most projects, I would rather have a client with obvious behavior than one with ten layers of abstraction. The version I want is usually boring: explicit timeout, a short list of retryable errors, limited attempts, backoff, jitter, useful logs, and special handling for operations that should not be duplicated. Nothing about that is clever. That is the point. Network failures are already unpredictable enough. I do not want the recovery logic to be unpredictable too. Final Thoughts Getting a 200 OK is the easiest part of building an API integration. The real work starts when the remote service does something you did not expect. I have found that the most reliable clients are not the ones that retry the most. They are the ones that have a clear opinion about failure. They know when to wait. They know when to try again. And, just as importantly, they know when to stop.

By Ally Garcia

Gossip on Cryptography: Part 3

In this blog, we will continue our discussion from the previous blog, Parts 1 and 2. If you have not read it, please read it once. So far, we have discussed Caesar cipher, Vigenere cipher, symmetric encryption, AES, convergent encryption, and IV. If all these terms sound familiar to you — great! If not, please go back and read Part 1 and Part 2 first. Now, in Part 2, we ended with a teaser that in the next blog we will talk about hashing and asymmetric algorithms. So let's get into it! First, Let's Talk About Hashing So far, everything we discussed was about encryption and decryption — you encrypt something, and you can decrypt it back. Simple. But what if I tell you there is a technique where you convert data into something, and you can NEVER go back to the original? Sounds weird, right? Why would someone do that? Let me give you a real-life example. Imagine you are the owner of a hostel. You keep a register at the gate. Every night at 10 PM, you take a photo of this register. Now, the next morning, if someone modifies the register (adds a fake entry or removes one), you can easily compare yesterday's photo with today's register and catch the change. Hashing works exactly like this photo. You give any data as input; the hash function produces a fixed-size string (called a hash or digest). If even ONE character in the original data changes, the hash output changes completely. A Simple Example Input: "Sahil" Hash (SHA-256): 9b4c...a32f (a 64 character string) Input: "sahil" (just lowercase 's') Hash (SHA-256): 7f3a...b91e (a completely different 64-character string!) See? Even one small change → completely different hash. This property is called the Avalanche Effect. Important Properties of Hashing Let's keep it simple. A good hash function has these properties: One way (Irreversible): You can go from "Sahil" → hash, but NOT from hash → "Sahil." It's a one-way street. Like making an omelet from an egg — you can't get the egg back from the omelet.Deterministic: The same input will ALWAYS give the same output. "Sahil" will always produce the same hash every time.Fixed size output: No matter how big your input is — whether it's one word or an entire 1000-page book — the output hash size is always the same (for SHA-256, it's always 64 characters).Avalanche effect: Even a tiny change in input means a completely different hash. We just saw this above. So, Where Is Hashing Used? Great question! Here are the most common places: 1. Storing Passwords This is the most common use case. When you set a password on any website, good websites never store your actual password. They store the hash of your password. So when you log in next time: You type your passwordThe website hashes itCompares it with the stored hashIf they match → Welcome! This is why when you click "Forgot Password" on most websites, they reset your password instead of showing you the old one. Because they literally don't know what your old password was! 2. File Integrity Check You download software from the internet. How do you know no one tampered with it during download? The website gives you the hash of the original file. After downloading, you calculate the hash of your downloaded file. If they match, the file is safe! This is used everywhere — Linux ISO downloads, software releases on GitHub, etc. 3. Digital Signatures We will cover this more in detail in coming parts! Popular Hashing Algorithms MD5 – Old, fast, but now considered weak. Avoid using it.SHA-1 – Also old, mostly deprecated now.SHA-256 – The current gold standard. Used everywhere. (Bitcoin also uses this!)bcrypt/Argon2 – Special hashing algorithms designed specifically for passwords. They are intentionally slow — which makes brute force attacks harder.PBKDF2 (Password-Based Key Derivation Function 2) – Another password-specific algorithm. It takes your password + a salt and runs a hashing function thousands of times in a loop (this is called key stretching). The more iterations, the harder it is to brute force. It is widely used and is the recommended choice in many government and enterprise security standards (like NIST). Wait — Can Someone Still Crack Hashes? Yes! There are ways to try. The most common one is called a Rainbow Table Attack. Here's how it works — imagine I am a hacker and I have pre-calculated the hashes of millions of common passwords: "password" → 5f4dcc..."123456" → e10adc..."admin" → 21232f... Now if I get your stored hash from a database breach, I just look it up in my table. If your hash matches any entry → I know your password! Solution? SALT! No, not the one you put in food In cryptography, a Salt is a random value that is added to your password before hashing. Your password: "mypassword" Random Salt: "xK9#mQ" Combined: "mypasswordxK9#mQ" Hash of this: (some unique hash) Now, even if two people have the same password "mypassword", because their salts are different, their stored hashes will be completely different! Rainbow Table attacks become useless. The salt is stored alongside the hash in the database (it's not a secret; it just needs to be unique per user). Now Let's Talk About Asymmetric Encryption Remember in Part 2 we discussed symmetric encryption — where the same key is used for both encryption and decryption? The problem with symmetric encryption is — how do you share the key safely? Imagine Rahul in Delhi wants to send an encrypted message to Priya in Mumbai. He needs to share the key with her first. But if he sends the key over the internet, a hacker can intercept the key and then decrypt all future messages. This is known as the Key Distribution Problem. Asymmetric encryption solves this beautifully. The Magic of Two Keys In asymmetric encryption, instead of one key, you have two keys: Public key – You share this with the WHOLE WORLD. Anyone can have it.Private key – This stays with you ONLY. Never share it with anyone. The magic is: Whatever is encrypted with the Public Key can ONLY be decrypted with the Private Key. And these two keys are mathematically linked to each other. Real Life Example — The Magic Mailbox Think of it like a special mailbox: The mailbox has a slot (public key) — anyone can drop a letter in it.But only YOU have the key to open the mailbox (private key) — only you can read the letters. Rahul wants to send a secret message to Priya: Priya shares her Public Key with Rahul (and the whole world — no problem!)Rahul uses Priya's Public Key to encrypt the messageThe encrypted message travels over the internet — even if a hacker intercepts it, they can't read itPriya uses her Private Key to decrypt the message No need to share any secret key beforehand! The problem of key distribution is solved! Most Popular Asymmetric Algorithm: RSA RSA (named after its inventors Rivest, Shamir, and Adleman) is the most famous asymmetric algorithm. It is based on a very simple mathematical observation: It is very easy to multiply two large prime numbers. But it is extremely hard to factorize the result back into those two primes. For example: Easy: 61 × 53 = 3233Hard: Given 3233, find the two prime factors (61 and 53) When the numbers are hundreds of digits long, even the fastest computers in the world would take millions of years to crack it. That's the security of RSA! RSA key sizes you will commonly see: 1024-bit (old, avoid), 2048-bit (current standard), 4096-bit (extra secure). Symmetric vs. Asymmetric — When to Use What? Symmetricasymmetric Keys Same key for encrypt & decrypt Different keys (public + private) Speed Very Fast Slow Key Sharing Problem Yes, it exists No, solved! Example Algo AES RSA Used For Encrypting large data Key exchange, Digital Signatures In the real world, both are actually used together! The typical flow is: Use asymmetric encryption to securely exchange a secret keyThen use symmetric (AES) encryption for the actual data — because it's much faster This combo is how HTTPS (the secure web) actually works! When you open any https:// website, this exact thing is happening in the background. That little lock you see in your browser? That's this. Terms We Have Learned So Far (Including Parts 1 & 2) CryptographyAlgorithmPlain textKeyCipher textSymmetric encryptionConvergent encryptionInitialization vector (IV)HashingHash/DigestAvalanche effectSaltRainbow table attackAsymmetric encryptionPublic keyPrivate keyRSA Please keep them in mind, as these are the generic terms used everywhere in the world of encryption and decryption. Coming in Part 4 (Part 4 is in progress — stay tuned!) In the next blog, we will gossip about some very interesting things like: Digital signatures – How do you prove that a message is really from who it claims to be from?PKI infrastructure – The backbone of trust on the internetSSL/TLS – What actually happens when you open an HTTPS website, step by stepEnvelope encryption – A very clever technique used by cloud providers like AWS and GCPAnd more... Stay tuned for Part 4! If you liked this blog, do give it a like and share it with someone who you think should learn this. Let's spread the knowledge! Read the previous parts here: Part 1 and Part 2.

By Sahil Aggarwal

Portable Intelligence Architecture: When the Runtime Becomes the Hard Problem

For two decades, we focused on moving data to the intelligence. Now, we’re seeing a massive shift: we have to move the intelligence to the data. That flip changes everything. Your host platform isn’t just an API gateway anymore; it’s an operating system. The Meeting That Wasn’t About Models The meeting that changed how I think about AI infrastructure had almost nothing to do with models. We spent months obsessing over model quality. Then, over a few weeks, the agenda quietly reorganized itself. We were talking about onboarding third-party units. We debated what happens when two versions of the same model disagree under replay. We worried about whether one tenant’s inference could starve a neighbor’s on a shared accelerator. We fought over who pays for a millisecond. At some point, I wrote down what was actually on the whiteboard: routing, versioning, isolation, admission control, resource accounting, governance boundaries, and latency budgets. That is not a machine learning agenda. That started to look more like an operating systems agenda. We had stopped solving an AI problem and started designing a runtime. It happened the way architecture usually happens, as an accumulation of decisions that only later reveal their shape. This article names and outlines that shape. I call it the Portable Intelligence Architecture, or PIA. It is not a product, a vendor category, or a rebranding of edge ML. It is an architecture pattern that several teams appear to be converging on independently, and an argument that the runtime layer deserves to be treated as a first-class architectural concern rather than an implementation detail discovered later in production. Twenty Years of Moving Data to Intelligence We’ve been living in an API-first world because, for a long time, the math was simple: intelligence was expensive and centralized, payloads were small, and network costs were rounding errors. It made sense to ship the data to the model. That premise was right for its time. If you’re running a proprietary fraud model that needs nightly retraining and a custom feature store, you don't ship that code to the caller. You expose an endpoint. The caller sends a few kilobytes, and you send back a score. That made sense at the time. Every abstraction we’ve built lately has just been a refinement of where that intelligence lives. Libraries. APIs. Microservices. Containers. Portable Intelligence Units. I see this as a ladder. Libraries were linked. APIs were called. Microservices were deployed. Containers were scheduled. Each step made the unit more self-contained and independent. A Portable Intelligence Unit (PIU) is just the next rung. It’s a versioned, resource-declared unit that brings the inference straight to the host where the context already lives, and at scale. The old API-first rule isn't dead, but it’s become conditional. In my experience, three things changed the game. Accelerators became first-class citizens. GPUs and LPUs aren't "specialized hardware" anymore. They're just another resource class for the scheduler. Once you can allocate inference hardware like memory, deploying a model locally becomes a standard infra task, not a massive research project. Models got smaller. Thanks to distillation and quantization, a killer model can now be a few gigabytes. When the model is smaller than the context it needs to digest, moving the model is the only logical choice. Context is live and now at scale. The signal that matters for enterprise decisions isn't a static prompt. It's live inventory, session data, and real-time supply conditions. That state is too heavy and sensitive to export. It stays where it is. Just think about all the session data the new LLM chats are generating. I first assumed portable intelligence would mostly mean shipping simple decision logic outward: rules, gradient-boosted trees, small classifiers, the kind of thing you push to the edge because it is cheap to push. My assumption was wrong. Modern accelerator economics mean we can run heavy-duty models right at the point of decision. The old constraints that forced us to keep things simple at the edge have almost vanished. Where APIs Alone Become Insufficient Remote inference isn't going away, but I've found it’s structurally insufficient for a certain class of high-performance systems. Here are the four forces we keep running into. Context gravity. We’ve known for years that you move computation to the data when the data is expensive to ship. But now, it’s about fidelity. If you try to export the live state to a remote endpoint, you have to freeze and flatten it. You don't just lose bandwidth; you lose the predictive relationships that make the model work.The millisecond wall. When you have a 10ms budget, a remote round trip isn't just "slow." It's a non-starter. You can't optimize your way out of a budget the network ate before your code even touched the CPU.Governance and silos. Between data residency laws and partner contracts, the data often physically cannot move. Moving the model is frequently the only way to stay compliant. The model crosses the fence; the data doesn't. Think GDPR, etc.The cost of scale. Per-call pricing is great when you're small, but at high volume, it'll kill your margins. We usually discover this late, and it’s a painful, expensive lesson to learn. These forces don't kill the API, but they force us to build a second plane where we deploy intelligence instead of just calling it. Remote inference Portable intelligence Context moves. Model moves. Boundary crossed by data. Boundary crossed by artifact. Cost unit is per call. Cost unit is per allocated compute. Failure mode is tail latency. Failure mode is operational drift. Portable Intelligence Architecture In simple terms, PIA is an architecture where you package inference into a portable, versioned unit and drop it into a host runtime that already owns the data. It has three parts. A Portable Intelligence Unit, or PIU, is the deployable artifact: a model plus its preprocessing, its declared resource envelope, its data contract, its version identity, and its governance metadata. A PIU is not “a container with a model in it.” The container is packaging. The contract is the architecture.The host is the runtime that admits, schedules, routes to, isolates, meters, and observes PIUs. It owns the context that made deployment worthwhile in the first place.The decision plane is the hot path where PIUs execute against live context under a latency budget. It is deliberately separated from the control plane, which handles admission, versioning, policy, and rollout on a different time scale. Running a model near data isn't new; we've been doing that with embedded code for decades. The real innovation here is the multi-tenant, contract-governed host. We're talking about units from different teams, on different release cycles, all sharing one runtime to make one decision. That’s a runtime problem, plain and simple. And luckily, operating systems have already given us the blueprint. Figure 1. The control plane governs the host runtime, which routes requests across Portable Intelligence Units and reduces their outputs into a decision. The Five Principles Move the smaller thing. The architectural question is never “should intelligence be central or distributed.” It is “which artifact is cheaper to move in bytes, in fidelity, and in legal exposure? Is it the model or the context?” Answer that honestly per workload, and the topology falls out. It will not be the same answer for every workload in the same system.The host is an operating system, not a gateway. A gateway routes requests. An operating system admits programs, isolates them, schedules them against finite resources, accounts for what they consume, mediates access to privileged state, and defines the interface through which they ask for more. Once you have multiple third-party units sharing accelerators under a shared latency budget, you are building the second thing whether you intended to or not. Building it deliberately is cheaper.Optimize the composition, not the component. This is the principle I would have argued against a year ago. Marginal model quality is usually not the binding constraint. Routing, admission, fallback, and sequencing of specialized units produce more end-to-end improvement than another point of accuracy on any single unit. The system-level win comes from composition, and composition is an orchestration property.Operational properties are the contract. Latency, versioning, governance, and cost are not things you tune after the design. They are the declared interface of a unit. A PIU that does not declare a p99 budget, a data-class permission set, and a resource envelope cannot be safely admitted, because the host has no basis for scheduling it. Treat these as first-class contract fields, or you will enforce them later with incident reviews.Assume re-entry. Single-shot inference is the easy case and increasingly the minority case. Design the interface between a unit and the host assuming that intelligence will call back into the runtime mid-reasoning for a lookup, a tool, or another unit. This is the hardest principle to satisfy, and I will be honest about the fact that we have not fully solved it. Composition Is Where the Value Is The single most consequential thing we learned was not that one sophisticated model could be deployed at the decision point. It was that several specialized units working together were worth substantially more than one general unit. The reasons are structural, not empirical: Specialized units are small, which makes them cheap to schedule and fast to load.They can be owned by different teams, or different companies, and released independently.They can be versioned independently, which means a regression is contained.The routing decision between them is itself cheap, so composition costs less than you would guess. The router is where the architecture actually lives. It decides which units see a request, whether they run in parallel or in sequence, what happens when one exceeds its budget, and what the fallback path is. Every hard question in this pattern eventually becomes a router question. The Host Becomes an Operating System The analogy is not decorative. It is predictive: it tells you which problem you will hit next. Operating system concept PIA equivalent Process Portable Intelligence Unit Scheduler Router and admission controller Memory protection Tenant and data-class isolation Resource accounting Per-unit compute metering and attribution System calls Callback interface from unit into host Package management Partner onboarding and unit registry Permissions Governance policy on data classes Device drivers Accelerator abstraction If you take one thing away, let it be this: keep your control plane and decision plane strictly separate. It’s the highest-leverage move you can make. The control plane handles admission, registry, version promotion, policy, entitlements, cost models, rollout, and rollback. The decision plane handles request context, routing, inference, composition, fallback, and emitting the decision. When you let control-plane junk leak into the decision path, like a registry call during routing, you’re coupling a minutes-scale system to a milliseconds-scale one. Your tail latency will let you know exactly why that was a mistake. The Contract Look at the manifest below. These fields aren't just metadata; they're everything the host needs to schedule, isolate, and govern a unit without ever needing to call the owner. Figure 1: PIU Manifest JSON JSON { "piu": { "id": "risk-scorer", "version": "4.2.1", "owner": "partner:northwind", "artifact": { "image": "registry.internal/piu/risk-scorer@sha256:9f2c...", "signature": "cosign:...", "format": "onnx" }, "resources": { "accelerator": { "class": "gpu.small", "count": 1 }, "memory_mb": 6144, "max_concurrency": 32 }, "budget": { "p50_ms": 4, "p99_ms": 18, "timeout_ms": 25, "on_exceed": "fallback:[email protected]" }, "data_contract": { "input_schema": "schemas/[email protected]", "compat": "backward", "required_fields": ["entity_id", "signal_vector", "channel"], "null_policy": "reject" }, "governance": { "data_classes": ["pseudonymous", "aggregate"], "prohibited_classes": ["pii", "cross_tenant"], "residency": ["eu"], "audit": "sampled:0.01" }, "economics": { "billing_unit": "accelerator_ms", "attribution": "tenant" }, "capabilities": { "reentrant": false, "max_callbacks": 0 } } } Two fields in that manifest are doing far more work than the rest. data_contract.compat is where partner integrations actually succeed or fail. capabilities.reentrant is where the open problem lives. Getting this onto Kubernetes is the easy part once you have the contract. The substrate is just plumbing; the contract is the architecture. Figure 2: Kubernetes Deployment Manifest YAML apiVersion: apps/v1 kind: Deployment metadata: name: piu-risk-scorer labels: piu.host/id: risk-scorer piu.host/version: "4.2.1" piu.host/tenant-class: partner spec: replicas: 6 selector: matchLabels: piu.host/id: risk-scorer template: metadata: labels: piu.host/id: risk-scorer piu.host/version: "4.2.1" spec: nodeSelector: accelerator.class: gpu.small containers: - name: unit image: registry.internal/piu/risk-scorer@sha256:9f2c... env: - name: PIU_MANIFEST value: /etc/piu/manifest.json - name: HOST_CALLBACK_SOCKET value: /var/run/piu/host.sock resources: limits: nvidia.com/gpu: 1 memory: 6Gi requests: memory: 6Gi readinessProbe: httpGet: path: /healthz/warm port: 8080 initialDelaySeconds: 20 volumeMounts: - name: host-socket mountPath: /var/run/piu volumes: - name: host-socket hostPath: path: /var/run/piu type: Directory Pay attention to the readiness probe. If a unit says it's ready before the accelerator memory is loaded, it'll start taking traffic it can't handle. That’s how you would end up exceeding your p99. Operational Lessons, Ranked By Cost Partner onboarding is the hardest problem, and it is not a packaging problem. I expected packaging friction. What we actually hit was data compatibility. Two teams agree on a schema, ship against it, and still fail because one side’s “session” means something subtly different from the other’s, or a field is nullable in practice but not in contract, or the feature distribution the unit was trained on does not match the distribution the host produces. Schema compatibility is necessary and nowhere near sufficient. What moved the needle was a developer protocol: an SDK, a conformance test suite, and golden datasets that a partner could run before ever touching our environment.Versioning models is easy. Testing across versions is not. Tagging a version takes an afternoon. Knowing whether version 4.2.1 behaves acceptably across every tenant, every context distribution, and every composition path it participates in is a combinatorial problem. Shadow traffic and production replay are the only honest tests I know of. Unit tests on models are theater.Multi-tenancy on accelerators is harsher than on CPU. Noisy-neighbor effects that are annoying on CPU are structural on shared accelerators, particularly once batching enters the picture. Batching couples tenants’ latency profiles: one tenant’s traffic shape now determines another tenant’s tail. Either isolate hard and pay for it, or accept the coupling explicitly and model it.Design the cost model before deploying sophisticated inference. This should be a boring statement, but it isn't. If you deploy first, you learn your unit economics from a bill, in arrears, after the architecture has ossified. Decide early what the billing unit is, whether it's accelerator-milliseconds, admitted requests, or allocated capacity, because that choice propagates into routing policy and eventually into what you can sell.Average latency is a vanity metric. p99 is the product. Dean and Barroso made this point over a decade ago, and it applies with extra force here because composition fans out. If a decision touches four units and each has a well-behaved tail, the composed tail is worse than any individual one. Budget the composition, not the components. Enforce timeouts at the router with a defined fallback, and treat fallback as a normal outcome rather than an error. Tradeoffs, Stated Plainly PIA buys latency, governance topology, and cost predictability at high volume. It costs the following: Operational surface area. You now operate a runtime. That is a permanent staffing commitment, not a project.Debugging across trust boundaries. When a composed decision is wrong, and three of the units belong to other organizations, root cause becomes a negotiation.Supply chain risk. Admitting third-party inference into your runtime is admitting third-party code into your runtime. Signing, scanning, and resource limits are table stakes, not maturity.Freshness. Centralized models update on one cadence. Distributed units update on many. Some drift is now a design parameter rather than an accident.Capacity planning. Accelerator capacity is lumpy, and lumpy capacity plus strict latency budgets means paying for headroom you do not use. When Not to Use PIA Do not build this if: Your volume is low. Per-call pricing is a gift at low volume; take it.Your latency budget is loose. If 200 ms is fine, call the API.There is one model, one owner, one release cadence. You have a deployment, not a runtime.Your context is small and legally exportable. Then the context is the smaller thing to move, and principle 1 tells you to move it.Your models are iterating weekly. Central deployment has a much shorter feedback loop, and early-stage model velocity beats architectural elegance every time. The pattern earns its complexity at the intersection of high volume, tight budgets, multiple owners, and immovable context. Outside that intersection, it is overhead with a nice diagram. The Future Is Agents Orchestrating Units The direction this is heading is not one enormous model at the decision point. It is an agent orchestrating many specialized Portable Intelligence Units, selecting and sequencing them dynamically based on the decision at hand. That future is architecturally coherent right up until it hits the problem we have not solved. Iterative workflows break the budget model. Single-shot inference has a clean contract: the host gives a unit context, the unit returns a result inside a declared budget. Re-entrant reasoning does not work that way. A unit pauses mid-reasoning, calls back into the runtime for a lookup or another unit’s output, and resumes. Now the budget is not a duration; it is a session with an unknown number of stages, holding accelerator memory the whole time. Every mechanism the host relies on gets harder: admission control cannot know the cost of admitting a request, scheduling has to handle units that are resident but idle, fair-sharing has to prevent one long chain from starving short ones, and tracing has to reconstruct a call graph that did not exist at admission time. Operating systems solved the analogous problems with preemption, quotas, and priority scheduling over roughly thirty years. I do not think we get a shortcut. But I do think naming the problem correctly is most of the work, and the correct name is scheduling, not prompting. Conclusion Portable Intelligence Architecture does not replace APIs. It complements them. Most systems will run both planes, and the interesting design work is deciding which decisions belong on which plane. What has changed is where the difficulty sits. For most of the last decade, the limiting factor in enterprise AI was model quality, and the industry organized itself accordingly. That is no longer where the constraint binds. The models are good enough for a large and growing set of enterprise decisions. The new bottleneck is the runtime. We need a layer that lets portable intelligence run safely and at scale across different owners and tight budgets. We stumbled into our runtime one whiteboard at a time. Trust me, it’s much cheaper to build it on purpose. This isn't an implementation detail; it’s a first-class architectural concern. References Gray, J. Distributed Computing Economics. Microsoft Research, 2003.Dean, J. and Barroso, L. A. The Tail at Scale. Communications of the ACM, 2013.Barroso, L. A., Clidaras, J., Hölzle, U. The Datacenter as a Computer. Morgan & Claypool.Sculley, D. et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.Dehghani, Z. Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly, 2022.Kubernetes documentation: Device Plugins and Dynamic Resource Allocation.ONNX: Open Neural Network Exchange specification.

By Punit Shah

Making Running Optional: Scaling AI Agents on Kubernetes With Agent Substrate

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text <actor-name>.<atespace>.actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

By Mayowa Fajobi

Extracting Entities and Relationships From Engineering Documents With spaCy

Engineering teams generate a lot of useful knowledge, but most of it is locked inside text. A service ownership note may tell you who owns an application programming interface (API), while a runbook may tell you which database a service relies on. An incident review may detail how one fault impacted the other systems. Each of these is individually useful. It’s when we are able to link together all of these facts that we get our greatest value. As an example, think about a developer who wants to know: Plain Text Which team owns the API that Checkout Service depends on? To answer this question, the developer would need to make several connections: Plain Text Checkout Service depends on Payment API. Platform Team owns Payment API. With a keyword search, you could locate documents that include “Checkout Service,” “Payment API.” With a retrieval-augmented generation(RAG) pipeline, you could locate relevant chunks and forward them to your LLM. If there isn’t some understanding of the relationships between services, APIs, databases, etc., then your RAG may still fail to identify a connection. This is where entity and relationship extraction can help. We want to move beyond viewing engineering documents as plain text; instead, we’d like to see the text as a source of structured facts, such as: Plain Text Checkout Service --DEPENDS_ON--> Payment API Platform Team --OWNS--> Payment API Payment API --STORES_IN--> PostgreSQL These facts can then provide the basis for knowledge graphs, dependency analysis, impact analysis, and GraphRAG pipelines. Here we will define a simple Python pipeline using spaCy to extract specific types of engineering entities and relationship triplets from text. Our objective is to produce a working prototype, rather than creating the ultimate extraction tool. We also hope that this provides a practical starting point for developers to test, view, and modify their own documentation. Why This Pattern Matters in Real Systems This pattern applies to multiple aspects of real-world engineering systems. Service Ownership Lookup A developer can query a service owner by looking at a service they are unaware of (as opposed to manual searches of a service catalog). Analyzing Dependencies Using a graph to find all other services impacted when a system (API, Queue, Database) goes down. Response During Incidents Relationship extraction can transform incident notes and runbooks into navigationally easier-to-use dependency maps during outages. GraphRAG Pipelines Entity/relationship extraction is required for GraphRAGs prior to being able to get the appropriate graph contextual information. Poor entity/relationship extraction will result in poor graph contextual information regardless of how good the LLM is. Discovering Architecture Many large organizations have their architectural knowledge dispersed across various documents, diagrams, and repositories. Entity/relationship extraction provides a method to make this knowledge searchable and reusable. We'll use a simple dataset in this tutorial, but you could easily expand on it using your organization's service catalog(s), repository metadata, cloud inventory, or documentation from real engineering systems. What We Are Building We will create a small Python script that reads engineering notes and produces two files: Plain Text entities.json triples.json The first file contains detected entities: JSON [ { "text": "Checkout Service", "label": "SERVICE" }, { "text": "Payment API", "label": "API" } ] The second file contains relationships: JSON [ { "subject": "Checkout Service", "relation": "DEPENDS_ON", "object": "Payment API", "source_text": "Checkout Service depends on Payment API during payment authorization." } ] This output can later be loaded into NetworkX, Neo4j, a vector database, or a GraphRAG pipeline. Project Setup Create a new folder: Shell mkdir spacy-entity-relationship-extraction cd spacy-entity-relationship-extraction Install a virtual environment, spaCy, and the small English model: Shell python3.12 -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install spacy # installs spacy python -m spacy download en_core_web_sm # installs english model The small English model is enough for this tutorial. For production work, you may need a larger model, domain-specific rules, or custom training data. Step 1: Define Sample Engineering Documents Create a file named extract_entities.py. Plain Text documents = [ "Checkout Service depends on Payment API during payment authorization.", "Payment API stores transaction metadata in PostgreSQL.", "Platform Team owns Payment API and manages its deployment pipeline.", "Recommendation Service calls Catalog API to retrieve product details.", "Catalog API indexes product data in Elasticsearch.", "Search Team owns Catalog API.", ] Examples like these are simple; however, they illustrate typical engineering documentation patterns: one System Depends Upon Another System,a service stores information in a database,a team is responsible for a service,one application calls an application programming interface (API). Step 2: Add Custom Entity Rules Most of the time, general-purpose named entity recognition (NER) models are trained to identify persons, organizations, locations, and other types of information such as dates. The engineering documentation our NLP tools will process contains a range of domain-specific entities, including services, APIs, databases, queues, teams, repositories, and cloud-based services. In order for these entities to be identified in the pipeline, it would make sense to incorporate domain-specific rules into our code. Python import spacy def create_pipeline(): nlp = spacy.load("en_core_web_sm") # Load English Model ruler = nlp.add_pipe("entity_ruler", before="ner") # Create Ruler patterns = [ {"label": "SERVICE", "pattern": "Checkout Service"}, {"label": "SERVICE", "pattern": "Recommendation Service"}, {"label": "API", "pattern": "Payment API"}, {"label": "API", "pattern": "Catalog API"}, {"label": "DATABASE", "pattern": "PostgreSQL"}, {"label": "SEARCH_INDEX", "pattern": "Elasticsearch"}, {"label": "TEAM", "pattern": "Platform Team"}, {"label": "TEAM", "pattern": "Search Team"}, ] ruler.add_patterns(patterns) # Add Patterns to Ruler return nlp EntityRuler enables you to include common domain-specific named entities from your company’s service catalog, team listing, repository listing, and database inventory, etc., without training a custom model. Step 3: Extract Entities Now add a function to extract entities from each document. Python def extract_entities(nlp, documents): results = [] for text in documents: doc = nlp(text) for entity in doc.ents: results.append( { "text": entity.text, "label": entity.label_, "source_text": text, } ) return results Run it: Python if __name__ == "__main__": nlp = create_pipeline() entities = extract_entities(nlp, documents) for entity in entities: print(entity) Expected output: Plain Text {'text': 'Checkout Service', 'label': 'SERVICE', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'} {'text': 'Payment API', 'label': 'API', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'} {'text': 'Payment API', 'label': 'API', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'} {'text': 'PostgreSQL', 'label': 'DATABASE', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'} The above output is much better than plain text. The output indicates which tokens within the document refer to engineering entities. Step 4: Normalize Duplicate Entities Because entities are almost always referred to by different names within the same document (eg., payment-api, Payment-API, payment service, etc.), you need an approach that uses aliases to link all of these references back into a single “concept” or node. The first approach to doing this would be creating an alias map. This could look something like this: Plain Text ALIASES = { "payment-api": "Payment API", "Payments API": "Payment API", "payment service": "Payment API", } The second part of this solution involves defining an additional helper function that will take each found entity reference and “normalize” it by replacing it with its corresponding concept label. The normalized value is then stored along with the original extracted data in order to be returned as output. Here's how we can do that: Python def normalize_entity_name(name): return ALIASES.get(name, name) Then update the entity extraction function: Python def extract_entities(nlp, documents): results = [] for text in documents: doc = nlp(text) for entity in doc.ents: results.append( { "text": normalize_entity_name(entity.text), "label": entity.label_, "source_text": text, } ) return results As previously stated, while this process appears to be relatively minor, normalizing entities is probably one of the most critical steps in developing a knowledge graph because if done poorly, what may appear to be a single system in the world may end up being represented as many separate nodes. Step 5: Extract Relationship Triples Now that we have identified all of our entities, we want to find the relationships. We will begin with the simplest way to do this using verb phrases. The simplicity of this method does not allow for much flexibility or learning from the data, as a trained relation extraction model would, but it has the advantage of being completely transparent and very easy to debug. Plain Text RELATION_VERBS = { "depends on": "DEPENDS_ON", "stores": "STORES_IN", "owns": "OWNS", "calls": "CALLS", "indexes": "INDEXES_IN", } Create an additional helper function that identifies entity locations within the text. Python def extract_relationships(nlp, documents): triples = [] for text in documents: doc = nlp(text) entities = list(doc.ents) if len(entities) < 2: continue for verb_phrase, relation in RELATION_VERBS.items(): if verb_phrase in text.lower(): subject = normalize_entity_name(entities[0].text) object_ = normalize_entity_name(entities[1].text) triples.append( { "subject": subject, "relation": relation, "object": object_, "source_text": text, } ) return triples The above code has no universal way to determine which entity is the subject or object; however, when you know your input data is structured like controlled engineering documentation, this method provides a good working example for a baseline. Run it: Python if __name__ == "__main__": nlp = create_pipeline() triples = extract_relationships(nlp, documents) for triple in triples: print(triple) Expected output: Plain Text {'subject': 'Checkout Service', 'relation': 'DEPENDS_ON', 'object': 'Payment API', 'source_text': 'Checkout Service depends on Payment API during payment authorization.'} {'subject': 'Payment API', 'relation': 'STORES_IN', 'object': 'PostgreSQL', 'source_text': 'Payment API stores transaction metadata in PostgreSQL.'} {'subject': 'Platform Team', 'relation': 'OWNS', 'object': 'Payment API', 'source_text': 'Platform Team owns Payment API and manages its deployment pipeline.'} For many internal engineering knowledge bases, this kind of controlled extraction is a good first step before adding more complex models. Step 6: Export JSON The final step is to save entities and triples. Python import json def write_json(path, data): with open(path, "w", encoding="utf-8") as file: json.dump(data, file, indent=2) Use it: Python if __name__ == "__main__": nlp = create_pipeline() entities = extract_entities(nlp, documents) triples = extract_relationships(nlp, documents) write_json("entities.json", entities) write_json("triples.json", triples) print(f"Wrote {len(entities)} entities to entities.json") print(f"Wrote {len(triples)} triples to triples.json") Complete Example Here is the full script: Python import json import spacy documents = [ "Checkout Service depends on Payment API during payment authorization.", "Payment API stores transaction metadata in PostgreSQL.", "Platform Team owns Payment API and manages its deployment pipeline.", "Recommendation Service calls Catalog API to retrieve product details.", "Catalog API indexes product data in Elasticsearch.", "Search Team owns Catalog API.", ] ALIASES = { "payment-api": "Payment API", "Payments API": "Payment API", "payment service": "Payment API", } RELATION_VERBS = { "depends on": "DEPENDS_ON", "stores": "STORES_IN", "owns": "OWNS", "calls": "CALLS", "indexes": "INDEXES_IN", } def create_pipeline(): nlp = spacy.load("en_core_web_sm") ruler = nlp.add_pipe("entity_ruler", before="ner") patterns = [ {"label": "SERVICE", "pattern": "Checkout Service"}, {"label": "SERVICE", "pattern": "Recommendation Service"}, {"label": "API", "pattern": "Payment API"}, {"label": "API", "pattern": "Catalog API"}, {"label": "DATABASE", "pattern": "PostgreSQL"}, {"label": "SEARCH_INDEX", "pattern": "Elasticsearch"}, {"label": "TEAM", "pattern": "Platform Team"}, {"label": "TEAM", "pattern": "Search Team"}, ] ruler.add_patterns(patterns) return nlp def normalize_entity_name(name): return ALIASES.get(name, name) def extract_entities(nlp, documents): results = [] for text in documents: doc = nlp(text) for entity in doc.ents: results.append( { "text": normalize_entity_name(entity.text), "label": entity.label_, "source_text": text, } ) return results def extract_relationships(nlp, documents): triples = [] for text in documents: doc = nlp(text) entities = list(doc.ents) if len(entities) < 2: continue for verb_phrase, relation in RELATION_VERBS.items(): if verb_phrase in text.lower(): subject = normalize_entity_name(entities[0].text) object_ = normalize_entity_name(entities[1].text) triples.append( { "subject": subject, "relation": relation, "object": object_, "source_text": text, } ) return triples def write_json(path, data): with open(path, "w", encoding="utf-8") as file: json.dump(data, file, indent=2) if __name__ == "__main__": nlp = create_pipeline() entities = extract_entities(nlp, documents) triples = extract_relationships(nlp, documents) write_json("entities.json", entities) write_json("triples.json", triples) print(f"Wrote {len(entities)} entities to entities.json") print(f"Wrote {len(triples)} triples to triples.json") Run it: Shell python extract_entities.py You should now see two output files: Plain Text entities.json triples.json Why This Matters to GraphRAG Systems GraphRAG systems rely on accurate entity and relationship information in order to perform well. Poor extraction means poor graph retrieval. The Large Language Model (LLM), while capable of reasoning about the context provided, cannot correct the underlying data problems. Therefore, entity extraction, entity normalization, relationship extraction, and source tracking are to be considered first-class engineering issues. Production Considerations While this tutorial uses a small set of rules to extract entities and relationships, larger-scale production systems require additional robustness through "guardrails". Use a service catalog when possible. If your organization has an existing internal listing of services, teams, repositories, and owners, then these items should be used as the sources of truth for extracting patterned entities. Track confidence. Do not treat all extracted relationships with equal weight. Inferences from unstructured documentation, such as a free-form document, likely have less confidence than those from structured documentation, such as a service catalog. Keep the source sentence. Each triple includes the original sentence text. This makes reviewing, debugging, and referencing answers much simpler. Review low-confidence edges. Graph errors grow rapidly. An error in one dependency can cause downstream traversal results to be misleading. Start simple before using an LLM. While LLMs can be helpful for extracting entities/relationships from unstructured data, they increase cost, latency, and variability. Rules + Domain Dictionary may be sufficient for predictable documentation like engineering documentation. Key Takeaways SpaCy is effective in both general NLP and domain-specific extraction of entities via custom entity rules. Domain documents like engineering documentation will have custom entity names that include SERVICE, API, DATABASE, TEAM, etc. The relationship triples represent an efficient path from the unstructured document to the knowledge graph. Entity Normalization is NOT optional. If you do not perform Entity Normalization on your extracted entities, then the single "Real-world" system will be represented by multiple isolated/disconnected nodes in your graph. In terms of performance with GraphRAG systems, quality of extraction is equal in importance as retrieval and prompting. Try It Yourself Add the following documents: Plain Text Billing Worker publishes events to Kafka. Data Platform Team owns Kafka. Then add the following to the entity patterns: Plain Text Kafka Billing Worker Data Platform Team Finally, add a new relationship type: Plain Text PUBLISHES_TO Your goal is to produce the following triple: Plain Text Billing Worker --PUBLISHES_TO--> Kafka This small exercise demonstrates the work that is needed to adapt entity and relationship extraction to your own engineering domain.

By Sriharsha Makineni