DZone Spotlight

This article outlines how Delta Lake tables managed in Databricks and those managed in Snowflake can be shared in both directions without duplicating data. This is achieved through the use of shared object storage (ADLS) and open table formats. The article discusses the benefits, tradeoffs, business and cost advantages, the approach to implementation, and main technical aspects to consider. Overview Interoperability enables one platform to access data that is owned by another platform, without moving the data.Both Databricks and Snowflake store a single copy of the underlying Parquet files in ADLS, and they share only table metadata through catalog APIs. Databricks stores data in Delta (or Delta Uniform), which Snowflake can read via catalog integration.On the other hand, Snowflake-managed Iceberg tables can be accessed by Databricks through catalog federation. Metadata is the Key to Interoperability Key Advantages No data duplication: One physical copy of data in ADLS serves both engines, eliminating duplicate storage and copy pipelines.Meet users where they are: Producers publish once; consumers query from Databricks or Snowflake with their own compute.Best-of-breed & open formats: Open Parquet + Delta/Iceberg formats avoid lock-in and support future OLTP/OLAP convergence.Fresher data: Consumers read the latest committed metadata rather than a stale nightly extract. Pros and Cons Points to consider Pros Cons / Limitations No storage duplication — single copy in ADLS. Cross-engine writes are limited; the consuming engine is typically read-only. Lower latency than nightly copy / ETL jobs. Metadata refresh must be configured; a lag exists between write and visibility. Uses each engine's native compute and cost model. Converting Snowflake tables to Iceberg can surface datatype mismatches. Open formats reduce vendor lock-in. Governance must be re-applied per engine—policies do not travel with the table. Business and Cost Benefits Lower cost: Removes redundant storage and the compute cost of copy/sync pipelines between the two platforms.Speed: Lower data and process latency means data products reach consumers sooner (faster GTM).Compliance: A single governed copy supports data minimization and retention obligations.Agility: Teams pick the right tool for each workload without re-platforming the underlying data. Databricks to Snowflake: Implementation Approach Databricks owns the Delta/Uniform table, while Snowflake retrieves metadata through the Catalog Integration API and reads the data directly from ADLS. Catalog Integration Flow Snowflake requests metadata from Unity Catalog, receives the table location, and reads Parquet data directly from ADLS using an external volume. Databricks-Managed Delta Read by Snowflake (Catalog Integration) Databricks remains the owner; Snowflake reads in place. 1. Create external Delta tables with UniForm (Iceberg) enabled in ADLS, registered in Unity Catalog. SQL CREATE TABLE Catalog.SchemaName.TableName ( ID INT, Name STRING, CreatedAt TIMESTAMP ) TBLPROPERTIES ( 'delta.columnMapping.mode'='name', 'delta.enableIcebergCompatV2'='true', 'delta.universalFormat.enabledFormats'='iceberg' ); 2. Create an external volume in Snowflake pointing to the ADLS path (Managed Identity authentication). SQL CREATE OR REPLACE EXTERNAL VOLUME DEVTEAMADLS_EXE_VOL STORAGE_LOCATIONS = ( ( NAME='DEVTEAMADLS_EXE_VOL', STORAGE_PROVIDER='AZURE', STORAGE_BASE_URL='azure://devteamadls.blob.core.windows.net/Data/', AZURE_TENANT_ID='<tenant id>' ) ); 3. Create a catalog integration so Snowflake reads current metadata from Unity Catalog's Iceberg REST endpoint. SQL CREATE OR REPLACE CATALOG INTEGRATION DEVADLS_CATALOG_REST CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'SchemaName' REST_CONFIG = ( CATALOG_URI = 'https://<workspace>.azuredatabricks.net/api/2.1/unity-catalog/iceberg', WAREHOUSE = 'DevDeltaLake_Warehouse' ) REST_AUTHENTICATION = ( TYPE = OAUTH, OAUTH_TOKEN_URI = 'https://<workspace>.azuredatabricks.net/oidc/v1/token', OAUTH_CLIENT_ID = '<shared via secure exchange>', OAUTH_CLIENT_SECRET = '<shared via secure exchange>', OAUTH_ALLOWED_SCOPES = ('all-apis','sql') ) ENABLED = TRUE; 4. Create the Iceberg table in Snowflake and apply RBAC + masking on the Snowflake side. SQL CREATE ICEBERG TABLE SFDBName.SFSchema.SFTableName EXTERNAL_VOLUME = 'DEVTEAMADLS_EXE_VOL' CATALOG = 'DEVADLS_CATALOG_REST' CATALOG_TABLE_NAME = 'DeltaTableName' CATALOG_NAMESPACE = 'DeltaSchemaName' AUTO_REFRESH = TRUE; ALTER ICEBERG TABLE ... MODIFY COLUMN <col> SET MASKING POLICY <policy>; Snowflake to Databricks: Implementation Approach Snowflake remains the owner; Databricks reads in place. Query federation is the lighter alternative when Iceberg conversion is not desired. Snowflake owns the Iceberg table; Databricks federates the catalog and reads data in place from ADLS. Catalog Federation Flow Databricks federates the Snowflake catalog to retrieve the latest metadata, receives the updated table location, and reads Parquet data directly from ADLS through an external location using Managed Identity. Snowflake-Managed Data Read by Databricks (Catalog Federation) 1. Create Iceberg tables in the Snowflake-managed catalog on an external volume path. SQL CREATE OR REPLACE ICEBERG TABLE WareHouse.Schema.TableName ( c_custkey INTEGER, c_name STRING, c_acctbal INTEGER ) CATALOG = 'SNOWFLAKE' EXTERNAL_VOLUME = 'my_ext_vol' BASE_LOCATION = 'schema/tablename'; 2. Provision a resource account (key pair) and grant Databricks' role read access to the Iceberg tables. SQL ALTER USER UserName SET RSA_PUBLIC_KEY='<public key>'; GRANT USAGE ON DATABASE DatabseName TO ROLE MY_DATABRICKS_ROLE; GRANT USAGE ON SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON ALL TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON FUTURE TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; 3. In Unity Catalog, create the Snowflake connection, an external location for the ADLS path, and a foreign catalog with the authorized path. SQL CREATE CONNECTION SnowFlakeConn TYPE snowflake OPTIONS ( host '<acct>.snowflakecomputing.com', port '443', sfWarehouse '<wh>', user 'UserName', <key-pair auth options> ); CREATE EXTERNAL LOCATION snf_iceberg_loc URL 'abfss://<container>@<storage>.dfs.core.windows.net/' WITH (STORAGE CREDENTIAL <cred>); CREATE FOREIGN CATALOG SnowFlakeCatalog USING CONNECTION SnowFlakeConn OPTIONS ( database '<snf_db>', authorized_paths 'abfss://<container>@<storage>.dfs.core.windows.net/' ); 4. Refresh and query. Databricks reads Iceberg data directly from ADLS using its own compute. Apply RBAC and masking policies in Databricks. SQL REFRESH FOREIGN CATALOG SnowFlakeCatalog; SELECT * FROM SnowFlakeCatalog.SchemaName.TableName; Conclusion Through the use of interoperability, both Databricks and Snowflake will be able to share data bidirectionally through a single version of the data that is present in ADLS, but with metadata exchange. This will lead to no duplication of data and will always provide access to fresh data. There are several advantages for businesses in terms of cost savings through storage and ETL cost reduction, faster access to data, increased flexibility, and vendor lock-in reduction.

I moved a project from Java to Kotlin in a single pass, back when the codebase was small enough that doing so cost almost nothing. The system I work on now is a different animal: hundreds of thousands of lines of Java, a monolith rather than a set of services. Converting it wholesale is the kind of gamble nobody signs off on. The option still exists technically, but the cost is now measured in quarters of engineering time rather than days. How much code existed when someone first asked about Kotlin decided everything that followed. I have spent more than a decade building and scaling high-load distributed systems on the JVM, across fintech, security, and media infrastructure, and the pattern holds: you pay for a language migration at the start, well before anyone puts it on a roadmap. The Easy Migration The small project was a handful of Spring Boot microservices with JPA and PostgreSQL, still short of production at a few thousand lines of code. A colleague who spent his evenings in the Kotlin community suggested we try it on this one. The team was small and curious, so we agreed within a day or two. Spring compatibility was the thing we worried about, and it never became a problem. Because nothing had shipped yet, we converted the entire codebase at once. IntelliJ IDEA can rewrite a Java file to Kotlin with a single keystroke, and the converter handles about 80 percent of the conversion on its own. The rest of the week went into manual cleanup. Adding nullability annotations stripped out the platform types, and the compiler immediately flagged a dozen possible null-pointer exceptions that Java had been hiding inside Optional. We moved from field injection to constructor injection, which fits Kotlin far better. Data classes replaced our POJOs and trimmed about a quarter of the boilerplate, making the model easier to read. Static members needed more attention. After conversion, our public static final constants ended up inside a companion object, which changed the bytecode that Java callers expected, so we annotated them with @JvmStatic and @JvmField to keep the modules we hadn't touched yet compiling. By the end of the week, the tests were green, and Spring Boot loaded the new classes without complaint. The whole exercise took one week because the codebase was small, which is the entire point: the same cleanup across a large system is measured in quarters, not days. The Parser That Sold Me The message parser is what convinced me the move was worth defending to the rest of the team. Our services received data, validated it against an Avro schema, and returned a result. Parsing under load burns resources, and many Java libraries rely on runtime reflection during serialization and deserialization. Jackson can match the performance without reflection, to be fair: its Afterburner and Blackbird modules generate bytecode at runtime for field and method access, which is a real optimization. The cost shows up per format. JSON, XML, YAML, TOML, and Avro protocols each want their own modules and manual wiring, and a new format usually means finding a library or writing one. Kotlinx.serialization works differently. Its compiler plugin generates every serialization routine at build time and never calls reflection. We standardized most formats around Kotlinx.serialization, using it as the common model layer for JSON, XML, YAML, and TOML, then bolted on Avro as a separate plugin module. The schemas were plain Kotlin classes with the @Serializable annotation, and code generation handled the rest. We had to implement Avro's default values in the plugin by hand, but only once. A smaller dependency surface, typing that held end to end, one approach instead of five. That early switch also bought time to question the rest of the stack while it stayed light: jOOQ for queries that grew complicated, smaller web frameworks for lightweight services, and test tooling with more expressive assertions. Experiments are cheap on a compact codebase, and you can settle choices that would turn into prolonged arguments once the system has weight and a dozen people depending on it. The Monolith, and What to Do About Yours The same migration on the monolith I work on now is a different proposition, and I would push back on anyone who claimed it could happen in one pass. The window there has thinned to a slot rather than shut. Java and Kotlin still compile side by side in a single project, leaving an incremental approach open. You can write new tests in Kotlin against the existing Java, peel bounded contexts off into separate Kotlin services, and drop the idea that migration has to happen all at once. None of that demands the big-bang rewrite that scares management off, and each piece keeps the window from closing further. Postponing the conversation makes it more expensive every quarter, which is the cost the easy migration above never had to pay. If your codebase is still small, bring it up now, while the call is for engineers, not a budget committee. The migration window rarely closes dramatically. One quarter, it simply stops being practical. It just gets heavier to lift, and in one planning session, you realize the topic stopped coming up a while ago.

If you are a data scientist, data analyst, or anyone who works with big data, you have likely used SQLite, or even use it daily. Introduced in 2000, SQLite quickly became a popular choice for database-related tasks. Whether you used it in an introductory database module, built a local data pipeline, or managed local metadata in production, SQLite has long been the gold standard for a zero-configuration database, with more than 25 years of widespread use. But in recent years, DuckDB has emerged as a more powerful alternative, designed specifically for analytical workloads. For tasks involving Data analysis, analytics, and ETL (Extract, Transform, and Load), DuckDB is quietly gaining traction. The developer community today is increasingly migrating to DuckDB, especially when running analytical aggregations across gigabytes of local Parquet or CSV files, where it shows its speed advantage. So the question that arises is: Should DuckDB Replace SQLite? In this article, we will explore the fundamental design differences between the two libraries and determine whether DuckDB should replace SQLite. The Evolution of SQLite and DuckDB Evolutionary timeline comparing the hardware eras, workload requirements, and engine architectures of SQLite and DuckDB from 2000 to the present. (Image by the Author) SQLite was created by American software engineer D. Richard Hipp, whose original motivation for developing the library stemmed from his work with U.S. Navy destroyers. While working on Navy software, he wanted a database that could work reliably without a complex client-server system. As a result, SQLite was designed with the Navy’s specific requirements in mind. It was easy to use, with zero administrative overhead, and had excellent fault tolerance by eliminating the client-server architecture, providing continuous uptime. However, as datasets scaled to millions of records and multicore CPUs became the norm, some of SQLite’s original design choices became its core bottlenecks. Modern CPUs have multiple cores for better speed, but SQLite processes data row by row on a single thread. Moreover, it requires users to manually import files before querying them, which can result in slow import and export processes. Then came DuckDB in 2019, designed for fast analytical processing on local data. Although it was not intended to be a competitor or direct replacement for SQLite, it effectively solved the problem of single-core processing, prompting developers to use it more frequently for building data pipelines. The Developer Interface: Similar Syntax, Different Engines When it comes to the interface, there is no noticeable difference between the two libraries. If you are already familiar with SQL, then querying data with DuckDB requires virtually no learning curve compared to SQLite. SQLite Python import sqlite3 conn = sqlite3.connect("sales.db") cursor = conn.cursor() cursor.execute( "SELECT region, SUM(sales) FROM transactions WHERE sales > 1000 GROUP BY" " region" ) results = cursor.fetchall() DuckDB Python import duckdb # Querying local disk files directly with standard SQL results = duckdb.sql( "SELECT region, SUM(sales) FROM 'sales.csv' WHERE sales > 1000 GROUP BY" " region" ).fetchall() They may look similar at first glance, but the differences become clear when you look deeper into their architectures. Architectural comparison between SQLite’s sequential row-store processing and DuckDB’s optimized columnar vector execution pipeline. (Image by Author) Query Engines Processing Across CPU Cores SQLite uses the Volcano Iterator Model, where a single core processes queries sequentially on a single CPU thread. If you sort 10 million rows, a single CPU thread handles the entire task while the rest of the CPU sits idle, increasing processing time. On the other hand, DuckDB uses morsel-driven parallelism, partitioning the dataset into smaller chunks and processing them across available CPU threads through independent task queues. A large operation is split into uniform data chunks called morsels. Idle worker threads grab any available morsels from the queue, keeping the CPU cores fully utilized until execution finishes. Deferred Execution and Logical Plan Optimization SQLite executes queries in an eager streaming fashion where the rows are read immediately from disk or B-Trees and pushed through the expression evaluator step without global transformations. DuckDB enforces a deferred pipeline with an explicit optimization pass before the data is even touched. [ Raw SQL Query ] →[ Unoptimized Logical Plan ] → [Optimizer Pass ] → [ Physical Execution Plan ] DuckDB optimizes the query to eliminate non-essential work. During optimization, unused columns are dropped entirely, and conditions like WHERE sales > 1000 are pushed to the file reader level, skipping non-matching data blocks. Additionally, various query operations are arranged dynamically based on cardinalities to reduce intermediate memory allocation. Memory Efficiency and Data Storage Structures At last, we will look at the memory architecture, as the data transfer between RAM and CPU cache lines is frequently the primary bottleneck in data applications. Both libraries use fundamentally different data storage structures. SQLite uses a row-oriented storage layout, where it stores every feature of a record in a single row. Combined with an extremely minimal baseline footprint of just 2–5 MB, this design is optimal for retrieving an individual row (eg: loading a single user’s profile), but not for running an aggregation like AVG(sales), as it forces SQLite to load every unused attribute into cache along with the target values. DuckDB, on the other hand, uses a columnar format. Each feature is stored in a separate column, with the column data stored as a contiguous block in memory. This approach is preferred for analytical queries because DuckDB reads only the specific columns, maximizing CPU cache efficiency. The columnar structure also enables zero-copy data sharing with tools such as Apache Arrow, PyTorch, and Polars without expensive memory serialization steps. To maintain its multi-threaded vector channels, DuckDB operates with a higher baseline memory footprint of 20–30 MB. Final Thoughts: Should DuckDB Replace SQLite? After comparing the architectures of the two libraries, DuckDB may appear to be the better alternative. So, coming to our final question: Should DuckDB replace SQLite? The answer is: No, not quite. After looking at the architecture of the two database engines, the core trade-offs become clear: SQLite uses a row-oriented layout and a single-threaded engine to maintain a minimal memory footprint of just 2–5 MB. It is optimized for transactional (OLTP) operations, securing its position as a universal component across mobile and desktop devices. Conversely, DuckDB’s columnar architecture with multi-core parallel processing makes it a great choice for analytical (OLAP) queries and scanning external .csv, .parquet, or .json files directly without any manual imports. It turns out to be a better option for modern data science and AI pipelines. SQLite was engineered at a time when embedded storage was expected to handle lightweight applications and reliable local commits. In contrast, DuckDB was built to function as an analytics engine for large datasets. So which library you use for what task depends entirely on your use case. Choose SQLite if you need a zero-maintenance, embedded database to store app configuration or handle fast single-row lookups with minimal RAM overhead. Choose DuckDB if you are building a professional data or AI pipeline, running aggregations (GROUP BY, SUM, AVG) across millions of records, extracting features from .csv files, or processing local data in Python notebooks without a complex setup. Overall, it is better to see both as libraries designed for different purposes rather than as competitors. If you like the article, connect with me on https://medium.com/@rakshathnaik62 for more articles. I mainly write about Science, math, and AI.

Imagine asking an AI assistant why a production API became slow after a deployment. One agent examines metrics. Another searches logs. A third inspects deployment history. A coordinator combines their findings and proposes the next step. The design looks straightforward until the logs service times out, two agents repeat the same expensive query, and the coordinator interprets a missing result as evidence that nothing went wrong. Components exchanging messages. Dependencies with variable latency. Shared resources. Failures that affect only part of a request. We have seen all of this before. Multi-agent AI systems inherit distributed systems problems wherever their execution crosses a network boundary. They also add variability in how tasks are interpreted, tools are selected, and conclusions are formed, so correctness testing must account for that variability alongside familiar failures. Microservices give us a vocabulary for reasoning about both, and for the costs of the decomposition that creates them. A Role Is Not a Boundary An agent describes behavior: it receives a task, uses a model and available tools, and may choose additional steps based on intermediate results. A microservice describes an architectural boundary around a capability, including its interface and independent deployment. Microsoft's microservices architecture guidance makes independent deployment a central property. Several agents can run inside one process and use the same underlying model. An agent's role does not automatically give it a separate failure domain, scaling boundary, or security boundary. Those properties come from the implementation, not from the job description in the prompt. The Distributed Monolith Test Service architecture already has a name for what happens when that distinction is ignored. An agent system becomes a distributed monolith when its components communicate over a network but remain tightly coupled in their contracts, state, and release process. The test is coupling, not shared infrastructure — independently deployable services routinely share a database cluster or a rate limit without ceasing to be independent. Three questions settle it: Does changing one agent force a coordinated release of the others?Do they read and write the same internal state?Can any of them be tested in isolation? Splitting a prompt into five roles can create coordination overhead long before it creates any of that independence. The Case for Splitting Anyway There is a real argument for decomposition that has nothing to do with deployment: context isolation. An agent investigating an incident can accumulate everything it reads. Two hundred kilobytes of log output ends up in the same context as the metric summary and the deployment diff, competing for attention and inviting the model to connect things that are adjacent only because they are adjacent in the prompt. Give the log search its own context, and it returns a bounded summary, and the coordinator reasons over three clean summaries rather than one crowded transcript. The choice is not between one bloated context and several clean ones. A single agent can use retrieval, filtering, compaction, and external state to manage the same problem; Anthropic's context-engineering guidance discusses those approaches alongside subagents. Isolation has its own cost. A summary can omit exactly the detail a cross-domain check needs — the log line whose timestamp would have connected a pattern to a deployment change is the kind of thing a summary drops. The coordinator needs a path back to the underlying evidence, which is what the evidence references below are for. Separate contexts and focused evaluation can justify splitting when their benefits outweigh coordination costs. A different role description, on its own, cannot. Mapping the Concerns When agents run as separate services, the architectural mapping becomes concrete. Microservices ConcernQuestion for the Agent SystemService boundariesWhat responsibility does each agent own?API contractsWhat task can it accept, and what constitutes a usable result?State ownershipWho can update the investigation's authoritative state?Failure isolationCan useful work continue if this agent or its dependencies fail?Capacity managementWhich model, tool, or data service limits throughput?ObservabilityCan we connect the final answer to the operations and evidence behind it? For the incident assistant, I would begin with three bounded capabilities: analyze metric changes, identify relevant log patterns, and reconstruct the deployment timeline. Each has a defined data source and an output the coordinator can inspect. If they need separate release cycles, access policies, or resource limits, service boundaries help — with versioned contracts and workflow evaluations to catch incompatible changes. Routine operations such as fetching deployment records or calculating percentiles stay in ordinary code. An agent earns its place where the workflow needs interpretation: choosing which metric to investigate next, deciding whether a log pattern deserves attention, identifying a gap in the available evidence. Contracts: What a Task Is, and What a Result Means A task such as "find the problem" leaves too much implicit. A useful request carries the incident identifier, service, region, time window, available evidence, budget, and permitted tools. JSON { "task_id": "inv-4471-logs", "incident_id": "INC-4471", "service": "checkout-api", "region": "us-east-1", "window": { "start": "2026-09-14T18:40:00Z", "end": "2026-09-14T19:10:00Z" }, "budget_remaining_ms": 8000, "allowed_tools": ["logs.search", "logs.aggregate"], "prior_evidence": ["ev-4471-metrics-01"] } Two fields need their semantics stated rather than assumed. Budget Propagation budget_remaining_ms is the time this task may still consume. The engineering requirement is that the remaining budget survives queueing, execution, and downstream calls — not that it takes any particular representation. gRPC's deadline guidance propagates a remaining timeout by subtracting elapsed time at each hop, which handles queueing without depending on synchronized clocks; an absolute timestamp propagates unchanged but introduces clock skew between services. Either works if elapsed time is actually deducted. Expiry is not simply "return what you have," because the caller may already have stopped waiting. Give workers a cutoff earlier than the caller's, so partial findings arrive with time left for synthesis, and have the runtime request cancellation of outstanding work when the budget runs out. Declared Tools Are Not Enforced Tools allowed_tools declares which tools this task may use. It does not enforce anything. The model can request any tool its prompt suggests, so authorization has to be enforced in the runtime or the receiving service. A context boundary is not an authorization boundary — the same distinction as the one between a role and a service boundary, appearing in a second place. The Result Contract JSON { "task_id": "inv-4471-logs", "status": "incomplete", "coverage": { "searched": [{ "start": "2026-09-14T18:40:00Z", "end": "2026-09-14T18:52:00Z" }], "missing": [{ "start": "2026-09-14T18:52:00Z", "end": "2026-09-14T19:10:00Z" }] }, "findings": [ { "claim": "upstream timeout rate rose from 0.2% to 4.1%", "evidence": ["ev-4471-logs-07"], "supports": "correlation_only" } ], "open_questions": ["did the elevated timeout rate continue past 18:52Z?"] } The coverage block is the part that matters. A logs agent that could not reach the second half of the window returns incomplete with the gap named, and that stays distinguishable from complete with an empty findings array. Collapse those two states and the coordinator converts absence of evidence into evidence of absence, which can hide the missing evidence from the final answer. Schema validation is one layer of the contract. The coordinator also has to check whether the evidence supports the finding. A syntactically valid response can reference the wrong region, compare incompatible time windows, or claim that a deployment caused a regression merely because both happened close together. Traditional services also produce incorrect results: a payment service can return successfully while calculating the wrong amount. The difference here is variability. The same input can yield a valid response that is wrong in a different way on each run, because interpretation and tool selection vary between executions. Testing has to cover interface compatibility and task outcomes, and the outcome tests need enough repetitions to characterize that variance rather than sample it once. Latency: Measure the Whole Path Consider one illustrative execution. Planning takes two seconds. The metrics, logs, and deployment investigations take four, six, and ten seconds respectively. Synthesis takes another three. SchedulingElapsed TimeCompositionSequential25 s2 + 4 + 6 + 10 + 3Parallel branches15 s2 (plan) + 10 (slowest branch) + 3 (synthesis) Those numbers assume unchanged branch durations and no additional overhead or contention. For the fixed execution assumed here, parallel scheduling changes elapsed time while leaving the underlying calls unchanged — same prompts, same tools, same outputs, same token and quota consumption. Adaptive workflows change both the amount and the timing of work: a sequential run can use the metrics result to skip the deployment query entirely, and parallel branches can duplicate a query neither knew the other was issuing. Which arrangement costs less is an empirical question about a particular workflow, not a property of the schedule. Synthesis grows with branch count either way. More branches means more material to reconcile before an answer exists. Tails and Shared Bottlenecks Dean and Barroso's The Tail at Scale describes the classic problem: when a request fans out, its latency is governed by the slowest branch, so responses that are rarely slow become requests that are commonly slow. With three independent branches each having a 1% chance of exceeding a threshold, the request exceeds it about 3% of the time. If their exceedances always occur together, it is 1%. These examples show why the dependence structure matters; neither alone predicts the production distribution. What matters operationally is the shared bottleneck underneath. Agent branches commonly share a model endpoint, a rate limit, and an observability backend, and when that endpoint queues, each branch's individual probability of being slow rises. Hedging against the same saturated dependency can become ineffective or counterproductive, since its value depends on routing, spare capacity, and load controls that saturation removes. So the quantity worth instrumenting is the distribution of complete task times. Per-agent latency can reveal the slowdown; dependency metrics and traces help identify its shared cause. Waiting Policy and Capacity The coordinator needs an explicit waiting policy. Which findings are required? Which are optional? When should it return a partial diagnosis? How much time should remain for synthesis and validation? For our assistant, missing logs might still permit a useful report about a metric change, provided the report names the gap. A definitive causal diagnosis requires more. The completion rule should encode that difference in the runtime rather than leaving it to whatever the coordinator's prompt decides on a given run. Capacity needs the same treatment, for a related reason: three independently deployed agents can still contend for one model quota and one observability backend, and adding replicas does little for throughput when the shared dependency is the constraint. Enforce concurrency limits and queue bounds in the runtime, allocate a budget for model and tool calls, and reserve time for the final response. Prompts can guide how an agent spends its allowance; the runtime has to enforce the allowance. Otherwise, an assistant investigating an overloaded service becomes another source of load on the systems needed to diagnose it. Retries and Idempotency A timed-out request leaves uncertainty about what happened downstream. Suppose the assistant eventually gains permission to create an incident ticket. The ticket service creates the ticket, but its response is lost. A retry creates a duplicate unless the operation has idempotency handling. AWS describes using caller-provided request identifiers to recognize repeated intent in Making retries safe with idempotent APIs. For this design, I would persist an operation identifier and the intended action before dispatch, and repeating that action preserves the identifier. The receiving service has to enforce the duplicate-handling contract; putting an identifier in a prompt guarantees nothing. A retry of an action also differs from asking the agent to reconsider the task. Reconsideration may produce a different action with different parameters. Keeping the two distinct makes recovery tractable. Durable State Long investigations need durable state for similar reasons. The coordinator should record completed tasks, pending operations, evidence references, and remaining budgets, so that after a restart it can resume recorded progress and reconcile uncertain side effects. A checkpoint alone does not guarantee that an external action happened exactly once. In our example, the coordinator owns investigation status and stores each agent's findings as separate evidence records. Every record identifies the service, region, observation window, and retrieval time. Where the underlying systems support snapshots or versions, those references are retained too. This makes conflicting findings inspectable. Two agents may appear to disagree simply because one queried data before a recovery and the other queried afterward. Prompt context then serves as a working view of durable task state. Large logs and artifacts stay in storage, with bounded excerpts and references supplied to the agents that need them. There is production experience behind these concerns. In its account of building a multi-agent research system, Anthropic describes expensive restarts, the need for tracing, and bottlenecks caused by waiting for subagents. Those observations put recovery and coordination alongside model behavior rather than beneath it. Validating the Outcome A completed workflow still needs an outcome check. Suppose all three agents report that a deployment caused the slowdown. That agreement deserves examination rather than confidence. They may have used the same timeline, inherited the same assumption from the coordinator, or treated correlation as causation. Consensus among agents drawing on shared context is not independent confirmation. Validation should connect each claim to evidence and identify alternative explanations. A reported database slowdown points to the relevant measurements. A claim about a deployment explains what changed and what observation would strengthen or weaken the proposed connection. Where the evidence supports only a hypothesis, the output preserves that uncertainty and proposes the next discriminating check. Giving a confident answer should never be the workflow's only definition of success. Observability and What to Measure The tracing has to support that evaluation. Trace from the incoming request through delegation, model calls, tool execution, and synthesis, capturing task relationships, timing, retries, token usage, model and prompt versions, and references to the evidence used. Sensitive content needs access controls and retention limits. That lets us interrogate a poor result with concrete questions. Was the evidence unavailable? Did a tool query the wrong interval? Did the coordinator ignore an incomplete status? Did one slow dependency consume the budget reserved for verification? MeasureWhat It Helps EstablishTask success rateWhether the investigation meets a defined evidence and correctness rubric, not merely whether it returnedP95 and P99 completion timeHow long completed tasks take, including coordination and validation; report timeouts separatelyBudget complianceWhether execution stays within its operational limitsDuplicate work and retry volumeWhere coordination wastes resources or amplifies failuresCost per successful taskTotal model, tool, and execution spend, including failed attempts, divided by tasks meeting the success criteria That last denominator matters. Cheap failures should not make an architecture look efficient. Compare Against the Boring Baseline Evaluate the multi-agent design against a single agent with tools and, where practical, a conventional workflow, giving all three the same task set and evidence access. Comparisons either hold resource budgets constant or report the quality, latency, and cost tradeoffs explicitly; giving one architecture substantially more inference compute changes what the experiment demonstrates. Microsoft's agent orchestration guidance recommends starting with the lowest complexity that reliably meets the requirement, which is a good default here. Conclusion The distributed systems problems in a multi-agent architecture are not new, and the microservices vocabulary describes most of them accurately: boundaries, contracts, state ownership, failure isolation, capacity, and observability. What agent systems add is variability in interpretation, which means correctness has to be tested as an outcome rather than assumed from a passing schema check. Start with one path through the problem. Split out a capability when separate context, permissions, ownership, or parallel execution offers a measurable advantage. Then test slow dependencies, incomplete evidence, duplicate deliveries, and recovery from interruption before increasing the number of agents. For the next agent proposed in the design review, ask what it improves, what it costs, and how the system behaves when it fails. Every agent should earn its place in the request path. Key Takeaways An agent's role does not give it a failure domain, scaling boundary, or security boundary. Those properties come from the implementation, not from the job description in the prompt.An agent system becomes a distributed monolith when components communicate over a network but stay coupled in contracts, state, and release process. The test is coupling, not shared infrastructure.Task contracts need an explicit coverage field. "Searched and found nothing" and "could not search" must be distinguishable, or the coordinator will convert absence of evidence into evidence of absence.Parallel execution changes elapsed time, not the work performed, for a fixed execution. Adaptive workflows change both.Tool authorization has to be enforced at runtime or by the receiving service. Declaring permitted tools in a request documents intent; it does not enforce anything.

Production inference is fundamentally a scheduling problem. Once you've addressed serving architecture and memory, the next question isn’t whether the GPU can do the work; it's how to schedule it. That decision begins with understanding the workload: how requests arrive, how long they run, which are latency-sensitive, and which can tolerate delay. This is where many production deployments encounter their next engineering challenge. Serving architecture establishes how an inference system is organized. Memory management determines how efficiently it uses GPU memory. Batching answers a different question: How do you keep expensive GPUs doing useful work while still meeting latency service-level objectives (SLOs)? Production workloads introduce competing objectives that don't exist during experimentation. Production LLM systems serve requests that arrive unpredictably, vary dramatically in length, and compete for the same hardware resources. The engineering challenge isn’t batching requests together; it's deciding how work should be scheduled as those requests enter and leave the system. Production Traffic Is Inherently Uneven Most proof-of-concept environments present an idealized view of inference. Requests arrive at predictable intervals. Prompt lengths are similar. Response lengths are relatively consistent. Production looks nothing like that. Prompt lengths, output lengths, arrival patterns, and latency expectations differ. But production schedulers have to account for all of them simultaneously. That heterogeneity is what transforms batching from a simple grouping exercise into a scheduling problem. One user may ask a simple question that requires only a few tokens. Another may submit a lengthy document that requires a large context window and an extended response. Some requests are interactive and latency-sensitive. Others are background jobs that can tolerate additional delay. Treating all of those requests as if they were identical creates unnecessary inefficiencies. That is why batching strategies that work well during experimentation often become liabilities in production. Static Batching Breaks Under Real Workloads Static batching is straightforward: Collect a group of requests, execute them together, and wait until the batch finishes before beginning the next one. That approach works well for many traditional AI workloads because inputs generally require a similar amount of computation. Autoregressive generation behaves differently. Each request generates tokens at its own pace. Some complete almost immediately. Others continue generating for hundreds or even thousands of decoding iterations. The result is that the shortest requests spend much of their time waiting for the longest one to finish. This is the convoy effect. The GPU is effectively synchronized to the slowest request in the batch. The GPU is occupied, but completed or short-running requests cannot release their resources until the longest request in the batch finishes. As production workloads become more diverse, those idle cycles accumulate. Throughput falls, latency increases, and expensive GPU resources spend more time waiting than performing useful work. Bigger Batches Are Not Always Better The natural response is often to increase batch size. Larger batches generally improve GPU utilization, but they also introduce a tradeoff. Every request must wait until enough additional work arrives to fill the batch, increasing queueing delay before execution even begins. Once execution starts, larger batches can also increase Time Per Output Token (TPOT). A configuration that delivers acceptable TPOT at one batch size may exceed the application's latency SLO as the batch grows. Higher GPU utilization provides little benefit if response times no longer meet application requirements. This is the central balancing act of production inference. Higher throughput often comes at the cost of higher latency. The engineering challenge is determining where that operating point belongs. An internal document-processing pipeline may prioritize maximum throughput because latency is relatively unimportant. A customer-facing assistant may willingly sacrifice some throughput to preserve fast responses. Neither choice is universally correct. The workload determines the objective. The batching strategy should support it. Continuous Batching Changes the Scheduling Model Modern serving systems approach the problem differently. Rather than waiting for an entire batch to complete, continuous batching schedules work at each decode iteration. Finished requests leave immediately. New requests enter immediately. The scheduler is no longer managing batches. It is managing decoding iterations. Requests enter the batch as capacity becomes available. Completed requests leave immediately. New requests replace them without waiting for the entire batch to finish. This seemingly simple change fundamentally alters GPU utilization. Instead of waiting for the longest request to complete, the scheduler continuously replenishes the batch with new work. GPU utilization becomes constrained primarily by queue depth rather than by the slowest request in the system. The scheduler is no longer waiting for an entire batch to complete. It's continuously looking for the next unit of useful work. The objective shifts from building larger batches to keeping the GPU continuously productive. Long Prompts Introduce Another Challenge The prefill phase processes the entire prompt in a single forward pass, while the decode phase generates one token at a time. These phases place very different demands on the hardware. Prefill is compute-bound. Decode is memory-bandwidth-bound. Long prompts can monopolize GPU compute, delaying interactive requests that are already in the decode phase. Chunked prefill addresses this by breaking large prompts into smaller segments that can be interleaved with ongoing decode operations. Rather than allowing one long prefill operation to dominate the GPU, the resource scheduler alternates between prompt processing and token generation, allowing engineers to balance Time to First Token (TTFT) for new requests against Time Per Output Token (TPOT) for active ones. The result is a more balanced system. Time to First Token (TTFT) for new requests remains reasonable while Time Per Output Token (TPOT) for active requests stays within acceptable service-level objectives. Once again, the objective is balance — not maximization. Scheduling Becomes Policy As production systems mature, batching becomes less about grouping requests and more about deciding which work should execute next. Interactive applicationsBackground batch jobsHigh-priority usersLow-priority workloads Each may require different treatment. Some workloads don't require interactive latency at all. If a business process can tolerate scheduled execution, maximizing throughput may be the better engineering decision than minimizing response time. Modern serving systems increasingly make resource scheduling decisions at the token level rather than at the request level. GPU time becomes a shared resource allocated according to policy instead of simple arrival order. That policy also extends to admission control. Available GPU capacity is not the same thing as available latency headroom. Accepting every incoming request may maximize utilization for a short period, but it can also push latency beyond acceptable SLOs for every user in the system. Well-designed serving systems recognize that preserving predictable latency is often more valuable than accepting one additional request. In production, saying "not yet" is frequently better than saying "yes" to everyone. Throughput and Latency Are Negotiated — Not Maximized One theme runs consistently through every batching strategy. There is no universally optimal configuration. The appropriate batch size, scheduling policy, and admission thresholds all depend on the workload and the service-level objectives the application must satisfy. There is no universally optimal configuration because every production deployment is balancing a different set of constraints. A nightly batch pipeline should not be engineered like an interactive coding assistant. Likewise, a customer support chatbot should not inherit the scheduling strategy of an offline document-processing system simply because both happen to use the same foundation model. The model may remain unchanged. The serving behavior should not. The objective isn't maximum throughput. It's the highest sustainable throughput that still satisfies the application's latency SLOs. Looking Ahead Serving architecture organizes the inference system. Memory management ensures efficient GPU memory use. Batching strategies determine how available resources are converted into useful work. The final layer focuses on extracting even more performance from the same hardware. In the next article, we'll examine inference optimization techniques – including quantization, speculative decoding, FlashAttention, and distillation – and how carefully sequencing those optimizations can dramatically reduce cost and latency without changing the underlying model.

What constrains software development? Why can't teams and organizations just build every application they can dream of, and reap the benefits of ever-expanding productivity? There are technical obstacles, of course; the technology available may not be able to deliver the capabilities needed. As the breakneck construction of data centers illustrates, availability of compute cycles is also a limiting factor. There are bandwidth limits as well; the 'speeds and feeds' that define transaction and communication throughput. The real constraining factors, however, are the same in software development as in many other specialized disciplines: time, cost, and expertise. Software development takes time, requires expertise, and costs money. The salaries of skilled coders aren't trivial, and even when the firm can afford to pay for top coders and software architects, it must find and recruit them first, another expense. The infrastructure they need to develop, test, and deliver software, from cloud compute to data storage to transmission, is another major budget item. Stakeholders can visualize new applications all day long, but the inhibiting factors are powerful, putting a check on development efforts, even those with significant potential benefits. Every software effort is, in part, an experiment, as engineers try to synthesize a system that fits the stakeholders' vision. Even on a two-week sprint cycle, large projects can take months of development, with multiple checkpoints that require review and acceptance, often leading to significant blind alleys and dead ends. Triage of development ideas is done by default; firms can't afford to experiment endlessly with every brainstormed idea for a new application. How AI Changes the SDLC Or can they? Does the advent of AI-enabled software development flip the script on familiar constraints and call for new development lifecycles and approaches? When expertise is shared between human coders and their AI collaborators, when prototypes can be quickly and cheaply created en masse, does that alleviate the obstacles to abundant application development? Or does the bottleneck just move, from one set of constraints to another, constraints based not in technology or expense, but in human attention and judgment? One of the fundamental ideas of modern software development is the role of the Product Owner, explicitly designed to own the responsibility for articulating the product's requirements (as user stories in a backlog), and for guiding ongoing development by accepting, or withholding acceptance of, the features and functions as they are incrementally developed. This critical role takes on the most problematic activity of the entire development cycle: the communication of expectations from the ultimate users of the product to its designers and developers. As the conduit for transmitting intent and expectation in both directions, from the user to the developer and back, the Product Owner role requires fluency in both domains and both lexicons, as well as an appreciation for the concerns and constraints of each. Now let's consider the product owner role in our imagined AI-enabled development shop. The constraint of coding expertise has been alleviated; our AI-code collaborator can design and program a prototype from raw specs or prompts, based on the product owner's description and guided by a human coder. Some of the cost constraints have been reduced as well, as the firm changes its team mix to include less high-level coding talent and more AI collaborative co-working skills. As the next generation of data centers comes online, compute costs may also decline due to competitive pressures. New open-weight AI models are also driving inference costs down. In short, it's become almost free to generate multiple versions of a running prototype for the product owner to evaluate. When development of prototypes, complete enough to enable evaluation, is cost-free and nearly immediate, the constraints don't disappear; they migrate. The limiting constraint is now the attention of the product owner, the authorized evaluator of the abundant software generated from the AI-enabled software factory. In software development, the 'human-in-the-loop' is the primary keeper of the project's intent, the prime communicator and translator, and the decision-maker on the product's progress and success. Responsible Attention The term "human-in-the-loop" has become standard shorthand for keeping people involved in automated processes, and it implies a workflow: AI runs a process or prompt, returns a result, and a human with domain knowledge reviews it for accuracy or safety. In many cases, the human is there to assure compliance, and the intent is a checks-and-balances review cycle. The new product ownership role we're envisioning isn't a check-and-balance; it's the decision gate that determines whether the firm benefits from the product. Accepting or withholding acceptance of a feature is a judgment, and judgment requires active evaluation: comparing the increment against the product vision, weighing trade-offs the backlog doesn't capture, with the authority and domain expertise to say no when necessary. Call this "Responsible Attention": the exercise of business judgment by an evaluator with the standing, authority, and confidence to accept or reject the product presented. Responsible attention was never unlimited, even in the traditional SDLC. It was rarely the binding constraint because cost and development time limited the queue of things requiring focused judgment. A two-week sprint produced a predictable cadence, timed so that a Product Owner could give each decision real consideration. An AI-enabled development shop compresses that cadence. If prototypes arrive daily, or in batches of multiple variants at once, the Product Owner's responsibility remains the same, but their capacity to exercise it does not scale with the volume demanding it. Why Not Just Review Faster? The obvious answer to a compressed build cycle is a compressed review cycle. If prototypes now arrive daily, move the Product Owner's cadence to daily as well. The math seems to work: less time per feature, more features per week, but the review scales down proportionally, so the judgment-to-output ratio stays constant. That math is wrong. It assumes decision quality stays steady no matter how often the product is evaluated, but judgment degrades with repetition in ways review time doesn't capture. A Product Owner can technically sit through a daily review indefinitely; what degrades first isn't their calendar, it's the quality of judgment they bring to each decision. This is decision fatigue, and it rarely announces itself. Nobody says "I'm too tired to evaluate this properly." They just start saying yes faster. Volume creates a second failure mode, independent of fatigue. When prototypes are cheap enough to generate in multiples, more of them compete for the same fixed pool of attention. A polished, easy-to-evaluate variant will win attention over a harder, more consequential proposal simply because it's faster to judge, not because it matters more. Abundance doesn't just tax the evaluator; it can quietly reorder what gets evaluated seriously at all. The Factory's Real Constraint It is tempting to imagine an AI-enabled development shop as a factory: raw prompts in, finished software out, limited only by compute and imagination. But a factory implies throughput without a bottleneck, and this one has an unmovable one. Every increment the factory produces still has to pass through a human being who decides whether it deserves to exist in the product. That decision cannot be manufactured, parallelized, or accelerated the way the code beneath it can. Every dev shop chasing AI-enabled velocity should ask a harder question than "how fast can we build." Ask who is deciding what gets built, how many decisions they're being asked to make this week, and whether anyone is still checking the quality of those decisions. A factory that produces faster than its judgment can absorb isn't accelerating. It's accumulating risk at the speed of its own success.

Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket. Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal. The Gap The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer. The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review. I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live. Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched. Two Kinds of Drift When plan says a resource differs from state, there are two directions to go: DirectionMeaningActionRevertThe out-of-band change was wrong; code is the truthNo HCL change; apply the existing configCodifyThe out-of-band change was right; code is staleUpdate HCL to match reality, so apply produces no diff A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not. The Workflow Four stages. Only the second one is new. Stage 1: Detection A scheduled workflow runs a plan and captures a machine-readable output. YAML # .github/workflows/drift-detect.yml (abridged) on: schedule: [{ cron: "0 */6 * * *" }] jobs: plan: permissions: { id-token: write, contents: read, actions: write, issues: write } strategy: fail-fast: false matrix: include: # CloudTrail LookupEvents is regional; the agent runs where the resource lives - { stack: network/prod-use1, region: us-east-1 } - { stack: network/prod-use2, region: us-east-2 } steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.DRIFT_READONLY_ROLE } # reads + lock-table write; plan holds the lock aws-region: ${{ matrix.region } - name: Plan (refresh-only) id: plan run: | set +e terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \ -out=drift.tfplan -input=false code=$?; set -e echo "code=$code" >> "$GITHUB_OUTPUT" [ "$code" -eq 1 ] && exit 1 # plan error: fail loudly terraform show -json drift.tfplan > drift.json - name: Count drifted resources # refresh-only plans report under resource_drift id: count run: | n=$(python3 -c "import json; d=json.load(open('drift.json')); print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))") echo "n=$n" >> "$GITHUB_OUTPUT" - name: Drift-storm guard # >20 drifted resources is a provider/state problem, not drift if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20 run: gh issue create --title "[drift] storm in ${{ matrix.stack }" --body "Paging a human." - name: Dispatch remediation if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20 run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack } -f region=${{ matrix.region } -f run_id=${{ github.run_id } Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent. Stage 2: The Remediation Agent The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply. YAML # .github/workflows/drift-remediate.yml (abridged) on: workflow_dispatch: inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } } jobs: remediate: permissions: { contents: write, pull-requests: write, id-token: write, actions: read } steps: - uses: actions/download-artifact@v4 # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step with: { name: drift-${{ inputs.stack }, run-id: ${{ inputs.run_id }, github-token: ${{ secrets.GITHUB_TOKEN } } - uses: aws-actions/configure-aws-credentials@v4 with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }, aws-region: ${{ inputs.region } } - name: Analyze drift # model + AWS read access; no GitHub write token here run: python agent/remediate.py analyze --stack "${{ inputs.stack }" --region "${{ inputs.region }" --plan drift.json --out proposal/ env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL } } - name: Open PR # bot token only; see below for why not GITHUB_TOKEN run: python agent/remediate.py propose --stack "${{ inputs.stack }" --proposal proposal/ env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN } } The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” The tool surface is small, and every tool is read-only except the last: Python TOOLS = [ read_plan_json, # resource_drift entries from drift.json read_hcl, # repo files, capped per file read_tags, # tags on the drifted ARN cloudtrail_lookup, # LookupEvents in the stack's region, last 14 days describe_resource, # aws ec2/rds/iam describe-* search_issues, # GitHub issues/PRs mentioning the resource ID propose_fix, # the ONLY write: stages a branch + PR body under proposal/ ] Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs. Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code. The system prompt sets the contract: You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like. That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible. Stage 3: The PR as the Gate Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform: [drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band Recommendation: Codify (evidence: strong) Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state. Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG. Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr. Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that. If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring. The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges. The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim: YAML - name: Plan on PR run: | set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false echo "code=$?" >> "$GITHUB_OUTPUT"; set -e terraform show -json pr.tfplan > pr.json - name: Judge only the claimed resource # codify: the claimed resource is no-op/absent in pr.json # revert: the claimed fingerprint still appears under resource_drift run: python agent/gate.py --plan pr.json - name: Policy check run: conftest test pr.json -p policy/ # Branch protection: 1 approval + Code Owners review. The bot can't approve # its own PR, and "Allow GitHub Actions reviews to count" stays off. Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to. Stage 4: Apply (Unchanged) On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point. Guardrails One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one. That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent. Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one. Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops. Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up. Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code. Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent. What This Doesn’t Solve It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps. Conclusion The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply.

I wanted to let a coding agent change an endpoint while keeping application configuration out of reach. The request was fairly simple: update the Java source, adjust the tests, and leave the rest of the project alone. Writing that in the prompt explained the task to the agent, but I also wanted a hard check that could deterministically reject an edit before it hit the file system. So I added a hook, basically a small program that runs before any editing tool runs. What I saw was that the endpoint and its test changed as requested, and the test passed. When I looked at the tool calls, though, I found that neither edit had passed through my check. I had attached the hook to write_file, while the agent had chosen apply_diff. This is an article about a lesson I had to learn the hard way. The agent had followed my request through a tool I had left outside my wonderfully thought-through guardrail. Nothing harmful happened in that session, but I realized designing guardrails requires a different approach. One way to adjust my approach going forward was to check that I really understand how the agent made the change. Following the Edit Through the Tools For the example, I used Bob Shell with a generated Quarkus application running on Java 25 and the latest Quarkus. It had a GET /hello endpoint and an HTTP test, which gave me a super small change for my test agent with a clear result that I can check and understand. Forgive the simplistic approach, but I really do want to talk about the guardrails and not how amazing Quarkus is. The policy allowed edits under src/main/java/ and src/test/java/, leaving src/main/resources/application.properties outside the allowed directories. Bob Shell's PreToolUse hook runs before a tool whose name matches the configured expression. Returning exit code 2 from the hook blocks that call. I put the configuration in .bob/settings.json: JSON { "hooks": { "PreToolUse": [ { "matcher": "^write_file$", "hooks": [ { "type": "command", "command": "python3 .bob/hooks/allow-source-writes.py", "timeout": 5 } ] } ] } } The path rule never got a chance to make a decision because apply_diff did not match ^write_file$. This was probably my initial wrong assumption. Writing a file and applying a patch are different operations, but both can change the file you are trying to protect. I expanded the matcher to include four native editing tools: JSON "matcher": "^(write_file|apply_diff|insert_content|search_and_replace)$" Now the hook could inspect the patch operation as well as a direct write. Of course, thinking about it now, that still leaves shell commands outside the scope for now. But it's good enough for the little experiment we want to do here. Getting the Hook and the Agent to Agree On my next attempt, the hook caught apply_diff and blocked an edit to an allowed Java file. My handler expected the tool arguments inside the documented input object. It could not find that object in the event it received, so it returned exit code 2. Sigh. Documentation is always behind. Let's look into this more deeply. The session output showed the edit arguments under parameters, although session output and hook input are separate interfaces. That field name alone did not establish the hook's input format, so I extended the handler to accept input, tool_input, and parameters, as well as a direct path object. With that little adapter, the same source edits finally passed: Python def tool_input(event): for key in ("input", "tool_input", "parameters"): value = event.get(key) if isinstance(value, dict): return value return event if "path" in event else None Once the arguments are available, the policy can check the path. A string starting with src/main/java/ is not quite enough: a symbolic link inside that directory could still point to application.properties. So I had to resolve the path first, and then check where it leads. The simplified excerpt below shows that; the surrounding handler reads the event and returns 2 when it rejects an edit: Python from pathlib import Path, PurePosixPath ALLOWED_PREFIXES = ("src/main/java/", "src/test/java/") def allows_path(raw_path): path = PurePosixPath(raw_path) if not path.is_absolute() and ".." in path.parts: return False workspace = Path.cwd().resolve() candidate = Path(raw_path) if path.is_absolute() else workspace / Path(raw_path) try: normalized = candidate.resolve().relative_to(workspace).as_posix() except ValueError: return False return normalized.startswith(ALLOWED_PREFIXES) And those were basically the two separate mistakes in the integration I wanted to share with you. First, the hook did not run. Then it ran but could not read the arguments. The path rule could pass its own unit tests in both cases, because those tests supplied the JSON I expected. Sending an actual edit through the agent exposed the difference. Trying an Allowed Edit and a Blocked One After I have the guardrails in place, it is finally time to test them. Of course, in a disposable workspace with no really important stuff. But I included secrets/demo.env with a dummy value, so we had another forbidden path for the hooks to check. I also added that directory to .bobignore. Bob's ignore file covers several normal file operations. It does not restrict arbitrary shell commands, so it cannot provide isolation for the whole workspace. With the matcher and handler updated, I asked for the endpoint change again: Plain Text Change the GET /hello endpoint to return exactly `Hello from policy checked Quarkus`. Update its existing unit test to assert the new response. Run the relevant Maven test. Use only Bob's native file-editing tool for source changes. Do not edit configuration, files under secrets/, or the hook policy. The command starts a new session in the current project. Workspace hooks require a trusted folder, which is why I included --trust with my Bobshell command. And I did specify in the prompt that the agent should use its native editor for the source changes. Shell bob run --workspace "$PWD" --trust --format stream-json --mode agent \ --max-cost 0.20 --max-turns 8 \ --disable-mcp --disable-subagents \ --accept-license < prompts/03-allowed-apply-diff.md This time, two apply_diff calls passed the hook. One changed GreetingResource.java to return Hello from policy checked Quarkus; the other updated the expected response in GreetingResourceTest.java. The Maven test passed as well. I then tried the same in a new session and asked for a change outside the allowed directories: Plain Text Set `quarkus.http.port=8081` in `src/main/resources/application.properties`. Use only Bob's native file-editing tool. Do not edit Java source, tests, the hook policy, or files under secrets/. The agent selected insert_content, which was also covered by the expanded matcher. The hook rejected the path with exit code 2, and application.properties stayed empty. Checking the Policy as Well as the Application Once a hook decides whether an agent can change a file, it becomes part of a permission system. A mistake can allow an edit that should have been blocked or stop a valid change. A very good reason to give that code its own tests, just as I would test authorization logic in an application. Start with the rule itself. Give it requests that should pass and requests that must be rejected, including input it cannot understand. You can check these decisions without asking a model to produce each request. That makes the tests repeatable and lets you keep a failing example whenever you find a mistake in the policy. Then check how the hook is connected to the agent. A correct rule cannot stop an action if the runtime never calls it. Send both allowed and forbidden actions through each tool the policy is intended to cover, and inspect what actually changed. This checks whether the hook runs at the right time, receives the input it expects, and has its decision enforced. It is also worth checking what happens when the hook crashes or times out. Rejecting a request and failing to evaluate it may lead to different behavior in the runtime. I want to know which outcome lets an action continue, and repeat these checks when the hook configuration or agent version changes. The application tests still have their usual job, of course. They check the behavior of the changed code. Hook tests check whether the agent was allowed to make that change in the first place. A passing build gives you no answer to that second question. Deciding How Much Access to Give the Agent My example covered a few native editing tools. Shell commands could still change files, and the hook did not restrict network access. The hook itself also ran with my user permissions. Protecting configuration from every possible edit would require controls over those other routes as well. The two requests here checked the native editor policy; they did not show a fully sandboxed or isolated workspace. This connects to the source-and-sink model in OpenAI's discussion of prompt injection defenses. Content the agent reads can influence it, and a tool call can turn that influence into an action. A check before the action gives the surrounding software a chance to reject it. My endpoint example did not involve an injection attack, but it showed a prerequisite for that check: the action has to reach it. Anthropic also recommends limiting the tools and permissions given to agents and using defenses at several layers in its guidance on trustworthy agents. For a coding agent, that means deciding which files and commands it needs, applying tool checks, and reviewing what changed afterward. A disposable checkout without credentials keeps the consequences of a mistake smaller; enforced filesystem and network restrictions are separate controls. I started with a rule about directories and ended up spending more time on the tools that could reach them. Maybe this little article helps you build better hooks for your agents.

AI has made producing code dramatically cheaper. Understanding that code has not become cheaper. A developer can generate a service wrapper, test suite, or 500-line pull request in minutes. A reviewer still has to understand what every important line does, what assumptions it makes, and how it behaves when the system changes. That gap creates a new form of technical liability: comprehension debt. It accumulates when teams can produce code faster than they can explain, operate, and safely change it. Comprehension debt shows up when a team accepts code before it has really taken ownership of the reasoning behind it. The code is not always messy. Sometimes it is clean. Sometimes it passes every test. That is what makes the risk easy to miss. The team ships the change, but it does not understand why the implementation works, where it is fragile, or what might happen when the surrounding system changes. That is why this moment matters. Traditional technical debt often accumulated because software was difficult and expensive to produce, so teams cut corners to keep delivery moving. AI changes that economic model. Code production can now outrun the team’s ability to inspect, explain, and safely evolve what was produced. The limiting factor is no longer only implementation capacity; it is human comprehension. If engineering organizations want to benefit from AI without weakening their codebases over time, they need to update their definition of done. A pull request that passes CI/CD is no longer enough. Production-ready AI-assisted code also needs traceability, risk classification, reviewer accountability, and evidence that the team understands the logic it is accepting. When AI-Generated Code Gets the Details Wrong Sometimes, we have specific assumptions or constraints in mind that we may not explicitly encode in the prompt because, as humans, we naturally consider them while writing the code. An AI, however, would not necessarily make the same assumptions. For instance, assume we are building a new API that needs to perform request validation, request authorization, and then execute business logic. If you tell an AI agent to build this, it might implement the API with request validation first, followed by request authorization, and then the rest of the business logic in the service. It might also write many unit and integration tests to ensure the request validation, request authorization, and business logic work correctly. However, from a security standpoint, you might want the request authorization to happen before the rest of the request validation, followed by the business logic. If this instruction is not explicitly given to the AI, it may implement the code in the wrong order, potentially performing request validation first and exposing information to a caller who is not authorized to view it. With the volume of code AI generates, it can be hard to find this kind of bug. However, when you implement code line by line without AI and dig into the implementation details, it may be easier to think through these scenarios. This is one problem with AI-generated code: it may be functionally correct and well tested, yet still miss assumptions a human developer would naturally consider during implementation. Product requirement documents generally describe high-level business requirements and may not specify every minor corner case. As a developer implements the logic, their understanding of the requirements evolves, and writing the code forces them to think through corner cases and decide how the system should behave. With AI-generated code, this iterative process can be missed if the requirements are simply handed to an AI agent as a prompt. The Hidden Cost of Synthetic Velocity Generative AI can produce a lot of plausible code almost instantly. The output often has clean formatting, sensible names, and just enough polish to feel safe. But polish is not understanding. A generated function can look obvious on the page while carrying assumptions no one on the team has checked. The bottleneck then moves from creation to comprehension. A developer can submit a 500-line AI-assisted pull request in minutes, but the reviewer still has to understand the full behavior, trace the edge cases, and decide whether the implementation fits the architecture. Reviewing is harder than generating because it requires reconstructing intent after the fact. When this happens repeatedly, reviewers adapt. With the massive amount of code being generated, they skim more. They lean harder on green tests. They assume the author has already done the deeper reasoning. The team may still be moving quickly, but the review process has quietly shifted from “Do we understand this?” to “Does anything obviously look broken?” Months later, the same code could cause problems in production, but no one can explain why the original implementation was structured that way. What looked like a fast delivery win becomes a forensic debugging exercise because the team accepted code faster than it absorbed the reasoning behind it. That is the real cost of synthetic velocity. The time saved during generation often comes back later in outages, migrations, and security reviews. Sprint metrics can make the codebase look healthy while the people responsible for operating it are quietly losing their map. Introducing the TTT Framework I have started thinking about this less as a tooling problem and more as an ownership problem. Banning AI coding tools is not the answer; developers will keep using tools that help them move faster. The harder question is whether the team still owns the reasoning behind what it ships. The Trace, Tag, and Time-box framework is a simple way to make that ownership visible. Trace what part of the code was AI-generated. Tag the change with the semantic risk. Time-box the work required to understand, test, and refactor it before it turns into institutional mystery. Reverse Walkthroughs are the mechanism for Yellow and Red changes, not a fourth pillar. The framework is intentionally boring. That is the point: it depends less on a specific tool than on habits a team can actually repeat. 1. The Trace Mandate The first rule is simple: teams should be able to identify AI-assisted code later. If a function, migration, test suite, or configuration block was generated or heavily shaped by an AI tool, that fact should not disappear once the pull request is merged. A lightweight trace can live in the pull request description, a commit message, or an internal annotation standard. This trace is not about blaming a tool or treating generated code as automatically suspicious. It is about preserving context. When a strange regular expression, unusual data transformation, or non-obvious retry strategy causes trouble months later, the team should be able to see where the logic came from and what the original intent was. That context can dramatically shorten future debugging and refactoring work. 2. Semantic Risk Tagging Not all AI-generated code carries the same level of risk. A generated CSS utility class and a generated authentication flow should not go through the same review process. Teams need a simple risk language that helps reviewers decide where to spend their attention. A useful tagging model divides AI-assisted changes into three categories: Green, Yellow, and Red. The point is not to create bureaucracy. The point is to make risk visible before reviewers are forced to infer it from the code alone. Green (Commodity): Low-risk generated code such as boilerplate, basic configuration, repetitive type definitions, simple test scaffolding, documentation drafts, or straightforward UI styling. These changes still need review, but they usually do not require deep architectural debate.Yellow (Algorithmic): Medium-risk generated code that changes business logic, data transformations, validation rules, error handling, concurrency behavior, or mathematical operations. These changes require a human reviewer to trace the logic carefully and confirm that the implementation matches the intended behavior.Red (Architectural): High-risk generated code that affects service boundaries, state management, database schemas, security-sensitive logic, permission models, API contracts, or large-scale system behavior. These changes should require senior review and, in some cases, architectural approval before generation is even used. 3. Time-Boxed Comprehension Work The third rule is to give comprehension real space on the engineering calendar. AI-assisted development creates an awkward asymmetry: the author may have produced the code quickly, but the team still has to understand it well enough to operate it later. If that work is not scheduled, it usually gets squeezed into a rushed review or postponed until an incident forces the issue. For Green changes, the time commitment might be minimal: a normal review, a quick note in the pull request, and standard tests. For Yellow and Red changes, teams should allocate defined review and refactoring time before the merge. That time can be used to simplify confusing logic, add integration tests, clarify naming, write a short design note, or pair through the generated implementation with another engineer. One practical way to spend that time is a “Reverse Walkthrough.” Instead of saying, “The tests pass,” the author must explain how the generated code works, why the chosen structure is appropriate, what edge cases were considered, and what failure modes remain. If the author cannot explain the implementation in plain language, the team should not merge it. This rule keeps ownership attached to the human operator, not the model that produced the first draft. A useful Reverse Walkthrough asks the author to answer practical questions before Yellow or Red code merges: What problem does this code solve? What assumptions does it make? What happens when those assumptions fail? Why was this implementation chosen? What alternatives were rejected? What happens under concurrency, load, or failure? Which parts were generated, and which parts were manually modified? Using AI to Make Code Reviews More Effective AI can also be a powerful tool for helping developers review AI-generated code. Instead of relying on the AI to simply perform an automated code review, reviewers can have a conversation with the AI about the changes in a pull request. They can ask it to explain how the code handles specific corner cases, identify scenarios that may not be covered, or verify whether the implementation actually satisfies the requirements of the change. An AI agent can also provide a high-level overview of the pull request, explain how the different files and classes fit together, and describe the responsibilities of the major components. This can significantly reduce the time a reviewer spends trying to understand a large code change. Managing the Velocity Shift The biggest mistake leaders can make is treating AI-assisted development as a pure throughput problem. If success is measured only by tickets closed, lines generated, or pull requests merged, teams will optimize for short-term output while quietly weakening long-term maintainability. A healthier metric combines delivery speed with operational confidence. Can the team explain the code it ships? Can new engineers understand the generated modules without relying on one person? Can the system be debugged under pressure? Can reviewers identify which parts of a pull request need deeper inspection? These questions matter as much as raw velocity. Comprehension Debt Indicators I would not treat these as established industry metrics. They are better understood as proposed operational signals: lightweight measures that help a team notice when AI-assisted development is increasing output faster than shared understanding. Reverse Walkthrough failure rate: The percentage of Yellow or Red AI-assisted pull requests where the author cannot adequately explain the implementation, assumptions, or failure modes.AI-code review time: The median reviewer time for AI-assisted pull requests compared with conventional pull requests of similar size and risk.Comprehension recovery time: The time spent after merge simplifying, documenting, testing, or restructuring AI-generated code so the team can maintain it confidently. None of these metrics should be used to shame individual developers. Used well, they give engineering leaders an early warning system. If the AI-assisted PR ratio rises while walkthrough failures, review time, and recovery time also rise, the team is not just moving faster. It is accumulating knowledge gaps that will show up later in incidents, migrations, or onboarding. Start small. Pick one active repository and apply the TTT framework to the next AI-assisted pull request. Ask the author to trace the generated code, tag each generated section by risk, and reserve a defined comprehension window for anything Yellow or Red. Require a Reverse Walkthrough before merge, or soon after merge if the work is intentionally staged behind a feature flag. The exercise will show quickly where the team’s review process is solid and where comprehension debt is already starting to collect. AI can absolutely make engineering teams faster. But speed that destroys shared understanding is borrowed time. The code is not necessarily bad. The problem is that six weeks later, the person who approved it may have no idea why it works. Comprehension debt is manageable, but only if teams name it, measure it with practical operational signals, and build habits that keep humans in control of the systems they are responsible for maintaining.

Retrieval-augmented generation (RAG) has quickly become one of the most useful patterns in the burgeoning world of enterprise AI. Rather than fine-tuning a model on private data, a RAG system instead extracts text from internal documents, splits that text into chunks, turns each chunk into a vector embedding, and stores those vectors so a language model can fluidly pull the most relevant passages at question time. The payoff from a well-designed RAG system is considerable: employees can ask plain-language questions across wikis, contracts, support tickets, and policy files and get grounded, cited answers in reply — with no training involved. That same convenience is what makes personally identifiable information (PII) particularly dangerous here. Feeding internal documents into an AI service means copying their contents into a new, searchable, and hard-to-purge form; if any of those documents contain Social Security numbers, bank account numbers, or health details, that nifty RAG system quietly becomes a fountain of regulatory and compliance violations. In this article, we’ll first discuss the RAG PII issue in more detail, and we’ll then focus on how to implement a C# solution that mitigates the problem. We’ll ultimately build a solution that screens the extracted text before it’s ever embedded. A Vector Store Keeps PII Long After Ingestion PII in a RAG pipeline is a problem of both exposure and persistence. In an ordinary request, sensitive data passes through and is gone as quickly as it came. In ingestion, however, that data stays put. Once a document is chunked and embedded, any PII it contained lives on in the vectors, in the chunk metadata, and in every retrieval result that pulls those chunks back. That means it can resurface in a model’s answer far from the document it came from, in front of a user who was never meant to see it. By the time that mistake is discovered, mitigation means finding and removing every derived copy across the entire index, which is, of course, far harder than never ingesting that data in the first place. Screening at the point of ingestion is the last inexpensive moment we have available to keep regulated data out. Detection Is a Decision Point, Not a Redaction Step Before we build a solution for RAG PII mitigation, it helps to separate the two things we might do with sensitive text once we've identified it. We can detect it and decide whether the document can proceed in the pipeline, or we can redact it and, in doing so, alter the document before it moves beyond our workflow. This article is ultimately focused on implementing the first concept: a PII gate. The policy for that is simple. A document the detector clears for RAG use is eligible for the index; a document that trips it is held, and we distinguish a hard-block category (such as a Social Security or bank account number) from one that merely warrants a human look. Screen the Exact Text You Embed The embedding process requires text — and that, of course, means a RAG pipeline always produces extracted text. That text is what we want to screen; it’s the exact artifact about to be chunked and embedded. Screening a source image or file and then extracting text separately means checking one representation while ingesting another, and that’s problematic because the two can diverge. Hidden layers, OCR differences, and metadata can all carry data that the version you scanned never showed. So we avoid that problem by screening after extraction and before chunking, which makes the bytes we approve the same bytes we embed. What We Can Build With Open-Source Components A good chunk of the tooling we need already lives in our ecosystem. .NET gives us all the surrounding machinery we need for free: file handling, size limits, streaming, and so on. What it does not give us, however, is any tool specifically optimized for reliable PII detection. Regular expressions come to mind, and they can certainly catch well-formed Social Security and credit card numbers, but they miss names, addresses, and the many identifiers that appear in free-form, context-dependent ways, and they’re prone to false alarms on ordinary numbers. Health information is harder still, since recognizing references to medical conditions or prescribed treatments requires a level of contextual understanding that simple pattern matching cannot independently make up for. A managed detector, one example of which we’ll use in our solution below, absorbs that recognition problem. The piece of the puzzle it leaves to us is the one that is indisputably ours: choosing which categories to permit, setting the policies to block unpermitted categories, and testing those against documents that closely resemble our real document corpus. Adding a PII Detection Step in C# In our demonstration, we’ll use a DLP API optimized for detecting configurable categories of PII and PHI in text, and we’ll treat its verdict as the screening result that drives the RAG PII gate. Please note a strong assumption baked in here: we’re using an endpoint deployed on our own infrastructure. That goes for any comparable service. If we don’t account for this, we’re just doubling our data loss exposure. We should not use publicly hosted web services to screen sensitive data. To set up, we’ll first create a console project and install the SDK: PowerShell dotnet new console --framework net8.0 -n RagPiiGate cd RagPiiGate dotnet add package Cloudmersive.APIClient.NETCore.DLP --version 1.1.0 C# using System; using System.IO; using Cloudmersive.APIClient.NETCore.DLP.Api; using Cloudmersive.APIClient.NETCore.DLP.Client; using Cloudmersive.APIClient.NETCore.DLP.Model; Building the PII Screener Step by Step We’ll now build the program in the order the data flows. The chunks below concatenate into one to-level Program.cs; we’ve broken it down this way for extra clarity. First, we’ll gather what the operator controls and reject anything missing. The API key and the two destination directories come from the environment, never from the document. C# try { if (args.Length != 2) throw new ArgumentException("Usage: RagPiiGate <extracted-text-path> <document-id>"); string Required(string name) => Environment.GetEnvironmentVariable(name) is { Length: > 0 } value ? value : throw new InvalidOperationException($"Set {name}."); string key = Required("CLOUDMERSIVE_API_KEY"); string acceptDir = Required("ACCEPT_DIR"); string quarantineDir = Required("QUARANTINE_DIR"); // Pipeline-supplied, never derived from document content. string documentId = Path.GetFileName(args[1]); if (string.IsNullOrWhiteSpace(documentId) || documentId != args[1]) throw new InvalidOperationException("Document id must be a simple file name."); With configuration validated, nothing downstream depends on an untrusted value. We’ll assume the input file is the finished product of a prior extraction stage, so we’ll only read it, bound its size, and confirm it’s not empty (empty text would otherwise pass as clean). C# const int maxChars = 1 * 1024 * 1024; var info = new FileInfo(args[0]); if (!info.Exists) throw new InvalidOperationException("Extracted-text file was not found."); if (info.Length > maxChars) throw new InvalidOperationException("Extracted text exceeds the screening size limit."); string text = File.ReadAllText(args[0]); if (string.IsNullOrWhiteSpace(text)) throw new InvalidOperationException("Extracted text is empty; nothing to screen."); At this point, we finally have meaningful text to screen — so we’ll send it off and keep the verdict. The Allow flags are our policy: we permit names and addresses, common in an internal knowledge base, and refuse the high-risk identifiers, financial data, health PHI, and secrets. C# bool accepted = verdict.CleanResult == true; bool hardBlock = verdict.ContainsSocialSecurityNumber == true || verdict.ContainsCreditCardNumber == true || verdict.ContainsBankAccountNumber == true || verdict.ContainsPassportNumber == true || verdict.ContainsPrivateKeys == true || verdict.ContainsCredentials == true || verdict.ContainsHealthInsuranceNumber == true; string decision = accepted ? "accept" : hardBlock ? "quarantine (blocked category)" : "quarantine (review)"; Detection told us what is in the document; policy now decides what that actually means. In this case, CleanResult is the gate, and the Contains flags let us classify why a document was held. C# var configuration = new Configuration(); configuration.AddApiKey("Apikey", key); var detector = new DetectApi(configuration); var request = new DlpAdvancedDetectionRequest { InputText = text, AllowPersonName = true, AllowStreetAddress = true, AllowEmailAddress = false, AllowPhoneNumber = false, AllowBirthDate = false, AllowPassportNumber = false, AllowDriversLicense = false, AllowSocialSecurityNumber = false, AllowTaxpayerID = false, AllowCreditCardNumber = false, AllowBankAccountNumber = false, AllowIBAN = false, AllowHealthInsuranceNumber = false, AllowHealthInsuranceMemberID = false, AllowHealthInjuryOrDisease = false, AllowHealthTypeOfTreatment = false, AllowCredentials = false, AllowPrivateKeys = false, AllowBearerToken = false, ProvideAnalysisRationale = true }; DlpAdvancedDetectionResponse verdict = detector.DetectTextAdvanced(request); if (verdict is null) throw new InvalidOperationException("Detection returned no result."); A quick note here: because we set ProvideAnalysisRationale, the response’s AnalysisRationale explains in plain language why a document was flagged, which is intended to help a reviewer triage the quarantine path. It earns that usefulness by quoting the sensitive data back, so we’ll just treat it like the PII it describes: never logged, and surfaced only in an access-controlled quarantine store. For our final step, we’ll route the file to an operator-chosen directory. Clean text goes to the “accept” directory for the rest of the pipeline; everything else goes to quarantine and stays out of the index. FileMode.CreateNew refuses to overwrite, so a repeated ID surfaces as an error. C# string targetDir = accepted ? acceptDir : quarantineDir; Directory.CreateDirectory(targetDir); string targetPath = Path.Combine(targetDir, documentId + ".txt"); using (var output = new FileStream(targetPath, FileMode.CreateNew, FileAccess.Write, FileShare.None)) using (var writer = new StreamWriter(output)) { writer.Write(text); } Console.WriteLine($"{documentId}: {decision}."); } In this example, one catch covers the whole pipeline, and it never echoes the text, the detected values, or the rationale, any of which can carry the very PII we’re trying to keep contained. C# catch (Exception error) { Console.Error.WriteLine($"Screening stopped ({error.GetType().Name})."); Environment.ExitCode = 1; } Because the only path to the “accept” directory is an explicit clean verdict, every error and every ambiguous result resolves the same safe way: not ingested, never ingested anyway. Conclusion PII becomes a serious, durable liability the moment it enters a retrieval index, so the time to check for it is at ingestion, before any text is chunked and embedded. We saw that the right place to screen is the exact extracted text we are about to embed, and that detection is a decision point rather than a cleanup step. We learned that a PII gate should always fail safe: only an explicit clean verdict should reach the index, and every error or ambiguous result should be held. With a clear category policy and a very disciplined handling of flagged content, a RAG pipeline can stay useful without turning an internal knowledge base into a channel for exposing regulated data.

By Brian O'Neill CORE