DZone Spotlight

Stolen credentials served as the entry point in 22% of breaches last year, and in attacks on basic web applications that figure climbs to 88%. Those numbers describe a password problem, and databases sit at the end of nearly every attack path. A username and password prove nothing about the machine presenting them. Mutual TLS closes that gap by requiring both sides of a connection to present certificates and prove who they are before a single query runs. Securing production database connections has convinced me that enterprises implement mutual TLS using readily available tools and established certificate management practices. What Certificate-Based Authentication Actually Closes Off A password travels, gets shared, gets phished, and gets left behind in a script. A certificate bound to a specific client does none of those things easily, which is why mutual authentication blunts three familiar attack patterns: stolen credentials replayed from an unfamiliar host, spoofed clients impersonating an application server, and lateral movement after an attacker gains an initial foothold. Enterprises usually maintain password rotation policies that are triggered after a set period or upon an employee's exit from the team. Certificate-based authentication safeguards the data in case a team misses rotating those credentials, because the password alone no longer grants entry. I ran into this while setting up an open-source alerting tool, where the password had to live in a config file or a session variable. The session variable fails when the tool auto-restarts during maintenance, leaving a hardcoded password or a decryption utility to mask it. A certificate addresses that problem because its lifetime can be governed by the organization's security policies. Once the certificate expires, a password alone is no longer sufficient to authenticate the client. Machine identities now outnumber human identities by more than 80 to 1 in the average organization, and each database connection string is one of them. The Rollout Decisions That Matter Most An mTLS program stands on three design choices. The first is the certificate authority, where an internal CA gives the team full control over issuance and revocation for database traffic that never leaves the estate. The second is cipher selection, which deserves more attention than it gets, because Oracle, MySQL, and MongoDB each negotiate TLS differently, and a cipher suite that works on one engine can fail the handshake on another. The third is rotation, and this is where programs die quietly. 81% of organizations have suffered at least two outages caused by expired certificates in a two-year window. Six months to a year is an ideal certificate lifetime across a polyglot estate, though the organization's security baselines govern, and estates handling critical PII or PCI data, or carrying past breach attempts, can justify a reduced lifetime. Renewing very frequently creates its own outages, because some systems require a reboot to bring new certificates into effect, a real challenge for heavily used 24/7 applications without a high availability solution ready. Keeping lifetimes consistent across Oracle, MongoDB, MySQL, and PostgreSQL helps manage the rotations, but databases are not all created in a single day, so expiry timelines differ and an inventory or dashboard tracking every expiry becomes essential. Above all, automating renewal wherever possible reduces the risk of downtime from an expired certificate. Securing the Monitoring Layer Itself Monitoring an encrypted estate raises a question teams often skip, which is how to keep the monitoring path from becoming the weak point. Many organizations use Prometheus to collect database metrics, with more than two-thirds of organizations running it in production, yet exporters may be deployed with unencrypted scrape endpoints if they are not configured to use TLS. One common deployment approach is to run the database exporter process on the database server itself. In Prometheus-based environments, configuring both the client connection (--config.my-cnf) and the exporter (--web.config.file) to use client certificates allows the monitoring pipeline to follow the same mutual authentication model as the database it monitors. This helps ensure that metrics are collected over authenticated, encrypted connections rather than introducing a weaker path into the environment. Choosing the Right Approach Organizations can implement mutual TLS using either commercial certificate management platforms or open-source tooling. The right approach depends on factors such as certificate volume, compliance requirements, auditing needs, and the operational resources available to manage the environment. Early in my career, I assumed enterprise-licensed products were the default choice for every deployment. Over time, I found that decision is more nuanced. Large environments managing tens of thousands of certificates may benefit from centralized lifecycle management, auditing, and governance features, while many organizations can successfully implement mTLS using open-source tools that meet their operational requirements. The priority should be selecting an approach that supports reliable certificate issuance, rotation, and revocation while integrating with existing security processes. Regardless of the tooling, a well-managed certificate lifecycle is what ultimately strengthens database authentication and reduces operational risk.

Most developers I've worked with write SQL every day. Very few of them are DBAs. According to the 2024 Stack Overflow Developer Survey — 65,000 developers across 185 countries — database administrators make up just 0.3% of the developer population. The tools built for SQL performance were designed for that 0.3%. I built QueryTuner for everyone else. I've spent 13 years as an application architect. In that time, I've watched the same situation repeat itself across teams: a query is slow, the developer who wrote it has to fix it, and the tools available to them are either way too expensive or way too generic. Enterprise monitoring agents like pganalyze or Datadog Database Monitoring cost hundreds of dollars a month and require installing an agent with full database credentials. Generic AI LLMs don't know whether you're on Oracle or MySQL. There's nothing useful in between. That gap is what QueryTuner tries to fill. The Core Constraint: No Database Connection The first decision I made was also the most important one. QueryTuner would not connect to any database. Every enterprise SQL tool requires credentials. In most organizations, getting credentials approved takes longer than just fixing the query manually. I wanted something a developer could try in 30 seconds without asking anyone for permission. The tradeoff is real. Without connecting to your database, QueryTuner can't see actual row counts, current index usage, or live execution plans. But it can analyze the SQL text itself — and most slow query problems come from a small set of well-known patterns. You don't need to connect to a database to spot a function wrapped around a column in a WHERE clause. The Heuristic Engine QueryTuner runs 12 deterministic rules against every query before anything else happens. These rules catch the patterns that cause most slow query problems in production: Functions on indexed columns are the most common. If you write WHERE YEAR(created_at) = 2024, the database has to call YEAR() on every row before it can filter. The index on created_at becomes useless. The fix is a range condition: WHERE created_at BETWEEN '2024-01-01' AND '2024-12-31'. The index works again. Leading wildcard LIKE patterns are the second most common. LIKE '%value' can't use a B-tree index. The database reads every row. Most developers don't know this until they see it in an execution plan for the first time. Correlated subqueries in the SELECT clause are the most expensive. If you have a subquery inside your SELECT list, it runs once for every row in the outer query. On a table with 50,000 rows, that's 50,000 separate database lookups. A LEFT JOIN does the same work in a single pass. Cartesian JOINs are the most dangerous. A JOIN without an ON clause multiplies every row in table A by every row in table B. On production tables with millions of rows, this can crash your database server. QueryTuner marks these as critical severity — the only finding type at that level. The heuristic engine runs in under 200 milliseconds. It always runs, regardless of whether the LLM layer is enabled. This was a deliberate design choice. I wanted the tool to be useful even when the AI component is unavailable. The LLM Layer After the heuristics run, users can optionally enable an LLM layer — HuggingFace or OpenAI. The LLM adds plain-English narrative, a rewritten query using CTEs, and flags for assumptions it can't verify without knowing the actual schema. The key design principle here: the LLM is additive. If it fails — cold start on the free tier, rate limit, network timeout — the user still gets complete structured findings from the heuristic layer. The tool does not degrade to an empty screen when AI is unavailable. The Dialect Problem This was the hardest part to get right. SQL is not one language. The correct way to create an index in production differs significantly across databases. In PostgreSQL, you use CREATE INDEX CONCURRENTLY to avoid locking the table during index creation. Without CONCURRENTLY, all writes block until the index is built. On a busy production table, that can mean minutes of downtime. In MySQL, the idiomatic form is ALTER TABLE orders ADD INDEX idx_name (column). The CREATE INDEX syntax also works, but ALTER TABLE integrates better with InnoDB's internal operations. In Oracle, you add NOLOGGING to skip the redo log during index creation. This makes it significantly faster, but you can't recover the index from redo logs if something fails mid-creation. Use it during maintenance windows only. In SQL Server, CREATE NONCLUSTERED INDEX ... WITH (ONLINE=ON) allows reads and writes to continue during index creation. This is an Enterprise edition feature. FILLFACTOR=90 leaves 10% of each page free for future inserts, reducing page splits over time. In SQLite, there's no concurrent DDL. Index creation locks the entire database file. The only mitigation is scheduling it during low-traffic windows. Generic advice — "add an index on customer_id" — is not enough. The statement a developer runs in production depends entirely on which database they're on. Getting this wrong can cause downtime. I solved this by centralizing all dialect-specific logic in a single file: dialect_config.py. This is a dataclass-based config with one entry per database. Each entry holds the index DDL template, optimizer rewrite syntax, LLM system prompt context, and maintenance commands for that dialect. When the tool generates a recommendation, it calls get_dialect(db_type) and gets everything it needs from one place. The practical benefit: adding a sixth dialect means adding one dataclass entry. No other files change. Schema-Aware Confirmed Recommendations By default, every index recommendation carries a confirmed: false flag. The tool is analyzing SQL syntax, not your actual database. It doesn't know whether the column exists, whether an index already covers it, or what the real table name behind an alias is. If you paste your CREATE TABLE statements alongside the query, that changes. QueryTuner parses the DDL, builds a schema map, and cross-references every detected column against it. If the column exists and no index covers it, the recommendation flips to confirmed: true. The DDL it generates uses your real table name — not a placeholder like <o_table>. Suggestions for indexes that already exist in your DDL are suppressed entirely. For a developer who is about to run a CREATE INDEX on a production database, that distinction matters. confirmed: true means the recommendation was verified against their actual schema. confirmed: false means it's a pattern-based estimate worth investigating. What I'd Do Differently The alias resolution logic — matching o to orders — is the weakest part of the system. It works for common patterns (single-letter aliases, prefix matches) but fails for arbitrary aliases. This is the first thing I'd improve with more time. The LATERAL join gap is the other known limitation. Correlated columns inside LATERAL joins are not detected. It's documented as an intentional xfail in the test suite and will be addressed when the execution plan parsing layer is built. Try It QueryTuner is open source under the MIT license. Live: querytuner.comSource: github.com/AutoShiftOps/querytunerAPI: POST /analyze — accepts query, dialect, optional schema DDL Feedback is especially welcome from Oracle and SQL Server practitioners. Those are the dialects with the least real-world battle-testing, and the production edge cases are where the tool needs the most work.

In the early days of the web, monitoring availability was simple: a server either responded to a ping, or it didn't. HTTP checks tightened that up a little — a 200 OK meant the dashboard turned green, and everyone assumed things were fine. That assumption doesn't really hold anymore, though. A modern app can return a picture-perfect 200 OK and still be completely unusable to an actual customer. Take an e-commerce site where the web server is healthy and responding in milliseconds. Somewhere behind it, a third-party inventory service has quietly died, or a CSS change buried the checkout button under a promo banner nobody tested for. Nobody can buy anything. Server's up. Business is down. Legacy monitoring can't see any of this — it was built to check the plumbing, not whether the person standing at the sink can actually get water out of the tap. Uptime Isn't an Infrastructure Metric Anymore In a monolithic architecture, the app and the database lived on one server, and uptime was basically a binary infrastructure question. That's not how most applications get built anymore. A typical SaaS product today is a single-page application backed by dozens of independent microservices spread across regions, plus a stack of external dependencies — an identity provider, a payment processor, a CDN, whatever else. If any one of those goes down, your own servers can be perfectly healthy while your users still can't get through a core workflow. Uptime, in that world, has to mean the continuous availability of the actual business workflow, not a response code. What Synthetic Monitoring Actually Does Synthetic monitoring uses automated clients to simulate real user traffic on a schedule, from multiple locations, around the clock — instead of waiting for a human to hit a broken flow and file a ticket. These aren't simple URL pingers, either. A synthetic monitor opens a real browser, renders the DOM, executes JavaScript, fills out forms, clicks through multi-step flows, and checks that the right data shows up on screen, all while watching the underlying API calls to make sure the backend agrees with what the UI is claiming. If a login flow that normally takes two seconds suddenly takes ten, or a button just stops responding, the monitor flags it right away, typically with a video of the failed session and enough diagnostic detail attached that someone can actually act on it, routed straight into whatever incident tool the team already uses. That's a fundamentally faster loop than "someone tweeted that checkout is broken." Where This Overlaps With QA: Shifting Right QA and production monitoring have traditionally been separate worlds — different teams, different tools, a handoff at the deployment line. That divide doesn't have much justification anymore. If a team's already built solid automated functional tests for CI/CD, there's no real reason to throw those away once code ships. The same script that validates a checkout flow pre-deploy can get repurposed to run every few minutes in production as a synthetic monitor — generally called "shifting right." Done well, it cuts duplicated engineering effort and gets QA and SRE working off the same definition of "healthy" instead of two different ones. Testing Beyond the Front Door: Complex User Journeys Basic uptime monitoring tells you the front door is open. Synthetic monitoring actually walks through the door, picks something up, applies a promo code, checks shipping, completes a transaction — the whole path, not just the entrance. That requires handling state, not just static checks. A monitor testing a healthcare portal needs to log in with synthetic credentials, get through MFA, pull a specific record, and confirm it belongs to the test account and nothing else. One testing a fintech transfer needs to confirm the UI shows success and then separately query the backend to make sure the balances actually moved, because a UI that says "success" while the ledger disagrees is arguably worse than an honest failure. Validating both the interface and the underlying state is what makes this useful for anything regulatory or revenue-critical. The Self-Healing Problem Running scripts against a live production environment is harder than running them in staging, because production changes constantly — new banners, UI experiments, shifting layouts. A rigid script breaks on cosmetic changes it shouldn't even care about, and that's how you end up with false alarms nobody trusts. This is where AI-assisted self-healing has become genuinely useful, rather than just a buzzword bolted onto a monitoring dashboard. If a button's ID changes from submit-order to confirm-purchase, a brittle script just fails. A self-healing monitor uses visual and semantic signals to relocate the element, finishes the check, and logs a low-priority note for someone to review later, instead of paging an engineer at 3 a.m. over what amounts to a rename. Alert Fatigue Is a Design Problem, Not a Tooling Problem Poorly tuned monitoring trains engineers to ignore it, and static thresholds are a big part of why. If an alert fires whenever a page takes longer than three seconds, a one-off network blip pages someone for a problem that resolves itself before anyone even looks at it. A better approach builds a dynamic baseline from historical performance data — per time of day, per day of week — and only escalates when something deviates meaningfully from that baseline. Often it's worth requiring confirmation from more than one geographic location before paging anyone at all, so a regional network hiccup doesn't wake someone up for nothing. Where This Matters Most E-commerce is the obvious one — downtime there is measured in dollars per second, and synthetic checks on cart logic, discount calculation, and payment gateway responses catch the silent revenue leaks a green uptime dashboard would never surface. Multi-tenant SaaS is a quieter version of the same problem: a single shared microservice failing can degrade the experience for every tenant at once, sometimes without anyone noticing for a while. Synthetic scripts that log in under different tenant configurations help confirm data isolation is actually holding and that SLAs are being met in practice, not just assumed on paper because nothing's screamed yet. Healthcare and fintech carry real regulatory weight on top of the operational risk. Synthetic checks that confirm patient records render correctly, or that a banking handshake with a clearing house completes securely, end up functioning as both an operational safeguard and a rough form of continuous compliance evidence — useful when an auditor eventually asks how you know. The Takeaway A green uptime dashboard doesn't mean much anymore if all it's checking is whether a server responds. The failures that actually cost money and trust — a hidden checkout button, a silently failing third-party integration, a broken multi-step flow — live above the infrastructure layer. Only something that behaves like a real user is going to catch them.

Analytics teams do not get too upset about small errors. If a product dashboard is off by half a percent on a Tuesday, nobody files a ticket. If your marketing funnel counts some web sessions twice, the overall trend is still okay. Everyone moves on. I spent a part of my early career in that world. It is a place to learn how to move fast, ship features, and use data to get a general idea. Then I started building pipelines that fed automated billing and revenue recognition systems. The rules changed completely. Financial-grade data is different. When a number goes on a customer invoice, drives a usage-based billing meter, or gets repeated by an executive to the board of directors, "roughly right" becomes a problem. The pipeline is not just informing a business decision - it is the decision. If it fails, someone has to answer for it to an external auditor. That change moving from analytics to shipping numbers people stake their reputations on — made me scrap my old way of doing things and rethink how I design data infrastructure. If you are building lakehouse platforms that have to scale out and remain completely defensible under scrutiny, here is what actually matters. The Reconciliation Gap Nobody Warns You About Here is the first painful lesson: correctness and scale do not work well together, and billing data is right in the middle. Usage-based billing means you are dealing with huge, high-volume event streams, API hits, compute-seconds, database operations, and converting those numbers into actual cash. The volume forces you toward distributed systems. The money demands accuracy. You cannot ship an infrastructure that's very fast but drops some events, and you cannot ship a framework that is perfectly consistent but takes a long time to close out a daily ledger. The place where this trade-off is hardest is late-arriving or out-of-order data. Imagine a streaming meter where an event happens at 11:58 PM. It does not hit your ingestion engine until 12:03 AM the next morning. If your daily aggregation pipeline already completed at midnight, that customer usage falls into the wrong billing month or disappears. Multiply that event by many transactions, and you have a massive reconciliation gap that your finance team will catch. Because of this, my absolute baseline rule for any pipeline touching revenue is that it must be 100% idempotent and completely reprocessable from source. I mean reprocessable in the sense that I can replay a raw event window from three weeks ago and land on the exact same decimal point. To do that, your transformation logic has to be completely deterministic and keyed entirely on business identifiers rather than system arrival times. In production, that usually looks like a merge statement driven by event and entity IDs: SQL MERGE INTO billing_usage_gold AS target USING staged_events AS source ON target.event_id = source.event_id WHEN MATCHED AND source.ingested_at > target.ingested_at THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * The SQL looks simple. The actual engineering discipline is ensuring that event_id remains stable, unique, and uncorrupted all the way back to the source application code. If you lock down that data contract, your downstream reconciliation nightmares mostly go away. Layering for Defensiveness, Not Aesthetics I am a pragmatist when it comes to the classic layered lakehouse. Many data teams adopt this setup just because it looks tidy in a slide deck. When you are dealing with financial pipelines, those layers serve a functional, defensive purpose. The raw layer needs to be entirely immutable and append-only. Think of it as a ledger of exactly what the world looked like when the event happened, timestamped, raw, and completely untouched. Never let transformation logic touch or rewrite this layer. When an auditor asks, "What exactly did the system report on November 14th?" this table holds the answer. It should not change just because you refactored a downstream SQL model six months later. The refined layer is where you handle the reality of data engineering: deduplication, type casting, schema enforcement, and core business rules. This is also where you have to build structural data-quality checkpoints. For architectures, that means ditching passive logs or soft warnings and leaning into automated testing frameworks like dbt to physically break things when they go wrong. If a data point turns into an invoice line item, a bad value should not log an error; it needs to kill the process. We handle this by setting our dbt data assertions to a hard error severity level: YAML # models/staging/staged_events.yml version: 2 models: - name: billing_usage_silver columns: - name: event_id tests: - unique: config: severity: error - not_null: config: severity: error - name: compute_seconds tests: - dbt_utils.expression_is_true: expression: ">= 0" config: severity: error By explicitly setting severity: error, a single duplicate event ID or a bizarre negative usage value will not just trigger a warning. It will kill the execution DAG instantly. Is it annoying to debug a stopped pipeline at 2:00 AM? Yes. I would much rather explain a delayed operational dashboard to an internal stakeholder than explain a fraudulent or inaccurate charge to a paying enterprise customer. The serving layer is your business-facing interface. It features grains, locked-down definitions, and the exact tables that feed your downstream billing engines, margin tools, and executive reporting. By the time any row hits this layer, it has survived every quality gate you can throw at it. Your analysts and finance partners can build on top of it safely, without rewriting core logic five different ways and coming up with five different answers. If It Isn't Observable, It Isn't Auditable People in data engineering tend to talk about observability like it's a nice-to-have optimization trick or a post-launch polish item. For financial systems, observability is literally the entire game. When you sit down with auditors or finance directors, they do not care if your Apache Spark clusters are running at peak efficiency. They want to know two things: How do you know this final number is correct, and can you prove it to me right now? Answering that honestly requires three things built directly into your infrastructure: Freshness monitoring that actually wakes you up. Silence does not mean everything is working. If a key serving table misses its scheduled data drop, you should not find out because a finance manager pings you on Slack. You need to wire freshness monitoring into a high-priority on-call rotation like PagerDuty. You have to catch the delay before the downstream billing window closes out.Lineage a human can trace. When a revenue metric looks weird on a summary, you need to be able to trace that specific number back through every single SQL transformation, join, and filter to the original raw event in minutes. Relying on "trust me I wrote the code" does not work. Automated, column-level data lineage maps turn an afternoon of code review into a two-minute look.Continuous data quality logging. Treat data quality metrics as a first-class production output. We track row-count variations, null rates, and distribution drifts on every run, logging them out to monitoring tables or platforms like Elementary. If your system ingestion drops out of nowhere, you need to know whether your customers actually stopped using the product or an upstream webhook silently broke. [Raw Event Ingestion] ⬇ Flows into:[Silver Layer] ➡ (Runs dbt Hard Schema & Unique Tests ➡ Fails? HALT & ALERT) ⬇ Flows into:[Gold Serving] ➡ (Triggers Continuous DQ & Freshness Monitoring ➡ PagerDuty / Slack Alerts) Compliance Is Just a Feature Wearing a Suit If you have ever been through a pre-IPO sprint or a standard Sarbanes-Oxley (SOX) audit, you know how exhausting it feels. The biggest mental shift is realizing that compliance guidelines are really just standard system requirements written in legal language. Auditors care about controls, lineage, reproducibility, and separation of duties. If you translate that into engineering terms, it means: your transformation code must be version-controlled and peer-reviewed, production deployments should happen via automated CI/CD pipelines instead of a local laptop terminal, data access needs to be tightly permissioned and logged, and you must be able to reproduce historical numbers on demand. Infrastructure-as-Code (IaC) handles all of this heavy lifting for you. When your cloud environments, access roles, and pipeline configurations live inside a Git repository, the question of "Who changed this permission, and when did they do it?" always has an unalterable answer. Teams that treat compliance as a chore end up panicking every single quarter. Teams that build these automated checks directly into their deployment workflow barely even notice the audit happening. It is the same amount of work either way; doing it continuously is just significantly cheaper. Unlocking Self-Service Without the Chaos The real reward for dealing with all this architecture is that you can finally let other teams get their own data without causing problems. "Self-service analytics" usually gets a bad name because companies often give raw, messy tables to a lot of people. As you would expect, everyone comes up with their own definition of what "gross margin" or "active user" means, and you end up with big arguments inside the company about whose spreadsheet is correct. A controlled and reliable serving layer completely changes this situation. When your definitions are fixed, consistent, and easy to see, your finance team can look at margins by market segment, your marketing teams can build expansion models, and your product managers can look at consumption trends. Everyone is getting their data from the same place. That is the moment your data engineering team stops being a bottleneck for the whole organization. Instead of spending your week answering special requests or running manual data extractions, you get to focus on building infrastructure that can handle a lot of work. Faster decision-making and clear visibility into operations do not come from a magic machine learning model. They happen because your underlying numbers are finally stable enough to act on without needing to check. A Few Things I Wish I Knew Earlier If you are currently moving from building product analytics to managing data that has real financial importance, remember that while your technical skills are still useful, your standards for engineering are not good enough. Design your systems so that you can repeat everything exactly, not just handle a lot of work. Make your data quality tools stop the pipeline if there is a problem instead of just giving a warning. Treat data history, system updates, and automated alerts as parts of your infrastructure rather than things you will do later. And stop thinking of compliance as a rule. A well-built pipeline is already mostly ready for audits anyway. The logic of distributed systems is hard. That is what we all talk about and study. The harder thing is accepting that when your data represents real money, "close enough" is not good enough.

A CI Runner That Shouldn't Have Died If you deploy AWS Lambdas through Terraform, you almost certainly use archive_file. With enough lambdas, a single terraform apply can kill the CI runner with OOM. The trickiest part is that you will not see any errors in Terraform output and have no clue what just happened. I noticed this when my lambdas started failing — every first terraform apply after a routine change. SIGKILL from the kernel OOM killer and nothing in Terraform logs. The strange part is that reapply sometimes worked — not always on the first try, but eventually it went through. I've named the ticket "Flaky CI," and two weeks of investigation was focused on the CI itself: runner memory, parallel jobs, Docker leaks. terraform apply was the last suspect — from my perspective, there was no way or reason for it to consume so much memory. If you've never wondered how Terraform providers work, it's actually pretty simple. Most of them are just API wrappers. They send HTTP requests, parse responses, and update state. archive_file is one of the exceptions — it works with real files on disk. This means that its memory usage is actually determined not by the number of defined resources, but by the total size of the data it should process. That's why the pattern went unnoticed for years — without knowing about the provider's insides, the issue looks like some CI flakiness. When I finally reached the source code, the answer was found in a few lines in zip_archiver.go file. What archive_file Actually Does archive_file data source creates a zip or tar archive from a directory or file. This is a standard pattern for lambdas: you point source_dir at the function code and pass the resulting archive to aws_lambda_function. YAML data "archive_file" "lambda" { type = "zip" source_dir = "${path.module}/src" output_path = "${path.module}/lambda.zip" } Nothing suspicious at first glance, but behind these lines is a call chain, which is worth a deeper look. When Terraform processes this data source, the provider calls archiveFile — it creates a ZipArchiver and iterates over files in source_dir. For each file, it calls the ArchiveFile method, which does the following: Go content, err := os.ReadFile(fname) // ... f, err := a.writer.Create(name) // ... _, err = f.Write(content) os.ReadFile reads the entire file into a []byte — one contiguous buffer in memory. Then that buffer is passed to the zip writer via Write. After the write, the buffer becomes garbage. This was a design choice from 2016, and at the time, it was reasonable. Terraform configurations archived small files — configs, scripts, and templates. A typical source_dir weighed something like kilobytes, so there was nothing to optimize at this point. That's why the simplest way to read a file was chosen — os.ReadFile. The code looks like a textbook example. But the context changed. Lambda zips today are 50-250 MB uncompressed. ML models, large dependencies (numpy, pandas, puppeteer), bundled assets. And teams deploy not one lambda but five, ten, or twenty through a single Terraform workspace. The code from 2016 didn't change. The scale of the data did. Why Can't the Garbage Collector Help The natural and reasonable question: doesn't Go's garbage collector reclaim memory between files? GC runs indeed — it just has nothing to reclaim. All ten archive_file data sources are independent — they have different source directories and no shared references (if you do not specify them directly). Terraform's graph walker places them at the same level and evaluates them concurrently. This is usually a good thing timewise, but not in this case, as all 10 buffers are alive at the same time. Each goroutine holds its 50 MB until zip write completes. The garbage collector scans the heap and identifies every buffer as still in use, so it reclaims nothing. Meanwhile, peak heap hits 10 x 50 MB = 500 MB (measured: 508 MB). If the model is right, peak memory should scale linearly with parallelism. Your CI runner's memory limit doesn't. Measuring the Pattern I've chosen two ways of measurement: a standard Go benchmark for precision (isolating the archiver) and a Terraform integration test for realism (a real provider during terraform plan). The headline: for 10x50 MB concurrent archives, peak heap drops from 508 MB to 8 MB -- a 98% reduction. Full results, heap growth during archiving, buffered versus streaming: 1x50MB: 50.8 → 0.8 MB (98% reduction)10x10MB: 108 → 8.1 MB (92% reduction)10x50MB: 508 → 8.1 MB (98% reduction) Real Terraform under terraform plan with parallelism matrix, peak RSS in MB: Implp=1p=2p=5p=10Buffered1232765791034Streaming173275384533 Buffered RSS scales linearly with parallelism. Streaming flattens the curve. One anomaly you could've noticed: at p=1, streaming shows a higher RSS than buffered. I'm fairly sure it's just noise. Single-archive runs finish fast, and sampling RSS every 100ms is too coarse to catch what's really happening in that window. The number that matters is p>=2, and that's where the pattern holds. On speed: Go benchmark wall time stays within about 3% across every scenario. So the streaming fix isn't quietly buying memory savings with a performance hit. You get the memory back for free. All measurements are reproducible: https://github.com/olegmmv/terraform-archive-memory-research. Putting these measurements together gives a three-stage picture of the memory cost: StagePeak Heap (10x50MB, p=10)StatusBaseline (current provider)1034 MBMeasuredWith input-side streaming533 MBMeasuredWith full pipeline streaming~320 KBArithmetic projection The third row isn't measured, but is arithmetic. I'll describe later why, but for now, just keep in mind that it shows what we'd see if a second os.ReadFile in the output path is also streamed. The Tar Archiver Already Streams The fix isn't speculative; just open a neighboring file in the same provider. In tar_archiver.go, addFile opens the file, defers close, and copies via io.Copy into tarWriter. No buffering — streaming by default. Go file, err := os.Open(filePath) // ... defer file.Close() // ... _, err = io.Copy(a.tarWriter, file) The zip_archiver.go path, though, chose the buffered approach: Go content, err := os.ReadFile(infilename) // ... _, err = f.Write(content) Same codebase and job to be done, but two different choices. archive/zip.Writer.Create returns an io.Writer that streams, with CRC-32 computed during the write via crc32.NewIEEE. There was never a technical barrier. The only thing needed for the fix now is applying the same pattern. The Streaming Fix Here is the diff: replace os.ReadFile with os.Open and Write with io.Copy: diff - content, err := os.ReadFile(infilename) + file, err := os.Open(infilename) if err != nil { return err } + defer file.Close() if err := a.open(); err != nil { ... - _, err = f.Write(content) + _, err = io.Copy(f, file) Everything else stays the same; the only thing that's different is the read-write pattern. This is the actual implementation behind the streaming numbers in the previous section. The streaming version does still allocate memory, of course — you can't get to zero. But it's way down: my benchmark put it at around 0.8 MB. This is due to archive/zip internal buffering: the io.Copy buffer, the deflate compressor state, and small zip metadata structures. One caveat worth flagging: this is the input side only. On the output path, the provider uses its own ReadFile function to compute checksums on the completed zip archive. The Second ReadFile: Output Checksums The Go benchmark showed a 98% reduction, but terraform plan with parallelism=10 only drops from 1034 MB to 533 MB -- about 50%. Where's the missing 48%? Once the zip lands on disk, the provider turns around and reads it straight back. That's what genFileChecksums does: it opens the output file and computes four hashes -- md5, sha1, sha256, sha512 -- for Terraform state. And each one of those hashes wants the full file content. So the provider pulls the entire output zip into memory, using the same os.ReadFile we've been dealing with all along. In my benchmark, the output zip comes out roughly the size of the input. The test data is random bytes, and Deflate can't do much with those. Real Lambda packages compress a lot better, but the pattern remains: the provider reads whatever the output size is back into memory. Run ten of these in parallel at 50 MB a pop, and you're already 500 MB deep, purely on checksums. The PR goes after the input side. It removes the os.ReadFile allocation during archive creation, and the effect is big. In straight Go benchmarks, heap usage drops by 98%, from 508 MB to 8 MB. Real Terraform runs are tamer, about half: peak RSS falls from 1034 MB to 533 MB. So where's that remaining 533 MB coming from? It's the second os.ReadFile, the one inside genFileChecksums, still reading the finished zip back into memory so it can hash it for Terraform state. Technically, you can stream the checksums too. hash.Hash already satisfies io.Writer, so nothing stops you from wrapping all four hashes in an io.MultiWriter and feeding them while the zip is being written. One pass, no second read. The catch is that it's a very different patch from the input-side one. genFileChecksums is structured around post-hoc reading. Making it streaming means restructuring how the provider integrates checksum computation with archive creation. That's state-management territory, not plain I/O. If both sides streamed, the only thing left to allocate would be io.Copy's default buffer. Ten goroutines, 32 KB each, and you land at 320 KB total. Throw in a sliver of zip writer state per goroutine, and that's basically it. The theoretical floor. What the PR actually does is the first half: input streaming, leaving that 533 MB residual behind. The output half, streaming through MultiWriter, is written down as future work. So one PR cuts the problem in half. Closing it out takes two. What It Costs in Practice At the Lambda deployment limit of 250 MB, ten concurrent archives push peak heap to roughly 5 GB -- well past most CI runner allocations. There are workarounds, each with a price tag. Dial parallelism down, and you trade throughput for memory. Spin up beefier CI runners, and you trade dollars for memory. Both get you unstuck, but neither addresses the root cause. The PR is up at https://github.com/hashicorp/terraform-provider-archive/pull/501. The fix is under ten lines of Go, so the investigation took much longer than the implementation. Some design choices age well, but some scale with your infrastructure.

When putting their model into production, every team or organization encounters the same issue. Failures go unnoticed for days at first because there is no monitoring. As teams begin to fix the issues, they identify areas where production results deviate from the training data, create dashboards for every metric, and set alerts for every threshold. This results in engineers being paged at two in the morning for a bug that fixes itself within an hour, and when an important alert arises, it goes unanswered due to alert fatigue, creating a pipeline that silently feeds garbage into the model. When a team learns to disregard 95% of the issues, they are very likely to disregard the remaining 5% that are actually important, and the solution to this isn’t less monitoring. The good solution to this problem is monitoring, which is tiered, routed, and pruned differently from the infrastructure monitoring that most teams already know. The Problem With Applying Old Monitoring Rules To AI Traditionally, application monitoring used to be binary, which is whether the application or service is up or down, latency is high or low, etc. But AI models don’t fail with these signs; they usually degrade over time. For instance, a recommendation model does not show exceptions when the user behavior shifts; it just silently gets worse at what it was supposed to do. A classifier model does not throw an error when its input distribution changes; it just returns answers confidently with increasingly wrong predictions. An AI application does not crash when it hallucinates; instead, it returns a normal HTTP 200 response with incorrect content. This creates two problems: When AI models fail, the reason for failure is invisible to classical infrastructure monitoring, which causes teams to bolt on multiple checks like data quality checks, drift detectors, and output scorers, each introducing a new source of noise. AI models are statistical in behavior and not deterministic, so setting threshold alerts on them leads to them firing constantly, and training teams have to tune the model. As a result, thorough AI monitoring does not make the application safer; beyond a certain point, it only makes things worse. What to Actually Monitor Monitoring issues that no one will ever take action on is often the first step towards alert fatigue. It is useful to consider it in four layers, each with its own owner and mode of failure. Infrastructure and service: Metrics like inference latency, throughput, Graphics Processing Unit (GPU)/Central Processing Unit (CPU) utilization, error rates, and cost per request and token consumption for anything calling a hosted large language model (LLM) API are classic operational metrics and can usually be monitored with the existing Application Performance Monitoring (APM) tools. Data quality: This is another important thing to keep an eye on because it can cause broken feature pipelines, upstream schema changes, input formats being changed without getting noticed, and null-rate spikes. These are usually the worst failures because you can't see them unless you're looking for them, and the model keeps making predictions based on bad data. Model quality: This can be tracked by looking at changes in the Confidence Score or how much the prediction distribution has changed from what was seen during training. This can be used instead of measuring accuracy because it's hard to tell right away how measures like accuracy are calibrating, because to measure accuracy, you would have to compare the predicted result to the actual correct answer, which doesn't always exist at the time of prediction. Generative artificial intelligence/large language model quality: Metrics like hallucination rate, coherence, factual grounding, toxicity, and susceptibility to prompt injection need different types of tooling to identify them because they are not like traditional metrics and would require human-in-the-loop sampling or an LLM as a judge for identifying them. The mistake many teams make is that they apply the same alerting techniques to all four layers, which is the infrastructure one, as that is the traditional way of setting up monitoring for applications, but issues related to data quality and model quality require a trend-based review. How to Alert Without the Noise Replace static thresholds with adaptive baselines. When systems learn a baseline from historical behavior and trigger alerts on deviations from it, like “alert if latency exceeds 200ms,” this ignores the daily and weekly traffic patterns, and the same is valid for data volume and null rates, which leads to a large number of false alarms being raised. So, teams that have made this switch from static thresholds to adaptive baselines have reportedly reduced noisy alerts by 60–90%. Introduce real severity tiers. When an alert is critical and poses an instant business risk, it is sent to an on-call engineer so that the problem can be fixed right away. Warnings about poor performance that are not critical are sent to a Teams chat channel during business hours, and signals about long-term trends land on the dashboard to be looked at from time to time. This helps to make sure that the notification's urgency matches its real urgency. Correlate and deduplicate before notifying. One change to the schema upstream can cause a dozen problems downstream. Sending a dozen alerts for one root cause either makes the team too busy or forces them to mentally group alerts together, which your tools should be doing for you. Route alerts to whoever can act on them. Misrouting is a common cause of tiredness. If the central platform team doesn't know about the business, they might ignore a spike they can't understand, and the domain team that would be able to understand it would never see the alert. Both problems are solved by linking alerts to the right person by domain, based on where the problem starts. Prioritize by business impact. A system that looks for unusual events handles all alerts the same way because it doesn't know which parts of your system are important to the business. When you think about how important each problem is before choosing how loud to alert, you get a lot fewer alerts overall, and a lot more of them are ones that you should actually act on. Conclusion It's important to understand that all of the ideas we've talked about work together; none of them can be used on their own. For example, adaptive thresholds only give out fewer alerts that aren't differentiated by severity. Without proper routing, severity tiers send the wrong messages about how important something is to the incorrect individuals. To avoid alert fatigue, teams need to take comprehensive actions, which include proper alert designs and organizational practices. They should also ensure that every alert can be acted on, which is better than monitoring everything, because AI monitoring only scales, and not having anyone see a model fail could have serious consequences. Good monitoring means building a system that sends alerts only when it matters, so when it does, people actually act on it.

“...premature optimization is the root of all evil…” Donald Ervin Knuth Introduction "Premature optimization is the root of all evil." Most software engineers know this, attributed to Donald Knuth, author of The Art of Computer Programming and one of the most influential figures in computer science. Many have also picked up the practical conclusion that followed: "let's make it work first, fix performance later." After all, it's easier to add another EC2 instance than to find the root cause. But here is what Knuth actually wrote: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." A little different, isn't it? The second sentence is almost never quoted — and that is convenient, because it turns a careful statement into a simple excuse. Sometimes for laziness. Sometimes because people assume that optimization means sacrificing readability: cryptic bit manipulation, obscure tricks, code that only the author understands at 2 am. I believe Knuth was indeed warning against that kind of optimization. But that assumption is wrong more often than people think. Good, clean code is frequently efficient code too — not by accident, but because choosing the right tool for the job tends to be both clearer and faster. The examples in this article are proof of that. Scope This article focuses on simple, cheap, and foolproof tips that can be applied universally — regardless of your architecture, framework, or domain. In my experience, they carry virtually no risk of making things worse. Architecture, design, networking, database connectivity, threading — these are deliberately out of scope. Not because they are unimportant, but because they are context-dependent. The right answer depends on your specific system, and each of these topics deserves its own article. Examples String Operations We are all familiar with built-in JDK string utilities like: equals(), startsWith(), endsWith(), contains(): Java s1.equals(s2); s1.startsWith(s2); s1.endsWith(s2); s1.contains(s2); Unfortunately, JDK provides only one function for case-insensitive comparison: Java s1.equalsIgnoreCase(s2) There are no functions for case-insensitive startsWith(), endsWith(), contains(). So, often we combine toLowerCase() or toUppserCase() with startsWith(), endsWith(), contains(): Java s1.toLowerCase().startsWith(s2.toLowerCase()); s1.toLowerCase().endsWith(s2.toLowerCase()); s1.toLowerCase().contains(s2.toLowerCase()); A little verbose and null-prone, but just fine if not on the critical path. However, this technique might cause some performance problems. Do not forget that String is an immutable class, so instead of just a char-to-char comparison between two strings, we create two additional strings that then must be garbage-collected. Considering that String is a wrapper over a char array, the memory allocation may become expensive. The solution is to use case-insensitive utilities provided by different libraries, e.g., Apache Lang3: Java startsWithIgnoreCase(s1, s2); endsWithIgnoreCase(s1, s2); containsIgnoreCase(s1, s2); Or, starting from version 3.18.0: Java Strings.CI.startsWith(s1, s2); Strings.CS.startsWith(s1, s2); Where CI exposes case-insensitive and CS — case-sensitive utilities. Many people like regular expressions and use java.util.Pattern class sometimes, not where it is really necessary. For example: Java Pattern.compile("^prefix.+suffix$").matcher(s).find() Instead of: Java s.startsWith("prefix") && s.endsWith("suffix") Or even: Java Pattern.compile("^prefix").matcher(s).find() instead of s.startsWith("prefix") Pattern.compile("suffix$").matcher(s).find() instead of s.endsWith("suffix") Pattern matching is significantly slower than trivial substring matching. The following table shows evaluation time for 1 million operations: Operation * 1 million times Time, ms s.equals("hello") 7 s.startsWith("hello") 6 s.endsWith("hello") 11 s.contains("hello") 24 s.toUpperCase().startsWith("HELLO") 65 s.equalsIgnoreCase("hello") 5 Pattern.compile("hello").matcher(s).find() 238 pattern.matcher(s).find() 31 What can we see from this table? Performance of equals() and startsWith() is similarendsWith() is 2 times more expensivecontains() is 4 times more expensive than equalsChanging case followed by startsWith() is 10 times (!) more expensiveCase-insensitive comparison functions do not have any performance penaltiesSearching for a substring using a precompiled pattern is about 20% more expensive than using a plain contains() method. Compiling the pattern and using it is almost 10 times more expensive than the plain contains() method. So next time you reach for Pattern.compile(), it is worth pausing for a second: is regex actually needed here, or is a plain string method both simpler and faster? If you really need a pattern, at least compile it in advance — better yet, declare it as a private static final class member. Collections Let’s assume that we want to know whether a given list contains the specific element: Java list.contains("red"); In fact, this call invokes code like this: Java int n = list.size(); for (int i = 0; i < n; i++) { if ("red".equals(list.get(i))) { return true; } } Starting from Java 8, we have a streaming API that just hides from us the same gory details: Java list.stream().anyMatch("red"::equals); This is perfectly fine when the list is short, changes frequently, or is searched only occasionally. But if the list is large, stable, and searched repeatedly, a HashSet is the right tool — offering average O(1) lookup instead of O(n). If you cannot change the original data structure, converting it once at initialization time and searching the Set from that point forward is almost always worth it. If both the guaranteed element order and the fast lookup are needed, we can either hold duplicated data structures — a list for ordering and a set for search or just use LinkedHashSet, which solves both problems. Another common case is case-insensitive search. We already saw above that the combination of toLowerCase() or toUpperCase() with comparison significantly reduces the performance. This can be solved by using TreeSet with custom comparator, e.g. String.CASE_INSENSITIVE_ORDER: Java Set<String> set = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); This gives you a sorted, case-insensitive set with no extra allocations - and the same approach works for TreeMap when your data is key-value pairs. Enum Lookups Everyone knows that an enum entry can be found by its name using a built-in method valueOf(s). However, what to do if the given string is lowercase while enum entries following the naming convention are called using capital letters? Some people use a combination of toUpperCase() and valueOf() that work just fine but have the penalty we discussed above. However, very often people prefer to create a special field representing a “custom” name, so the simple enum like: Java enum Color { RED, GREEN, BLUE } Turns into: Java enum Color { RED("red"), GREEN("green"), BLUE("blue"), … } Let’s mention that this design has at least two disadvantages: Duplicate data: The custom name is the same as a built-in but in a different case, which can be solved much more easily. This allows using really custom names that, according to my experience, in most cases are not needed and just create so-called “edge cases” that, in turn, in most cases are just a signal of bad design and might cause a lot of “stupid” bugs. However, let’s continue. How do people often use this custom name? Java public static Color ofColor(String color) { return Arrays.stream(values()) .filter(c -> c.color.equals(color)) .findFirst() .orElseThrow(() -> new IllegalArgumentException("No enum constant %s.%s".formatted(Color.class.getName(), color))); } The implementation looks pretty nice, but this approach means that each call of ofColor() iterates over the list. Yes, in most cases enums are not huge, so the list is short, but anyway, why do this if we can just create a map from the custom name to the enum entry once during initialization and then use it with O(1) complexity? The following example solves both problems at once: it uses a case-insensitive map where the key is the standard name() of the enum entry during initialization: Java private static final Map<String, Color> colors = Arrays.stream(values()).collect(toMap(Enum::name, e -> e, (existing, replacement) -> replacement, () -> new TreeMap<>(CASE_INSENSITIVE_ORDER))); So, now the method ofColor() becomes trivial: Java public static Color ofColor(String color) { return Optional.ofNullable(colors.get(color)) .orElseThrow(() -> new IllegalArgumentException("No enum constant for " + color)); } One can argue that a map-based implementation is not always possible because sometimes the lookup criteria are too complex to be reduced to a simple key. Although I agree in general, I can say in turn that in many (if not in most) cases this is still possible. So far, the lookup key was a simple string. But what if the search criteria is a range rather than an exact value? Consider a more physically accurate model of colors as ranges of electromagnetic waves. Java public enum Color { BLUE(450, 495), GREEN(495, 570), RED(620, 750); …} How to implement the method ofWaveLength(int waveLength)? The straight-forward way is to iterate over the values of the enum and compare the given wave length with the range for each entry, i.e. implement O(n) search. But we can do better using NavigableMap, which is designed exactly for this kind of range query: Java private static final NavigableMap<Integer, Color> wavelengthMap = Arrays.stream(values()) .collect(Collectors.toMap( color -> color.minNm, color -> color, (existing, replacement) -> existing, TreeMap::new )); Unfortunately, the search method is not as trivial as in the previous example, but still very simple and fast: Java public static Color ofWaveLength(int nm) { return Optional.ofNullable(wavelengthMap.floorEntry(nm)) .map(Entry::getValue) .filter(value -> nm <= value.maxNm) .orElseThrow(() -> new IllegalArgumentException("No enum constant for wavelength: " + nm + " nm")); } Now, let’s compare the performance. Operation * 1 million times Time, ms valueOf(s) 34 valueOf(toUpperCase(s)) 78 Iteration with equals() 40 Color.ofColor() iteration 166 Color.ofColor() map 20 Color.ofWaveLength() map 32 The table shows that: As expected, toUpperCase() reduces performance twiceIteration with call of equals is a little bit more expensive than valueOf() although the enum has only three members and will grow linearly as the enum grows. The more members enum has, the more time iteration takes. Map-based implementation is even faster than one based on the built-in valueOf(). Stream-based iteration (ofColor() iteration) is surprisingly slow. Stream setup overhead (boxing, lambda dispatch, spliterator initialization) is non-trivial for tiny collections Pre-Intitialization The principle here is: do not do something several times if you can do it once. The most trivial example is string or numeric constants: Java private static final String FILE_NAME = "config.json"; private static final int MAX_VALUE = 10_000; However, the same principle applies to heavier objects — and that is where it really matters. Let’s take a look at logging. Most people are used to writing the following “magic” line at the beginning of each class (unless we use Lombok’s @Slf4j annotation): Java private static final Logger logger = LoggerFactory.getLogger(MyClass.class); Are all these modifiers (private static final) really needed? Some people try to save typing time: Java private final Logger logger = LoggerFactory.getLogger(MyClass.class); Moreover, if the logger is not static, we can do even more: Java private final Logger logger = LoggerFactory.getLogger(getClass()); This line looks better because it is error-proof: the class here is not hard-coded, so this line can be copied as-is from one class to another or inherited from the base class. So, what’s the problem? The problem is that retrieving the correct logger is potentially expensive due to synchronized registry lookups. Doing this on every instantiation adds up. A friend of mine told me that once in the company where he worked, this change in some critical path improved performance so much that they managed to reduce the AWS cluster by about one hundred large EC2 machines. The same rule applies to pattern compilation. As the benchmark table showed, compiling a pattern on every method call is nearly ten times slower than reusing a precompiled one. The result of Pattern.compile() should always be stored in a static final field. The only exception is the case when the regular expression is generated dynamically, but we should do our best to avoid such a design. Very often we have to format or parse dates. Traditionally I used SimpleDateFormat. What can be more obvious than this: Java private static final String FORMAT = "yyyy-MM-dd HH:mm:ss"; private static final DateFormat format = new SimpleDateFormat(FORMAT); Frankly speaking, I did this many times following the principle I stated above: there is no reason to create the instance every time we need it if we can create it only once. The problem is that SimpleDateFormat is not thread-safe, so sharing the same instance among different threads can cause the problem. Even worse: we can live with this bug for years without knowing about it, since it only happens under high load and in some cases can just produce slightly wrong results that can be lost in an ocean of valid data. So, should we create instances of SimpleDateFormat every time we need it and cause CPU and GC to work hard? Fortunately, starting from Java 8, we can use DateTimeFormatter instead: Java private static final DateTimeFormatter formatter = DateTimeFormatter.ofPattern(DATE_FORMAT); This class is thread-safe, so we can share its instance among different threads and get consistent results. Conclusion We started with a quote that is almost always cited incomplete. Knuth never said ignore performance — he said don't sacrifice clarity for speculative gains, while reminding us not to pass up opportunities in that critical 3%. The examples in this article live in that 3%. None of the performance issues described here should ever appear in production code. They are not hard to avoid — they require no profiler, no benchmarking framework, no architectural discussion. Just the habit of reaching for the right tool. And that habit pays off. Choosing equalsIgnoreCase() over toLowerCase().equals() is cleaner and faster. A static final logger is simpler and cheaper. A pre-built enum map is more readable and O(1). Good code and efficient code are not in conflict here — they are the same code. The only thing required is the habit of pausing for a second and asking: am I doing this n times when once would do? All code examples from this article are available on Gist.

We have spent the last two years learning how to ground a single AI agent in enterprise data. That was the easy part. Coordinating a fleet of them turns out to be a different problem entirely. Multi-agent systems ask questions our current platforms weren't built to answer. How do two agents share state without contradicting each other? Whose credentials are used when Agent A calls Agent B? What audits the decision when an agent triggers another based on a probabilistic inference? Most enterprise architectures shrug at all of these. They were built for humans reading dashboards, not autonomous consumers acting on inference. The result is a quiet architectural crisis. I see multi-agent pilots pass demo review and then fall over the moment they meet real production traffic. It's rarely the model. It's that the system has nowhere to govern reasoning itself. I've come to believe a new architectural layer is emerging as the answer. I call it the Reasoning Control Plane. It sits alongside the data, application, and security planes every enterprise architect already knows. It governs how autonomous agents share context, authenticate to each other, expose their decisions to observation, and behave when the stakes are high. Every previous era of enterprise architecture eventually produced a new plane when a new class of consumer showed up. Agents are that new class, and the plane hasn't been named yet. The Planes We Already Know All mature system architectures that have shipped in the last thirty years are organized into control planes and data planes. A control plane governs. A data plane executes. The pattern is so ubiquitous by now that architects reach for it reflexively when a new domain needs structure. Zoom out, and enterprise architecture runs on three planes: The Data Plane governs how information is stored, moved, and queried. Data warehouses, Lakehouses, Streaming Data.The Application Plane governs how code executes and services communicate. APIs, Orchestrators, Workflow engines.The Security plane governs identity, access, and audit trail. IdPs, Policy engines, SIEMs. Each of these assumes a specific kind of consumer. A human as the end-user. An application making deterministic calls. A user authenticating to a resource. Autonomous agents fit none of those assumptions cleanly. An agent needs to consume the data plane for grounding, invoke the application plane for effects, and satisfy the security plane's policies. Fine, we can wire that up. But the reasoning that an agent does across those three planes has no home. When one agent triggers another based on a probabilistic decision, what governs that? When two agents share a "customer" concept, what enforces that they mean the same thing? When an agent takes a regulated action, what audits the rationale? There's no plane for that. Not yet. Introducing the Reasoning Control Plane The Reasoning Control Plane is the architectural layer that governs how autonomous reasoning gets coordinated, constrained, and observed across an enterprise's agentic systems. It's not about where the inference happens. Models can be anywhere. It's where the enterprise expresses what reasoning is permitted, how it's grounded, how it's audited, and what happens when it fails. Position it above the traditional three planes. It consumes services from all of them: the data plane for grounding, the application plane for effectors, the security plane for identity. But it exposes new primitives that none of the older planes provided on their own. Those primitives are what agents actually need to work together: A shared semantic context so agents mean the same thing when they say "customer" or "at risk"Agent-to-agent access controls so one agent's actions stay bounded when it delegates to anotherObservability of non-deterministic workflows so decisions can be reconstructed after the factDeterministic guardrails on actions that must never be free-planned If you've built a multi-agent pilot that worked once and then failed inconsistently on a second run, one of these four is missing. The Reasoning Control Plane is where they belong together. The Reasoning Control Plane governs shared context, delegated authority, decision evidence, and high-stakes actions across enterprise agent systems. Dimension 1: Shared Semantic Context Multi-agent systems break down first at the level of shared meaning. Agent A's understanding of "the customer" isn't Agent B's. Agent A's definition of "at risk" was trained against the churn model. Agent B's was defined against the credit model. When they collaborate, they compound the ambiguity, and nobody notices until an action lands in the wrong place. Structured semantic layers have existed for years in the analytics world. They exposed shared metrics and dimensions to BI tools, so "revenue" meant the same thing across every dashboard. The Reasoning Control Plane needs the same thing, but built for agents instead of humans. Machine-first, so it returns schemas and structured concepts, not charts. Composable, so agents can assemble context on the fly. Versioned, so an agent can tell which definition of "at risk" it's operating against. If your multi-agent design has no shared semantic surface, every agent redefines the world for itself. That works for one agent. It doesn't survive the second. Dimension 2: Agent-to-Agent Access Controls Traditional identity and access management assumed one human authenticating to one system. Agent-to-agent access breaks that model. When Agent A delegates to Agent B, whose credentials are used? Whose scope? What happens when Agent B invokes Agent C on the same request? Most current implementations answer this the wrong way. They give every agent a service account with broad permissions and hope for the best. That works until an agent hallucinates a request outside its intended scope. Then it works catastrophically well, because the service account executes the mistake with full authority. The Reasoning Control Plane needs a different primitive. Scoped, delegable, time-bounded authorization that follows the reasoning chain. When Agent A delegates to Agent B, the token B receives should be narrower than A's own. Bounded to the specific task. Expiring quickly. Auditable back to the originating human intent. None of this is new in identity engineering. OAuth's scoped tokens and step-up authentication are close analogs. What's new is applying the same rigor at the agent boundary, treating every delegation as a potential blast-radius event and constraining it accordingly. I keep asking why we don't have this yet in mainstream agent frameworks. The honest answer, I think, is that the frameworks were built by ML engineers, not identity engineers. The two worlds haven't merged. They will, but it's going to take another year of production incidents to force the marriage. Dimension 3: Observability for Non-Deterministic Workflows Traditional application performance monitoring made an assumption that's dead for agentic systems. Same input, same code path. Two runs of the same agent against the same input can now produce different plans, different tool calls, different outcomes. That doesn't mean the system is unobservable. It means observability itself has to be redesigned from the ground up. The Reasoning Control Plane needs to capture what traditional APM never did. The plan the agent chose. The context it considered. The tools it invoked. The confidence it expressed at each step. The alternatives it rejected. This isn't a superset of tracing. It's a different discipline. It looks less like OpenTelemetry spans and more like a per-request, per-agent decision journal that lets an operator reconstruct what happened after the fact and, more importantly, generalize from patterns of failure. Here's the thing I've learned from every incident review I've done in this space: your multi-agent system will act unexpectedly, and you'll want to know why. If you didn't build the plane's observability from day one, you can't answer the question. You can guess. You can't answer. At one Enterprise I worked with, a sales agent drifted its discount recommendations 8% to 10% higher than policy over 2 weeks. Every discount had passed the workflow's guardrails individually. But because we had built decision-level observability from Day 1, we could replay everything the agent had reached for: the retrieved comparables, the sample deals, the confidence scores. Within few hours, we traced the drift to a promotional campaign from two Quarters back still in the retrieval index. The same instrumentation has since caught two other drifts before they reached revenue. Bolting observability on after the first incident doesn't work either. The information you need was in the model's context at inference time. Once that request is done, the context is gone. If you didn't capture it, you can't recover it. The plane has to instrument this from day one. Dimension 4: Deterministic Guardrails for High-Stakes Actions The last dimension is the recognition that not every step of an agent's workflow should be reasoned about. Some steps have to be scripted. Bolted down. Refusing to change based on anything the model has to say. Take an agent that helps close a sales deal. Recommending discount tiers? Fine, reason about it. Actually applying the discount to a signed contract? That has to be deterministic. Policy-bounded. Approval-gated. Executed by code that never asks a model what to do. This is where many current agent frameworks fall short, and I'll be blunt about it. They give you the tools to let an agent do anything and expect you to constrain it in the prompt. That isn't architecture. That's hope. A real guardrail lives outside the model's context. As code. As a policy engine. As a circuit breaker. It is impervious to prompt injection and model drift. If the model can see it, the guardrail is negotiable, and negotiable guardrails aren't guardrails at all. I've never seen a production multi-agent system survive without this discipline. Every one that tried to constrain behavior purely in the prompt ended up with an incident within six months. That may sound harsh, but the pattern is remarkably consistent. Deterministic guardrails are the architectural expression of a simple principle: reasoning proposes, policy disposes. The Reasoning Control Plane declares up front which actions belong to reasoning and which belong to policy. The guardrail layer is where you enforce the split. Where Multi-Agent Designs Break Down Nearly every failed multi-agent pilot I've reviewed traces to one of these four dimensions being absent or half-built. No shared semantic context produces coherent-sounding but internally contradictory outputs. No scoped access controls produce security incidents. No decision observability produces mysteries that never get diagnosed. No deterministic guardrails produce compliance events. The Reasoning Control Plane's diagnostic value is that each dimension can be scored independently. Ready, partial, or absent. The weakest dimension caps what the system can safely do. You inherit your worst dimension, not your average, and no amount of investment in the other three lifts the ceiling. That's the single most important thing to internalize about multi-agent architecture. What to Instrument First Architects who buy this framing usually ask which dimension to build first. The right answer depends on where you are, but the sequence I've seen work is: semantic context, then observability, then access controls, then guardrails. Reasoning Control Plane in sequence: semantic context, observability, access control, and guardrails Semantic context is first because it unblocks everything else. Without it, no other layer has a stable substrate to reason about. Observability is second because you can't improve what you can't see. Every subsequent design decision gets easier when you can trace real behavior. Access controls come third because they contain blast radius as autonomy grows. Guardrails come fourth because they're the most application-specific. The right ones depend on knowing your regulated actions, and you rarely fully know those until you've shipped a pilot. The Reasoning Control Plane isn't a product you buy. It's a discipline you adopt, layered across the data, application, and security planes you already run. No single vendor will market it as a coherent category for another year or two. But it's emerging as the architectural piece that separates multi-agent systems that survive from the ones that quietly break. The organizations that recognize this now will build the infrastructure their agents actually need. The rest will keep debugging demos in production, wondering why the model is the problem when it never really was.

Key Takeaways In regulated industries, cloud migration success is determined less by technology selection and more by how deliberately you decouple risk vectors — compliance risk, organizational hesitation, user adoption gaps, and integration changes — so no single failure can derail the whole program.You can successfully migrate an application to AWS while keeping data on-premises by routing through a REST API abstraction (e.g., IBM’s DB2 REST API layer) paired with dedicated AWS security groups controlling cloud-to-on-prem traffic, allowing the data migration to proceed on its own compliance and trust-building timeline.The most dangerous compliance gap in regulated applications isn’t declared sensitive fields — it’s free-form text fields where users may inadvertently type SSNs, credit cards, or other regulated identifiers; proactive tokenization in the application’s write path closes this gap before any audit finds it.Long-tenured business users carry a decade of UX muscle memory that QA testing cannot replicate; allocating real production validation time (such as a 15-day dark deployment cohort) is essential when migrating systems users have relied on daily for 10+ years.Before starting a regulated cloud migration, ask which risk vector each architectural decision is decoupling and whether your team is aligned on why — this single question reframes "cloud migration" from a technology project into a coordinated risk-management exercise. Introduction Most published writing on legacy-to-cloud migration treats it as a technical exercise: pick the stack, plan the cutover, flip the switch. In regulated industries, that framing fails — and the failure mode isn’t a missed deployment window. It’s a stalled program, a failed compliance audit, or a client who pulls back from the cloud strategy entirely. A cloud migration in healthcare insurance is as much about regulatory risk management, organizational trust-building, and user adoption as it is about microservices and Fargate. Get the technology right and miss the risk choreography, and the project doesn’t ship. I led the first WebSphere-to-AWS migration in the health division of a Fortune 50 insurer — a multi-year program touching PHI data, long-tenured business partners, and downstream services concurrently migrating to the cloud. Over that program, six architectural patterns emerged as decisive. Not for the technology they enabled, but for the risks they made manageable. None are individually novel. What’s distinctive is how they work together — as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. Pattern 1: Strangler Fig With Dark Deployment When migrating critical production systems to the cloud, the temptation is a hard cutover — flip the switch at 2 AM on a Sunday and hope for the best. We chose a different path: a 15-day dark deployment on AWS production, accessible only to a designated cohort of business partners. Three factors drove this decision. 1. First-mover risk in the department. This was the first WAS-to-AWS migration in this Fortune 50 insurer’s health division. There was no internal precedent to draw from — no playbook, no lessons learned from a prior AWS rollout. A "big bang" cutover would have exposed our full user base to whatever unknowns we hadn’t anticipated. Dark deployment let us pioneer the path with limited blast radius. 2. Regulatory exposure on PHI data. The application processes Protected Health Information. Any data integrity issue — a missed field, a misformatted record, a sync gap — could have triggered regulatory scrutiny. By exposing the new AWS environment to a small group of business partners first, we could validate end-to-end data flow in real production conditions without putting the full user base or compliance posture at risk. 3. UX learning curve. We had explicitly rejected a lift-and-shift approach. The new application wasn’t just re-hosted — the UI had been redesigned, the APIs restructured, and user workflows updated. Even excellent technical execution couldn’t eliminate the learning curve our users would face. Dark deployment gave us 15 days of real-world UX observation: where do users hesitate, what do they misunderstand, which workflows feel awkward? By the time we cut over publicly, we had already addressed the rough edges. The result: When we replaced the WAS production URL with the AWS production URL, end users perceived the change as a routine UI update, not a foundational technology migration. Pattern 2: Decouple Application Migration From Data Migration The default assumption in cloud migration is that application and data should move together. We made the opposite choice: migrate the application to AWS while keeping the underlying DB2 data on-premises. Three factors made this the right call. 1. PHI/HIPAA compliance complexity. The application processes Protected Health Information governed by HIPAA. Moving regulated healthcare data to a new environment raises a long list of compliance questions — encryption-at-rest configurations, audit logging, access control policies, business associate agreements with the cloud provider, breach notification readiness. None of these are insurmountable, but they take months of compliance review. Treating data migration as a separate workstream with its own compliance approval cycle was significantly less risky than bundling it into the application cutover. 2. Client comfort and trust-building. Cloud migration is as much a psychological transition for the client as a technical one. Moving an application to AWS is one decision; moving sensitive data off the client’s own infrastructure is a much larger one — it changes their security perimeter, their incident response posture, and in some cases their regulatory filings. Insisting on moving both at once would have either delayed the program waiting for full executive comfort, or risked a "no" on the entire initiative. Application-first let us demonstrate the new architecture working successfully before the data migration conversation began. 3. Parallel team enablement. Decoupling created room for a separate analytics team to independently assess which data could move to the cloud, on what timeline, and under what compliance framework. The application architecture was designed from day one to support a hybrid future — partial data on AWS, other data on-prem — so the analytics team’s work didn’t block application progress. How the technical decoupling works. The natural temptation when keeping data on-prem is to expose a direct database connection from the AWS application back to the on-prem DB2 instance. We rejected that — opening database ports across the cloud-to-on-prem boundary is a security liability, a latency problem, and a fragile dependency. Instead, we used IBM’s DB2 REST API layer to expose data access through authenticated HTTPS-based service calls. The AWS application talks to data through an API, not a database connection. This abstraction also positions the application to seamlessly switch to AWS-resident data later, without any application code change — only the API endpoint moves. Network-layer security follows the same decoupling principle. We provisioned dedicated AWS security groups on the Fargate side specifically for the IMS and DB2 connections back to the on-premises environment — only requests from those approved security groups can traverse the firewall to the on-prem data tier. Combined with the REST API abstraction, this gives us both application-layer (authenticated HTTPS) and network-layer (security-group-controlled) protection across the cloud-to-on-prem boundary. The result: A successful cloud migration with regulatory exposure isolated to a single workstream, and a forward path that doesn’t force the client into uncomfortable decisions before they’re ready. Pattern 3: EJB Monolith → Containerized Microservices on Fargate The original application was a Java EJB monolith running on WebSphere. The "lift-and-shift" temptation would have been to containerize the existing EJB code as-is into AWS Fargate — preserving the architecture, just moving the deployment substrate. We rejected that and instead decomposed the monolith into bounded REST microservices. Three reasons drove this decision. 1. Downstream services were also migrating. The application integrated with 5–7 SOAP-based services owned by adjacent teams — agreement service, customer service, sensitive data masking, and others. Those teams were simultaneously migrating their own services from WAS to AWS, which meant interface contracts, protocols, and endpoints would inevitably change. Inside an EJB monolith, every downstream integration change forces a recompile-redeploy-retest cycle of the entire application. Inside microservices, only the integration adapter for the affected service needs to change. With multiple active migration interfaces, the flexibility difference compounds quickly. 2. EJB development velocity is structurally slow. Even routine changes to EJB code require a full WAR/EAR build, redeployment to the WAS instance, and a heavy test cycle. The technology wasn’t designed for the iteration speed we needed to support a multi-year migration alongside actively changing downstream dependencies. Microservices on Fargate gave us a development model — fast container builds, independent deployments, isolated test environments — that matched the pace of the work. 3. Future data migration optionality. As noted in Pattern 2, the underlying data was kept on-premises for now, but a phased data migration to AWS was planned. By isolating database calls and IMS calls into dedicated microservices, the change required when the data eventually moves is localized — swap one service’s data access logic rather than reworking the monolith. The architecture is positioned for the data move whenever the client is ready. How we sized the decomposition. The boundaries followed natural integration points: each external SOAP integration became its own bounded microservice with a thin REST API. Data access calls (DB2 via REST, IMS) were isolated into dedicated services. The frontend talks to a coordination layer that orchestrates calls across these services. The result was a clean set of containerized microservices on AWS Fargate — each independently deployable, scalable, and testable. The result: A modernization that didn’t just relocate the code, but restructured it to absorb the inevitable changes coming from adjacent migrations across the organization — without recompile-redeploy-retest pain. Pattern 4: Frontend Decoupling via S3 + CloudFront The original WAS application followed the classic tightly-coupled pattern: JSP pages rendered server-side, deployed alongside the backend, scaling and updating as one unit. We made an architectural break in the migration — the frontend became a fully independent single-page React application hosted on Amazon S3 and served via CloudFront. Three factors made this the right call. 1. Independent deployment cadence. Frontend and backend evolve at different speeds. UI tweaks — copy changes, validation logic, visual updates — are frequent and low-risk. Backend API changes are slower and require careful coordination with downstream service migrations. Decoupling them means UI changes can be deployed instantly through a separate UI pipeline (different Git repository, different infrastructure, different release cadence) without touching the backend microservices. A small label change no longer requires a full backend deployment. 2. Adopting an accessibility-first enterprise UI library. Alongside our migration, an internal innovation track was building a shared component library to unify UX patterns across the organization’s applications — consistent typography, controls, brand elements, and critically, accessibility as a first-class concern: full screen reader support, keyboard navigation, sufficient color contrast, and ARIA-compliant semantics. JSP-based legacy pages couldn’t meaningfully integrate this kind of library. By rebuilding the frontend as a React single-page application, we adopted the library fully — and incorporated rigorous accessibility testing into every release cycle. Users who rely on assistive technologies (screen readers, alternative input devices, magnification) get full application access. For an application processing PHI in a regulated industry, this proactive accessibility-first approach is itself a substantial improvement over the legacy app. 3. Global performance through edge caching. S3 alone would have served the static assets, but we layered CloudFront on top to push content to edge locations closer to users. Business partners access the application from different geographic regions; CloudFront cuts load times by serving cached assets from the nearest edge, not the S3 origin in a single AWS region. This is a substantial UX improvement that simply wasn’t possible with WAS-hosted JSPs. How the architecture flows. User requests hit CloudFront, which serves cached React bundles, HTML shells, and static assets from the nearest edge. The React application then makes authenticated REST API calls back to the backend microservices on AWS Fargate. The frontend has no awareness of which microservice serves any particular request — it talks to a coordination API layer that handles orchestration. The result: A UI architecture that’s faster (edge-cached), cheaper (no application servers for the frontend), easier to update (independent pipeline), more inclusive (accessibility-first), and aligned with the broader enterprise UX modernization effort. Pattern 5: Business Partner Real-Production Validation Cohort Pattern 1 described the deployment mechanism — a 15-day dark deployment exposing AWS production to a limited cohort. Pattern 5 is about who was in that cohort and why we deliberately chose real business partners over our QA team for production validation. Two factors shaped this decision. 1. Decades of muscle memory in the existing UX. Our business partners — long-tenured users of the application — had been using the legacy UI for 10–15 years. They knew every workflow, every shortcut, every quirk. The new React application introduced not just a new visual style but new patterns from the organization’s modern component library. Even with rigorous accessibility and usability testing in QA, a brand-new UI in front of users with a decade of habits guaranteed friction. The 15-day validation cycle gave those users time to acclimate to the new patterns and surface UX issues that only show up at the speed of real daily work — keyboard shortcuts they used unconsciously, screens they navigated to multiple times an hour, validation logic that affected their flow. QA testers, by definition, don’t have that muscle memory. 2. First-of-its-kind migration with concurrent change. This was the first WAS-to-AWS migration in the health division, and we’d simultaneously re-architected the UI, the API layer, and incorporated changes from downstream services that were also mid-migration. With that many concurrent changes, even thorough QA can’t realistically simulate the full combinatorial space of real production usage — real customer data, real edge cases, real integration timing, real load patterns. Putting real business partners on the actual AWS production environment for 15 days was our safety net: anything QA missed, the cohort would surface, and we could fix it before broad cutover. Beyond the cohort: maturing the delivery pipeline. A secondary benefit of running an extended validation window was that it gave the engineering team time to mature the CI/CD pipeline alongside the application. By the second application in the migration program, we’d evolved the cohort approach into a full blue/green deployment model on AWS — building organizational learning alongside the application portfolio. The validation pattern isn’t static; it strengthens with each subsequent migration. The result: a validation approach that combined deep domain familiarity (real business partners) with controlled exposure (limited cohort, real production) — catching the issues QA can’t, well before public cutover. Pattern 6: Defensive Tokenization for Sensitive Data in Free-Form Fields In regulated industries, the obvious sensitive data — SSN fields, credit card fields, account number fields — gets protected automatically. The dangerous category is the unstructured data: a free-form text field where a user can type anything. In our application, users entered "health notes" — narrative text describing customer interactions. The risk: nothing in the application schema prevents a user from typing an SSN, a credit card number, a driver’s license, or other regulated identifiers directly into that note. Once stored, that PHI/PII data is sitting in a free-text column with no encryption-at-rest tailored to it, no masking on display, no controlled access — and our compliance posture changes accordingly. We addressed this proactively by integrating an internal sensitive-data-masking service into the application’s write path. Before any free-form text reaches the data layer, the masking service scans the input, identifies regulated identifiers (SSN-pattern strings, credit card numbers via Luhn check, driver’s license formats), and applies tokenization — replacing the identifier with a non-reversible token or masked representation. The original value never lands in the database in plaintext. Three things made this a deliberate architectural pattern, not an afterthought: 1. It was incorporated before the formal risk assessment, not in response to it. Risk assessment was a new exercise for the team — none of us had been through one for AWS-hosted PHI before. Rather than wait for the assessment to flag the free-form field as a finding, we performed our own data classification first, identified the free-form notes as a regulated-data risk vector, and integrated the masking service pre-emptively. When the formal risk assessment ran, this control was already in place. 2. We reused an existing internal service, not built a new one. The masking service already existed in another WAS-hosted application within the broader life/health portfolio. Instead of re-implementing tokenization logic, we adopted the existing service — saving development time and inheriting the existing security review and operational maturity of that service. Migrations are a good moment to identify reusable internal capabilities rather than reinvent them. 3. It addresses a class of risk most compliance reviews don’t anticipate. Compliance checklists focus on declared sensitive fields ("the SSN field," "the account number field"). They rarely interrogate free-form text fields, because those fields aren’t supposed to hold sensitive data. But in practice, users type whatever they need to type — and what they type is what your application stores. Proactive defensive tokenization closes that gap. The result: free-form notes that look normal to users, but whose backend storage is sanitized of any regulated identifiers the user may inadvertently include. The application’s compliance posture is robust to user behavior, not just to user intent. Conclusion: The Through-Line Is Decoupling Looking back across the six patterns, the through-line isn’t any specific technology — it’s a posture: deliberate decoupling of risk vectors so that no single failure, regulatory finding, organizational hesitation, or user adoption gap can derail the whole migration. Pattern 1 (Strangler Fig with Dark Deployment) decouples cutover risk from broader rollout.Pattern 2 (Decouple App from Data) decouples application migration from the data-and-compliance timeline.Pattern 3 (EJB → Microservices) decouples downstream integration changes from our own deployment cadence.Pattern 4 (Frontend on S3/CloudFront) decouples UI release cadence from backend release cadence.Pattern 5 (Business Partner Validation Cohort) decouples real-world UX surprises from public rollout.Pattern 6 (Defensive Tokenization) decouples user behavior risk from data-layer compliance posture. None of these patterns are individually novel. What’s distinctive is choosing them together, as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. The result was a migration that didn’t surprise our compliance team, didn’t surprise our users, and didn’t surprise our auditors — which, in a regulated industry, is the kind of unsexy outcome that defines success. If you’re starting a similar program, the question isn’t which of these patterns to adopt. It’s: which risk vector are you decoupling, and is your team aligned on why?

A scheduled job that needs ninety to one hundred eighty seconds to produce a single output file looks harmless until the day you ship a new build while it is still running. The deployment controller drains the old task and starts a replacement. For a window of two or three minutes, both replicas are alive, both read the same input snapshot, and both intend to write the same logical output. Without idempotent output keying, they write it twice, and the second write has no obligation to agree with the first. Any consumer that reads during that window can pick up state assembled from two different runs. This is not a theoretical race. It shows up in any system where a long-running task publishes to shared storage, and the orchestrator uses rolling deployments, which is to say most production batch pipelines. The failure is quiet. Nothing crashes. Logs show two successful task completions. The corruption lives entirely in the output, and it surfaces later as a downstream decision made on data that never existed as a coherent snapshot. Why Rolling Deployments Break Long-Running Tasks The root cause is a mismatch between two time scales. A rolling deployment is designed around request handlers that finish in milliseconds, so a few seconds of overlap between old and new replicas is invisible. A task that runs for minutes does not fit that assumption. When the controller starts the new replica, the old one is often most of the way through its work, holding partial results in memory and heading toward the same destination key. The orchestrator considers both healthy. It has no concept of the work each task is doing, only of the process lifecycle. Most teams reach first for at-least-once scheduling with a fixed output path. The task computes its result and writes to a known location; the newest write wins. That model is fine when only one task ever runs. Under deployment overlap, it produces last-writer-wins on a destination that two writers reached through different code paths or different partial reads. If the new build changed how a field is aggregated, the surviving file depends on which replica finished last, which is nondeterministic. Distributed locks are the next instinct, and they trade one failure mode for another. A lease in a coordination service such as etcd or ZooKeeper can stop two tasks from writing at once, but a task that holds a lease for three minutes and then suffers a stop-the-world pause or a network partition forces a choice. Either the lease expires and a second task proceeds, which is the exact duplication you wanted to prevent, or the lease is held conservatively, and a crashed task blocks all progress until an operator intervenes. Locks move the problem; they do not remove it. The durable fix does not try to prevent overlap. It makes overlap harmless. Detecting Divergent Writes Before They Reach Downstream Consumers You cannot fix what you cannot see, and duplicate writes are close to invisible by default. On a store that keeps only the latest object, the second write erases the evidence of the first. The first instrumentation step is to turn on object versioning for the output prefix, which costs storage but converts a silent overwrite into an inspectable history. With versioning on, a duplicate write is detectable as more than one version of the same key inside a single scheduled window. That alone is not a defect: an idempotent rewrite of identical bytes is benign. The real signal is divergence: two versions of the same logical output whose checksums differ. The scan below walks every version under a window prefix, groups by key, and reports only keys whose versions carry more than one distinct entity tag (ETag), the marker that two runs produced different bytes for the same window. Plain Text #!/usr/bin/env bash # Scans an object store for duplicate, DIVERGENT writes to the same logical # output window: the signature of two task replicas racing during a deploy. # Works against any S3-compatible store (AWS S3, MinIO, Ceph RGW). It only # reports, so it is safe to run against production. set -euo pipefail BUCKET="${1:?usage: detect_divergence.sh <bucket> <prefix>}" PREFIX="${2:?usage: detect_divergence.sh <bucket> <prefix>}" # Object versioning is what makes a duplicate write visible at all: without it, # the second write silently overwrites the first and you lose the evidence. versions_json="$(aws s3api list-object-versions \ --bucket "$BUCKET" --prefix "$PREFIX" \ --query 'Versions[].{Key:Key,ETag:ETag,Time:LastModified}' \ --output json)" # A key with one version, or several versions sharing an ETag, is benign. A key # with MULTIPLE DISTINCT ETags means two runs produced different bytes for the # same window: a real correctness defect, not a cosmetic duplicate. echo "$versions_json" | jq -r ' group_by(.Key)[] | {key: .[0].Key, etags: ([.[].ETag] | unique), writes: length} | select((.etags | length) > 1) | "DIVERGENT \(.key) writes=\(.writes) payloads=\(.etags | length)"' # Exit non-zero if any divergence was found, so a deploy gate can block. divergent="$(echo "$versions_json" | jq ' [ group_by(.Key)[] | select(([.[].ETag] | unique | length) > 1) ] | length')" echo "scanned prefix=$PREFIX divergent_keys=$divergent" test "$divergent" -eq 0 Run this on a schedule and wire the exit code into a deployment gate. A nonzero result during or just after a rollout is a direct measurement of the bug, not an inference from downstream symptoms. The divergence rate climbs sharply with task duration. A job under thirty seconds rarely overlaps a rollout, while a job in the two- to three-minute range will overlap nearly every deployment that lands during its run. Idempotent Output Keying and Atomic Publish The structural fix has two parts. First, derive the output key from the inputs rather than from wall-clock time or a process identifier. Two replicas working the same scheduled window must compute the same key, so that duplication targets one object instead of two. Second, publish that object atomically, so a reader never sees a partial write and a duplicate publish becomes a no-op rather than a second racing write. Start with the key. Build it from the fields that define the unit of work: the pipeline name, the closed time window being summarized, and a schema version that you bump only when the output format changes. The schema version earns its place during exactly the moment under discussion. A new binary mid-deploy that emits a new format gets a different key, so it does not collide with the old binary's output. Rust use sha2::{Digest, Sha256}; // Two task replicas that pick up the same scheduled window build the SAME // RunSpec. That property is what the whole scheme relies on. #[derive(Clone)] struct RunSpec { pipeline: String, window_start_epoch: u64, // closed window, deterministic per schedule tick window_len_secs: u64, schema_version: u32, // bump only when the OUTPUT FORMAT changes } impl RunSpec { // Content key derived purely from inputs. Identical inputs -> identical key, // which is what lets two overlapping runs target one object, not two. fn output_key(&self) -> String { let mut h = Sha256::new(); h.update(self.pipeline.as_bytes()); h.update(self.window_start_epoch.to_be_bytes()); h.update(self.window_len_secs.to_be_bytes()); h.update(self.schema_version.to_be_bytes()); let digest = h.finalize(); format!("{}/{}/state-{:x}", self.pipeline, self.window_start_epoch, digest) } } The key is content-derived, so identical inputs yield an identical key, and a changed format yields a new one. The second piece is publishing without a destructive overwrite. The pattern that holds up is to write to a unique temporary object, flush it to durable storage, then promote it into the final key with an operation that is atomic at the storage layer. On a single filesystem, that promotion is a rename. On an object store it is a conditional put that fails if the key already exists, or a multipart completion. Rust use std::fs; use std::io::Write; // Atomic publish: write to a unique temp object, fsync, then promote into the // final key with an operation that is atomic at the storage layer. On one // filesystem that is rename(2). On an object store it maps to a conditional // PutObject (If-None-Match) or a multipart completion, NOT a streamed append. fn atomic_publish(key: &str, payload: &[u8], writer_id: &str) -> std::io::Result<bool> { let final_path = store_root().join(key); fs::create_dir_all(final_path.parent().unwrap())?; // Skip-if-exists: a duplicate run that finds the object already there does // no work and produces no second write. Handles the common finish-early case. if final_path.exists() { return Ok(false); } let tmp = store_root().join(format!(".tmp-{}-{}", key.replace('/', "_"), writer_id)); let mut f = fs::File::create(&tmp)?; f.write_all(payload)?; f.sync_all()?; // durable before it becomes visible // Two writers can both pass the exists() check; rename is still atomic, so // the object is whole, and the payloads are byte-identical because the key // is content-derived. It does not matter which one lands. fs::rename(&tmp, &final_path)?; Ok(true) } Skip-if-exists handles the common case where one replica finishes well ahead of the other. The harder case is two writers that both pass the existence check before either commits. Atomicity at the promotion step is what saves you: the object is always whole, and because the key is content-derived, both candidate payloads are byte-identical, so it does not matter which one lands. Readers need one more guarantee. They should never have to guess which key is current. Publish each generation under its own immutable key, then advance a single pointer with a compare-and-swap (CAS), so consumers follow the pointer and always read a complete generation. A losing writer detects the conflict and backs off instead of regressing the pointer to an older or duplicated generation. Rust use std::fs; // Readers follow a single pointer, so they always observe one COMPLETE // generation, never a partially written one. fn publish_generation(key: &str, payload: &[u8]) -> std::io::Result<()> { let p = store_root().join(key); fs::create_dir_all(p.parent().unwrap())?; fs::write(p, payload) // immutable, content-addressed } // Optimistic compare-and-swap: only advance the pointer if it still holds the // value the writer last observed. A losing writer (a duplicate from the deploy) // detects the conflict and backs off instead of regressing to an older or // duplicated generation. Maps to a conditional write (If-Match on an ETag) in a // real object store or a small consistent key-value store. fn cas_pointer(expected: Option<&str>, next: &str) -> std::io::Result<bool> { let ptr = store_root().join("latest"); let current = fs::read_to_string(&ptr).ok(); let matches = match (current.as_deref(), expected) { (None, None) => true, (Some(c), Some(e)) => c == e, _ => false, }; if !matches { return Ok(false); // someone else moved it; do not clobber } fs::write(&ptr, next)?; Ok(true) } Trade-Offs: Content Keys vs. Locks, and What Teams Pay Content-derived keys with atomic publish cost more storage and more writes than a single fixed path. Every generation is retained until a lifecycle policy expires it, and versioning multiplies object count during the overlap windows you are now able to observe. For a pipeline producing one object per minute, the added cost is small, a few percent of the storage line in most setups, and it buys an output history you can audit and roll back. Against distributed locks, the comparison is starker. A lock-based design adds a hard dependency on a coordination service in the write path, which means its availability becomes your availability and its tail latency becomes your tail latency. The keying approach has no such dependency at write time. Its correctness comes from determinism and atomic promotion, both properties of code and storage you already run. The cost is discipline: every input that affects the output must be folded into the key, or two genuinely different results can collide under one key, and you reintroduce silent corruption from a new direction. The methodology that makes this safe to adopt is incremental rollout validated by the detection scan. Deploy the keyed publish path to a single region first, then run the divergence scan across a full deployment cycle before widening. A clean scan across one rollout is strong evidence the keying covers every input that matters. The verification below runs two overlapping replicas of the same task and asserts that exactly one object results and its contents match what either replica intended. Rust // Verification: two replicas of the SAME logical task, as happens when an old // pod and a new pod both fire during a rolling deploy. Exactly one object must // result, and its bytes must match what either replica intended. fn overlapping_runs_converge() { let spec = RunSpec { pipeline: "border-state".into(), window_start_epoch: 1_726_000_000, window_len_secs: 60, schema_version: 3, }; let key = spec.output_key(); let payload = build_payload(&spec); let wrote_old = atomic_publish(&key, &payload, "old-replica").unwrap(); let wrote_new = atomic_publish(&key, &payload, "new-replica").unwrap(); assert!(wrote_old ^ wrote_new, "exactly one replica writes the object"); assert_eq!(walk(&store_root()).len(), 1, "overlap converges to one export"); } #[test] fn identical_inputs_yield_identical_keys() { let a = RunSpec { pipeline: "p".into(), window_start_epoch: 100, window_len_secs: 60, schema_version: 1 }; assert_eq!(a.output_key(), a.clone().output_key()); } Teams that skip this work do not see failures immediately, which is what makes the omission dangerous. The pipeline runs clean for weeks, then a deployment lands during a long task and a single corrupted generation flows downstream. By the time anyone traces the bad decision back to its source, the offending object has been overwritten, and the logs show two clean completions. The keying and atomic publish pattern turns that entire class of incident into a no-op, and the detection scan turns the residual risk into a number you can watch.