Testing, Deployment, and Maintenance

A few weeks ago, I disabled key authentication on an Azure storage account we used for Terraform state management. It was one of the key security recommendations in Microsoft Defender for Cloud. It made sense to use RBAC-only permissions, enforce PIM approvals for the Infrastructure team, and avoid storing static credentials in config files, where leaks are possible. This is exactly the kind of control you want for state files, which contain the keys to your entire cloud environment. But I missed an important line in the azurerm backend config. If use_azuread_auth = true is not explicitly set, the provider uses key-based authentication by default. Since key authentication had been disabled, terraform init failed and the pipeline broke. The actual fix was easy, but finding what was wrong, not so much. JSON terraform { backend "azurerm" { resource_group_name = "rg-tfstate-prod" storage_account_name = "sttfstateprod001" container_name = "tfstate" key = "platform/prod.tfstate" use_azuread_auth = true } } This is not the kind of detail every engineer should have to remember in every repository. It belongs in the module. That is the gap I am talking about: the security decision was correct, but the delivery path still allowed the wrong configuration. The same pattern shows up elsewhere: storage accounts left open, IAM roles with excessive permissions, credentials committed to repositories, diagnostic settings missed, or Terraform modules that still allow insecure defaults to slip through. Security Enters Too Late, and Everyone Pays For It There is a common pattern: a developer builds a feature, security reviews it and flags something, the developer reworks it, the release gets delayed, and someone gets the blame. The cycle repeats until everyone is frustrated. The cost side of this doesn't get enough attention. Catching a vulnerability while you're still writing the code is a relatively quick fix. Finding the same issue in production is a different situation entirely: incident response kicks in, there may be regulatory questions to answer, and the reputational impact is difficult to measure. The further right security sits in the delivery process, the heavier each failure gets. Most teams are inadvertently set up to find problems at the point where they cost the most. What Shifting Left Actually Looks Like People toss around 'shift left' so much that it’s lost its punch. Here’s what it actually looks like in practice: Plan: Include threat modeling in sprint planning and spend 30 minutes on it rather than managing it in a separate process or document.Code: Use IDE plugins to flag insecure patterns in real time while you code and pre-commit hooks to run secrets detection before committing the code. The developer finds out immediately, not weeks later in a review.Build: Run SAST on every commit to catch injection risks, insecure cryptography, and hardcoded secrets/credentials before code is deployed to a shared environment.Test: Let DAST probe the application in staging as an attacker would. SAST reads code, and DAST attacks the running system. One finds what the other misses.Deploy: Scan your IaC before applying changes, check container images for CVEs, and use OPA policy gates to verify signing, permissions, and network policies before anything reaches production. Running security through each of these stages means issues come up when they are still manageable, rather than after they have already caused damage. Installing Tools Is Not a Program How DevSecOps failures look in practice: Tools like Checkov and Semgrep are configured in the pipeline, and by next month, the developers have written suppression rules for the findings so the feature can be shipped. The tools keep running, but no one is checking their outputs. Three things matter more than which tools you choose: Tuning: SAST generates false positives because it doesn’t know what’s happening at runtime. Run a co-triage session with a developer and a security engineer; work through the first 50 findings; fix the problematic rules; or write a justified suppression. After a couple of sessions, developers start trusting the output because it becomes more accurate and actionable.Signal engineering: Let critical and high CVEs block the pipeline immediately, while medium and low go to a dashboard with remediation SLAs. Developers will find ways to bypass the findings instead of fixing them if you block the commit for every medium, which will end up in a bigger mess than you started with. Ownership: Send findings straight to the person who can fix them, and give them enough info to act. A centralized security queue is where urgency goes to die. The Terraform backend scenario I opened is the exact example. The security decision to use RBAC only and disable key authentication was absolutely the right one. But here’s the catch: use_azuread_auth = true was not enforced during provisioning. If a hardened module had that flag set by default, that misconfiguration simply couldn’t have happened. That’s the real difference between having a security policy and actually building a security platform. The Platform Team Is the Structural Answer Adding more process to a structural problem doesn’t fix it. What’s required is a different model entirely. A real platform team treats the internal platform as a product, with engineers as its customers. Their job is to make secure, compliant delivery the path of least resistance: golden path templates, a shared CI/CD toolchain, secrets management, and self-service provisioning, all built with guardrails from the start. When teams repeatedly provision similar workloads — containerized APIs, data pipelines, Kafka consumers – the same security configuration decisions recur. Golden path templates address this by embedding those decisions up front. Encryption at rest is already configured, IAM permissions are scoped to what the workload actually needs, logging and network policies are in place, and the backend authentication flags in the Terraform modules are set correctly from the start. A developer selects the right template, fills in the required fields, and provisions. The repository they get back already has security gates running in the pipeline. There is no separate step to secure it afterward. Figure 1: A secure golden path platform embeds security controls into the default delivery path. This is what removes the need for individuals to get every detail right under pressure. In my experience, even when you know the correct configuration, you can still miss something in the moment. The platform handles that by making the secure option the default. In many organizations, platform teams work best when they sit within Engineering rather than reporting directly into the CISO function. If they are seen mainly as a compliance function, product teams may treat them as another gate to work around. Security should define the policies and risk boundaries; Engineering should build and operate the platform that makes those policies usable. Where to Start: Sequence the Platform, Don’t Boil the Ocean The most common mistake is trying to implement everything at once. Every scanner, every policy gate, every access control change lands in one big push. It creates noise before it creates trust, and teams lose confidence in the tooling before it has a chance to prove its value. Sequence it instead. Months 0 to 3: secrets scanning as a pre-commit hook, SAST in CI, IaC scanning before Terraform apply, and a security champions program with one dedicated developer per squad. Low friction, immediate signal, nothing that unnecessarily blocks delivery. Months 3 to 6: DAST in staging, container image scanning, OPA policy gates, and SCA on every build. At this point, the platform needs to make a clear distinction: critical and high findings stop the pipeline; everything else goes into a remediation backlog with defined ownership and SLAs. Months 6 to 12 mark the point at which platform security matures into deeper controls: workload identity, privileged access management, zero-trust network policies, and a real-time compliance dashboard. Never trust, always verify, and assume breach stop being principles on a slide and become defaults in the environment. Don't wait for a fully staffed platform team or executive sponsorship. The Terraform backend fix I mentioned earlier eventually became a hardened provisioning module used by the wider infrastructure team, turning a one-off incident into a reusable secure pattern. No one needs to remember the flag because the platform handles it automatically. That's what security as a platform property actually looks like. Not a gate at the end. A system that makes the right thing the easy thing, by default, every time.

The bug report was received as a customer complaint. An AI agent responsible for managing vendor onboarding had sent a rejection email to a supplier the company had been trying to close for three months. Nobody had authorized it. Nobody had configured it to reject vendors in that category. The agent autonomously made the decision after analyzing a compliance document and cross-referencing it with an internal policy database. By the time the complaint arrived, the reasoning chain that produced the decision had been discarded. The agent had no memory of why it did what it did. The logs showed the action but not the thought. That story is fictional in its specifics but accurate in its structure. This phenomenon represents a class of problems that teams deploying AI agents in production are encountering with increasing frequency: the agent performed an action, the output is visible, but the intermediate reasoning, including the sequence of context retrievals, model calls, tool invocations, and decisions that led to the output, is either absent, incomplete, or stored in a format that renders post hoc investigation nearly impossible. Traditional observability was not designed for systems that exhibit cognitive processes. Why Agent Observability Is Structurally Different Conventional service observability is built around a relatively stable model: a request enters a system, passes through a defined set of operations, and produces a response. The execution path may be complex, but it's deterministic and bounded. You can instrument each step, correlate the signals with a trace ID, and reconstruct exactly what happened for any given request. AI agents break this model in at least three ways. First, the execution path is not determined at design time — it emerges from the agent's reasoning. An agent deciding which tools to call, in what order, based on what it reads in a retrieved document, is making structural decisions at runtime that a static trace can't fully capture. The spans exist, but the semantic reason a particular branch was taken lives inside a model call that returned natural language, which most tracing systems treat as an opaque blob. Second, agent systems frequently involve state that persists across requests: memory stores, retrieved context, and conversation history, which means the behavior of the system at time T is partially determined by things that happened at times T-1 through T-n. Debugging a poor decision often requires reconstructing not just the current request but the accumulated state that shaped it. Most observability stacks are not built for these scenarios. Third, multi-agent systems introduce the problem of causal attribution across agent boundaries. When Agent A passes a task to Agent B, which delegates a subtask to Agent C, which calls a tool that returns erroneous data, and that incorrect data propagates back up the chain to produce a wrong output from Agent A, the causal chain is real but fragmented across three separate execution contexts. Without deliberate design, you'll have three separate traces with no shared context that links them. The Minimum Viable Agent Trace The starting point for any serious agent observability implementation is defining what the minimum viable trace looks like for a single agent execution. In practice, this means capturing five things that standard OpenTelemetry spans don't cover by default. The first is the full prompt context, not just the user message but the complete input to each model call, including the system prompt, retrieved documents, tool outputs injected into the context, and the conversation history. The information is costly to store and verbose, but you need it to understand the model's reasoning. Sampling helps here: store full prompt context for a percentage of executions, prioritizing those that result in high-stakes actions or errors. The second is the model's reasoning output before tool calls. If your agent framework supports it, capture chain-of-thought or scratchpad outputs of the model's intermediate reasoning before it decides to call a tool or produce a final answer. This is the closest thing to a stack trace for a reasoning system. Without it, you can see that a tool was called but not why. The third is a tool called "provenance" for each tool invocation, recording not just the inputs and outputs but which part of the reasoning chain triggered it. Fourth is the agent's decision points: moments where the agent chose between multiple possible actions. Fifth is cross-agent delegation context: when one agent hands off to another, the receiving agent's trace must carry a reference to the delegating agent's trace ID. Python # Minimal agent span instrumentation using OpenTelemetry from opentelemetry import trace import json tracer = trace.get_tracer('agent.core') def traced_model_call(agent_id, prompt_context, step_label): with tracer.start_as_current_span(f'agent.model_call.{step_label}') as span: span.set_attribute('agent.id', agent_id) span.set_attribute('agent.step', step_label) # Store truncated prompt for cardinality control span.set_attribute('agent.prompt_hash', hash(str(prompt_context))) span.set_attribute('agent.prompt_len', len(str(prompt_context))) # Full prompt stored separately in blob storage, keyed by trace+span ID store_prompt_context( trace_id=format(span.get_span_context().trace_id, '032x'), span_id =format(span.get_span_context().span_id, '016x'), context =prompt_context ) response = call_model(prompt_context) span.set_attribute('agent.output_len', len(response)) span.set_attribute('agent.tool_calls', extract_tool_calls(response)) return response The pattern above separates high-cardinality content (the full prompt) from the trace span itself, storing it in blob storage keyed by trace and span IDs. This keeps the tracing backend manageable while preserving the ability to retrieve full context for any specific execution. The prompt hash allows you to detect when two executions were given identical contexts, which is useful for identifying cases where the same input produced different outputs, which is a diagnostic signal in itself. Multi-Agent Correlation: The Delegation Chain Problem Here's where things got genuinely complicated in a system I was involved with: we had three agents — a planning agent, a research agent, and a writing agent that collaborated on generating reports. Each was instrumented individually and produced clean traces. But when a report came out wrong, reconstructing which agent's decision caused the problem required manually cross-referencing three separate trace trees, none of which had a shared parent. The fix was implementing what we called a "workflow ID," a UUID generated at the entry point of any multi-agent task and propagated explicitly to every agent that participated in that task, regardless of how many hops away from the origin they were. This workflow ID was added as a span attribute on every agent span and as a field in every log line produced during the task. With it, querying all spans and logs associated with a single end-to-end agent workflow became a single filter, not a manual correlation exercise. Python # Propagating workflow context across agent boundaries from dataclasses import dataclass from opentelemetry import trace, context, propagate @dataclass class AgentWorkflowContext: workflow_id: str # stable across all agents in a task parent_agent: str # which agent delegated this task delegation_depth: int # how many hops from the origin agent def delegate_to_agent(target_agent, task, wf_ctx: AgentWorkflowContext): child_ctx = AgentWorkflowContext( workflow_id = wf_ctx.workflow_id, # same ID propagates parent_agent = wf_ctx.parent_agent, delegation_depth = wf_ctx.delegation_depth + 1 ) span = trace.get_current_span() span.set_attribute('workflow.id', child_ctx.workflow_id) span.set_attribute('workflow.depth', child_ctx.delegation_depth) span.set_attribute('workflow.parent_agent', child_ctx.parent_agent) return target_agent.run(task, child_ctx) The delegation depth attribute turned out to be more useful than expected. In one debugging session, seeing that a particular tool call was happening at delegation depth 4 — four hops from the original request immediately flagged that the agent system had gone significantly deeper into a recursive subtask chain than intended. Without that attribute, the trace looked like any other tool call. Semantic Logging: What Happened vs. Why Standard logging captures what happened. For agent systems, you also need to capture the agent's stated reasoning at key decision points. This doesn't require exotic infrastructure; it requires a logging discipline that treats the model's reasoning output as a first-class log field rather than as data to be discarded after use. In practice, this means that when an agent produces a reasoning step leading to a significant action — such as calling an external tool, delegating to another agent, producing a final output, or deciding to abandon a task — the full reasoning text should be logged alongside the action. Tag it with the workflow ID, the agent ID, and a decision type label. This produces a semantic audit trail that lets you answer the question, "Why did the agent do X?" without having to reconstruct it from indirect evidence. The objection is storage cost, and it's legitimate. Reasoning outputs from LLMs are verbose. Storing them for every execution at scale is expensive. The practical answer is tiered retention: store full reasoning logs for executions that result in errors, high-stakes actions (anything that sends an external communication, modifies a record, or triggers a financial transaction), or random sampling of normal executions for baseline calibration. For the rest, store only the decision label and the action taken. This keeps costs manageable while preserving investigative capability for the cases that matter. What I'd Do Differently In hindsight, the single most important decision to make before deploying an agent in production is defining what a 'high-stakes action' means for that specific agent and ensuring those actions always produce full semantic logs regardless of cost. Initially, we did not define logging requirements; instead, we treated logging as uniform across all action types, which resulted in issues when an agent took an unexpected external action, and we lacked a reasoning log to explain it. I'd also invest earlier in a replay capability: the ability to take a logged prompt context and re-run the agent over it with a modified model or prompt configuration to verify that a fix actually changes the behavior that caused a problem. Without a replay capability, any changes you make are based on hope rather than verification. With it, you can verify that the reasoning path actually differs before deploying. When should you not build this level of observability? If you're prototyping or running an agent in a low-stakes, easily reversible context, the overhead of full semantic logging and workflow ID propagation is probably premature. Build it before you go to production with consequential actions, not after. The cost of retrofitting it once an unexplained agent decision has already caused a real problem is significantly higher than building it in from the start. Key Takeaways Standard distributed tracing captures what happened in agent systems but not why. Semantic logging of reasoning outputs at decision points is the missing layer; treat it as first-class infrastructure, not optional verbosity. Propagate a workflow ID across all agents in a multi-agent task. Without it, correlating signals across agent boundaries requires manual effort that fails under incident pressure. Separate high-cardinality prompt content from trace spans. Store the full prompt context in blob storage keyed by trace and span ID, and reference it from the span. This preserves investigative capability without bloating your tracing backend. Please define high-stakes actions prior to deployment and ensure they consistently generate complete semantic logs. The executions you most need to investigate are exactly the ones where missing reasoning context is most detrimental. Conclusion Observability for AI agents is not a solved problem. The tooling ecosystem is immature, the standards are still forming, and most teams are improvising solutions on top of infrastructure designed for deterministic services. That's not a reason to skip it; it's a reason to be deliberate about what you build, because the defaults will leave you blind at exactly the wrong moment. The deeper challenge is that agent observability isn't just a technical problem. It's also an accountability problem. When an AI agent takes a consequential action, someone needs to be able to answer the question of why, not just for debugging purposes, but for the humans affected by the decision and for the organization responsible for the system. A vendor who received a rejection email deserves a better answer than "the agent decided that." The infrastructure to produce that answer has to be designed in, not bolted on. The open question I keep returning to: as agent systems become more capable and their reasoning chains longer and more complex, at what point does the volume and opacity of their decision-making exceed our practical ability to observe and understand it? We may be building systems that are genuinely difficult to audit, not because of missing tooling but because of fundamental limits on human comprehension of long reasoning chains. What does accountability look like then?

A pipeline can finish successfully, schemas can match, and null checks can pass, while the business is still looking at yesterday's truth. Freshness deserves its own quality model. The pipeline succeeded. The schema matched. Required fields were present, ranges were sane, and the dashboard refreshed on schedule. Every quality check was green. The number on the screen was still wrong, because it was built from data that stopped updating two days ago and nobody noticed. This failure is common, and it is quiet. Most data quality programs are built to answer one question: is this data valid? They check for nulls, types, ranges, uniqueness, and referential integrity. Those checks are necessary, and they catch a real class of problems. They also share a blind spot. A record can be perfectly valid and completely stale. Validity is about whether the data is well-formed. Freshness is about whether it is current enough to trust. They are different properties, and a pipeline that measures only the first will keep serving old truth with a green status next to it. Freshness Is Not Correctness Structural quality asks whether a row is shaped correctly. Freshness asks whether the row should still be believed given how much time has passed. A transaction record from Tuesday is structurally identical whether it is read on Wednesday or three weeks later. Its validity never changes. Its usefulness for a decision that assumes current data changes completely. This is why freshness belongs in the quality model rather than in a separate operations dashboard. Most quality dimensions that teams already track, such as completeness, accuracy, consistency, uniqueness, and validity, describe the data as it sits. Freshness describes the data relative to now. Leaving it out of the quality model means the platform can report high quality on data that is too old to act on, which is not a contradiction the business will find reassuring. The concept that ties this together is the freshness gap: the distance between when an event actually happened and when a consumer can first see it. Structural checks never measure this gap, because both a fresh record and a stale one are equally valid. The gap is the part of quality that only time reveals. Why Pipelines Hide Staleness The reason staleness stays hidden is that pipeline success and data freshness measure different things, and teams routinely treat the first as a proxy for the second. A job can complete successfully while delivering nothing new. Common paths to a green pipeline over stale data include: The source sent no new files. The job ran, found the same input as yesterday, processed it correctly, and reported success. Nothing failed. Nothing updated either.Only some partitions arrived. The pipeline loaded the partitions it received and completed. The missing region or date range is not an error to a job that was never told those partitions were mandatory.A late-arriving upstream delayed the real data. The scheduled run fired on time against data that had not landed yet, so it processed an incomplete or old snapshot and finished cleanly.The dashboard cached a stale table. The pipeline updated the table, but the serving layer or BI tool returned a cached result, so the freshest data never reached the screen.A backfill overwrote current data with an older snapshot. A correction job ran a historical range and, through a scope error, replaced newer records with older ones. Every row is valid. The table went backward in time. None of these trip a structural check, because in every case the data that is present is well-formed. The problem is not the shape of what arrived. It is the age of what arrived, and whether anything arrived at all. Freshness Needs Its Own Contract Freshness cannot be governed by a single global rule, because different datasets have different tolerances. A five-minute delay is a crisis for fraud detection and irrelevant for a historical archive. Tying freshness to the pipeline schedule is the common shortcut, and it is wrong, because the schedule describes when the job runs, not when the data is expected to be current for a specific use. The fix is to define freshness expectations per dataset, anchored to the business decision the data supports rather than to the cadence of the job that produces it. DatasetFreshness expectationWhy it mattersFraud eventsUnder 5 minutesDecisions are made in real timeDaily balancesBy 7 AM ETMorning reporting depends on itMonthly finance closeBy business day 3Tied to the reporting cycleHistorical archive24 to 48 hoursLow operational urgency Each expectation is a contract. It states what current means for that dataset, and it gives monitoring something concrete to check against. Without it, freshness is a matter of opinion, and the first time anyone forms an opinion is usually after a stale number has already reached a decision. Measuring Freshness Correctly The technical heart of freshness is that there is no single timestamp called "the time." A record carries several distinct times, and confusing them is how freshness monitoring gives false comfort. Four matter: Figure 1. A record carries four distinct times. Structural checks see only the published value. The freshness gap, which is event time to publish time, is the delay no structural check measures. The relevant times to consider are: Event time: When the thing actually happened in the source system. A purchase was made, a sensor fired, an address changed.Ingestion time: When the record entered the platform. The moment it landed in the queue or the raw zone.Processing time: When the transformation ran over it. The point where it was cleaned, joined, and shaped.Publish time: When it became queryable by a consumer. The moment the serving table or dashboard could return it. The freshness gap that matters to the business is publish time minus event time, because that is the total delay between reality and what a consumer can see. A pipeline that measures only processing time, "the job ran at 06:00," reports a healthy number while the events it processed are hours old, because the delay lived upstream, before ingestion, where the job never looked. Measuring the wrong timestamp is worse than not measuring, because it produces a confident freshness metric that is disconnected from reality. A dataset can show a two-minute processing lag and a six-hour event-to-publish gap at the same time. The first number looks great on a status page. The second is the one the business feels. What Freshness Failures Look Like In practice, freshness failures usually look healthy from the outside. The job finishes, the schema matches, and the records pass validation. The failure lives in the time dimension: no new source data arrived, only some partitions landed, a dashboard served a stale cache, or a backfill moved the table backward. Structural validation sees rows that are well-formed. Freshness monitoring sees that the published dataset no longer reflects the current state of the business. Passing one tells you nothing about the other. A Practical Freshness Pattern Making freshness a first-class quality dimension does not require a new platform. It requires treating the age of data as something the pipeline measures, records, and alerts on, the same way it already treats nulls and types. A workable pattern: Carry timestamps through the pipeline. Preserve event time from the source, and stamp ingestion, processing, and publish times as the record moves. The freshness gap cannot be measured if the timestamps needed to compute it were discarded early.Record freshness in a small audit table. For each dataset and run, store the maximum event time published and the publish time itself. This gives a queryable history of how current each dataset actually was, run over run.Attach a freshness SLA to each dataset. Encode the per-dataset expectation from the contract above as a checked threshold, not a comment in a runbook.Alert on the gap, not on job status. Trigger when publish-time-minus-event-time crosses the dataset's threshold, independent of whether the job reported success. This is the alert that catches the source-sent-nothing case, which job monitoring cannot see.Make freshness visible downstream. Surface the last known freshness next to the data itself, so a consumer can see that a dashboard is running on data from two days ago before they act on it. Track compliance as a percentage over time rather than as a pass or fail on a single run. A dataset that met its freshness SLA 99 percent of the time last month, and is trending down, is a more honest signal than a single green check, and it is the number a business owner can actually reason about. A minimal audit table makes this concrete. One row per dataset per run is enough to compute the gap, compare it against the SLA, and keep a history: ColumnMeaningdataset_nameDataset being monitoredrun_idPipeline run identifiermax_event_timeLatest event included in the published datapublish_timeWhen the dataset became availablefreshness_gap_minutesPublish time minus max event timesla_minutesFreshness threshold for the datasetsla_statusPass or fail for this run The gap column is the one structural checks never produce, and the status column is what the freshness alert reads rather than job success. The Green Check was Measuring the Wrong Thing Validity and freshness are independent. A pipeline can watch one perfectly and never look at the other, which is exactly how a dataset ends up well-formed, internally consistent, and two days out of date with a passing status next to it. The structural checks were doing their job. They were just never the checks that would have caught this. Freshness needs its own contract, its own timestamps, and its own alert that fires on the age of the data rather than the exit code of the job. Decide what current means for each dataset, watch the distance between event time and publish time, and put that distance in front of the people making decisions. A pipeline finishing was never the same claim as the data being fresh, and the sooner a platform stops treating the first as proof of the second, the fewer stale numbers reach a meeting.

A CI/CD pipeline that runs without errors creates a sense of correctness. The job is green. The deployment happened. The infrastructure must reflect what it should. This logic feels sound, and it breaks down in a specific way worth understanding. The pipeline knows what it was told to do at the time it ran. It does not know whether that was the right thing to do. And it cannot tell you whether the state it produced is still aligned with what the organization actually needs, because it has no persistent model of intended state to check against. It ran, it applied, it exited. When a pipeline becomes the de facto source of truth, the actual authority over infrastructure state is distributed across pipeline history, environment variables, conditionals, and whatever was in the repository at the time of the last run. That is a fragile place to store operational intent. What a Source of Truth Actually Is A source of truth is not a pipeline. It is a versioned, human-readable record of what the infrastructure should look like, maintained independently of any single run. The pipeline reads from it and acts on it. The pipeline is not it. The distinction matters more as the environment grows. A small team with one pipeline can often get away with pipeline-as-truth for a while. The fragility is manageable when the team holds the mental model and the surface area is small. That tolerance disappears at scale, during incidents, or when team composition changes. Consider what happens during an incident. The question is not “what did the pipeline last apply?” It is “what should this environment look like right now?” If the answer requires tracing pipeline history or reading environment variables buried in CI configuration, the source of truth is inaccessible when you most need it. The Common Drift Pattern Drift in this context is not always configuration drift in the Terraform sense, where a resource exists in cloud state but not in code. It is more subtle: a gap between what the pipeline applies and what the organization intends, which is never written down explicitly. The pattern develops gradually. A variable gets hardcoded into a pipeline stage because changing the input file was inconvenient. A condition gets added to skip a resource in a specific environment. A flag gets added to suppress a Terraform error that kept reappearing. Each change is rational at the time. None of it is recorded as a decision about intended state. The pipeline becomes progressively more opinionated, and that opinion is encoded in logic, not in a readable model anyone can inspect. This is why a pipeline that passes every run can still represent a system in drift. The runs are consistent. The intent they encode has deviated from what the organization would say it wants, if someone asked. The drift also accumulates knowledge risk. A new engineer joining the team cannot read the pipeline and understand what the environment is supposed to be. They can only understand what the pipeline will do if they run it. Those are not the same thing, and the difference matters when something needs to change under pressure. A Simple Corrective Structure The fix does not require a new platform or a new toolchain. It requires separating the declaration of intent from the act of applying it. In Terraform terms, this looks like a clear split between variable definition files that describe environments and the pipeline logic that applies them. Shell environments/ production.tfvars staging.tfvars dr.tfvars pipeline/ apply.sh The .tfvars files are the source of truth. They describe the intended state of each environment. The pipeline reads them and applies. Any change to what an environment should look like goes through the variable file first, through version control, through review. The pipeline reflects intent; it does not define it. This pattern extends to more complex inventory structures. HybridOps uses a similar model, where environment inventory files define the operational state and pipeline jobs are written to consume those definitions rather than encoding intent directly in pipeline logic. The result is that looking at the inventory or variable files gives a direct answer to “what should this environment look like?” without tracing pipeline history or reading conditional logic. Reviewing Your Own Setup A practical way to check whether your pipeline has become the source of truth: if the pipeline repository were deleted today, could a new engineer reconstruct the intended state of each environment from the infrastructure code alone? If the answer is no, or not without significant effort, the pipeline is carrying intent that belongs in declared configuration. That is the part worth extracting. It is also worth asking how many decisions about intended state live in pipeline conditionals. Every if ENVIRONMENT == "production" block in a pipeline script is an opinion about what production should look like. That opinion deserves to be in infrastructure code, reviewed like any other infrastructure change, not buried in deployment logic that only runs at apply time. A pipeline that applies changes cleanly is genuinely useful. It becomes a liability when it is also the only place intent is recorded. The reliability of the system comes from the fact that intent is declared somewhere stable, not from the fact that the pipeline is fast. Build pipelines that read intent. Store intent somewhere readable. That division is simple to state and consistently worth enforcing.

When Nothing Is Broken — But the System Still Fails You hit a failure. Tests are failing, or the system behaves in a way that doesn’t make sense. You check the code first. Nothing obvious. Then logs. Still nothing conclusive. You retry. Same result. At that point, the instinct is simple: Something inside the system must be broken. But in many cases, nothing is. Every individual component is behaving exactly as it was designed to behave. The code executes, the service responds, and the infrastructure appears healthy. Yet the system still fails. This is what makes these situations difficult — because the failure doesn’t originate inside a component. It emerges from the way components interact. When “Correct” Behavior Still Produces Failure A surprising number of integration failures follow the same pattern: API behaves exactly as documentedService returns valid dataClient processes input correctlyInfrastructure reports healthy status And yet, the system still fails. This is not a problem inside any single layer. It’s a problem of misaligned assumptions between layers. These assumptions are rarely visible in code, and they tend to exist implicitly in how systems are designed and used: How data is structured and interpretedHow serialization and deserialization are handledWhat defaults are assumed across layersWhere validation responsibility actually lives As long as these assumptions align, the system behaves predictably. The moment they drift, failures begin to appear — even when every component still looks correct in isolation. Boundary Failures: Where Systems Stop Agreeing Some failures don’t fit traditional debugging categories. They're called boundary failures. These occur at the interaction between systems — not within them. From the outside, everything appears valid: Contract is defined correctlyService responds successfullyClient code compiles and executesData itself is technically correct But the systems are no longer interpreting behavior the same way. That divergence is enough to break the system. A Real Example: Contract vs. Runtime Drift Consider a simple integration. An API contract defines the response as: JSON { "settings": { "theme": "dark", "notifications": true } } A generated client expects: C# public class SettingsResponse { public string Theme { get; set; } public bool Notifications { get; set; } } Over time, the API evolves. The runtime response changes: JSON { "settings": { "preferences": { "theme": "dark", "notifications": true } } } From the API’s perspective: Response is validRequest succeedsUnderlying data is still correct But from the client’s perspective: Expected fields are no longer mappedDeserialization produces null or default valuesDownstream logic continues using incorrect state This doesn’t result in a clean failure. Instead, it surfaces as subtle issues: Fields that previously contained values now return empty or nullDefaults silently replace real values without triggering alertsApplication flows execute successfully — but produce incorrect outcomesBehavior becomes inconsistent across environments depending on serialization or configuration Nothing crashes. No exception directly identifies the problem. The system appears operational — but is quietly incorrect. This is the key characteristic of boundary failures: the system doesn’t fail loudly — it drifts into failure And that failure cannot be owned by a single system. It exists in the assumptions connecting them. A Short Debugging Walkthrough In practice, this kind of issue rarely presents itself clearly. The first signal is usually indirect. You notice a downstream feature behaving incorrectly — missing values, inconsistent outputs, or logic that appears to “work” but produces the wrong result. Initial checks don’t reveal anything unusual: The API call succeedsStatus codes are correctLogs show no errors At this point, the issue often looks like a business logic bug. The investigation typically goes deeper into the application: Validation rulesTransformation logicClient-side handling Nothing stands out. Only after inspecting the raw response payload does the problem become visible. The structure doesn’t match what the client expects. Fields are present, but nested differently. The data exists, but is no longer mapped. From there, the rest becomes clear: Deserialization silently failsModels populate with defaultsDownstream logic operates on incomplete data The system never actually “breaks.” It simply starts producing incorrect results. And until that mismatch is identified, debugging tends to move in the wrong direction—deeper into code instead of across system boundaries. Why Boundary Failures Are Hard to Detect Traditional failures give visible signals: ExceptionsStack tracesAlertsHealth check failures Boundary failures rarely do. Instead, engineers observe: Inconsistent behavior across environmentsPartial correctness where some flows work, and others don’tMissing or transformed data without clear causeNondeterministic outcomes that are difficult to reproduce The system looks healthy from an operational standpoint, but behaves incorrectly. Modern architectures amplify this problem. Systems increasingly depend on: Generated SDKsLayered abstractionsDistributed servicesAsynchronous processing Each layer introduces: AssumptionsTransformationsPotential mismatches As systems scale, the number of boundaries grows much faster than the visibility into them. Another Example: Execution Context Drift Boundary failures are not limited to APIs. They also occur across execution environments. A service runs locally with: Shell MODE=debug In that environment: Logging is verboseValidation is relaxedBehavior appears predictable In CI or production: Shell MODE=production Now: Validation rules changeDefaults behave differentlyLogging is reduced or removedTiming and concurrency behavior may shift Same application. Same codebase. Different assumptions about execution. No bug inside the core logic. But a clear mismatch at the boundary between environments. The Wrong Question Slows Debugging Most debugging starts with: “What is broken?” That assumption leads investigation inward into code, frameworks, and implementation details. But for boundary failures, this direction is often misleading. A more useful question is: What assumption no longer holds across the boundary? That changes how you approach the problem. Instead of focusing on a single component, you analyze interactions: What each system expectsWhat actually happens at runtimeWhere those expectations diverge That’s where the failure usually exists. Practical Lessons When debugging failures that don’t behave predictably: Don’t assume the framework or tool is broken firstInspect real runtime data rather than relying on abstractionsCompare actual payloads with expected contractsMake implicit assumptions between systems explicitVerify execution context differences across environmentsInvestigate interactions before diving into internal logic In many cases, the issue is not buried deep inside a component. It’s sitting at the boundary. Closing Modern systems fail at the seams, not because components are incorrect, but because independently correct systems stop agreeing. And when everything looks correct—but the system still behaves incorrectly— that boundary is often where the real problem lives.

There is a number that engineering organizations love to report, and that engineering leaders love to receive: 100% code coverage. It has the satisfying quality of completeness. But completeness of what exactly? It implies that every line has been tested, every branch examined, every condition verified. It looks like the mathematical proof of a job well done. It is not, however. And the gap between what that number promises and what it delivers is, in many organizations, the single most expensive misunderstanding in the quality program. Code coverage measures the proportion of code that tests execute, not the proportion of behavior that tests verify. These are profoundly different things, and conflating them produces systems that are well-covered, yet dangerously under-tested. The engineers know the coverage number. The executives trust the coverage number. And the system fails in ways that the coverage number was incapable of detecting. What Code Coverage Actually Measures Code coverage tools work by instrumenting your codebase. They insert tracking markers at every line, branch, and condition. They run your test suite and record which markers were triggered. The coverage percentage is the proportion of markers that fired at least once during the test run. Notice what that definition contains and what it does not contain. It contains: which lines executed. It does not contain: whether the execution produced a correct result, whether the assertions in the tests actually verified the behavior, or whether the inputs used during execution were representative of the conditions the system will encounter in production. A test that calls a function and asserts nothing — or asserts the wrong thing — still generates coverage. A test that calls a function with a single, safe input still generates coverage for every line that input traverses. The coverage tool has no way to distinguish between a test that rigorously verifies behavior and a test that merely visits code. What Coverage Measures vs. What It Does Not MEASURES: Which lines of code were executed at least once during the test suite run MEASURES: Which branches (if/else paths) were taken at least once DOES NOT MEASURE: Whether the behavior of those lines was correct DOES NOT MEASURE: Whether the assertions in the tests were meaningful DOES NOT MEASURE: Whether the inputs used were representative of production conditions DOES NOT MEASURE: Whether the system behaves correctly under adversarial, boundary, or unexpected inputs DOES NOT MEASURE: Whether the test suite is enough This distinction is not a technicality. It is the entire problem. And the fastest way to see it is to look at code. The 100% Coverage Illusion: A Demonstration The following example is a payment processing function. Python # payment.py def process_payment(amount, card_number, currency='GBP'): """ Process a payment transaction. Returns a dict with 'status' and 'transaction_id'. """ if amount <= 0: raise ValueError('Amount must be positive') if currency not in ['GBP', 'USD', 'EUR']: raise ValueError(f'Unsupported currency: {currency}') # Mask card number for logging masked = card_number[-4:].rjust(len(card_number), '*') # Calculate processing fee (2.9% + 30p) fee = round((amount * 0.029) + 0.30, 2) total = amount + fee # Simulate gateway call transaction_id = f'TXN-{card_number[-4:]}-{int(amount*100)}' return { 'status': 'success', 'transaction_id': transaction_id, 'amount': amount, 'fee': fee, 'total': total, 'masked_card': masked } A payment processing function. Straightforward. Plausible. In production somewhere right now. A Test Suite That Achieves 100% Coverage Python # test_payment.py import pytest from payment import process_payment def test_valid_payment(): result = process_payment(100.00, '4111111111111111') assert result['status'] == 'success' def test_negative_amount_raises(): with pytest.raises(ValueError): process_payment(-10.00, '4111111111111111') def test_zero_amount_raises(): with pytest.raises(ValueError): process_payment(0, '4111111111111111') def test_unsupported_currency_raises(): with pytest.raises(ValueError): process_payment(100.00, '4111111111111111', currency='JPY') Four tests. Every line, branch, and condition in process_payment() is executed. The coverage tool is satisfied. The Coverage Report Running this suite with pytest-cov produces the following output: Python $ pytest test_payment.py --cov=payment --cov-report=term-missing ================================================================= platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 collected 4 items test_payment.py .... [100%] ----------- coverage: platform linux, python 3.11.4 ----------- Name Stmts Miss Cover Missing ---------------------------------------------- payment.py 14 0 100% ---------------------------------------------- TOTAL 14 0 100% 4 passed in 0.12s Four tests passing. 100% code coverage. Every statement executed. Every branch taken. The CI pipeline is green. The coverage badge on the repository is green. The engineering manager's dashboard is green. Now let us examine what these tests do not verify — what the 100% coverage number is actively concealing. What the 100% Coverage Is Hiding The test suite above exercises every line. It verifies almost nothing about behavior. Here is a systematic account of the failures it will not catch. Failure 1: The Fee Calculation Is Never Verified The processing fee calculation — the line that determines how much money is actually charged — is executed by test_valid_payment() but never asserted against. The test checks that status is 'success'. It does not check that the fee is correct, that the total is correct, or that the relationship between amount, fee, and total is arithmetically sound. # This calculation runs. It is never checked. fee = round((amount * 0.029) + 0.30, 2) total = amount + fee # For amount=100.00: # fee should be: (100.00 * 0.029) + 0.30 = 2.90 + 0.30 = 3.20 # total should be: 100.00 + 3.20 = 103.20 # Now introduce a bug: fee = round((amount * 0.29) + 0.30, 2) # 0.29 instead of 0.029 # fee becomes: 29.30 # total becomes: 129.30 # Coverage: still 100%. Tests: still passing. Customer: charged 29% instead of 2.9%. The bug is a single decimal point. The coverage tool cannot see it. Neither can any of the four tests. Failure 2: The Card Masking Is Never Verified The masked card number is returned in the response and presumably used in receipts, logs, and customer communications. The masking logic runs during test_valid_payment(). It is never asserted against. # This runs. It is never checked. masked = card_number[-4:].rjust(len(card_number), '*') # For card_number = '4111111111111111' (16 digits): # masked should be: '************1111' # Introduce a bug that exposes the full card number in logs: masked = card_number # accidentally log the full number # Coverage: still 100%. Tests: still passing. # PCI-DSS compliance: violated. Customer data: exposed. A PCI-DSS violation that 100% coverage cannot see, because coverage does not check what is returned — only that the line ran. Failure 3: The Transaction ID Embeds Unmasked Data The transaction ID construction is never examined. As written, it embeds the last four digits of the card number — which may or may not be acceptable depending on where transaction IDs are stored and logged. But more critically, if the construction logic changes in a way that embeds more card data, no test will catch it. # Transaction ID construction — executed, never verified. transaction_id = f'TXN-{card_number[-4:]}-{int(amount*100)}' # Change to accidentally embed more card data: transaction_id = f'TXN-{card_number}-{int(amount*100)}' # Coverage: 100%. Tests: passing. # Audit log: now contains full, unmasked card numbers. Failure 4: Floating-Point Currency Handling Is Untested The function handles currency arithmetic using floating-point numbers. Anyone who has worked with financial systems knows that floating-point arithmetic and money are a dangerous combination. The tests use clean round numbers (100.00, -10.00). They never probe what happens with amounts like 99.99, or 0.01, or values that produce floating-point rounding artifacts. # What actually happens with some real-world amounts: # amount = 99.99 # fee = round((99.99 * 0.029) + 0.30, 2) # fee = round(2.89971 + 0.30, 2) # fee = round(3.19971, 2) = 3.20 <- acceptable # amount = 19.99 # fee = round((19.99 * 0.029) + 0.30, 2) # fee = round(0.57971 + 0.30, 2) # fee = round(0.87971, 2) = 0.88 <- acceptable # But with Decimal arithmetic, the story differs. # The function uses float, not Decimal. # For high-volume systems, accumulated rounding errors # across thousands of transactions produce discrepancies # that appear in reconciliation reports months later. # Tests with clean inputs: passing. # Coverage: 100%. # Finance team's reconciliation nightmare: upcoming. The tests never probe the arithmetic with values that expose floating-point behaviour. Coverage does not notice. Failure 5: No Testing of What Happens When the Gateway Fails The function simulates a gateway call. In a real implementation, this would be a network call to a payment processor. The simulation always succeeds. The tests never ask: what happens when the gateway times out? What happens when it returns an error? What happens when it returns a malformed response? The coverage tool reports 100% on a function that has never been tested under its most important real-world condition: failure. Coverage Report and Confidence 100% line coverage: confirmed. Every line of process_payment() executes during the test run. Fee calculation correctness: unverified. A decimal point error charges customers 29% instead of 2.9%. Card masking correctness: unverified. A one-line change exposes full card numbers in logs. Transaction ID safety: unverified. A refactor can embed unmasked card data in audit logs. Floating-point precision: untested. Financial reconciliation errors accumulate silently. Gateway failure handling: untested. The function has never been tested under the condition it will most frequently encounter in production during incidents. The coverage number: 100%. The confidence it should provide: close to zero. Coverage of Behavior vs. Coverage of Code The demonstration above makes the distinction concrete. Now it can be stated precisely. Code coverage measures the proportion of source code statements, branches, or conditions that are executed during a test run. It is a property of the test suite's interaction with the code. Behavior coverage measures the proportion of the system's meaningful behaviors. Like the things it is supposed to do, and the things it must not do — that are verified by the test suite. It is a property of the test suite's relationship to the system's specification and risk profile. A test suite can achieve 100% code coverage while covering almost none of the system's meaningful behaviors — as the payment example demonstrates. Conversely, a test suite with 60% line coverage, if designed against the system's risk profile, can verify the behaviors that matter most and leave only low-consequence code unexecuted. This is not an argument against measuring code coverage. Coverage data is useful: it identifies code that is never exercised by any test, which is a meaningful signal. Uncovered code is a known unknown — you have no evidence about its behavior whatsoever. But covered code is not the same as thoroughly tested code. This distinction must be understood by anyone using code coverage metrics to make decisions. dimensioncode coveragebehavior coverage What it measures Proportion of lines/branches executed Proportion of meaningful system behaviors verified What 100% means Every line was executed at least once Every significant behavior has been verified — including error cases, boundaries, and failure modes What 0% means No code was executed (tests did not run)No meaningful behavior has been verified — even if tests pass What tools can measure it Automated, precise, available in all CI pipelines Requires human judgment, risk analysis, and specification review What it predicts Which code paths the tests reach. Not correlated with defect detection effectiveness The likelihood that important defects have been detected. Correlates with production quality outcomes Its most dangerous failure mode False confidence from tests that execute code without verifying it Scope blindness — failing to identify which behaviors are meaningful in the first place What Good Coverage Thinking Actually Looks Like Rejecting the coverage illusion does not mean abandoning the code coverage measurement. It means using coverage data correctly — as one signal among several, interpreted in the context of risk, rather than as a proxy for quality. The following approach is not a framework to mandate. It is a way of reasoning that produces better decisions than a percentage target. Step 1: Identify the Behaviors That Matter Before writing a single test, the question is: what does this system do that, if wrong, would cause harm? For the payment function, sample behaviors include charging the correct amount, protecting card data, generating accurate transaction records, and handling failures gracefully. These are the behaviors that tests must cover. They are determined by the system's risk profile, not by its line count. # Behaviour inventory for process_payment() # CRITICAL (failure causes financial or compliance harm): # - Fee calculation produces correct result for all valid inputs # - Total = amount + fee, always # - Card masking produces no more than last 4 digits in any output # - Transaction IDs contain no sensitive data # - Rejected amounts (<=0) never produce a transaction # - Unsupported currencies never produce a transaction # IMPORTANT (failure causes degraded service): # - Gateway timeout returns a defined error state # - Gateway error returns a defined error state # - Malformed gateway response is handled without exception leak # STANDARD (failure causes minor friction): # - Masked card format is consistent # - Transaction ID format is consistent # A test suite designed against this inventory will look # very different from a test suite designed to hit 80% coverage. # It will also tell you far more about whether the system is safe to run. A behavior inventory. Written before tests, not derived from coverage reports after them. Step 2: Write Tests Against the Behavior Inventory # Tests designed against the behavior inventory class TestFeeCalculation: """CRITICAL: fee must be exactly 2.9% + £0.30, rounded to 2dp""" def test_fee_standard_amount(self): result = process_payment(100.00, '4111111111111111') assert result['fee'] == 3.20 assert result['total'] == 103.20 def test_fee_small_amount(self): result = process_payment(1.00, '4111111111111111') assert result['fee'] == 0.33 # (1.00 * 0.029) + 0.30 assert result['total'] == 1.33 def test_fee_high_value_transaction(self): result = process_payment(9999.99, '4111111111111111') assert result['fee'] == 290.30 # (9999.99 * 0.029) + 0.30 assert result['total'] == 10290.29 def test_total_equals_amount_plus_fee(self): """Invariant: total must always equal amount + fee exactly""" for amount in [0.01, 1.00, 19.99, 99.99, 1000.00]: result = process_payment(amount, '4111111111111111') assert result['total'] == round(result['amount'] + result['fee'], 2) class TestCardDataProtection: """CRITICAL: no output may expose more than last 4 digits""" def test_masked_card_hides_all_but_last_four(self): result = process_payment(100.00, '4111111111111111') assert result['masked_card'] == '************1111' assert '411111111111' not in result['masked_card'] def test_transaction_id_contains_no_sensitive_data(self): result = process_payment(100.00, '4111111111111111') # Only last 4 digits permissible in transaction ID assert '411111111111' not in result['transaction_id'] def test_no_field_in_response_contains_full_card(self): card = '4111111111111111' result = process_payment(100.00, card) for key, value in result.items(): assert card not in str(value), \ f'Full card number found in field: {key}' Tests designed against the behavior inventory. They may not achieve 100% line coverage. They verify the things that matter. Step 3: Use Coverage Data to Find Gaps, Not to Set Targets After writing tests against the behavior inventory, run the coverage report — not to check whether you have hit a target, but to identify code that no test reaches. Uncovered code is a signal that deserves investigation. It may be dead code that should be deleted. It may be an error path that no test exercises. It may be a code path that your behavior inventory missed. # Using coverage as a gap-finder, not a target # After running behaviour-driven tests, the coverage report shows: # # payment.py Stmts Miss Cover Missing # ------------------------------------------------------- # payment.py 22 3 86% 45-47 # # Lines 45-47 are the gateway simulation block. # They are uncovered because no test exercises the failure path. # # This is the coverage report doing its job correctly: # it has identified a gap in the behaviour inventory. # The response is to ask: 'What happens on lines 45-47, # and should we have a test for it?' # NOT: 'How do we get from 86% to 90%?' 86% coverage that identifies a meaningful gap is more useful than 100% coverage that conceals one. The Mutation Testing Alternative If code coverage is an unreliable measure of test quality, is there a better one? There is, and it is called mutation testing. It is more computationally expensive than coverage measurement, but it measures something that coverage cannot: whether your tests are capable of detecting changes in the code's behavior. Mutation testing works by automatically introducing small, deliberate changes — mutations — into the source code, then running the test suite against each mutated version. If a mutation causes a test to fail, the mutation is "killed" — the tests detected the behavioral change. If all tests still pass despite the mutation, the mutation "survived" — the tests failed to detect a change in behavior that a developer could easily introduce. # Original code fee = round((amount * 0.029) + 0.30, 2) # Mutation 1: change operator fee = round((amount * 0.029) - 0.30, 2) # survived: no test checks fee value # Mutation 2: change constant fee = round((amount * 0.29) + 0.30, 2) # survived: no test checks fee value # Mutation 3: negate condition if amount >= 0: # original: amount <= 0 raise ValueError('Amount must be positive') # survived? only if boundary untested # A high mutation score means your tests are sensitive to behavioral changes. # A low mutation score — even with 100% line coverage — means your tests # are not detecting changes that matter. Mutation testing reveals what coverage cannot: whether your tests would catch a developer accidentally changing the logic. Mutation testing is not a replacement for thoughtful test design. It is a diagnostic tool that exposes weak assertions and under-specified tests with a degree of precision that coverage metrics cannot approach. A test suite with 85% line coverage and a 90% mutation score is demonstrably stronger than a test suite with 100% line coverage and a 40% mutation score. For most organizations, mutation testing is not yet part of the standard CI pipeline — it is computationally expensive and requires configuration effort. But even running it periodically, on a sample of the codebase, provides more meaningful information about test quality than a continuous coverage percentage ever will. The Questions Executives Should Be Asking Coverage percentages appear in engineering reports, sprint reviews, and board-level quality dashboards. They are communicated as evidence of quality. Most of the people receiving them do not know that they are receiving a measure of code execution, not confidence. This is not a technical problem. It is an information design problem. The people making decisions based on coverage numbers have never been given the vocabulary to question them. The following set of questions changes that. They are designed to be asked by any engineering leader, with or without a technical background. They probe the dimensions of quality that coverage metrics conceal. The Executive's Coverage Questions 1. What behaviors does this number not measure? Ask the team to identify the five most important things the system does and confirm that each of those behaviors has dedicated tests with meaningful assertions. 2. What is our mutation score? If the team cannot answer this, the coverage percentage is the only quality signal they have — and it is a weak one. 3. What are the most important failure modes for this system, and are they tested? A system's failure modes are usually more important than its success paths. Financial corruption, data exposure, and service unavailability all require dedicated tests. Coverage numbers do not distinguish these from a test of a utility function. 4. What is uncovered, and why? The answer to this question is more useful than the coverage number itself. Uncovered code is a map of known unknowns. Understanding why those regions are uncovered reveals risk. 5. How does our coverage number change when we exclude assertion-free tests? Most teams will not have run this analysis. Asking for it creates a productive forcing function. 6. When did we last find a defect through our test suite rather than through production? This is the ground-truth question. If the last production incident involved code with high coverage, the coverage number needs to be interrogated, not reported. These questions do not require technical depth to ask. They require only the understanding that coverage measures execution, not confidence. Understanding that the gap between those two things is where many production defects live. The Uncomfortable Organizational Truth Coverage metrics persist not because they are accurate but because they are easy. They are generated automatically by widely available tools. They produce a single number that is simple to track, to report, and to include in a dashboard. They create the impression of accountability without requiring the harder work of defining what accountability for quality actually means. The organizations that have replaced coverage targets with behavior-oriented quality measures consistently report the same initial reaction from their teams: the work becomes harder to measure but easier to reason about. Engineers stop asking "have I covered this code?" and start asking "have I verified the behavior this code is supposed to produce?" The shift is subtle in vocabulary but very significant in practice. A harder truth is that behavior coverage is often more difficult to achieve. It requires someone to think about what the system is supposed to do, what it must not do, and what the consequences of failure in each area would be. This is skilled work. It is the kind of work that, in most organizations, is either not assigned to anyone or is assigned to QA teams under a job description that asks them to find bugs rather than define the behavior surface of the system. Fixing the coverage illusion requires that someone has the authority and the charter to define what behavior coverage means for a given system. It requires that engineering teams are held accountable for achieving it, even though it cannot be measured with a single percentage. Even though it requires more judgment than automation, and even though it is harder to put on a dashboard than a green badge. Wrapping Up So you've written your code and your tests, and you achieve 100% code coverage. What does that tell you about your tests? It tells you very little. Are they enough or not? You don't know if they are enough or not just by looking at code coverage. This metric measures which lines of code a test suite executes. It does not measure what portion of the behavior space of those lines has been verified. A test suite can achieve 100% line coverage while leaving critical financial calculations unverified, card data unprotected, and failure modes completely untested. The distinction between coverage of code and coverage of behavior is not semantic. It is the difference between a number on a dashboard and evidence of system quality. Coverage data is useful as a gap-finder to identify code that no test reaches. However, it is dangerous as a quality proxy, because it conceals the difference between tests that execute code and tests that verify behavior. Coverage targets make this worse, not better, because they create an incentive to optimize for the number rather than for the evidence. The rational response to a coverage target is to achieve it by writing meaningful tests that cover the behavior surface of the system. conceptprecise definition Code coverage The proportion of source code statements, branches, or conditions executed at least once during the test run. Measured automatically. Does not indicate correctness. Behaviour coverage The proportion of the system's meaningful behaviors — success paths, error conditions, boundaries, failure modes — that are verified by the test suite. Requires human judgment to define and assess. Mutation score The proportion of automatically-introduced code mutations that the test suite detects. A direct measure of the test suite's sensitivity to behavioral change. More meaningful than line coverage as a quality signal. Assertion-free test A test that executes code but makes no meaningful claim about its output. Generates code coverage without generating evidence. The most common cause of 100% coverage coexisting with catastrophic defects. Coverage target A minimum coverage percentage enforced at CI level. Incentivizes coverage optimization rather than evidence generation. Does not improve quality; frequently degrades it by displacing investment from meaningful tests to coverage-padding tests. Behaviour inventory A pre-test enumeration of the behaviors a system must exhibit, categorized by consequence of failure. The correct foundation for a test suite.

By Stelios Manioudakis CORE

In Q1 2026, three major agent infrastructure platforms dropped in nine weeks: OpenAI Frontier, the AWS Stateful Runtime, and Anthropic's Claude Managed Agents. What happens eighteen months from now when the model we built on gets deprecated, or we need to renegotiate pricing? I ran each platform through the same evaluation. Most teams I've talked to miss the question that actually matters: will your infrastructure survive a model change? Here's the five-pillar framework I use to find out, with real tests and code for each one. The Harness Is Not the Model Most platforms bundle two different things under one product name: the reasoning model and the harness. The model reasons. The harness does everything else that makes it an agent capable of executing multi-step tasks: Working and persistent memory – what the agent retains during a task and what carries over between sessionsTool execution – registering available tools, intercepting calls before they run, handling failuresSkills – reusable multi-step behaviors built on top of toolsOrchestration – coordinating multiple agents, routing subtasks, collecting resultsGovernance – access controls, human-approval gates, audit trails When the harness is tightly coupled to one model, you've made an architectural bet. That's not necessarily wrong, but you should make it on purpose. Five Portability Tests Run all five before you commit. 1. Memory: Who Does It Belong To? Run a two-session test. Session one: have the agent learn something concrete. A service that should always be read-only. A user preference. An entity it should recognize. Kill the session. Start a fresh one and ask whether it remembers. Then export the raw memory store: Shell # Export memory from your agent platform agent-cli memory export --session-id=<id> --output memory_dump.json # What you want to see: plain readable JSON cat memory_dump.json Portable memory looks like this: JSON { "entity": "postgres_prod", "type": "database", "note": "Read-only account. Never write.", "created": "2026-04-10T09:14:22Z", "tags": ["production", "restricted"] } Locked memory looks like this: JSON { "_type": "openai.memory.EphemeralObject", "_model_ref": "gpt-5.4-turbo-internal", "_blob": "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcY..." } If you need the provider SDK to deserialize it, it belongs to them. Persistent memory that survives a migration must be readable in a text editor without any provider dependency. If it's opaque binary or references internal provider objects, it won't survive. 2. Tools: MCP or Provider Subclass? Check how your tools are defined. MCP is now the de facto standard for provider-neutral tool registration. A tool defined in MCP works with any harness that implements the protocol. MCP-compatible (portable): JSON { "name": "query_database", "description": "Read-only SQL query against the analytics DB.", "inputSchema": { "type": "object", "properties": { "query": { "type": "string" }, "timeout_ms": { "type": "integer", "default": 5000 } }, "required": ["query"] } } Provider-locked (rewrite required when you leave): Python # This class dies with your provider contract class QueryDatabaseTool(openai.BaseTool): name = "query_database" def run(self, query: str) -> str: return db.execute(query) Plain JSON or YAML you can hand to another harness is portable. I've seen teams underestimate the tool-layer rewrite by a factor of three. It compounds fast when you have 30+ tools. For a practical look at how MCP connects agents to any API, the pattern holds across every provider. 3. Skills: Do They Survive a Model Swap? Skills are where lock-in accumulates without anyone noticing. A skill is a reusable multi-step sequence — something like search-summarize-route or draft-review-send. The problem is subtle: skills get built against one model's output format and response conventions. They work perfectly until they don't. Run this smoke test against a cheap model before you commit: Python SKILLS_TO_TEST = ["search_and_summarize", "draft_review_route", "incident_triage"] def smoke_test_skill(skill_name, model="gpt-4o-mini"): """ Not checking quality. Checking whether it completes. """ try: result = agent.run_skill( skill=skill_name, model_override=model, timeout=30 ) print(f"[PASS] {skill_name} on {model}: completed in {result.duration}s") return True except (ParseError, StepTimeoutError) as e: print(f"[LOCKED] {skill_name} on {model}: {e}") return False for skill in SKILLS_TO_TEST: smoke_test_skill(skill, model="gpt-4o-mini") smoke_test_skill(skill, model="mistral-7b") If it crashes: locked. If quality drops but it completes: portable. Those are completely different problems. 4. Orchestration: Grep Your Own Codebase Orchestration is where coupling gets expensive and invisible. If your planning agent parses sub-agent outputs using provider-specific response fields, swapping one model silently breaks everything downstream. The errors show up three layers away from the actual model call. Shell # Run this against your orchestration layer right now grep -rn "\.choices\[0\]" ./agents/ grep -rn "\.message\.content" ./agents/ grep -rn "openai\." ./orchestration/ grep -rn "anthropic\." ./orchestration/ If those grep results are long, you're coupled. Portable orchestration parses against schemas you control: Python # Locked: parsing a provider response object directly raw = agent_response.choices[0].message.content result = json.loads(raw) # Portable: your schema, not theirs result = TaskOutput.model_validate( parse_task_output(agent_response, schema=SUBTASK_SCHEMA) ) The fix isn't dramatic, but catching it late means touching a lot of code under pressure. For teams running multi-agent workflows with AWS Step Functions, this schema boundary becomes even more critical when agents span different provider runtimes. 5. Governance: Export and Read It The global race to govern AI agents has made governance a first-class concern — but most teams still treat it as an afterthought until a compliance conversation forces the issue. Governance covers what tools can be called, what needs human sign-off, and what goes into the audit log. It should live in your harness, not baked into a provider's permission system. Export it and look at what it references: Shell agent-cli governance export --format=yaml > governance_config.yaml cat governance_config.yaml Portable governance config: YAML policies: - name: restrict_production_writes applies_to_roles: [sre_agent, oncall_agent] deny: actions: [database.write, infrastructure.delete] resources: ["prod/*"] require_approval: - action: infrastructure.restart approvers: [oncall-lead] audit: log_all_tool_calls: true retention_days: 90 Locked governance config: YAML openai_platform: policy_set_id: ps_abc123xyz workspace_id: ws_prod_999 iam_role_binding: roles/openai.agentOperator permission_set: OPENAI_PROD_RESTRICTED If your config references provider IAM primitives or platform-specific IDs, it stays behind when you leave. You're rebuilding from scratch. The role of AI in IAM is evolving fast — your governance layer needs to be portable enough to keep up. How the Platforms Actually Scored OpenAI Frontier/AWS Stateful Runtime Well-built if you're staying on GPT. Memory in OpenAI infrastructure, tools through OpenAI SDK, orchestration layer assumes GPT conventions. The April 2026 Agents SDK update added workspace portability via Manifest and four memory tiers. Genuine improvements. But the docs say it plainly: designed for OpenAI models. The reference examples all use gpt-5.4. Know what you're signing up for. Claude Managed Agents The architecture is interesting. Three independent interfaces: session log, brain layer (Claude), and code execution sandbox. Each one can fail or be replaced without breaking the others. Anthropic published their reasoning: harness code encodes model limitations, and those limitations become technical debt as models improve. They built the interfaces to be swappable. The catch is the brain interface runs Claude. Changing that is a migration, not a config change. Claude Platform on AWS (GA May 11) Solves the governance and procurement problem better than anything else I tested. Auth through AWS IAM, billing through AWS Marketplace, audit logs in CloudTrail alongside your existing AWS services. Your governance policies for Claude agents live in a system your security team already knows. What it doesn't solve: data is processed by Anthropic outside the AWS boundary, so no Bedrock regional residency. And it's still Claude. The five tests above don't change. Open-source (LangChain Deep Agents, Letta, CrewAI, Microsoft Agent Framework) Model selection is a config variable. Memory in open formats. Tools in MCP. Governance is harness-owned. Less polished, more infrastructure work. That's the honest tradeoff. If you expect the model layer to keep moving, this is where you start. For teams already building compound AI systems for scalable workflows, the open-source stack plugs in naturally. Checklist Before You Commit These take a few hours. A migration takes months. Export persistent memory, open it in a text editor without a provider SDKCheck tool definitions: MCP/open schema vs. provider SDK subclassRun your top three skills against a non-primary model. Do they complete?Grep orchestration code for provider-specific response object referencesExport governance config, check whether it references provider IAM primitivesAsk: if this provider relationship ends tomorrow, which assets do I actually own? The platforms that shipped this year solve real problems. If you need production-grade agents fast and you're not switching models anytime soon, the managed options are good. The five tests tell you exactly what you're trading. Run them before you commit, and the decision is deliberate. Skip them, and you'll find out later at a worse time.

In a world where testing is mainly test execution, it is reasonable to expect that when people test, they simply expect to find bugs. "We test to find bugs" is the answer given in job interviews. An answer repeated in onboarding materials, embedded in KPI frameworks, and implicitly assumed in every conversation about when software is ready to ship. It is so thoroughly taken for granted that questioning it sounds absurd, like questioning whether hammers are for hitting things. Testing does find bugs since it involves test execution. But testing involves much more than execution. Framing testing as just a bug-finding activity results in a number of consequences that this article will discuss. The History To understand why the bug-finding framing is so persistent, it helps to understand where it came from. It did not emerge from careful thinking about what testing is for. It emerged from the early days of software development. In the sequential, phase-gated models of software development that dominated from the 1960s through the 1990s, testing was a phase. It came after coding. Its purpose was to check whether what had been built worked as specified. The people who did it were called testers, and the outputs of their work were bug reports. The entire apparatus — the phase, the role, the output — was organized around the assumption that defects were artifacts to be found and removed, like impurities in a manufactured component. This was a coherent model for its time and context. It mapped reasonably well onto hardware-adjacent software development. Change was expensive, requirements were relatively stable, and the cost of late discovery — while significant — was at least bounded by the pace of development cycles measured in months or years. Then software development changed. It was all about speed of development. Cycles accelerated. Requirements destabilized. Systems grew interconnected. The gap between what could be specified in advance and what users actually needed widened dramatically. The contexts in which the bug-finding model made sense dissolved — but the model remained. It was too embedded in tooling, organizational structures, hiring practices, and professional identity to dissolve with them. What remained was a model designed for a slow, sequential world, applied to a fast, iterative one. The phase became a sprint ceremony. The tester became a QA engineer. The bug report became a Jira ticket. The vocabulary updated. The underlying assumption, that testing is what you do to find defects in something already built, did not change, however. Bug Finders in the SDLC A bug may exist anywhere in the SDLC. A missing or ambiguous requirement. A design that fails to handle important scenarios. Incorrect or incomplete code. A flawed or missing test that allows defects to escape. A deployment or configuration error that causes the software to behave differently in production. So, in theory, bug finders could be employed all around the SDLC. But in practice, when testing is framed as bug finding it often becomes a late-stage filter. This determines what teams hire for, how they structure work, what they measure, how they explain incidents, and where they invest. It's not a minor detail! When organizations test to find bugs, features are typically designed, developed, and code-reviewed. A hand-off model is employed whereby, when development is complete, the feature moves to the QA team. The QA team writes test cases based on the requirements document. They execute the tests. They find several bugs. The bugs are returned to the developers. The developers fix the bugs. The feature returns to QA for regression testing. It passes. It ships. Under such settings, bugs are often exposed late, during the testing process after development. When information about a defect arrives late, decisions have already been made on the assumption that the defect does not exist. Those decisions must be revisited, revised, or lived with. The longer the latency, the more decisions have accumulated on a false foundation. Hiring Bug Finders One of the most direct and least-discussed consequences of the bug-finding framing is what it does to hiring. When an organization believes testing is about finding bugs, a good tester is a good bug-finder. But bug-finding, as a job description, selects for a specific and narrow set of skills. The typical bug-finding hire is technically literate but not technically deep. Comfortable at designing/executing test cases, skilled at documenting defects clearly, experienced with the tooling of defect tracking. These are real and useful skills, but they are a small snapshot of the value that testing can produce. Testing requires the capacity to ask questions that have not been asked before. It requires understanding of system architecture, since you cannot generate useful information about a system that you do not understand. It requires risk reasoning — the ability to identify which regions of the system's behavior space carry the most consequence if they are wrong. It requires communication skills oriented not toward defect documentation but toward translating. Translating technical evidence into decisions that technical and non-technical stakeholders can act on. It requires curiosity: a genuine interest in what the system is actually doing, rather than in whether it matches an expected outcome. The Hiring Mirror Job descriptions reveal the framing. "Find and report defects" → bug-finding model. "Generate evidence about system behavior and communicate risk" → information-generation model. Most job descriptions in the industry describe the first. Most organizations need the second. The gap between them is filled, imperfectly, by individuals who developed the second set of skills despite a system that did not ask for them. Team Structures Information about software behavior is generated — or should be generated — all around the SDLC. By developers writing unit and integration tests. By product owners reviewing acceptance criteria. By architects thinking through failure modes. By security engineers probing for vulnerabilities. By data analysts examining production telemetry. The bug-finding framing implicitly assigns all of this to testers. The team structures produced are almost universally recognizable, because they follow directly from the assumption that testing is a downstream, verification activity. This has major implications. The Bottleneck Problem When QA is the terminal stage before release, it becomes a bottleneck by construction. Development velocity is limited not by the rate at which developers can produce code but by the rate at which QA can process it. Organizations respond to this by automating testing and/or hiring more QA engineers, which increases throughput without addressing the underlying problem. In some cases, organizations have also reduced the time allocated to testing. This reduces thoroughness without anyone explicitly deciding to accept that risk. The Ownership Problem So, developers write code and testers verify that the code works OK. Does this imply that QA owns quality? Organizations that answer yes to this question open the door to other dangers. If testers own quality, then developers have a reduced incentive to ensure the correctness of their own work. After all, it's someone else's job. The empirical result is predictable and well-documented: defect rates increase when developers operate in an environment where defect detection is handled downstream. The Adversarial Problem In organizations where the hand-off model is combined with metrics that reward development velocity and measure QA performance by defect counts, a perverse incentive structure emerges. Development is rewarded for speed; QA is rewarded for finding bugs. The implicit incentives push these functions into an adversarial relationship — developers resenting QA for slowing release, QA resenting developers for producing buggy code. This adversarial dynamic is not a cultural problem. It is an incentive problem produced by a framing error. structurewhat it reveals about the framing Separate Dev and QA teams with hand-off Testing is a distinct downstream activity. Quality is owned by QA QA measured by bugs found Success is defined as defect detection, not information generation 'QA sign-off' as a release gate Testing is a filter, not a continuous discipline Testers assigned to features after development Testing is verification of completed work, not a parallel information stream Embedded QA in cross-functional teams Testing is a continuous contribution to the development process Shared Definition of Done including test evidence Quality is a property of the team's output, not a downstream check Developers writing and owning test suites Information generation is distributed to the point of production The Release Process The release process is where the framing becomes most visible. This is because the release process is the moment at which the organization is forced to answer the question: do we know enough to ship? The answer to that question is operationalized as: have we found enough bugs? The proxies are test execution rates, pass/fail ratios, open defect counts, and severity distributions. A release is approved when the open defect count is below a threshold, the regression suite passes, and the severity of known issues is judged acceptable. The assumption embedded in this process is that the tests have found the bugs, the bugs have been fixed, and what remains is known and manageable. This assumption, however, is almost never fully warranted. Experienced release managers know it. Every release approval is accompanied by a degree of unspoken uncertainty — a sense that the tests have probed what they have probed, and that what they have not probed remains unknown. This uncertainty is rarely made explicit, because the framework does not have a vocabulary for it. The question "what do we not know about this system's behavior?" often has no formal place in a release process organized around defect counts. The question is not "have we found the bugs?" It is "have we generated sufficient evidence about the risks that matter most, and is the residual risk in the untested regions acceptable given what we know about this system's usage and consequences?" This is a harder question to answer, but it is the right question. It produces better decisions since it forces the organization to name its assumptions rather than hide them under a metric. Incident Post-Mortems When something breaks in production, the organization is forced to explain how it happened. Post-mortems tend to converge on a specific narrative: the defect was present but was not found by testing. The conclusions that follow are predictable — more tests, better coverage, improved test cases for this class of defect. Sometimes a retrospective guilt is attached to a specific test or test type that should have caught the issue. Such a narrative can be dangerous for the cohesion of teams. It is also incomplete. It asks what the filter missed, when the right question is why the filter was the primary defense against this class of failure. It treats the absence of a specific test as the root cause. However, the actual root cause may be that the organization's testing model was structurally incapable of generating the information that would have prevented the incident. To put simply, why does testing only follow development? A few legitimate questions for post-mortems are: What information did we have about this region of the system's behavior before the incident?Why was that information insufficient — was it absent, was it present but not acted on, or was it structurally impossible to generate with the testing approach we were using?At what point in the SDLC was information about this failure mode accessible, and why did it not reach the people who could have acted on it?What does this incident tell us about the shape of our evidence — where are the gaps in what we know about our system's behavior? These are different questions from "which test should have caught this." They produce different answers and different remediation paths. And they are more honest, because they acknowledge that the incident may not always be a failure of test execution. It was a failure of information generation at some point in the SDLC, and the post-mortem's job is to find that point. The Blame Allocation Problem When testers are bug-finders, there can be an uncomfortable dimension to post-mortems that deserves naming directly. When an incident occurs, the implicit question "who was supposed to find this bug?" resolves to the testers. This is expected given the framing. If testing is the activity by which bugs are found, and this bug was not found, then the people responsible for testing bear some responsibility for the failure. This is counterproductive since post-mortems should be blameless. The blamelessness in post-mortems is more important than how we frame our jobs, but most importantly, the two interact with each other. If you find that blaming is part of your post-mortems, then I suggest first identifying why. Why do you need blame in your post-mortems? As you walk through the path to get rid of it, if framing your jobs stands in the way, then a good idea is to rethink the fundamentals. Fundamentals like: What is quality? How do we develop our code? Why do we test? Who owns quality, and how do we learn in software products/projects? Try to get the wider picture possible. Wrapping Up The belief that testing finds bugs is a consequential framing error in software engineering. Not because it is false — testing does find bugs — but because it is so incomplete that organizing a quality program around it produces systematic, predictable, and expensive failures. Testing is much more than bug finding. Testing generates information about software behavior. That information is the raw material of every quality decision made in the SDLC. The earlier that information is generated, the cheaper it is to act on. The later it arrives, the more decisions have accumulated on a false foundation, and the higher the cost of correction. After all, the answer to questions like "what is testing?" propagates through every dimension of how organizations work: who they hire, how they structure teams, what they measure, how they make release decisions, and how they explain incidents. Changing the framing is not a cosmetic exercise. It is a structural change that requires deliberate action at the level of process, metrics, and professional development. If we frame testing as information generation and try to see the wider picture, for example, as in this article, then things could be different in many ways, as shown below. Dimensionbug-findinginformation-generation Purpose of testing Find and remove defects Generate evidence about system behaviour throughout the SDLC Timing Late-stage filter after development Continuous discipline from requirements to production Ownership Owned by QA Distributed across the team; QA provides depth and system perspective Hiring Defect documentation skills Risk reasoning, system understanding, evidence communication and soft skills Team structure Separate dev/QA with hand-off Embedded, cross-functional, shared quality ownership Release decision Gate based on defect count Risk assessment based on evidence quality and coverage Post-mortem 'Which test missed this?' 'Where did information generation fail in our SDLC?' Success metric Bugs found, pass rate Escape rate, detection stage, risk coverage Cost profile Low visible cost early, high hidden cost late Higher visible cost early, lower total cost overall

By Stelios Manioudakis CORE

Modern software delivery is complex. Developers are responsible not only for writing code that meets business requirements — both functional and non-functional — but also for navigating a long chain of supporting steps. From containerization, testing, configuration, security, deployment, and monitoring, each stage often relies on specialized tools and teams. When these processes aren’t standardized, every project risks reinventing the wheel. The result is inconsistency, delays, and frustration. For example, requesting a new test environment might require submitting detailed tickets to a DevOps team, slowing timelines and draining energy. As organizations scale, so does the complexity — and the pain of delivery. Platform engineering addresses these challenges by creating shared, reliable foundations. It provides self-service tools, standardized workflows i.e., golden paths, and built-in guardrails, enabling teams to focus on what matters most: writing code and shipping features. This article explores what platform engineering is, why it matters, and how it helps organizations move faster while reducing developer burnout. It also examines common challenges and how to avoid turning platforms into yet another layer of complexity. Platform Engineering Definition Platform engineering is a practice of building and maintaining an internal, self-service platform that makes it easy for development teams to build, deliver, and operate software. Key principles of platform engineering are: Self-service (with guardrails) → developers should be able to build, deploy, and operate services independently — without filing tickets for routine tasks. Also ensuring automated guardrails for compliance and cost control.Golden paths, not golden cages → provide opinionated, well-supported paths that make the right thing easy — without preventing teams from choosing alternatives when needed.Product mindset → treat the platform as a product. Define users (developers), gather feedback, measure adoption, and iterate based on value delivered.Reduce cognitive load → abstract away infrastructure and operational complexity that does not directly contribute to the developer’s core task, i.e., building and shipping business logic. It's imperative to note that Platform Engineering is not DevOps, but DevOps scaled through product thinking — treating developers as customers and the platform as the product. Adoption Journey Large organizations often face delivery challenges that rarely make it into executive summaries. Issues like developer friction, inconsistent and/or duplicate tooling, and fragmented workflows are deeply embedded in day‑to‑day operations. Their impact — delayed releases, inefficiencies, and frustration — may be visible, but the root causes often remain hidden or disconnected from leadership narratives. Thus, the first step is discovery and validation. Organizations must surface real pain points through design thinking workshops, targeted surveys, analysis of past initiatives, and continuous community/user feedback. These insights form the foundation for defining a clear and grounded Platform Mission Statement — one that aligns platform capabilities with genuine organizational needs. Once the mission is clear, enterprises move toward unified platforms that standardize common tools and processes. This consolidation reduces duplication and improves reliability. Guiding Principles should be maintained via ADRs as a standardized platform for uniformity. Also, it's recommended to have decentralized decision-making to avoid bottlenecks and maintain long-term sustainability. To achieve this, use a community-driven approach via various guilds. As maturity grows, self‑service enablement becomes the focus — developers can provision infrastructure, build & deploy applications, perform verification, and integrate monitoring with minimal friction. This can be achieved via Internal Developer Platforms (IDP) and Internal Developer Portals. Finally, mature organizations embrace continuous improvement. The platform evolves like a product — guided by developer feedback and metrics. The feedback loops should act to adapt the platform and be evolutionary in nature. Being preventive or proactive, rather than reactive, goes a long way in achieving a successful platform implementation. Internal Developer Platforms (IDP) vs. Internal Developer Portals A common source of confusion in platform engineering is the distinction between an internal developer platform (IDP) and an internal developer portal. While these concepts are related and often work in tandem, they serve distinct purposes and have different architectural and user experience implications. Internal Developer Platform (IDP) An IDP is the “engine room” of platform engineering. It is a cohesive set of tools, frameworks, and automation scripts that standardize and automate the provisioning, deployment, and management of infrastructure and services. Key components typically include: Self-service infrastructure provisioning → Developers can request and manage resources (VMs, databases, clusters) via APIs or CLI tools, eliminating ticket-based workflows.Unified deployment and orchestration → Standardized CI/CD pipelines, container orchestration (e.g., Kubernetes), and Infrastructure as Code ensure consistent, reliable releases.Centralized configuration and secrets management → Version-controlled settings, automated secret management, and policy enforcement across environments.Automated monitoring and observability → Integrated metrics, logs, and tracing provide real-time visibility into system health.Security and compliance automation → Policy-as-code frameworks (e.g., OPA) enforce security and compliance at every stage. IDPs are typically built and maintained by platform engineering teams and are consumed by developers and operations teams to accelerate software delivery and reduce operational risk. Internal Developer Portal An internal developer portal is the “front door” to the platform. It provides a user-friendly interface (often a web dashboard) that aggregates documentation, service catalogs, APIs, and organizational guidelines. Key features include: Service catalog → Centralized inventory of services, APIs, and infrastructure, supporting discoverability and ownership tracking.Integration ecosystem → Unified view of the development toolchain, integrating with version control, CI/CD, observability, and project management tools.Self-service workflows → Guided forms and wizards for routine operations (e.g., provisioning, deployments), with built-in approval workflows and RBAC.Onboarding and knowledge sharing → Centralized documentation, onboarding guides, and community Q&A features to accelerate ramp-up and collaboration.Metrics and scorecards → Dashboards tracking service health, maturity, and compliance, providing actionable insights for improvement. Portals are typically used by application developers, product teams, and managers to discover services, access documentation, and initiate self-service workflows. When to Use Each (or Both) Start with an IDP when the primary pain points are manual infrastructure provisioning, inconsistent environments, or the need for standardized automation.Start with a Portal when discoverability, onboarding, and knowledge sharing are the main challenges, or when existing tools are underutilized due to lack of visibility.Combine Both for maximum impact: the IDP provides the backend automation and guardrails, while the portal exposes these capabilities through an intuitive, developer-friendly interface Team Topologies and Platform Engineering The success of platform engineering is deeply influenced by organizational structure and team interactions. The “Team Topologies” framework provides a powerful lens for designing team structures that optimize for effective platform adoption Four Fundamental Team Types Stream-Aligned Teams → Aligned to a flow of work from a business domain (e.g., a product or service). They own the end-to-end delivery and operation of features.Platform Teams → Build and maintain internal platforms that provide reusable services and capabilities to stream-aligned teams, reducing their cognitive load.Enabling Teams → Help stream-aligned teams overcome obstacles, adopt new technologies, or fill skill gaps.Complicated Subsystem Teams → Own subsystems that require deep specialist knowledge (e.g., advanced algorithms, core infrastructure). Three Team Interaction Modes Collaboration → Teams work together for a defined period to discover new solutions.X-as-a-Service → One team provides a service that another team consumes with minimal interaction.Facilitation → One team helps another team acquire new skills or capabilities. Relevance to Platform Engineering Platform Teams as Product Teams → Platform engineering teams should operate as product teams, treating stream-aligned teams as customers, gathering feedback, and iterating on platform featuresReducing Cognitive Load → The primary goal of the platform team is to reduce the cognitive load on stream-aligned teams, enabling them to focus on delivering business value rather than infrastructure concernsClear Interfaces and Boundaries → Well-defined APIs, documentation, and support channels ensure that platform capabilities are discoverable and consumable as a service, minimizing dependencies and handoffsContinuous Adaptation → Team boundaries and responsibilities should evolve as business needs and technologies change, with feedback loops guiding organizational adjustments Conway’s Law and Its Applicability to Platform Engineering Conway’s Law, articulated by Melvin Conway in 1967, states that “organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” In the context of platform engineering, this law has profound implications. How Conway’s Law Shapes Platform Design Organizational Silos Lead to Siloed Platforms → If engineering teams are organized in silos (e.g., by business unit or product line), the platforms they build will reflect this fragmentation, resulting in duplicated tools, inconsistent practices, and integration challengesCross-Functional Collaboration Enables Cohesive Platforms → Successful platform engineering requires cross-functional teams that span development, operations, security, and compliance, ensuring that the platform addresses the needs of all stakeholders and avoids becoming a new silo.Intentional Organizational Design → To achieve coherent, scalable platforms, organizations must deliberately design their communication structures and team interactions to support shared standards, rapid feedback, and continuous improvement. Real-World Example - Monolith to Microservices → Organizations transitioning from monolithic architectures to microservices often reorganize teams around domains or services. If team boundaries are not aligned with desired system boundaries, the resulting architecture may become fragmented or inconsistent, reflecting the underlying communication patterns rather than optimal technical design Platform Engineering as Both Solution and Symptom → Platform engineering often arises as a response to the silos and fragmentation created by previous organizational structures. However, if not implemented with a product mindset and cross-team alignment, platform engineering can inadvertently create new silos, perpetuating the very problems it seeks to solve Metrics and KPIs to Measure Platform Success Measuring the impact of platform engineering is essential for demonstrating value, securing buy-in, wider adoption, and guiding continuous improvement. Common Metrics DORA Metrics → Deployment frequency, lead time for changes, change failure rate, mean time to recovery (MTTR).SPACE Framework → Satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow.Adoption Rates → Percentage of teams and services using the platform.Time to Onboard → Time required for new developers to become productive.Operational Metrics → Incident rates, uptime, resource utilization, and cost savings.Developer Satisfaction: Surveys, Net Promoter Score (NPS), and qualitative feedback. Feedback Mechanisms Surveys and Office Hours → Regular check-ins with platform users to gather qualitative and quantitative feedback.Telemetry and Usage Analytics → Automated tracking of platform usage, feature adoption, and workflow bottlenecks. Limitations and Common Challenges of Platform Engineering Despite its benefits, platform engineering is not without challenges and limitations. Organizational Challenges Resistance to Change → Teams may be reluctant to adopt new tools or workflows, especially if they perceive a loss of autonomy or increased complexity.Alignment and Buy-In → Achieving consensus on standards, priorities, and platform direction can be difficult in large or distributed organizations.Skill Gaps → Building and maintaining platforms requires expertise in infrastructure automation, CI/CD, security, and developer experience, which may be lacking in existing teams. Technical Challenges Overengineering → Building overly complex or rigid platforms can lead to low adoption and maintenance burden.Integration Complexity → Aggregating data and workflows from diverse tools and systems requires careful planning and robust integrations.Legacy Systems and Technical Debt → Integrating with or migrating from legacy tools and architectures can be time-consuming and costly. Cultural Challenges Product Mindset → Treating the platform as a product, with continuous feedback and iteration, is essential but often overlooked.Avoiding the “Golden Cage” → Mandating platform adoption without addressing real developer needs can lead to resentment and shadow. Measurement and ROI Lack of Metrics → Many organizations fail to measure platform adoption, impact, or ROI, making it difficult to justify continued investment or guide improvements. Products Over Projects In platform engineering, the distinction between a project mindset and a product mindset is crucial. Project Mindset Focus → Deliverables, deadlines, and completion.Approach → Work is structured around a defined scope with a start and end date.Outcome → Once the project is “done” the team moves on, often with limited ongoing ownership.Risk → Platforms built this way may stagnate, as continuous improvement and user feedback loops are not prioritized. Product Mindset Focus → Long-term value, user experience, and continuous evolution.Approach → The platform is treated as a living product with ongoing investment, iteration, and support.Outcome → Teams own the platform end-to-end, ensuring it adapts to changing business and developer needs.Benefit → Encourages innovation and alignment with evolving enterprise strategy. In short, with a project mindset, tasks are done as “Deliver and finish,” while with a product mindset, it's “Deliver, own, and evolve.” Relationship Between Platform Engineering, DevOps, and SRE While platform engineering, DevOps, and site reliability engineering (SRE) share common goals, they address different layers of the software delivery lifecycle. DevOps Focus → Cultural and organizational transformation to break down silos between development and operations, emphasizing collaboration, automation, and continuous delivery.Practices → CI/CD, infrastructure as code, shared responsibility for quality and reliability. SRE Focus → Applying software engineering principles to operations, with a strong emphasis on reliability, scalability, and incident response.Practices → Service-level objectives (SLOs), error budgets, automated monitoring, and incident management. Platform Engineering Focus → Building and maintaining internal platforms that provide standardized, automated, and self-service capabilities for development teams.Practices → Product mindset, self-service, golden paths, policy-as-code, and platform-as-a-product. How They Interact Platform engineering provides the foundation on which DevOps and SRE practices can scale, standardizing workflows, embedding automation and compliance, and enabling self-service for developers and operations teams.DevOps and SRE teams collaborate with platform engineers to ensure that the platform supports reliability, scalability, and continuous improvement. Case Studies Netflix used their platform to solve developers’ challenges to manage multiple services and software, knowing which tools exist, and switching contexts between tools. Zalando leveraged their platform to unify the developer experience, promote compliance by default, and improve how the company operated over time. Carlsberg’s Gaia platform automated infrastructure provisioning, embedded compliance, and provided self-service capabilities, reducing manual work and accelerating project delivery. The platform’s success was attributed to cross-functional collaboration, a product mindset, and continuous feedback from developers eBay’s Velocity initiative (2020) boosted engineering productivity, cutting deployment times from 10 days to 1–2 and enabling same‑day mobile releases. Despite technical success, cultural resistance, outdated tech choices, and poor strategic execution prevented business growth. Conclusion Platform engineering represents a paradigm shift in how modern organizations build, deliver, and operate software at scale. By abstracting complexity, standardizing workflows, and empowering developers through self-service and automation, platform engineering accelerates delivery, improves reliability, and optimizes costs. However, its success depends on more than just technology — it requires intentional organizational design, a product mindset, continuous feedback, and a careful balance between standardization and flexibility. As the discipline matures, organizations that invest in platform engineering as a strategic capability will be best positioned to thrive in an increasingly complex and competitive digital landscape. The golden rule of platform engineering → Treat your platform as a product, your developers as customers, and adoption as a metric — not a mandate. Platform 2.0 — The AI‑Native Evolution Platform 2.0 represents the next stage of platform engineering, where artificial intelligence becomes a built‑in capability rather than an add‑on. It transforms platforms from static automation frameworks into adaptive, learning ecosystems that continuously optimize themselves. Core Principles Intelligence at every SDLC stage → AI augments design, development, testing, deployment, and operations with predictive and generative capabilities.Continuous learning → Feedback loops from telemetry and user behavior refine architecture and performance automatically.Autonomous optimization → Platforms self‑tune resources, detect anomalies, and evolve configurations without manual intervention.Human‑AI collaboration → Engineers focus on strategic design and governance while AI handles repetitive and analytical tasks. Platform 2.0 enables faster delivery, higher reliability, and smarter scalability, redefining platform engineering as an AI‑native discipline — intelligent, adaptive, and perpetually evolving. References and Further Reads Platform Engineering — WikipediaWhat is Platform EngineeringInfoQ trend report Platform Engineering as early adoptersThoughtWorks tech radar recommends adopting Platform Engineering as early as Apr/Oct 2021.Team TopologiesDZone survey where nearly half of respondents indicate using platform engineering.

By Ammar Husain CORE

AWS Glue makes it easy to get a PySpark pipeline running quickly. It is significantly harder to build one that stays maintainable as logic grows, performs reliably at scale, and does not quietly accumulate operational debt over time. Most Glue pipelines start simple and become difficult to manage gradually — formulas get hardcoded, modules grow without boundaries, output files proliferate, and before long a single job is doing too many things in ways that are hard to test, hard to debug, and expensive to change. This article presents a set of design principles drawn from production Glue ETL pipelines processing billions of rows. Each principle is independent — you do not need to adopt all of them to benefit from any one. But together they form a coherent approach to building Glue pipelines that are modular, observable, cost-efficient, and built to last. Principle 1: Externalize Logic Into Config, Not Code The single most impactful structural decision in a Glue pipeline is where business logic lives. When formulas, dataset references, column selections, and filter conditions are hardcoded in PySpark, every change requires modifying job code, redeploying, and re-validating the full pipeline. A one-line formula change carries the same deployment risk as a structural refactor. Over time, this creates a strong disincentive to make changes, and the pipeline calcifies. The better pattern is to treat the Spark job as a generic executor and externalize all business-specific declarations into configuration. Formulas are declared as config entries with operands, rounding rules, and output names. Dataset loading behavior — which table, which columns, which filters, whether to cache — is declared per source rather than scripted per job. Schema shapes for complex types are declared explicitly rather than inlined. JSON { "source_table": "headcount_actuals", "database": "finance_db", "select_columns": ["site", "badge_type", "headcount", "fiscal_week"], "filters": [{"column": "is_active", "value": "Y"}], "rename": {"hc_count": "headcount"}, "cache": true } When a new dataset is needed, a new config entry is added — no Spark code changes. When a formula changes, the config entry is updated — no job redeployment required. The job itself becomes stable and generic; only config changes as business requirements evolve. This principle pays increasing dividends over time. Pipelines with externalized logic are faster to modify, safer to deploy, and easier to hand off because the business rules are readable independently of the execution engine. Principle 2: Design Modules With Explicit Boundaries A Glue job that does everything in one place is easy to write and hard to maintain. As pipelines grow, the instinct to add more logic to an existing job accelerates technical debt faster than almost any other decision. The more durable pattern is to decompose computation into modules with explicit input and output contracts. Each module receives one or more DataFrames, applies a focused set of transformations, and produces a named output DataFrame. Modules communicate exclusively through in-memory DataFrame references — there is no disk I/O between stages, no shared mutable state, and no implicit dependency on execution order beyond what the data flow itself requires. Utilities follow the same boundary principle, organized into two layers. Generic pipeline utilities handle cross-cutting concerns — file writing, dataset loading, filtering, deduplication, pivot operations — and are shared across all modules. Module-specific utilities implement transformation logic scoped to a single module and are never invoked outside it. This structure means adding a new module requires only writing its scoped utilities and wiring it into the pipeline. The generic layer is never touched. Existing modules are never at risk from new module development. The downstream benefit is testability. Each module with clean boundaries can be validated independently using mocked PySpark DataFrames with no Glue environment required. Engineers can run pytest locally against individual modules, iterate quickly, and deploy only after local validation passes. Principle 3: Choose Your Job Topology Deliberately A common default in complex pipelines is to split computation across multiple Glue jobs, using S3 as the handoff layer between stages. This is sometimes the right choice — but it should be a deliberate decision, not an instinct. Multi-job topologies make sense when stages have genuinely different compute profiles, when intermediate outputs need to be reused independently by other consumers, or when a stage failure should not force a full recompute from the beginning. In these cases, job separation gives you independent retry boundaries, independent DPU sizing, and the ability to schedule stages on different cadences. Single-job topologies — where the full pipeline runs within one Spark session — make sense when all computation is tightly coupled, modules share the same input datasets, and intermediate outputs have no standalone value. Running everything in one session eliminates cold start overhead for intermediate stages, avoids the cost of serializing data to S3 and deserializing it back between jobs, and keeps the execution model simple to reason about: one trigger, one job, one result. The question to ask is whether the stages truly need to be independent. If intermediate S3 persistence adds coordination complexity without adding value — no independent consumers, no differential retry requirements, no meaningful DPU difference between stages — then collapsing to a single job is usually faster, simpler, and cheaper. If stages have real independence requirements, splitting them is the right call and the operational overhead is justified. Neither topology is inherently superior. The mistake is defaulting to one without evaluating the trade-offs for the specific pipeline at hand. Principle 4: Overlap Writes With Computation When Latency Matters Overlapping writes with computation is a well-established technique in high-performance computing, deep learning training, and heavy database operations. The core idea is to hide the slow latency of I/O operations by running them in the background while the CPU or GPU continues processing data. Rather than waiting for a write to complete before starting the next computation, both proceed simultaneously — I/O latency is absorbed into computation time rather than added on top of it. In Glue ETL pipelines, the same principle applies directly. In a pipeline where multiple output DataFrames are produced, the naive write strategy — complete all computation, then write all outputs sequentially — has two compounding problems. First, it creates a peak memory spike: all computed results are held in memory simultaneously while writes proceed one by one. Second, it serializes work that does not need to be serial: every millisecond spent waiting for S3 acknowledgment is a millisecond the Spark executors are idle. This is worth addressing only when latency is a meaningful constraint. For low-frequency batch jobs running overnight with no user-facing SLA, sequential writes are perfectly adequate. But for pipelines where users or downstream systems are waiting on results — or where job duration directly affects infrastructure cost — overlapping writes with computation delivers measurable wall-clock reduction. The two-phase write strategy implements this directly. Outputs from early modules are written to S3 in background threads immediately after those modules complete, running in parallel with later computation stages. By the time all computation finishes, a significant portion of the output data has already landed in S3. Remaining outputs are then flushed concurrently in a second phase. The implementation leans on Python's concurrent.futures.ThreadPoolExecutor to manage background write threads while the main Spark session continues computation on the driver. A generic write orchestration utility can wrap this pattern so individual modules never need to manage thread lifecycle directly — they simply declare their output and the utility handles scheduling, thread management, and error propagation. Python from concurrent.futures import ThreadPoolExecutor, as_completed def write_phase_a(write_tasks): with ThreadPoolExecutor(max_workers=len(write_tasks)) as executor: futures = {executor.submit(task["fn"], task["df"], task["path"]): task["name"] for task in write_tasks} for future in as_completed(futures): name = futures[future] future.result() logger.info(f"[Phase A] Write complete: {name}") The practical effect is that peak memory pressure is distributed over the job's lifetime rather than concentrated at the end, and total wall-clock time is reduced by the overlap between I/O and CPU-bound computation. For pipelines with many output datasets and a latency SLA to meet, the savings compound significantly. Principle 5: Right-Size Output Files With a Reusable Writer Utility Right-sizing output files is the practice of tuning file sizes to balance disk I/O performance, network transfer speeds, and downstream processing efficiency. Too many small files and downstream readers spend more time on metadata operations and S3 API calls than on actual data reads. Too few large files and parallelism suffers — readers cannot split work efficiently across threads or nodes. The target is consolidated, evenly sized files that match the read patterns of downstream consumers. Spark's default output behavior writes one file per partition, and partition counts are typically tuned for computation throughput rather than output shape. A job optimized for shuffle performance might produce hundreds of partitions, each containing a few megabytes of output data — perfectly reasonable for Spark internals, but harmful for any reader that comes after. This small file problem compounds over time as output partitions accumulate in S3 and the Glue Catalog metadata grows with them. The fix is a reusable writer utility that decouples output file sizing from Spark's internal partition count. Rather than accepting the default, the utility estimates the DataFrame's actual size, calculates the appropriate number of output files for a target file size — typically 128MB to 256MB per file — and coalesces partitions before writing. Python def write_optimized(df, output_path, partition_cols, target_file_size_mb=128): estimated_size_mb = df.rdd.map(lambda row: len(str(row))).sum() / (1024 * 1024) optimal_partitions = max(1, int(estimated_size_mb / target_file_size_mb)) df.coalesce(optimal_partitions) \ .write \ .partitionBy(*partition_cols) \ .parquet(output_path, mode="overwrite") Making this a shared generic utility rather than inline logic in each module has two practical benefits. First, it enforces consistent file sizing behavior across all outputs in the pipeline — no module accidentally writes thousands of tiny files because an engineer forgot to coalesce. Second, it centralizes the tuning knob: when the target file size needs to change — because downstream query patterns shift or a new consumer has different read characteristics — it changes in one place and applies everywhere. Right-sized output files improve Athena scan performance, reduce per-query S3 API costs, keep Glue Catalog partition metadata manageable, and make the output data easier to consume for any downstream system reading from S3. This is a low-effort, high-payoff improvement that applies to virtually every Glue pipeline writing to S3. Principle 6: Use Complex Types to Defer Denormalization SQL-based pipelines are constrained to flat, fully denormalized row structures at every intermediate stage because SQL has no native complex type support. This forces denormalization to happen early, inflating data volume at every subsequent join and aggregation. PySpark has native support for structs, maps, and arrays. Using these types at intermediate stages allows related values to be grouped logically without inflating row counts. A row that would require five denormalized rows in SQL can be represented as a single row with a struct or array column in Spark. Denormalization is then deferred to the final output layer only — applied once, at write time, for consumers that require flat structures. Everything upstream of the final write benefits from reduced volume, fewer shuffles, and faster joins. This principle is particularly impactful in pipelines with multi-level aggregations or wide schemas where dozens of metrics attach to the same dimensional key. Keeping those metrics grouped in a struct until the final output stage reduces the effective row count and join complexity throughout the pipeline. Principle 7: Build Observability Into Every Stage Glue jobs that fail silently or surface errors as opaque stack traces at the end of a long execution are expensive to debug. The investment in step-level observability pays back quickly the first time something goes wrong in production. The minimum viable observability pattern is row count logging at every materialization point. After each module completes and after each write, log the output row count with a descriptive label. This gives a running picture of data volume through the pipeline and makes it immediately obvious when a transformation has dropped rows unexpectedly or produced more rows than expected. Python def log_step(df, step_name): count = df.count() logger.info(f"[{step_name}] Row count: {count:,}") return df Pair this with a try/except/finally pattern at the job level that ensures spark.catalog.clearCache() is always called on exit — whether the job succeeds or fails — to release cached DataFrames and avoid memory leaks across retries. Python try: run_pipeline() except Exception as e: logger.error(f"Pipeline failed: {e}") raise finally: spark.catalog.clearCache() CloudWatch captures all logs automatically. When a job fails, the row count trail shows exactly where in the pipeline the problem occurred, making triage faster and reducing the time between failure and fix. Principle 8: Isolate Executions for Concurrency Pipelines that share compute resources across simultaneous executions create contention that is difficult to predict and expensive to manage. The common response — queue-based serialization — adds operational complexity without solving the underlying resource constraint. AWS Glue's execution model eliminates this problem structurally. Each job execution gets its own isolated DPU allocation. There is no shared compute pool. Ten simultaneous executions consume ten independent DPU allocations and do not interfere with each other in any way. Designing for this means treating each execution as fully independent: no shared state, no cross-execution coordination, no assumption about what other executions are running. Combined with idempotent writes — using overwrite mode so a retry produces the same result as the original execution — the pipeline becomes safe to run concurrently at any scale without additional coordination logic. The cost model reinforces this. Glue bills per DPU-second of actual compute consumed. An execution that takes eight minutes on 240 DPUs costs the same whether it runs alone or alongside a hundred other executions. There is no premium for concurrency and no shared pool to provision for peak load. Putting It Together These eight principles are independent but complementary. A pipeline that applies all of them is modular enough to develop in parallel, observable enough to debug quickly, cost-efficient enough to run at scale, and stable enough to maintain over time without accumulating structural debt. The quickest wins for most existing pipelines are Principles 1, 5, and 7 — externalizing logic into config, right-sizing output files with a shared utility, and adding row count logging at every stage. Each can be applied incrementally without restructuring the full pipeline. The remaining principles become more valuable as pipeline complexity grows and concurrency requirements increase. The underlying thesis is simple: a well-designed Glue pipeline should be easy to change, easy to test, easy to debug, and cheap to run. None of those properties require exotic infrastructure. They require deliberate design decisions applied consistently from the start.

By Janani Annur Thiruvengadam CORE