Testing, Deployment, and Maintenance Resources

DZone's Featured Testing, Deployment, and Maintenance Resources

The Telemetry Tax: Architecting Zero-Allocation Event Observability at 15B+ Daily Event Scale

Throughout my career, observability and monitoring systems have played a very important role in system stability, availability, and scalability when architecting, whether a system handles millions or billions of events per day. Having architected core infrastructure across communication platforms, enterprise compliance pipelines, and high-throughput transaction systems, I have repeatedly seen how background instrumentation can silently become a bottleneck under heavy production load. Telemetry and observability come with their own challenges. Instrumenting telemetry without degrading live business application performance, increasing request latency, or inflating cloud costs is often deprioritized initially but becomes very important as the system scales. Furthermore, it is a much harder problem to solve than processing domain events themselves in production. When an event-driven platform scales past 15 billion events per day, peaking between 200,000 and 400,000 TPS (transactions per second), the infrastructure costs and performance overhead of logging and tracing libraries create a severe operational bottleneck known as the Telemetry Tax. This article explains why addressing this telemetry tax is important and also provides you with brief insight into how you can start handling it with architectural changes at the initial level. The Multiplier Effect In a very scalable, high-throughput microservices architecture, business transactions rarely execute as isolated operations. An incoming event arriving at an API gateway typically passes through five to ten downstream services, caching layers, database instances, queues, message brokers (e.g., Kafka), etc. If every microservice hop generates three telemetry events in addition to the core business logic events, assuming two metric updates and a single log line, the resulting metric data scales exponentially: Regular workload (Core business Logic events): 15,000,000,000 events per dayMetrics and logs: 100,000,000,000 signals per dayStorage estimation: 15+ TBs of uncompressed JSON payloads daily Without specialized memory management and proper architecture, this massive volume of background instrumentation introduces critical failures in live production systems. Example: Garbage collection: Short-lived heap allocation for JSON log stringsThread Lock Contention: Sync metric exporters hold mutex locks while completing the request to send telemetry metrics, adding 10-20ms of latencyCascading downstream failures: Failure in downstream telemetry collector or failure in processing existing logs and metrics causes issues or backlog in upstream transaction workers, which may result in queue back-pressure and API timeouts. Engineering a Zero-Allocation Telemetry Pipeline To eliminate garbage collection pauses and thread lock delays, high-throughput microservices must decouple metric and log generation from core business logic, utilizing a middleware concept with pre-allocated memory pools and asynchronous buffers. Under heavy execution loads, standard string concatenations from logging and JSON marshaling for telemetry signals escape to the heap during Go's escape analysis. When millions of goroutines allocate short-lived objects on the heap per second, Go's runtime is forced into frequent concurrent sweep phases, consuming 15-20% of available CPU purely on garbage collection. By utilizing sync.Pool, we pre-allocate backing byte arrays that survive across the request lifecycle. When a worker goroutine finishes formatting a payload, the buffer pointer is reset (buf[:0]) and returned to the pool without invoking runtime.newobject . This keeps memory allocation in the critical execution path at zero bytes. Go package telemetry import ( "context" "sync" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" ) var tracer = otel.Tracer("zero-alloc-telemetry") // BufferPool pre-allocates memory var bufferPool = sync.Pool{ New: func() interface{} { b := make([]byte, 0, 1024) return &b }, } type RingBufferPipeline struct { telemetryChannel chan *trace.Span } func NewPipeline(bufferSize int) *RingBufferPipeline { return &RingBufferPipeline{ telemetryChannel: make(chan trace.Span, bufferSize), } } func (p *RingBufferPipeline) InstrumentEvent(ctx context.Context, eventID string) { // Retrieve pre-allocated byte slice from pool bufPtr := bufferPool.Get().(*[]byte) buf := (*bufPtr)[:0] defer func() { *bufPtr = buf bufferPool.Put(bufPtr) }() // Non-blocking trace span initialization ctx, span := tracer.Start(ctx, "ExecuteTransaction", trace.WithAttributes(attribute.String("event.id", eventID))) defer span.End() buf = append(buf, []byte("event_processed:")...) buf = append(buf, eventID...) // Async non-blocking dispatch select { case p.telemetryChannel <- span: default: // Add metric here to protect API response SLOs } } How to Control Ingestion Costs With Tail-based Adaptive Sampling Collecting 100% of telemetry traces across 100+ billion daily events leads to unsustainable tool ingestion costs (Regardless of the tool you use, e.g., Datadog, OpenTelemetry, Splunk). Standard Head-based sampling (deciding whether to keep a trace at the start of the request) drops fatal error traces while keeping millions of redundant HTTP 200 success traces. By deploying OpenTelemetry collectors configured with Tail-based adaptive sampling, trace spans are held in a 500 ms sliding memory buffer prior to routing: HTTP 200 / Successful Executions: Sampled at 0.1% to maintain baseline latency metricsHTTP 5xx / Latency exceptions (>200ms) / Server Errors: Retained at 100% for debugging, or any other analysis Production Performance Benchmark (Approx.) Refactoring telemetry infrastructure from sync logging to zero-allocation ring buffers and tail-based adaptive sampling helps with substantial performance improvements. (Note: The numbers in the table below are rough estimations based on high-scale modeling and past operational experience. Actual metrics may vary depending on your system and other aspects of architecture) Performance MetricSync TelemtryZero-allocation Adaptive TelemetryIngestion Throughput35k events/sec400k events/secp99 API response latency200ms40msAverage Monthly Cost$35-45k+$15-20kCPU overhead15-20% total CPU time spent on GC2-3% total CPU time spent on GC These estimations illustrate that the telemetry tax is not an inevitable cost of scale, but a consequence of applying synchronous, allocation-heavy architectural patterns to hyper-scale architectures. By refactoring memory management at the application level and applying adaptive filtering at the collector layer, engineering teams can achieve deep operational visibility while protecting system performance and cloud infrastructure budgets. Key Takeaways for System Architects Decouple the path: Never allow telemetry exporters to execute synchronously on the core business logic pathPre-allocate memory: Use memory buffer pools to eliminate garbage collection pauses during high-throughput event processingSample at the tail: Evaluate trace retention based on the execution outcome rather than making static decisions at the request ingress. More

The Silent Container Death: A TCP Dial That Never Times Out

By Alexander Fo

A pod goes into CrashLoopBackOff. You pull the logs expecting a stack trace, a panic, an error string - anything that points you somewhere. Instead, you get one line: Plain Text Loading config... And then nothing. No error. No exit message. The container is just gone, and a few seconds later it’s back, prints the exact same line, and disappears again. Magic. This is the story of chasing that silence to its root cause. TCP connection that was never going to succeed, and never going to fail either. At least not on any timescale a Kubernetes health check was willing to wait for. The Setup We were migrating backend services from a legacy message queue to Kafka. The new consumers ran side by side with the old ones in a “shadow mode.” That let us compare behavior before the real cutover. Part of that work meant pointing a dev environment at a managed Kafka cluster. (Think AWS MSK — the specifics don’t matter here.) We also updated the broker endpoint in config. In shadow mode, we send to both old and new queues, but only one of them processes the message. The other queue infrastructure just logs what it receives. The change looked trivial: swap one connection string for another, restart the pods, watch them come up. Instead, every pod that touched Kafka went straight into a crash loop. The only clue was that single “Loading config” line. Repeated forever. Why “No Error” Is the Error The instinct when a service crashes is to look for what it logged right before dying. Here that instinct is a trap. The absence of any further log output isn’t a hint — it’s the symptom itself. Two things had to be true simultaneously for this to happen: Something blocked the process long enough that Kubernetes’ health checks gave up on it and sent SIGKILL.Whatever the process wanted to log about being blocked never made it out of its internal buffers before the kill. That second point matters more than it looks. Go’s standard logger writes to os.Stdout. How a container runtime attaches to that stream determines whether output appears immediately or sits in a buffer. Buffering is common under load, or when the write target isn’t a real TTY. Consider a process blocked inside a library call, say dialing a broker. It never gets back to the point in its code where it would flush or print the next line. SIGKILL doesn’t give a process the chance to clean up. Whatever was sitting in a buffer is gone. From the outside, a service that’s actually deep in a hung network call looks identical to one that exited silently. Both just print “Loading config” and stop. The lesson here generalizes past Kafka. If a container’s logs stop dead with no error and no clean shutdown message, assume a hang-then-kill. Not a fast crash. Until proven otherwise. Reaching for the Network Layer Once “look at the application logs” stopped being useful, the next step was to get underneath the application entirely. Shelling into a node and watching the raw traffic (tcpdump) tells you what actually happened at the OS level. So does tracing the process’s syscalls with strace. Neither depends on whether the application ever got to log anything about it. What that showed: a TCP handshake that started and never finished. A SYN packet went out toward the broker; no SYN-ACK ever came back, and critically, no RST came back either. That distinction is the whole story. Connection refused is fast and loud. The remote host, or a firewall in front of it, actively sends back an RST packet. Your client’s connect() call fails almost immediately.Connection blackholed is slow and silent. Packets go out, and nothing comes back. The OS has no way to know if the remote end is down, unreachable, or just very far away. So it retransmits the SYN a few times with exponential backoff, then gives up. The kernel’s default TCP connect timeout can be well over a minute. In this case, the broker endpoint we’d configured was a private, VPC-internal address. It was reachable from some parts of the network, but not from the specific node group these pods landed on. No security group or routing rule was actively rejecting the connection – the packets were simply going nowhere. That’s the worst kind of network failure to debug from inside an application. Everything about it looks like the process is just slow, right up until it isn’t. Where the Health Check Made Things Worse None of this would have been quite so opaque if the failure had surfaced immediately. But the service’s startup path connected to Kafka before reporting itself healthy. On top of that, the Kubernetes startup probe carried a generous timeout, meant to avoid flapping on slow boots. That combination left the platform with no opinion about what was wrong. It just saw a container that hadn’t become healthy in time. So it did the only thing it can do here: kill it and try again. The pod restart count climbed. The backoff delay between restarts grew too - Kubernetes doubles it after repeated failures, up to roughly five minutes. Every fix we tried afterward seemed to take forever to take effect. That’s because we were still watching a container that hadn’t actually restarted yet. It was just waiting out its backoff window. Deleting the pod outright forced an immediate restart. That turned out to be the fastest way to test each hypothesis, rather than waiting for the backoff timer. The Fix, and the More Useful Part The actual fix was almost anticlimactic: switch to the broker’s public endpoint. In a real production setup, you’d instead fix the VPC routing or peering. That makes the private endpoint reachable from every node group that needs it. Once the TCP path was real, the connection succeeded instantly, and the crash loop stopped. The useful part isn’t the fix. It’s the checklist that could have saved us time: Handy Checklist Treat “one log line then silence” as a hang, not a crash. A clean crash logs an error. A silent one usually means something upstream killed the process mid-blocking-call.Go to the network layer early, not last. tcpdump or strace on the affected node will show you a stuck SYN in seconds. That’s far faster than adding print statements and waiting through several crash-loop cycles.Know the difference between “refused” and “blackholed” in your bones. An RST means someone answered and said no. Check credentials, ports, and application-level config. Silence means the packet never arrived. Check routing, VPC peering, security groups, and whether you’re using the right endpoint for the network you’re actually in.Set explicit, short connect timeouts in your client libraries. Don’t let a startup path inherit the OS’s default TCP connect timeout. The OS optimizes that default for general robustness, not for failing fast during a health check window.Make sure your logger flushes before anything that can block indefinitely. If a call to an external system can hang, log “attempting to connect to X” first. Then make sure that line is actually out the door, synchronously if necessary, before making the call. It costs you nothing when the call succeeds and saves you hours when it doesn’t.When you’re mid-debug, delete the pod instead of waiting out the backoff. Kubernetes’ exponential backoff on repeated CrashLoopBackOff restarts is helpful in production and actively annoying when you’re iterating on a fix. None of this is exotic — it’s TCP fundamentals and container basics that everyone technically knows. What makes it worth writing down is how convincingly a blackholed connection disguises itself as an application bug. Right up until you stop looking at the application and start looking at the wire. More

AWS 7R Migration Strategies: A Decision Framework for Engineering Teams

By Jerzy Kopaczewski

Why Databricks and Snowflake Speak the Kafka Protocol: Ingestion vs Architecture

By Kai Wähner

CORE

Git Blame Isn’t Enough: Building Verifiable Provenance for AI-Generated Code

By Uthej Mopathi

CORE

Beyond HTTP Handoffs: Build Durable Agent-to-Agent Services With Temporal Nexus

Agent systems are increasingly decomposed into specialized services: a planning agent delegates research, a research agent invokes retrieval and synthesis, and a compliance agent validates the result before an action is allowed. Open standards such as A2A formalize communication between independent agents, but transport interoperability is only part of the production problem. Requests can be lost after acceptance, retries can duplicate expensive work, callers can disappear while remote tasks continue, and multi-agent chains can become difficult to reconstruct. Temporal Nexus addresses a different layer. It connects Temporal applications through durable service contracts, allowing an agent capability to behave less like a fragile HTTP handoff and more like a reliable distributed operation. When an Agent Call Becomes a Distributed Transaction Boundary A conventional agent handoff often looks like ordinary RPC: serialize a task, send it to another service, wait for a response, and retry failures. That model works for short, stateless interactions. It becomes fragile when delegated work lasts minutes or hours, crosses ownership boundaries, depends on databases or model providers, or triggers side effects. Retry behavior, deduplication, cancellation, timeout ownership, and result delivery then become part of the application protocol. Nexus moves those concerns into Temporal’s execution model. A Nexus Endpoint routes a request to a target Namespace and Task Queue while hiding those implementation details from the caller. The caller depends on a named service contract rather than the handler’s Workflow type, queue, or deployment topology. Temporal describes the relationship as peer-to-peer: caller and handler Workflows remain independent executions while Nexus provides the durable boundary between them. This distinction matters for agent platforms because a durable agent service should expose a business capability, not internal orchestration mechanics. A “research” operation can remain stable even if its implementation changes from one Workflow to a graph of Activities, model calls, human approval, or additional Nexus calls. Temporal supports multi-level Nexus composition, with each hop represented as a separate durable operation. A Nexus Contract Fits the Agent Capability Boundary In the Java SDK, a Nexus Service defines the callable contract and Nexus Operations define individual capabilities. A compact contract for a research agent can remain deliberately narrow: Java @Service interface ResearchAgentService { @Operation AgentResult research(AgentTask task); } The contract carries only the input and output required by the collaboration boundary. Temporal’s Java guidance recommends service APIs containing operation names plus serializable input and output types; JSON and Protobuf are practical choices when multiple SDK languages are involved. The caller therefore does not need access to the handler’s prompts, memory representation, tool graph, or Workflow implementation. Long-running agent work should normally use an asynchronous Nexus Operation backed by a Workflow. Temporal reserves synchronous operations for reliable, predictably low-latency paths that finish within the ten-second handler deadline. Asynchronous operations can represent long-running work and return an operation token for tracking completion without holding a conventional request open. In Temporal Cloud, the maximum Nexus Schedule-to-Close timeout is 60 days. A handler can expose an agent Workflow without introducing an HTTP controller or polling API: Java @OperationImpl public OperationHandler<AgentTask, AgentResult> research() { return WorkflowRunOperation.fromWorkflowMethod( (ctx, details, task) -> Nexus.getOperationContext() .getWorkflowClient() .newWorkflowStub( ResearchAgentWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId("research-" + details.getRequestId()) .build()) ::run); } WorkflowRunOperation.fromWorkflowMethod maps the Nexus Operation to a Workflow execution. Temporal documents the Nexus request ID as stable across retries, making it suitable for constructing a deduplication-oriented Workflow ID when no stronger business identifier exists. A business identifier is generally preferable because it aligns duplicate suppression and operational correlation with domain semantics. Durability Changes the Meaning of Retry and Cancellation The most important Nexus behavior is execution semantics. Once a caller Workflow schedules an operation, Temporal atomically hands the command to Nexus machinery. Delivery uses at-least-once execution, automatic retry, rate limiting, concurrency limiting, load balancing, and circuit breaking. A handler can therefore be invoked more than once for the same operation, which makes idempotency essential for external side effects. Temporal also documents stronger duplicate protection by backing the operation with a Workflow whose ID reuse policy rejects duplicates. That behavior is especially relevant to agents because model inference may be nondeterministic and repeated tool calls can duplicate actions such as ticket creation, notifications, or database mutations. Durable execution does not remove the need for idempotency keys at external boundaries. Instead, it provides a stable execution context in which prior decisions and completed steps can be recorded and replayed consistently. Cancellation also becomes a protocol-level feature rather than a best-effort HTTP convention. Canceling a caller Workflow propagates cancellation to pending Nexus Operations and their backing handler Workflows. Termination is different: terminating the caller abandons pending operations and does not send cancellation to the handler Namespace, so remote work can continue. Graceful cancellation is therefore the safer control path for agent chains that may need cleanup or compensation. Timeouts express separate service-level expectations. Schedule-to-Start bounds how long an operation may wait to begin, Start-to-Close bounds asynchronous execution after start, and Schedule-to-Close bounds the complete lifecycle. Nexus retries retryable failures within those limits, while non-retryable failures resolve the operation and become visible to the caller. The Caller Stays Simple While the Runtime Carries State From a caller Workflow, the remote agent looks like a typed service stub. Retry loops, callback plumbing, and result polling do not need to appear in business code: Java private final ResearchAgentService research = Workflow.newNexusServiceStub( ResearchAgentService.class, NexusServiceOptions.newBuilder() .setOperationOptions( NexusOperationOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofHours(2)) .build()) .build()); public AgentResult delegate(AgentTask task) { return research.research(task); } The endpoint mapping is configured when the caller Workflow implementation is registered, allowing deployment configuration to bind a service name to a Nexus Endpoint without changing business logic. Temporal recommends a collocated pattern by default, where Nexus handlers live beside the Workflows they expose, and a router-queue pattern when routing requires separate scaling, permissions, or deployment ownership. Operationally, Nexus creates a stronger debugging surface than an opaque chain of HTTP requests. Caller histories record Nexus scheduling, start, completion, failure, timeout, and cancellation events. Bidirectional links connect caller events to handler Workflow histories, pending operations expose retry state, and OpenTelemetry integration can visualize call graphs across Nexus Operations, Activities, and Child Workflows. Security follows the service boundary. In Temporal Cloud, Nexus Endpoints use caller-Namespace allowlists, Workers authenticate with mTLS or API keys, and cross-Namespace Nexus traffic is secured by the platform. Nexus payloads use the same Data Converter model as Workflows and Activities, including codec-based encryption. Cloud Nexus Endpoints are not general public HTTP endpoints; they are reached through Temporal SDK execution inside the account. Durable Agent Services, Not Just Durable Requests Temporal Nexus does not replace agent interoperability protocols such as A2A. A2A standardizes how independent agentic applications discover capabilities, delegate tasks, and exchange results, while Nexus connects Temporal applications through durable operations. The boundary is complementary: an external A2A-facing layer can provide ecosystem interoperability, while Temporal Workflows and Nexus provide durable execution for agent services running inside Temporal. The deeper shift is from treating agent delegation as message delivery to treating it as durable work. HTTP can move a request across a network, but production agent collaboration also needs remembered state, bounded retries, deduplication, cancellation propagation, durable completion, access control, and traceable execution across service boundaries. Nexus puts those semantics directly into the invocation model. For long-running or side-effecting agent systems, that changes the handoff from an unreliable gap between services into a first-class execution boundary that can survive process crashes, worker outages, and delayed completion without losing the thread of the operation.

By Akhil Madineni

CORE

Predict, Repeat, Improve: Deterministic Simulation Testing Explained

It’s 2 AM. Your phone buzzes, the on-call alert flashes, and suddenly you are staring at a production outage that makes no sense. Following the logs, you get a hint: when you rerun the same scenario in staging, everything behaves perfectly. None of the quality gates/QA pipelines catch it, chaos experiments didn’t reproduce it, and now a ghost is chased that only appears when the system is under real-world pressure. Distributed systems are notorious for these “phantom failures” — rare timing-dependent bugs that surface unpredictably and vanish just as quickly. They are the kind of dreaded incidents that keep engineers awake at night because they are unreproducible. Take a real-world example: A service once crashed because two nodes tried to become leader at the exact same millisecond. In staging, the timing never aligned that way, so the bug remained invisible. But in production, under heavy load, it just happens, sending the system into chaos. Engineers spent days trying to recreate the failure, but without a deterministic replay, it's pure luck to get a reliable reproduction. Even when it happens, engineers may not be sure what caused it or how to reproduce it deterministically. Enter deterministic simulation testing (DST). DST builds a fully controlled, re-playable simulation of your system’s world — nodes, clients, clocks, network delays, partitions — so that even the most elusive bugs can be identified, captured, replayed, and studied. In this article, we will uncover how DST can transform those unpredictable 2 AM incidents into predictable, debuggable coordinates — giving you a new way to tame the chaos of distributed systems. Deterministic Simulation Testing Definition Deterministic simulation testing (DST) is a software testing methodology that places the system under test within a fully controlled, simulated environment. All sources of non-determinism — system clock, thread scheduling, network, disk — are intercepted and made deterministic. The key property is that, for a given initial seed and configuration, the entire execution is reproducible. The same sequence of events, faults, and outcomes will occur on every run with that seed. Key Concepts Breaking down the above definition, below are the key concepts for DST: Determinism → The system’s behavior is purely dependent upon its initial state and the seed. All non-deterministic sources are simulated to achieve determinism.Simulation → The system is run in a virtual environment that can simulate faults and control the passage of time. E.g., controlled clock skew introduced across various nodes, added network delays to achieve out-of-order event delivery.Reproducibility → Any failure or bug found during simulation can be reliably reproduced by rerunning the simulation with the same seed.Scenario Exploration → By varying the seed and/or simulation parameters, DST systematically explores a vast range of possible execution paths and failure scenarios. How DST Works Let's consider a simple scenario where two users update and read the same record in a very short interval. User 1 updates a record with a new value at time instance T0, and User 2 reads the same record at time instance T1. Note that the interval between T0 & T1 stays the same. In an ideal case (i.e., scenario 1), the new value is updated or written immediately, i.e., without any delay. Thus, when User 2 reads the same record at T1, it is able to read the latest value. In scenario 2 suppose the write is delayed due to network partitioning, disk write etc. User 2 thus sees the old value of the record even if it reads the value at the same time instance T1. Although a stale read may look trivial, it may lead to workflow halt, process crash, etc. in a complex real-world system. Imagine what could happen in a real-world system where: Multiple processes are scheduled for execution, within and across nodes.Multiple network calls are made between several nodes.Multiple operations are performed by several distributed processes on a single disk. Traditional testing strategies or frameworks are inherently constrained and thus can’t simulate such delays or faults. Because of this, it's nearly impossible to identify, catch, reproduce, or debug issues arising from such situations — rendering them unreliable or, at best, non-deterministic. To achieve determinism, the testing framework must take total control over the environment to intercept and manage all external interactions as described below: Controlled scheduling → Instead of relying on the operating system’s unpredictable thread/coroutine scheduler, the simulator provides its own deterministic scheduler. Thus ensuring various scheduling combinations are simulated.I/O mocking → All network calls, disk writes, and clock queries are routed through the simulator, allowing it to inject latency, drop packets, or change the time (e.g., clock skew).Single-threaded execution → Many DST frameworks run the entire distributed system stack within a single thread, completely stripping away the chaotic, unrepeatable nature of multi-threading. Thus, by eliminating real-world “flakiness,” DST allows developers to reproduce chaotic distributed system bugs with perfect precision, thanks to its inherent ability to replay any failing execution: Seed-based replay → The same seed reproduces the exact sequence of events, making debugging tractable.Time-travel debugging → Some platforms (e.g., Flashback) allow stepping backward and forward through execution, inspecting state at any point for a granular view of the system. DST Implementation Approaches and Architecture Patterns Below are two approaches for DST. Pluggable Non-Determinism Design the system so that all non-deterministic components (clocks, I/O, etc.) are pluggable. This strategy is used by TigerBeetle. This requires: Abstracting all system interactions behind interfaces.Providing both real and simulated implementations.Ensuring that the same codebase can run in both production and simulation by swapping implementations at startup. Pros: Deep control and minimal divergence between test and production code. Suitable for greenfield systems. Cons: Requires significant upfront design and is challenging to retrofit into existing systems. Deterministic Hypervisors and Emulation A more recent and flexible approach is to run unmodified binaries inside a deterministic hypervisor or emulation layer. This strategy is used by Hermit and Weave. Pros: Can test existing systems without code changes; language-agnostic; simulates the entire stack. Cons: May have performance overhead; some system behaviors may escape determinism if not fully intercepted. Benefits System employing DST benefits as below: Identify and reproduce rare failures → DST allows engineers to replay the exact sequence of events that led to a bug. This eliminates the frustration of “flaky” issues that appear inconsistently, making debugging far more reliable. Moreover, DST helps find bugs in execution paths unreachable by example-based tests.Improved developer productivity → Bugs are easier to reproduce, debug, and fix; less time spent on war rooms and emergency triage.Improved confidence in correctness → DST validates critical invariants (like consensus, failover, or transaction consistency) under controlled simulations. Engineers gain assurance that core distributed protocols behave as expected even under stress. Thus, preventing rare bugs from reaching production, increasing system uptime and user trust.Scalable debugging for complex systems → In microservice or event-driven architectures, DST helps tame the exponential growth of possible interleavings by focusing on deterministic seeds. This makes large-scale debugging more tractable. Challenges and Limitations While DST provides unparalleled confidence, it requires significant architectural investment. Retrofitting DST into existing systems may require significant refactoring. It can be highly intrusive, requiring developers to write custom code or frameworks, as production code often cannot rely on external third-party libraries that invoke un-mocked I/O or system calls. Moreover, DST requires careful modeling of external systems to avoid missing integration bugs. Ensuring sufficient coverage without combinatorial explosion is a major challenge — especially in modern systems with multiple integration points. Below are gaps in tooling that limit DST outcomes: Language and platform support → Not all languages and runtimes have mature DST frameworks.Hypervisor limitations → Deterministic hypervisors may not support all system calls or hardware features. DST Comparison and Applicability DST vs. Chaos Engineering DST is proactive and enables perfect reproducibility. It is best suited for development and pre-production, catching bugs before they reach users. It can simulate production chaos in minutes, and every failure is a permanent regression. Chaos engineering is reactive, non-deterministic, and validates the behavior of real deployments. It is essential for catching issues arising from real infrastructure, misconfigurations, or dependencies that simulation cannot model. However, it cannot guarantee coverage or reproducibility, and carries the risk of impacting users. In essence, both DST and chaos engineering are complementary to each other and are necessary for comprehensive reliability. DST in Functional vs. Performance Testing Functional Testing DST is ideally suited for functional testing. Validates correctness under all possible interleavings, failures, and workloads.Checks invariants, safety properties, and liveness under stress.Finds rare, timing-dependent bugs that are invisible to example-based tests. Performance Testing DST is not primarily designed for performance testing. The simulated environment does not reflect real hardware performance, network latency, or throughput.Time is virtualized and compressed; I/O is in-memory.Performance metrics (latency, throughput) measured in simulation may not correspond to real-world values. However, DST can be used to: Validate performance-related invariants (e.g., absence of deadlocks, progress under load).Simulate pathological scenarios (e.g., extreme contention, resource exhaustion) to observe system behavior. Recommendation Combine DST for functional correctness with real-world performance and benchmarking suites for comprehensive validation. DST Applicability to AI/ML and Agentic Systems AI/ML systems, especially those based on large language models (LLMs) and agentic workflows, are fundamentally non-deterministic. This makes traditional testing and debugging extremely difficult, with “heisenbugs” that vanish when observed. DST can be adapted to AI/ML systems by creating controlled, simulated environments for agents to operate in. Or using a hybrid approach of combining deterministic components (rule-based logic) with LLM-driven reasoning, using seeds to replay failures. Case Studies FoundationDB, with its deterministic simulator tool, achieved legendary reliability by running trillions of simulated CPU-hours, finding and fixing every known bug before production.TigerBeetle built a Viewstamped Operation Replication simulator (VOPR) to simulate financial transaction systems, catching subtle bugs in consensus and replication.Ethereum Merge used Antithesis to test the transition to Proof-of-Stake, simulating multiple client implementations in a deterministic environment. Conclusion Deterministic simulation testing (DST) represents a paradigm shift in the testing and validation of distributed systems. By enabling exhaustive, reproducible exploration of the vast state space of concurrent, failure-prone systems, DST empowers engineers to find and fix the rarest and most pernicious bugs before they reach production. Its integration with property-based testing, fuzzing, and fault injection, combined with advances in deterministic hypervisors and simulation frameworks, has made DST accessible to a growing range of systems and organizations. While DST requires significant engineering investment, careful system design, and ongoing maintenance, its benefits in reliability, developer productivity, and user trust are profound. As distributed systems continue to grow in complexity and AI/ML systems become more agentic and autonomous, the need for rigorous, deterministic validation will only intensify. The future of DST lies in deeper integration with formal methods, smarter state-space exploration, and broader applicability to AI/ML and hybrid systems. Organizations that embrace DST, alongside complementary techniques like chaos engineering and formal verification, will be best positioned to deliver robust, trustworthy, and resilient distributed systems in the years ahead. DST Tools and Frameworks Deterministic simulation testing (DST) tooling is still a niche but growing ecosystem. Each has a unique focus — ranging from language-level deterministic runtimes to full-stack hypervisor-based reproducibility. Based on the specific needs a single or combination of them can be picked up. References and Further Reads Taming Chaos — DSTSquashing the Heisenbug with DSTAntithesis — DSTPhil Eaton — DSTRedstone — DST FrameworkResonate — DSTJespen | TickLoom

By Ammar Husain

CORE

A Deep Dive into the Microsoft Foundry Document Intelligence SDK: From PDF to Structured Data

Most "RAG over PDFs" pipelines have a step nobody talks about much: something has to turn a scanned invoice, a multi-column contract, or a photographed receipt into text a model can actually reason over. On Microsoft's stack, that something is usually the Document Intelligence SDK, formerly Form Recognizer, and it's worth understanding on its own terms rather than treating it as a black box that happens before the interesting part starts. This is a hands-on deep dive into that SDK specifically. Not a tour of every Foundry Tools SDK — Vision and Speech and Content Safety each deserve their own treatment, but a real build using Document Intelligence: extracting layout as clean markdown, pulling structured fields out of a known document type, classifying documents before routing them, and training a custom extraction model on your own labeled data. The Mental Model First Two clients, and three kinds of model, cover almost everything this SDK does: DocumentIntelligenceClient runs analysis. Every call goes through one method, begin_analyze_document, and a model_id parameter decides what kind of analysis happens. It's a long-running operation, so every call returns a poller.DocumentIntelligenceAdministrationClient manages models. This is where you build custom extraction models and classifiers, list what's already been trained, and delete what you don't need anymore.Prebuilt models (prebuilt-layout, prebuilt-invoice, prebuilt-receipt, prebuilt-idDocument, prebuilt-read, and others) handle common, well-known document shapes out of the box. No training required.Custom extraction models, trained on your own labeled documents, handle document types nobody prebuilt a model for: your specific contract template, your specific intake form.Classifiers solve a different problem entirely: given a document of unknown type, which model should even look at it? This matters more than it sounds like it should, since most real document pipelines receive a mix of types, not one known shape. Prerequisites A Document Intelligence resource (or a multi-service Foundry resource, which includes it), giving you an endpoint and either an API key or Entra ID access.Python 3.9+ with the SDK installed. Python pip install azure-ai-documentintelligence azure-identity Python from azure.ai.documentintelligence import DocumentIntelligenceClient from azure.core.credentials import AzureKeyCredential endpoint = "https://YOUR-RESOURCE.cognitiveservices.azure.com" client = DocumentIntelligenceClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY")) For anything past local experimentation, swap the key for DefaultAzureCredential and an RBAC role scoped to the resource, the same pattern every other Foundry-adjacent SDK in this series has used. Step 1: Layout Extraction, Straight to Markdown This is the single most useful call in the whole SDK if your end goal is feeding documents into a RAG pipeline. prebuilt-layout doesn't just extract text; it understands headings, tables, and section structure, and it can hand all of that back as GitHub-flavored markdown instead of a flat text blob. Python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentContentFormat with open("contract.pdf", "rb") as f: poller = client.begin_analyze_document( "prebuilt-layout", AnalyzeDocumentRequest(bytes_source=f.read()), output_content_format=DocumentContentFormat.MARKDOWN, ) result = poller.result() print(result.content[:500]) result.content is now a markdown string, headings as #, tables as GFM pipe tables, page structure preserved. That matters more than it sounds like it should: a table flattened into plain text loses its row and column relationships, and a model reasoning over that text has to reconstruct structure it was never actually given. Markdown output keeps the structure intact. Step 2: Pulling Structured Fields From a Known Document Type For document types Document Intelligence already knows, invoices are the clearest example; you get named fields back with a confidence score per field, not just raw text. Python with open("invoice.pdf", "rb") as f: poller = client.begin_analyze_document("prebuilt-invoice", AnalyzeDocumentRequest(bytes_source=f.read())) result = poller.result() for doc in result.documents: vendor = doc.fields.get("VendorName") total = doc.fields.get("InvoiceTotal") if vendor: print(f"Vendor: {vendor.value_string} (confidence: {vendor.confidence:.2f})") if total: print(f"Total: {total.value_currency.amount} (confidence: {total.confidence:.2f})") That confidence score isn't decoration. It's the field you should actually branch on in production code; more on that in the production section below. Step 3: Add-On Capabilities You'll Want More Often Than the Docs Suggest A few optional capabilities aren't on by default, since they add processing cost, but are worth turning on deliberately rather than discovering you needed them after the fact: Python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentAnalysisFeature with open("shipping-label.pdf", "rb") as f: poller = client.begin_analyze_document( "prebuilt-layout", AnalyzeDocumentRequest(bytes_source=f.read()), features=[DocumentAnalysisFeature.BARCODES, DocumentAnalysisFeature.FORMULAS], ) BARCODES extracts barcode and QR code payloads directly, useful for shipping labels and inventory documents where the barcode carries the actual identifier the text doesn't repeat. FORMULAS pulls out mathematical expressions as LaTeX, relevant if you're processing scientific or financial documents where a formula matters more than the surrounding prose. There's also a high-resolution mode for documents where small print matters, at the cost of slower processing. Step 4: Build a Classifier to Route Mixed Document Types Real intake pipelines rarely receive one document type. A classifier solves the "what am I even looking at" problem before you commit to an extraction model. Python from azure.ai.documentintelligence import DocumentIntelligenceAdministrationClient from azure.ai.documentintelligence.models import ( BuildDocumentClassifierRequest, ClassifierDocumentTypeDetails, AzureBlobContentSource, ) admin_client = DocumentIntelligenceAdministrationClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY")) poller = admin_client.begin_build_classifier( BuildDocumentClassifierRequest( classifier_id="support-doc-classifier", doc_types={ "invoice": ClassifierDocumentTypeDetails( azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-invoices-container>") ), "contract": ClassifierDocumentTypeDetails( azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-contracts-container>") ), }, ) ) classifier = poller.result() You need at least five sample documents per category to train a classifier at all, and more than that for anything you'd trust in production. Once it's built, classifying an incoming document is a single call: Python with open("unknown.pdf", "rb") as f: poller = client.begin_classify_document("support-doc-classifier", AnalyzeDocumentRequest(bytes_source=f.read())) result = poller.result() for doc in result.documents: print(f"Classified as: {doc.doc_type} (confidence: {doc.confidence:.2f})") Step 5: Build a Custom Extraction Model for Your Own Document Type When a document type isn't invoices, receipts, or any of the other prebuilt shapes, train your own. This needs a set of labeled training documents in Blob Storage, produced through the labeling tool in Foundry's document intelligence studio or programmatically. Python from azure.ai.documentintelligence.models import ( BuildDocumentModelRequest, AzureBlobContentSource, DocumentBuildMode, ) poller = admin_client.begin_build_document_model( BuildDocumentModelRequest( model_id="acme-service-agreement-v1", build_mode=DocumentBuildMode.TEMPLATE, azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-training-container>"), description="Extraction model for Acme's standard service agreement template.", ) ) model = poller.result() Two build modes matter here, and they're not interchangeable. TEMPLATE mode is faster to train and works well when your documents follow a consistent visual layout, the same form filled out differently each time. NEURAL mode handles structural variation better, different layouts that still represent the same document type, at the cost of needing more training examples and longer build time. Start with TEMPLATE unless your documents genuinely vary in structure, not just content. One naming constraint worth knowing before you hit it: a custom model ID can't start with prebuilt-, since that prefix is reserved for Microsoft's own models across every resource. Where This Fits in the Bigger Picture This is the detail that trips people up once they've also worked with the Foundry SDK or Agent Framework elsewhere in this series: Document Intelligence doesn't go through your Foundry project endpoint at all. It has its own resource, its own endpoint (resource.cognitiveservices.azure.com), and its own authentication scope. That's what "Foundry Tools SDK" actually means as a category, prebuilt AI services with tool-specific endpoints, distinct from the Foundry SDK's unified project endpoint that Agent Framework and the Responses API build on. The practical upshot is the pipeline most teams actually want: run prebuilt-layout over incoming documents, get markdown back, and hand that markdown to a Foundry IQ Knowledge Base as a File Knowledge Source. Document Intelligence handles turning the PDF into clean, structured text. Foundry IQ handles chunking, embedding, and retrieval on top of it. Neither service needs to know the other exists; they just happen to compose well because Markdown is a reasonable interchange format for both. Production Considerations Before You Commit Don't trust a field just because it came back. A field with a confidence score of 0.41 should not silently flow into a downstream system as if it were as reliable as one scored 0.98. Set a threshold, route low-confidence extractions to human review, and log the confidence distribution over time so a model quietly degrading on a document template change doesn't go unnoticed.Classifier training minimums are a floor, not a target. Five documents per category is what the service requires to build at all. It is not enough to trust a classifier's accuracy in production. Budget for real evaluation data, held out from training, before routing real documents based on classifier output.TEMPLATE vs NEURAL is a real tradeoff, not a default to leave unexamined. Picking NEURAL by default because it sounds more capable means slower training and a higher training-data bar for a benefit you may not need if your documents are already visually consistent.Preview API versions and regional availability move independently of the SDK version. A given SDK release doesn't guarantee every feature is available in every region. Check current regional availability for newer capabilities (certain add-ons, newer prebuilt models) before designing around them.Markdown output is currently scoped to prebuilt-layout. Don't assume other prebuilt or custom models will hand back the same content format; check per-model support before building a pipeline that assumes Markdown everywhere.Cost scales with pages and capability, not just call count. Add-on features like high-resolution mode and custom model training both carry their own cost beyond the base per-page analysis price. Model this before committing to a design that turns on every add-on by default. Where This Leaves You The Document Intelligence SDK is easy to undersell because the interesting part of most AI applications feels like it's happening somewhere else, in the model, in the retrieval layer, in the agent's reasoning. But the quality ceiling of everything downstream is set right here, at the point where a physical or scanned document either does or doesn't become text a model can actually use well. Layout extraction to markdown, confidence-aware field extraction, classifiers for mixed intake, and custom models for your own document shapes cover the large majority of real document-processing needs, and all four are a few lines of SDK code once you know which one you need. The judgment call was never really about the API. It's about matching the right one of these four tools to what's actually in your inbound documents. References Microsoft. "azure-ai-documentintelligence README." Azure SDK for Python. github.com/Azure/azure-sdk-for-python/blob/main/sdk/documentintelligence/azure-ai-documentintelligence/README.mdMicrosoft Learn. "Document Intelligence layout model." learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layoutMicrosoft. "Migration guide, azure-ai-documentintelligence." Azure SDK for Python. github.com/Azure/azure-sdk-for-python/blob/main/sdk/documentintelligence/azure-ai-documentintelligence/MIGRATION_GUIDE.mdMicrosoft Learn. "Get started with Microsoft Foundry SDKs and endpoints." learn.microsoft.com/en-us/azure/foundry/how-to/develop/sdk-overviewMicrosoft Learn. "What is Foundry IQ?" learn.microsoft.com/en-us/azure/foundry/agents/concepts/what-is-foundry-iq

By Jubin Soni, FBCS

CORE

How to Test POST API Requests With Playwright TypeScript

Testing POST API requests is an important skill for modern QA and automation engineers working with backend services and microservices. In this article, we’ll explore how to test POST API requests with Playwright and TypeScript, focusing on sending a request body using different approaches. By the end, you will learn how to send POST API requests using the following approaches for adding a request body: JSON Object/Array(data)Stringified JSONJSON FileFaker library Application Under Test We will use the POST /addOrder API of the RESTful e-commerce demo application for this demo. The API schema is provided below: JSON { "user_id": "string", "product_id": "string", "product_name": "string", "product_amount": 0, "qty": 0, "tax_amt": 0, "total_amt": 0 } ] Testing POST API Requests With Playwright TypeScript Playwright provides a powerful request API that allows us to create and manage HTTP request contexts. Let’s walk through how to send POST API requests step-by-step using different approaches for passing the request body: Request Body as JSON Object/Array Let’s send a POST API request with a static JSON Array and verify that the response status code is 201. TypeScript test("POST order details API with static JSON Array", async ({ request }) => { const response = await request.post("http://localhost:3004/addOrder/", { data:[{ user_id: "1", product_id: "82", product_name: "Cadbury Bar", product_amount: 12, qty: 2, tax_amt: 1, total_amt: 25 }, { user_id: "2", product_id: "80", product_name: "MilkyBar", product_amount: 10, qty: 1, tax_amt: 1, total_amt: 11 } ], }); expect(response.status()).toBe(201); }); Code Walkthrough A Playwright test case for testing the POST API request is created using the request API, which is a built-in Playwright APIRequestContext fixture used to send HTTP requests. Sending the POST Request: The request.post() sends a POST request to the “http://localhost:3004/addOrder/” endpoint. The response received after sending a POST request is stored in the response variable.Passing the Request Body (Static JSON Array): The data property is used to send the request body, which accepts data in JSON Array and JSON object formats. Since the POST /addOrder API accepts an array of order details, allowing multiple orders to be submitted within a single JSON array, the test provides the data in JSON array format.Validating the Response: The response.status() retrieves the HTTP status code. The assertion ensures that the status code 201 is returned, indicating that the orders were successfully created. Request Body Using JSON.Stringify() JSON.stringify() can be used while sending raw payloads, testing malformed JSON, or sending custom-formatted JSON. However, the Content-Type header should be set to application/json so the server correctly interprets the request body as JSON data. Let’s write the same POST /addOrder test using JSON.stringify() and verify that a status code 201 is returned in the response. TypeScript test("POST order details API using JSON.Stringify", async ({ request }) => { const orderData = [{ user_id: "5", product_id: "64", product_name: "Cadbury Mini", product_amount: 5, qty: 3, tax_amt: 1, total_amt: 16 }]; const response = await request.post("http://localhost:3004/addOrder/", { data: JSON.stringify(orderData), headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); The orderData array is defined to contain one order object. Each field represents the order details such as user_id, product_id, product_name, and so on. This is a normal JavaScript object at this stage, not JSON yet. The JSON.stringify(orderData )converts the JavaScript object into a JSON string. As we manually stringify the request payload, we should explicitly set the Content-Type header to application/json. It tells the server to treat the request body as JSON data. Playwright sends the POST API request using the post() method and then validates that the response returns a 201 status code. Request Body as a JSON File Using a JSON file as a request body is a handy approach when testing POST API requests with Playwright TypeScript. It supports large payloads and allows the same file to be easily reused across multiple tests. The following orders.json file will be used as a request payload in the POST /addOrder API: JSON [ { "user_id": "1", "product_id": "79", "product_name": "5 star 10gm Chocobar", "product_amount": 5, "qty": 1, "tax_amt": 0.5, "total_amt": 5.5 }, { "user_id": "2", "product_id": "71", "product_name": "Lindt Milk Chocolate", "product_amount": 15, "qty": 3, "tax_amt": 2.5, "total_amt": 47.5 } ] The following configurations should be in place before we proceed to write the test using a JSON file as the request payload. tsconfig.json JSON { "compilerOptions": { "module": "NodeNext", "moduleResolution": "NodeNext", "resolveJsonModule": true } } We need to ensure that the JSON file is imported first and then attached to the data parameter while sending the POST request. TypeScript import orders from '../test_data/orders.json' with {type: 'json'}; test("POST order details API using JSON file", async ({ request }) => { const response = await request.post("http://localhost:3004/addOrder/", { data: orders, }); expect(response.status()).toBe(201); }); This test imports test data from an external orders.json file and uses it as the request body for a POST API call in Playwright. The request.post() method sends the imported JSON directly as the payload, and the test verifies that the API returns a 201 status code. Using a JSON file makes it easier to manage, reuse, and update request payloads without modifying the test logic, improving the maintainability and readability of API tests. Request Body With Faker Library Using the Faker library to create a request body for a POST API helps generate realistic and dynamic test data, reduces hardcoded values, and improves test coverage. It is especially useful for simulating real-world scenarios and avoiding duplicate data issues. However, it also has limitations. Since Faker generates random data, it may lead to inconsistent test results if not properly handled, and debugging failures could become more difficult without fixed or reproducible inputs. To use the Faker library, we need to install it first using the following command: Plain Text npm install --save-dev @faker-js/faker Next, let’s create two helper functions: one to create order objects with the required order details, and another to generate an array of orders based on the count provided by the user. TypeScript import { faker } from '@faker-js/faker'; export function createOrderDetails() { const productAmount:number = faker.number.int({ min: 1, max: 100 }); const qty:number = faker.number.int({ min: 1, max: 5 }); const taxAmt:number = faker.number.int({ min: 2, max: 10 }); const totalAmt:number = (productAmount*qty)+taxAmt; return { user_id: faker.number.int({ min: 1, max: 50 }), product_id: faker.number.int({ min: 1, max: 100 }), product_name: faker.commerce.productName(), product_amount: productAmount, qty:qty, tax_amt: taxAmt, total_amt: totalAmt }; } The createOrderDetails() function returns the order details with the required fields. It also calculates the total amount by summing the total product value and tax amount. The values in all fields are updated randomly using the appropriate methods provided by the Faker library. TypeScript import { faker } from '@faker-js/faker'; export function createRandomOrders(count:number) { return faker.helpers.multiple(createOrderDetails, {count}); } The createRandomOrders(count) method accepts a count parameter and returns randomly generated orders that can be directly used in the test. TypeScript return faker.helpers.multiple(createOrderDetails, {count}); This is where the work happens. multiple() is a helper function provided by Faker. Its job is to execute another function multiple times and collect all the results into an array. It takes two arguments: A function to execute repeatedly.An options object specifying how many times to execute it. It asks Faker to call createOrderDetails() exactly count times, collect all the generated orders into an array, and return that array to the caller. TypeScript test('POST order details API using Faker library', async({request}) => { const orderData = createRandomOrders(5); const response = await request.post("http://localhost:3004/addOrder/", { data: orderData, headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); This test sends a POST request to the /addOrder endpoint API using dynamically generated test data from the Faker library. The createRandomOrders(5) function generates an array of 5 random order objects, which are passed as the request body using the data property. The test then verifies that the API responds with a 201 status code, confirming that the orders were successfully created. Summary In this tutorial, we explored multiple approaches to testing POST API requests using Playwright with TypeScript, including sending static JSON payloads, using JSON.stringify, importing data from external JSON files, and generating dynamic test data with the Faker library. We also covered how to handle headers correctly and validate API responses using status code assertions. In my experience, using external JSON files is a practical approach for testing POST API requests because it lets us send bulk data from a single file. Similarly, the Faker library can also be used to generate dynamic test data; however, in some cases, using a third-party library may not be permitted. Ultimately, the software team chooses the approach that best fits their testing strategy. Happy testing!

By Faisal Khatri

CORE

Mistaking Code Production for Engineering Progress: AI Productivity Myths Part 1

A few months ago, one of our teams celebrated a milestone in their quarterly review. AI adoption was up. The productivity dashboard showed that developers were generating, on average, 40% more code per sprint. The tech lead showed this as a major win. Three weeks later, I was on a call for a production issue. The challenge was that errors were coming in three different formats depending on which endpoint you hit. The alerting was blind to a category of failure it had always caught before. We had a centralized exception handler. It logged context, mapped it to the right HTTP status, and pushed these alerts to our observability stack. When we investigated, we found that recent AI-assisted PRs had started introducing their own try-catch blocks inline. Each one caught exceptions locally, logged in a slightly different format, and returned a slightly different error shape. Some swallowed the exception instead of letting it propagate up to the handler that would have alerted us. Each one of those PRs was correct. Every one passed review, including the reviews I did myself. We were all checking for correctness, and consistency isn't the kind of thing that shows up in a git diff. Cleaning it up took most of a sprint. And that productivity dashboard? It counted the original generation and the cleanup as output. Twice the code, twice the "productivity," for a net loss of engineering time. The dashboard was going up while the system got worse underneath it. That's the mistake I keep seeing where teams confuse code generation with engineering progress. The LOC Trap, Reloaded Fred Brooks called this out in The Mythical Man-Month decades ago. He explicitly mentioned that measuring programming productivity by lines of code is nonsensical. Everyone agreed and then somehow forgot. So why did we rebuild the exact same dashboard the moment AI arrived? We are using the same flawed metric, with AI branding, and presenting it to boards. When a team lead reports that AI tools helped produce 40% more code, the follow-up I want to hear is “Did we actually need 40% more code?” Usually the answer is no. What we needed was the same outcomes with less effort, and effort in software lives overwhelmingly outside the act of typing. I’ll come back to that. There is a difference this time, and it is worth naming. Teams aren’t defending line counts out loud anymore. The dashboards have moved on to merged pull requests, agent tasks completed, and suggestions accepted. It is the same instinct in a unit that sounds more respectable in front of a board, counting the artifacts of work and reporting the count as progress. Why Senior Engineers Delete Code Here’s a pattern you’ll recognize if you have led engineering teams for any length of time. Your best engineers often produce fewer lines of code than anyone else on the team. Some of their most impactful weeks come out to negative line counts. That’s expertise rather than laziness. I haven’t fully worked out why the instinct for deletion over addition takes years to develop, but it does. GitClear has been measuring this rather than speculating about it. Their 2026 analysis covers 623 million changes from 2023 to 2026, and the figure that stopped me had nothing to do with volume. Moved code, their proxy for refactoring, dropped from 21% of all changes in 2022 to 3.8% by the middle of this year. Duplicated blocks are up 81% across the same window. Cross-file function calls, which is roughly what reuse looks like in a diff, are down 35%. Those three together describe a codebase that has stopped being rearranged when work goes in, and very little gets moved or deleted. When someone takes 2,000 lines of tangled logic and replaces it with 200 clean ones, that looks like a loss on any volume metric. It is an enormous win for the system, and it is exactly the activity that has gone quiet. One correction I owe, since I quoted the earlier version of this research at people for the better part of a year. GitClear’s 2024 report predicted two-week code churn would double in the AI era. It didn’t double. It went up 15%. The headline projection was too aggressive, and the part almost nobody quoted — refactoring falling off a cliff — turned out worse than predicted. Senior engineers get this intuitively. Every line of code is a liability, because every line has to be read, understood, tested, and maintained. So the best solution often makes code disappear, and that takes different forms like a well-chosen abstraction that kills duplication or a config change that removes a custom implementation. Sometimes it’s just a conversation with the PO that drops the requirement entirely. Now think about what AI coding metrics would say about this. An engineer spends a day understanding a system, realizes three services can collapse into one, and deletes 4,000 lines. By every AI productivity metric in use today, that engineer had a terrible day. In reality, they may have saved the organization months of future pain. Gergely Orosz tells a revealing story about what happens when you optimize for the wrong signal. When Uber introduced diff-count metrics, engineers started creating more, smaller changes to look productive. They flooded CI systems, driving up costs. The metric improved and engineering got worse. We are setting ourselves up for the same trap with AI-generated LOC. AI’s Tendency Toward Verbose Implementations This gets worse when you look at what AI coding tools actually excel at, which is producing plausible code quickly. That skill carries a built-in bias toward verbosity. Sometimes more code is genuinely the right call. Explicit beats implicit. A verbose but readable implementation can be better than a clever one-liner that nobody understands at 3 am when production is on fire. I’m not arguing for code golf. But AI-generated verbosity is a specific kind of bad, because it is default verbosity from ignorance of context rather than chosen verbosity for clarity. This distinction matters a lot more than I initially thought. Ask an AI assistant to implement a feature, and you’ll get a complete, working solution, longer than what an experienced developer would write, because it optimizes for correctness and completeness in isolation. It may not pick up the utility you wrote last month to do exactly this. It doesn’t realize the framework provides a one-liner if you structure the problem slightly differently. It can’t tell the difference between “I should be explicit here for readability” and “I am reinventing something that already exists three directories over.” The handler drift I opened with is the cleanest example I have of it. The AI did exactly what it was asked, every single time. Each PR added code, each one passed review on its own terms, and nothing was wrong inside any of them. What we lost lived across them. One handler gave us one error shape, and one error shape gave our alerting something to fire on. No single diff broke that, and all of them together did. GitClear tracks error-masking constructs, which is the failure mode buried in there, and they are up 47% since 2023. Our inline handlers were exactly that. I’d like to think we were an unlucky outlier, and the data says we were ordinary. I’m still not sure how you review for a property that isn’t visible in the file in front of you. Complexity as the Hidden Cost Code volume isn’t a perfect proxy for system complexity, and I acknowledged that above. But it is a directional one, and in aggregate it holds. When your codebase grows by 30-40% in a quarter without a corresponding growth in functionality, complexity is almost certainly growing with it. And system complexity is the single biggest thing determining how fast your team can move over time. I haven’t found a way around that in eighteen years. Every line of code carries ongoing costs that nobody puts on a dashboard: Cognitive load for anyone working nearbyTest coverage, without which it becomes a ticking riskReview time on every future change to itDependencies it drags in that need constant updatingMigration effort during every platform change When AI tools grow your code volume by 30-40%, each of these costs grows. The productivity gain at the moment of writing is real, and I’m not denying that. But it can be entirely eaten up by the downstream cost of maintaining a bigger, more complex system. Then there is the study I keep coming back to, and the update to it that I nearly missed. In mid-2025, METR found that experienced developers using AI coding tools took 19% longer on real-world tasks. The setup was specific. It had 16 open-source contributors working in their own repositories, code they’d lived in for years, 246 real issues averaging about two hours each, with Cursor Pro and Claude. In February 2026, METR published an update that takes a fair amount of it back. They are redesigning the experiment, and the reasons aren’t flattering to the original. Developers who would no longer work without AI declined to take part at all. Somewhere between 30% and 50% of participants avoided submitting exactly the tasks where they expected to want AI. The pay rate for the follow-up work dropped from $150 an hour to $50, which made recruitment worse again. Their own summary is that the newer data amounts to very weak evidence in either direction, and that developers are probably more sped up now, in early 2026, than the early-2025 estimate suggested. So the 19% was never a fact about AI-assisted development. It was a measurement of sixteen people in one setting, and the people who ran it now think it read low. What survives is the part that makes me think harder. Those developers believed they were 20% faster. Whatever the true effect was, it wasn’t the effect they perceived, and not one of them could feel the gap while it was happening. Reviewing and integrating suggestions consumed time none of them accounted for. Selection bias moved the headline number around, but it does not explain away a room full of experienced engineers being wrong about their own week. I could never square that finding with my own experience, because I do feel faster on certain tasks. The update moves me off the fence, slightly. Maybe I wasn’t fooling myself. I would still put no weight at all on my own estimate of how much faster I am, and that’s close to the only thing here I am confident about. None of that is an argument against the tools, only against measuring them by how much code they produce. The Strongest Number Against Me If you want to argue the other side, the best evidence available today is Microsoft’s. Early in 2026, they rolled Claude Code and GitHub Copilot CLI out across the organization and studied what happened. Tens of thousands of engineers, four months, and the ones who adopted merged roughly 24% more pull requests than the counterfactual said they would have. That is not a lab, and it is not a small group of volunteers. It’s the largest measurement of agentic coding tools anyone has published, the effect is large, and it points the right way. I take it seriously. I also notice what the unit is. The authors get there ahead of any critic. Their paper says a merged PR isn’t the same as the value it delivers, which is the argument of this entire post, conceded inside the study that is supposed to answer it. Twenty-four percent more merged pull requests is consistent with 24% more delivered value. It is equally consistent with the same work arriving in smaller slices, which is what happened at Uber the moment diff count landed on a dashboard. There are narrower caveats, and I won’t pretend I have chased all of them. The comparison is against engineers who already had AI in their IDE, so what it measures is the increment from adding an agent rather than the effect of AI from zero. Four months is not long enough for maintenance cost to turn up. And engineers chose for themselves whether to adopt. None of that makes the study wrong. It is a good measurement of pull request volume, and pull request volume behaves the way lines of code always did. It is easy to report and showcase, if somebody decides that moving it matters. Blind to what the system underneath is doing. What This Actually Means If You Are Leading a Team If you’re six months into your AI investment and your main evidence of ROI is that the output counter went up, whether that counter says lines or pull requests or tasks completed, that should worry you more than it reassures you. You might be measuring the accumulation of future cost and calling it present-day value. Here is what I would look at instead. Cycle time as value reaching production sooner, not just code getting written faster? Those two come apart more often than anyone expects. Rework rate counts if you are fixing more bugs in AI-assisted code? If generated code carries a higher defect rate, your productivity gain is a mirage. Cognitive complexity trend is when your application is getting harder to understand. Tools like SonarQube measure this. If complexity is climbing faster than it did before AI adoption, you’ve got a compounding problem. Developer effort distribution is where the time is actually going. If writing dropped from 20% to 10% of developer time while code review grew from 15% to 30%, you have moved the burden rather than reduced it. None of that is exotic, and it is not just my read. DORA’s 2026 work on the ROI of AI-assisted development arrives at something similar from a different direction. The return on these tools tracks the strength of the engineering system around them rather than the tools themselves. The things that decide it are unglamorous. Whether code review has any slack left in it. Whether people trust the test suite enough to act on a red build, which is the one I have never seen anybody audit. How much sits between a merge and production. Where those are weak, faster generation fills a queue and waits there, and the measurement science is still catching up to the tooling. The Uncomfortable Question Here is what I would ask any engineering leader who reports AI productivity gains based on code output: “If your best engineer spent last week deleting 3,000 lines of AI-generated code and replacing them with 300 lines that do the same thing better, would your dashboard show that as a win or a loss?” On a line count, that week is a catastrophe. On a PR count, it is one merged pull request, which is what a typo fix is worth too. Neither number has any way of seeing what actually happened. If your dashboard shows a loss, you are measuring the wrong thing, and you are rewarding your team for building a larger, slower, more fragile system in exchange for a chart that goes up and to the right. The goal was never more code. It was better systems that deliver business value and are maintained cheaply. Next in this series: Optimizing Benchmark Tasks Instead of Real Delivery Work — why the fact that coding is not the bottleneck makes most AI productivity claims irrelevant to actual delivery speed.

By Gaurav Gaur

CORE

The Request Timed Out, But the Payment Succeeded: Building Retry-Safe Mobile APIs

Mobile networks are notoriously unreliable. A common scenario is when a user taps "Pay Now" in an app: the payment request reaches the server and is processed, but the network response never reaches the phone. The client assumes the request failed and retries, leading to the charge running twice. This is precisely the kind of bug that idempotency solves. Idempotency means that repeating the same operation has no additional effect, and the second attempt should recognize it's a duplicate and do nothing new. In practice, mobile engineers must treat payment or order APIs as idempotent by attaching unique operation identifiers to requests and deduplicating them on the backend. With this approach, even if the network drops a response or the user double-taps a button, the user is charged only once. Achieving this involves coordination between the app and the server. On the client side, every payment or mutation request is given a persistent unique ID. For example, the app might generate a new UUID when the user submits a payment, save that operation in a local "outbox" or queue, and include the ID in the HTTP request: Swift let opID = UUID().uuidString var request = URLRequest(url: URL(string: "/api/payments")!) request.httpMethod = "POST" request.setValue(opID, forHTTPHeaderField: "Idempotency-Key") request.httpBody = /* JSON payload of the payment */ This custom header (Idempotency-Key) carries the operation's identity to the server. (Modern APIs often expect this header to ensure that POST is treated safely.) The client's logic must be persistent; before sending, it writes the operation (ID and payload) into local storage (e.g., SQLite or SharedPreferences) so that it can recover and retry if the app closes or the network is down. This pattern, sometimes called the outbox pattern, means the user's intent is recorded immediately. A background dispatcher can then drain the queue on network availability or app restart; it tries each pending request. If a send fails (timeout, no internet, 5xx error), the entry remains in the queue for a later retry. This ensures at-least-once delivery and the server will eventually see the request, even if the app crashes or the network is flaky. But at-least-once alone would cause duplicates, so the server must be ready. When the backend receives the request with its Idempotency-Key, it first checks a deduplication store (for example, a database table keyed by this ID). Java String key = request.getHeader("Idempotency-Key"); PaymentResponse prev = idempotencyStore.lookup(key); if (prev != null) { // We have processed this request before - return the original response return prev; } // No record of this key; proceed with processing PaymentResponse result = processPayment(request.getBody()); // Store the result before returning it idempotencyStore.insert(key, result); return result; If the key already exists, the server simply returns the stored result without charging again. This ensures that the second (or third) time the client re-sends, the user doesn't get double-charged. Stripe's API, for instance, works exactly this way: it saves the outcome of the first request for a given idempotency key, and any retry with the same key returns the same result. In effect, the combination of at-least-once delivery (the client keeps retrying) plus idempotent handling on the server yields an effectively-once outcome. The server's idempotency store can be implemented with a simple database table that records each key and the operation's result. For instance, a processed_payments table might use the idempotency key as a primary key or unique constraint. The service then does an atomic INSERT ... ON CONFLICT DO NOTHING (PostgreSQL syntax) or equivalent. If the insert succeeds, the code proceeds with the payment and stores the result; if it fails because the key already exists, it knows this is a duplicate and can fetch the prior result. Wrapping the insert and the business operation in one database transaction avoids a race condition, as either both the key and payment record are written, or neither is. In SQL terms: SQL BEGIN; INSERT INTO payments(idempotency_key, user_id, amount) VALUES (:key, :userId, :amount) ON CONFLICT (idempotency_key) DO NOTHING; -- Check how many rows were inserted: IF (INSERT was successful) THEN -- This is the first time seeing this key; perform the payment CALL process_payment(...); -- The payment service may record a transaction ID, etc. COMMIT; ELSE -- Key already existed: rollback any partial work ROLLBACK; -- Retrieve and return the original payment result END IF; Even if two identical requests arrive concurrently, the unique constraint ensures only one succeeds in its insert. The other can detect the conflict and simply return the saved response. The system design sandbox guide describes this approach as "The database enforces uniqueness and no separate check needed. This works well when the idempotency record belongs in the same database as the business data, since you can wrap both in a single transaction." On the mobile side, it's also wise to guard against duplicates before the request is even sent. A simple in-memory or on-disk set of "seen" IDs can help reject retry loops after a crash or double tap. In Swift: Swift final class OperationDeduplicator { private var seen: Set<String> = [] func shouldProcess(_ id: String) -> Bool { return seen.insert(id).inserted } } This OperationDeduplicator returns true only the first time an ID appears. Persisting this set across app launches (for example in Core Data or a file) makes the app resilient to a crash after the payment is sent but before the response arrives. On relaunch, the app knows it already handled that operation and won't enqueue it again. It's important to integrate these pieces smoothly. A typical mobile flow might look something like this: the user submits a payment form, the app immediately generates a new opID (a UUID) and creates an operation record { id: opID, payload: {amount, items, ...} }. This record is saved locally. Then a background task picks it up, attaches opID as the Idempotency-Key header, and sends it. If the network call times out, the record stays queued. When the app regains connectivity or restarts, the dispatcher tries again. Because the same opID is used each time, the server knows to treat all retries as one. Only after the server successfully processes the payment does the app remove the operation from its queue. This pattern ensures retries and crashes do not cause duplicate side effects. Some systems even use more granular controls. For example, if the backend involves multiple microservices, one service might call others, and each service should propagate the same idempotency key or a related correlation ID so that the entire transaction remains idempotent. Distributed tracing can help debug how a request flowed through the system. Ultimately, the goal is to capture the entire user action from UI tap through backend processing and make sure it's only applied once globally. This often means also having the backend return the same HTTP status and response body on every retry, so the client never gets an unexpected error. Developers should simulate network failures and verify that retries do not cause double effects. Most important is to observe real production behavior, as logs or traces with the operation ID can tie multiple client attempts to a single transaction. If everything is correct, the system achieves effectively-once behavior where the payment occurs exactly once no matter how many times the client tries. As systemdesignsandbox summarizes, "at-least-once delivery + idempotent consumer = effective exactly-once". In practice, this means mobile apps can assume failures are not fatal and they can safely retry with the same key, knowing the server will protect against duplicates. Conclusion In summary, retry-safe mobile operations require treating each user action as an idempotent transaction. The client must persist a unique operation key and reuse it across retries, while the server must detect previously processed keys and prevent duplicate side effects. Combining durable client operations with server-side idempotency allows payments and other critical transactions to survive timeouts, crashes, and unreliable networks without being executed twice.

By Uthej Mopathi

CORE

Software Quality Habits and AI

Software quality is about habits. The quality of our releases depends on the quality of our habits. Nobody ships a defect-riddled release because they didn't want quality. They ship it probably because the small daily behaviors that would have prevented it — the small habits — quietly stopped happening. One skipped test, one PR approved without being read, one “we'll fix it later” at a time. This article makes a quick introduction to those habits, small and large. It explains how small habits become large over time and how good and bad habits compound. I also discuss a specific danger I think we're underestimating: that AI, used without understanding and judgment, doesn't just fail to build good-quality habits. It can actively erode the ones that we may already have. Small Habits Small (or atomic) habits are small enough to survive a bad day and cheap enough to repeat without thinking about it. None of them is impressive in isolation. That's exactly why they're the unit that everything else is built from. Writing and testing: Writing the failing test before writing the fix, not afterChecking the null case, the empty list, the empty string, the zero, the timeoutWriting one assertion that actually checks behaviorDeleting a test that no longer tests anything meaningful instead of leaving it as decorationRunning the full suite locally before pushing, not just the file you touchedAdding a regression test the same day a bug is fixed, not “when there's time” Code review and collaboration: Actually reading a diff line by line before approving itLeaving a comment when you don't understand something, instead of rubber-stamping itAsking “what happens if this call fails” on every PR that adds a network or disk callRequesting a second reviewer on anything touching auth, payments, or data migration, without being told toReviewing your own diff once before asking anyone else to Naming, structure, and documentation: Naming things so the next person doesn't need you to explain themUpdating the doc or README in the same PR as the code change, not in a follow-up that never comesWriting a one-line commit message that says why, not just whatDeleting dead code the moment you notice it, instead of leaving it “in case”Logging enough context at the point of failure that you won't need to reproduce it just to read a log Everyday discipline: Reading the error message fully before searching for itReproducing a bug before claiming to have fixed itFlagging a flaky test the first time you see it, instead of re-running until it passesSaying “I don't know why this works” out loud instead of merging it anyway Large Habits Large habits need sustained effort and organizational will, not just individual discipline. They're expensive to install and easy to let quietly decay. I haven't worked with many teams that actually have the large habits that they think they have. A test suite that's trusted enough that a red build actually stops a merge, every time, with no manual override cultureA blameless postmortem process that produces real changes, not just a document nobody rereadsA quality gate in the release pipeline that blocks a release rather than getting routed around under deadline pressureAn onboarding program that transmits testing and review culture to new hires, not just repo access and a Slack inviteArchitecture reviews that ask “how will this fail” and “what happens at 10x load” before “does this work”A living catalog of architectural decisions and the reasoning behind them, kept current enough that people actually consult itA production monitoring and alerting setup tuned enough that on-call trusts the alerts instead of muting themA deprecation and technical-debt process with real budget and real deadlinesChaos engineering or systematic failure-injection practiced routinely, not once after a bad incidentA culture where raising a quality concern is rewarded even when it slows a release down How Small Habits Become Large A large habit is what a small habit looks like after a few hundred repetitions and a bit of organizational scaffolding around it. A few concrete examples: Writing one unit test with every change, consistently, for a year, is what a real unit test suite is made of. Unit test suites often accumulate, PR by PR, from the atomic habit of not merging untested logic. A test suite guards against regressions once many individual tests are maintained and are trusted enough so that a failure blocks a merge.Leaving one honest review comment when something is unclear, repeated across hundreds of PRs, is what eventually produces the right review culture. A culture where junior engineers feel safe asking questions and senior engineers feel obligated to answer them carefully. No policy document creates that; it's the residue of a habit practiced long enough to become the norm.Writing down why a bug happened, every time, in a shared place, is what a real blameless postmortem process is made of. The first few write-ups are just notes. After enough of them accumulate, cross-referenced and occasionally reread, they become an institutional memory the team consults before making the same category of mistake twice.Updating a doc or writing a short note about a decision each time a decision is made is what eventually becomes a living architectural record. This is something that new hires can read to understand not just what the system does, but why it's shaped the way it is.Flagging one flaky test instead of re-running it is what, multiplied across a team and a year, is the difference between a CI pipeline people trust and one people route around. The large habit — a quality gate that actually informs us about releases — cannot exist unless enough individuals have consistently practiced the small habit. The direction of causation matters here. You cannot batch-install a large habit. You can write the policy, buy the tool, mandate the process — but if the atomic habits underneath it aren't being practiced by individuals, the large habit is just a costume. The postmortem process without honest small write-ups is meaningless. The quality gate without individuals who trust failing checks is a formality people learn to bypass. Large habits are the compounded, aggregated shape of thousands of small habits; you build them from the bottom up, or you don't really have them at all. Habits Compound — In Both Directions A team that writes one more edge-case test per PR than it used to, for two years straight, ends up somewhere completely different from a team that writes one fewer. Neither team can point to the day quality became a strength or a liability. It's a rounding error every day and a number of dissatisfied customers over a year. The bad-habit version compounds just as reliably. “We'll add the test later” becomes later never comes. That's how test suites that nobody fully trusts are built. That's also how red CI that gets ignored becomes a quality gate that's causing problems instead of solving them. The organization now believes it has quality controls it doesn't actually have — and that belief gap is dangerous. Where AI Genuinely Helps Currently, AI is legitimately good at collapsing friction. Writing code and drafting test scaffolding or suggesting the edge case a tired engineer misses at 6 p.m. Summarizing a diff so a reviewer actually reads it instead of skimming, flagging the null case nobody handled — these are atomic-habit amplifiers. They make good habits cheaper to perform, and cheap good habits are often the ones that survive. As AI keeps improving, I expect it to be good at more and more aspects of the SDLC. Where It Becomes Dangerous “AI is good and bad” is a cliché I don't want to hide behind, so here's the specific mechanism. The danger isn't that AI writes bad code or hallucinates an API. It's that when AI fully closes a loop — writes the fix, writes a test that passes, resolves the incident — it can quietly remove the exact process that used to build the judgment we rely on to know when something is fragile. Before AI: an engineer hits a bug, doesn't understand it immediately, tries something, fails, forms a theory, tries again, and eventually understands the system well enough to fix it. More importantly, well enough to recognize the next three places the same class of bug will show up. The fix is important, but it is not the most valuable output of that process. The mental model built while failing is the most valuable output. When AI closes the loop opaquely, even if the bug is really fixed, the mental model may never get built. Engineers will lose knowledge while their judgment and critical thinking weaken. The ability to map incidents to root causes, to understand a system in more depth as we analyze it again and again — all that is gone if we simply accept an AI output without understanding. Terence Tao on AI in Mathematics Fields medalist Terence Tao raised a similar argument about mathematics. His claim isn't that machine-generated proofs are worthless but that the field of mathematics is stagnating because it is entering a crisis of value and understanding. Good, fruitful open problems are mined in a non-renewable fashion, Tao mentions. He goes on to explain: "In short, the indiscriminate use of powerful solution-extraction tools can achieve the immediate short-term goal of solving problems at hand, but at the cost of sustaining the ecosystem for the next wave of progress, or in understanding the progress already obtained." In essence, Tao isn’t objecting to AI’s capability; he is objecting to AI’s opacity. Using the Navier–Stokes regularity problem as his example, Tao described a scenario where an AI system runs an entire search process privately and simply hands back a finished proof. The problem may get solved, but the field gains almost nothing. And this is because the value of a hard open problem is not in the answer itself. The value lies in the decades of people's failed attempts: the tools that we had to invent along the way, the adjacent structure that we mapped while stuck, the wrong turns that became new subfields on their own. Solve a problem transparently, and the field grows by understanding and learning. Solve it as a sealed black box, and you collect the answer but skip the part that was actually producing the mathematics. Let's think about that for a moment from an engineering perspective. Aren't open problems, and how well we understand them, one of the main reasons for engineering innovation? The struggle to keep distributed systems consistent and available produced the CAP theorem. By understanding the CAP theorem, consensus protocols like Paxos and Raft now exist in reliable infrastructure. Trying to run software at a scale that no team could reason about manually gave rise to container orchestration and service meshes. Such open problems gave rise to site reliability engineering as a discipline in its own right. The difficulty of trusting code that could never be exhaustively tested gave birth to property-based testing, fuzzying, and formal verification tools like TLA+. Realizing that testing everything is often meaningless and inefficient (if possible at all) shaped software testing as a discipline. Chaos engineering exists because a team treated “our systems fail in ways we can not predict” as a problem worth living inside rather than a problem to be solved instantly without understanding the solution. Open Problems And Software Quality Our version of “open problems” is production incidents, hard-to-reproduce bugs, and the uncomfortable stretches where nobody's quite sure why the system behaves the way it does. Historically, those gaps close the slow way: someone traces the failure, builds a mental model, writes it up, and the team's collective judgment gets a little better. That's our version of the “adjacent structure mapped while stuck” — debugging notes, a workaround that became a design pattern, the engineer who six months later says “oh, I've seen this shape before.” An AI that resolves an incident by generating a fix nobody on the team fully understands solves the problem in the narrowest possible sense and gives back nothing else. Multiply that across a year of incidents, and you get an organization that ships, that's green on every dashboard, and that has steadily lost the tribal knowledge that used to make it resilient. The loss won't show up as a regression in this quarter's metrics. It shows up the next time a hard problem appears, and nobody in the room has the instinct for it — because the instinct was never built. It was skipped. To keep the loop intact, we could: Treat an AI-generated fix like a junior engineer's PR, not a senior engineer's judgment call — someone accountable still explains, in their own words, why it works before it mergesDon't let AI's speed quietly cancel the postmortem — if it was a real incident, someone still writes down what happened, even if the fix took ten minutes instead of two daysUse AI to compress the mechanical part of the habit loop — scaffolding, boilerplate — but keep the diagnostic “why did this actually fail” a human exercise, at least until an AI's explanation has been checked and not just found plausibleWatch for the belief gap: a green pipeline built partly on unexamined AI fixes can look identical to one built on understood fixes, until it doesn'tWhen onboarding, keep exposing new engineers to the failed attempts, not only the final AI-polished diff — the failed attempts are still where judgment gets transmitted Wrapping Up Identity is built by repeated evidence because in software we are what we repeat. We are our habits. We become someone who doesn't skip tests by repeatedly not skipping tests. Quality cultures work the same way at the team level. One repetition at a time, small habits accumulating into large ones. AI can make each of those repetitions cheaper, and that's genuinely valuable. What it must not be allowed to do is to remove our learnings from repetitions themselves and hand back only the outcome. As mathematics loses something specific and non-recoverable when a hard problem is solved opaquely, software quality is exposed to the same risk — on a much shorter feedback cycle, in production, right now.

By Stelios Manioudakis

CORE

Building a Practical Cloud-Native Golden Path: A Guide to Kubernetes-Based Service Delivery, Self-Service, and Developer-Friendly Defaults

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale. Every engineering organization that I have worked with eventually faces the same issue, which is that each team ships services differently. One team used Helm, another wrote raw manifests, and a third would have built a custom Bash script. As these different approaches accumulate, the supporting deployment steps often end up scattered across multiple Wiki pages that quickly go stale. New engineers then spend their first two weeks copying configuration values from an old repository and hoping they still work. A golden path fixes this without turning the platform team into a gatekeeper. It provides users with a standardized workflow for the shortest and most obvious route from a fresh repo to a production workload. This guide walks you through designing a minimum viable golden path, where guardrails belong, and how to keep it useful after v1. Choose the First Golden Path Start with one workflow to standardize first; the strongest candidate is usually the workflow your teams ship most often, or one that teams experience the most friction with. In many organizations, that workflow is a stateless HTTP service exposing a REST or gRPC API endpoint, deployed to Kubernetes and owned by one application team. For this walkthrough, we will use orders-api, a stateless HTTP service on Kubernetes, as our reference throughout this article. The intended users are application developers, not platform engineers — those who create the golden path itself. The path starts with a create-service command in a CLI or a form in an internal developer portal. It should end when the service is running in production with logs, metrics, ownership, and on-call rotation attached. Keep the first version deliberately narrow. A workload that needs GPU nodes, a queue-driven scaling model, or a stateful sidecar can wait. Trying to capture every exception at the beginning turns a practical delivery path into a long platform program. A golden path’s success criteria are qualitative, not quantitative. Analyze the first release by user adoption and experience. Are teams using standardized workflows instead of copying an old repository? Can a new engineer understand the end-to-end deployment process without asking around? Are on-call handoffs easier because services have the same operational shape? The answers to these questions matter more than looking at any adoption numbers displayed on a dashboard in the first few months. Define What the Path Standardizes A golden path is a curated set of decisions that are made once and reused consistently across services: The workload template should provide a Dockerfile, fully maintained base image, Kubernetes manifests, probes, resource requests and limits, a Pod Disruption Budget (PDB), autoscaling defaults, and consistent labels.The delivery pipeline should build, test, scan, sign, and publish the image.The platform defaults should include namespace rules, quotas, network policies, ingress, TLS, logging, metrics, tracing, and basic alerts. The path should not own product decisions; teams will still choose their language, framework, business logic, schema, feature flags, test strategy, and service-specific objectives. This boundary is very important. If we over-standardize, developers will work around the platform, and if we under-standardize, every instance will start with a different set of commands and dashboards. Also make sure the path is easy to find. One internal documentation page, one command, and one entry in the developer portal are enough. If a developer has to ask which template to use, the path has already failed and created friction. The table below shows the differences between shared standards the path owns and decisions each service team owns. Shared Standards vs. Team-Owned Decisions shared standard team decision Dockerfile, base image, patching cadence Language and framework choiceDeployment manifests, probes, resource requests/limits, PDB, Horizontal Pod Autoscaler Business logic, schema, feature flags Build, test, scan, sign, and publish pipeline Test suites specific to the service Namespaces, quotas, network policies, ingress, and TLS defaults Non-standard scaling (queue-driven consumers, GPU jobs) Logging, metrics, tracing, and alerting defaults Business-specific dashboards and SLOs Turn Common Requests Into Self-Service Actions Once the path is created and available to users, review the top 10 tickets your platform team receives. Look for repeated requests such as creating namespaces, adding a database, registering a DNS name, rotating a secret, or creating another environment. These are all good candidates because the desired outcome is already understood, and the steps are mostly predictable. For the Orders API golden path, the platform team can provide the following self-service actions and apply guardrails based on the risk from each change: Fully automated. These actions are reversible and have a limited blast radius. Creating a development namespace for orders-api, spinning up a preview environment on a PR, or rotating a non-production secret happens on demand without a human involved to review.Light review. Actions that change cost, security exposure, or shared infrastructure should require a light review. Provisioning production Postgres for orders-api opens a pre-filled change request that needs one approval. A new public DNS record on a shared domain is reviewed through a one-click approval on a pre-filled PR.Approval mechanism. Every self-service action generates a PR against a config repo, pre-fills the values, tags the reviewer, and merges on approval. The change flows through the same pipeline as code, and every action leaves an audit trail because it’s a git commit. The self-service interface should offer supported choices instead of exposing raw cloud APIs. For example, allowing every team to choose any PostgreSQL version, instance class, or backup schedule can leave the platform team operating 30 different database configurations. A better approach is to provide a small, opinionated set of options such as small, medium, and large. This gives developers enough flexibility while keeping the operational model understandable. For our Orders API, the developer-facing configuration can stay small: YAML # svc.yaml name: orders-api owner: team-orders tier: standard # small | standard | high runtime: http dependencies: - kind: postgres size: small # opinionated preset, not raw config on_call: orders-oncall The configuration captures the developer’s intent, while the golden path translates each request into an approved action with the right guardrail and a clear record of what happened. The table below shows how this works for the Orders API. Orders API Self-Service Actions, Guardrails, and Evidence Step Self-Service Action Guardrail Evidence Create service Run svc new via CLI or submit a portal form Template pinned to current version; namespace quotas applied Repository created with owner metadata; entry in service catalog Add dependency Pick from opinionated list (small/medium/large DB) One-click PR review for prod-tier resources Merged PR against config repo with reviewer name Deploy to prod Merge to main triggers promotion Progressive rollout with auto-rollback on error/latency signals Deployment record with canary metrics and rollback status Rotate secret Run svc rotate-secret New version issued; old version revoked after grace window Audit log entry linked to requester Create a Consistent Path From Code to Deployment Every service on the golden path should move through the same basic stages: pull request → merge to main → staging → production. The exact tooling can vary, but the meaning of each stage should not. At the PR stage, CI runs unit tests, linting, the container build, and security checks. Produce an immutable image tagged with the commit identifier, but do not deploy it to production.On merge to main, the same image is promoted to staging automatically. Rebuilding at each stage creates uncertainty because the artifact tested is no longer guaranteed to be the artifact released. Run integration and smoke tests in this stage.Promoting the image to production reveals the delivery guardrails. Start with a small percentage of traffic (5-10%), monitor health signals, and continue increasing traffic to 25%, then 100%. Roll back automatically when error rate, latency, or probe failures cross agreed thresholds. A developer should not have to recreate this logic in every repository — it should be baked into the deployment tooling. A failed orders-api canary would look like this end to end: The pipeline promotes the new image to 5% of production pods.The error rate for the /orders endpoint rises sharply during the observation window.The deployment controller restores the previous image and drains the new pods based on the rollback threshold.The pipeline posts a message in the orders-oncall service channel with a link to the failing dashboard and offending commit identifier (SHA).An incident record is created automatically only when rollback fails, or the service remains unhealthy. Teams may skip a stage for a documented case (e.g., configuration-only change), but the exception should be an explicit setting with an owner, not an informal workaround. Plain Text # pipeline stages (pseudo) on_pr: [test, lint, build, scan, sign] on_merge: [promote_to_staging, integration-tests] on_green: [canary-5, wait-signals, canary-25, wait-signals, full-rollout] On_regress: [auto-rollback, notify-oncall, record-failure, open-incident] Observability and Day-1 Operational Defaults Even if its pods are running, a service is not ready until the owning team can determine whether it is healthy and knows what action to take when it is not. The golden path should therefore create the minimum operational surface at the same time as the service. The template includes the following list on day one: Structured logs to the central log store, with request ID and trace identifiersRequest rate, error rate, latency percentiles, and saturation metricsDistributed traces with a platform-managed sampling defaultA standard dashboard created from the service nameAlerts for high errors, high latency, restart loops, and resource pressureLiveness and readiness checks connected to a health endpoint Ownership should also be captured during service creation. Ask for the team, on-call rotation, and support channel, then reuse those values in alert routing, the service catalog, and the runbook. Generate a simple runbook with sections dedicated to common failures such as stalled deployments, elevated errors, and pod eviction. A partially completed runbook with a familiar structure is far more useful than a blank page, and consistency here pays off during an incident. Keep the Golden Path Useful Over Time Exceptions are inevitable, so record the failure reason, owner, and expiry date rather than letting the exception become a permanent member. At review time, either the service returns to the path or the platform team decides the pattern is common enough to support. Treat templates and defaults like product code: review changes, version them, and provide a propagation method. When a base image or manifest default changes, open a change against each service instead of relying on teams to notice a document update. Silent drift is one of the fastest ways to lose developer trust in the path. Track a small set of signals such as the time from service creation to first production deployment, template version distribution, open exceptions, and the percentage of new services created through the path. Pair those numbers with developer feedback. A slow step that teams repeatedly bypass tells you where the next path improvement belongs. A new template version without a propagation plan becomes a fork. Extend the path when a pattern is used by three or more teams, but keep it narrow while it is still one team’s edge case. Plain Text # template bump propagation (pseudo) on template_release(new_version): for svc in services_on_path(): open_pr(svc, bump_template = new_version, auto_merge = svc.opts.auto_bump, reviewer = svc.owner) Making the Golden Path Useful in Practice A golden path succeeds when it is easier to follow than to work around. Start with one common workflow, standardize what is shared, and leave product choices with the service team. Make routine actions self-service, place checks in the delivery flow, and include observability from the first deployment. Usage signals can then inform future improvements to the path. A small path that ships, earns trust, and changes steadily will have a greater impact on engineering speed than a broad platform program that remains unfinished. Resources: CNCF TAG App DeliveryOpenTelemetry General Semantic ConventionsKubernetes Pod Security StandardsBackstage Software Templates“Building a CI/CD Pipeline With Kubernetes” by Naga Santhosh Reddy VootukuriKubernetes Security Essentials, DZone Refcard by Yitaek HwangPlatform Engineering Essentials, DZone Refcard by Apostolos Giannakidis This is an excerpt from DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale.Read the Free Report

By Naga Santhosh Reddy Vootukuri

CORE

Testing Business Programs Without Constructing Domain Objects

Unit testing business logic often requires surprisingly little business data. Suppose we want to test a program that loads an order, calculates its total, and rejects it when the amount exceeds a limit. The decision we want to verify is simple: The program loads the specified order.It asks for the order’s total.If the total is too high, it does not approve the order.It returns FALSE. Yet a conventional Java unit test may have to construct an Order. That order may require a customer, line items, currencies, prices, tax information, identifiers, and other objects that have nothing to do with the decision being tested. Builders, fixtures and mocking frameworks reduce the typing, but they do not eliminate the underlying problem: the test must participate in the internal representation of the domain model. BUBAS takes a different approach. Domain objects are opaque to a BUBAS program. Because the program cannot inspect them, a unit test does not need to construct them. It needs only a token. The Business Program Consider this BUBAS program: SQL PROGRAM ApproveOrder(orderId INTEGER, limit DECIMAL) RETURNS BOOLEAN DECLARE purchase Order DECLARE total DECIMAL purchase = LOAD_ORDER(orderId) IF NOT ORDER_WAS_FOUND(purchase) THEN LOG_EVENT "ERROR", "no such order: " + orderId RETURN FALSE END IF total = ORDER_TOTAL(purchase) IF total > limit THEN LOG_EVENT "INFO", "over limit: " + total RETURN FALSE END IF APPROVE purchase RETURN TRUE END. Order is a Java domain type registered by the application embedding BUBAS. The program can store an Order in a variable and pass it to operations that accept an Order, but it cannot access its fields or invoke its methods. There is no expression such as: SQL purchase.customer.account.balance If the program needs information about an order, the application must expose an operation for obtaining it: SQL total = ORDER_TOTAL(purchase) This restriction is primarily an encapsulation mechanism. The business program depends on the vocabulary of its domain rather than on the internal structure of Java objects. It also has an important consequence for testing. Replace the Object With Identity Here is a BUNIT test for the over-limit case: Gherkin PROGRAM OverLimitIsRejected "LOAD_ORDER" WITH ARGS(42) RETURNS "o1" "ORDER_TOTAL" WITH ARGS("o1") RETURNS 1500.00 "APPROVE _" IS MOCKED ARGUMENT "orderId" IS 42 ARGUMENT "limit" IS 1000.00 RUN RESULT IS FALSE "APPROVE _" WAS NOT CALLED END. The string "o1" is not an order serialized as text. It does not contain an order number, a total or any other property. It is a test token representing one opaque Order. The first mock says: Plain Text "LOAD_ORDER" WITH ARGS(42) RETURNS "o1" When the program calls LOAD_ORDER(42), BUNIT returns the token "o1" in place of the real Java object. The program stores it in purchase. Later it calls: Java ORDER_TOTAL(purchase) The second mock recognizes that same token and returns 1500.00. The program cannot tell that "o1" is not a real Order. It has no operation with which to inspect the object. It can only pass the value back through the vocabulary supplied by the host application. For this test, identity is all the domain object needs. We Are Testing the Conversation A BUBAS business program contains decisions and orchestration. Algorithms, persistence, infrastructure, and domain-object implementations remain in Java. Its unit test should therefore concentrate on questions such as: Which domain operations were invoked?With what arguments?What values did those operations return?Which branch did the program select?Which operations were deliberately not invoked?What result did the program produce? In the example, we do not test how ORDER_TOTAL calculates a total. That belongs in the Java test for the implementation of ORDER_TOTAL. We test what the business program does when ORDER_TOTAL reports 1500.00. This division gives us two focused tests rather than one oversized test: Java tests verify the individual domain operations.BUNIT tests verify how a business program coordinates them. The BUNIT test documents the business scenario directly. An order identified by 42 exists, its total is 1500.00, the approval limit is 1000.00, and the program must not approve it. The test does not explain how to manufacture an object graph that produces those facts. Opacity Buys Mockability Mocking domain objects in a general-purpose language is often difficult precisely because the production code can observe so much about them. It may call methods, inspect nested objects, compare values, serialize the object or pass it to code that expects a particular implementation. A substitute must reproduce every observable property used along the tested path. An opaque BUBAS value has only the observations provided by the registered vocabulary. If the vocabulary exposes ORDER_TOTAL, then the mock controls the answer to ORDER_TOTAL. If it does not expose the customer’s internal account object, neither the program nor the test needs to know that such an object exists. The object boundary and the testing boundary are the same boundary. This is stronger than merely saying that business programs should avoid inspecting domain objects. They cannot inspect them unless the embedder deliberately provides an operation that does so. Consequently, a token can stand in for any opaque value as long as the mocks define how the exposed operations respond to it. Multiple objects require only multiple identities: Plain Text "LOAD_ORDER" WITH ARGS(42) RETURNS "o1" "LOAD_ORDER" WITH ARGS(43) RETURNS "o2" "ORDER_TOTAL" WITH ARGS("o1") RETURNS 1500.00 "ORDER_TOTAL" WITH ARGS("o2") RETURNS 200.00 The test describes the distinctions that matter without constructing either order. The Test Uses the Real Language A dangerous form of mocking creates a second, simplified interface used only by tests. Eventually, the production vocabulary changes while the test vocabulary does not. BUNIT does not compile the business program against a parallel language. The program under test is compiled against the real sealed BUBAS language. Mocking happens later, at dispatch. Therefore, the test cannot silently keep using an operation that no longer exists in the production language. Nor can it casually return a value of the wrong BUBAS type. Before executing a test, BUNIT checks the mocks and the test configuration. It can report problems such as: A mock declared with the wrong number of arguments;A mock returning a value incompatible with the real operation;An argument supplied for a parameter the program does not accept;A mocked command that should initialize a variable but does not provide its value. The test reports these errors before the business program runs. The test remains artificial — as every unit test is — but it is artificial inside the actual language contract. Do Not Assert Everything A test becomes fragile when it records every interaction, whether or not that interaction matters to the scenario. BUNIT allows an expectation to specify only the relevant part of a call. For example: Plain Text "LOG_EVENT _, _" WAS CALLED WITH ARGS("INFO", CONTAINS("over limit")) The test requires an informational log message containing "over limit". It does not require the complete message to remain byte-for-byte identical. Similarly: Plain Text "APPROVE _" WAS NOT CALLED expresses the important negative requirement without inventing an Order merely to compare it with another Order. The purpose is not to reproduce the execution trace. It is to state the observable facts that define the business case. What This Does Not Test Opaque tokens do not prove that the Java implementation of LOAD_ORDER returns the right order. They do not prove that ORDER_TOTAL calculates taxes correctly or that APPROVE commits a transaction. Those operations require their own Java unit and integration tests. BUNIT tests the program at the orchestration boundary. This makes it possible to test business decisions without databases, service containers, or complete domain-object graphs, but it does not replace testing below or beyond that boundary. Nor does BUNIT make every Java application automatically testable. The application developer first has to expose a suitably designed vocabulary. If one enormous operation performs loading, calculation, approval and notification internally, BUNIT can mock that operation but cannot test the decisions hidden inside it. Testability therefore provides feedback about vocabulary design. Operations should represent meaningful domain capabilities at the level where business programs genuinely make choices. The Deeper Result Opaque domain types may initially look like a limitation. The program cannot examine its own values freely. It has to ask the vocabulary to interpret them. That limitation creates a clean separation: Java owns domain representation and implementation.BUBAS owns orchestration and decisions.BUNIT replaces domain capabilities at that same boundary.Tokens replace complex objects with identity when identity is all the test requires. The production program becomes independent of domain-object structure. The unit test inherits that independence. We do not need a fake Order with a fake customer containing fake line items whose prices happen to add up to 1500.00. For this business decision, we need only to say: Plain Text "ORDER_TOTAL" WITH ARGS("o1") RETURNS 1500.00 The business program never needed to know what was inside the order. Neither does its test. The detailed code and the BUBAS framework are available as open source at https://github.com/verhas/bubas.

By Peter Verhas

CORE

How to Verify Response Data in API Testing With Playwright TypeScript

One of the most important parts of API test automation is validating the response body to ensure data integrity. This step plays a key role in functional API testing, as it helps confirm that the API is returning the right data in the expected format. Response body validation isn’t limited to a specific request type; it applies equally to POST, GET, PUT, and PATCH APIs. The same validation approach can be used for any API response to verify the data returned by the service. Playwright offers multiple ways to validate response bodies. In this tutorial, I’ll walk you through these approaches to help you efficiently perform assertions on the response data using best practices. Checkout the previous tutorial blog to learn about Installation, the demo application, and how to send GET API requests with Playwright. How to Verify the Response Structure Response structure checks ensure that an API consistently returns data in the expected format, protecting the contract between backend services and their consumers. They help catch breaking changes early, such as missing or renamed fields, even when the API still returns a successful status code. TypeScript test("GET Order details and perform structure check", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, failOnStatusCode: true, }); const responseBody = await response.json(); expect(responseBody).toHaveProperty("message"); expect(responseBody).toHaveProperty("orders"); expect(responseBody.orders[0]).toHaveProperty("id"); expect(responseBody.orders[0]).toHaveProperty("product_name"); }); This test focuses on validating the structure of the API response. It validates that the response body contains the expected top-level keys and that each order object includes the required fields. Basic Assertions The basic assertions validate API success and data presence, making them a good first layer of verification before deeper structure or data-level checks. TypeScript test("Get order details and perform basic level verification", async ({ request, }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: 1, }, failOnStatusCode: true, }); const responseBody = await response.json(); expect(responseBody.message).toBe("Order found!!"); expect(Array.isArray(responseBody.orders)).toBeTruthy(); expect(responseBody.orders.length).toBeGreaterThan(0); }); This test performs a basic level check to confirm that the endpoint works as expected and returns the expected data in the response. After parsing the response body, the assertions focus on the following essential basic-level checks: TypeScript expect(responseBody.message).toBe("Order found!!"); The above line of code verifies that the API returns the expected message text in the response body. TypeScript expect(Array.isArray(responseBody.orders)).toBeTruthy(); This line of code ensures that the orders field in the response is an array, validating the basic response format. TypeScript expect(responseBody.orders.length).toBeGreaterThan(0); This part of the test confirms that at least one order is returned in the orders array, ensuring the response contains required data. How to Verify Response Data With Details Validating the actual data returned in the response is essential to ensure that the API response contains the correct values. TypeScript test("Get order and verify order details", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, failOnStatusCode: true, }); const responseBody = await response.json(); const order = responseBody.orders[0]; expect(order.id).not.toBeNull(); expect(order.id).toBeDefined(); expect(order.user_id).toEqual("1"); expect(order.product_id).toEqual("79"); expect(order.product_name).toEqual("5 star 10gm Chocobar"); }); The following code ensures that the response has a valid identifier and it is not missing or empty. TypeScript expect(order.id).not.toBeNull(); expect(order.id).toBeDefined(); This check is required because the API generates the order ID when a new order is created in the system. It ensures that the “id” field has a valid value generated and assigned to it, since this “id” is used to retrieve, update, or delete order data. TypeScript expect(order.user_id).toEqual("1"); expect(order.product_id).toEqual("79"); expect(order.product_name).toEqual("5 star 10gm Chocobar"); These statements assert that the order details are retrieved correctly for the respective request. The “user_id” - “1” was sent in the request, and verifying it in the response, along with the other order details such as “product_id” and “product_name,” ensures that the correct data is returned. How to Verify Response Data by Matching Objects and Arrays Playwright allows response data verification by matching objects and arrays partially within the API response. This approach is useful because it makes tests more flexible and confirms that the API returns the correct data structure and values. TypeScript test("Get order and verify matching object and array", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: 1, }, failOnStatusCode: true, }); const responseBody = await response.json(); expect(responseBody).toMatchObject({ message: "Order found!!", orders: expect.arrayContaining([ expect.objectContaining({ product_id: "79", product_name: "5 star 10gm Chocobar", product_amount: 5, qty: 1, tax_amt: 0.5, total_amt: 5.5, }), ]), }); }); In this test, the toMatchObject assertion verifies that the response contains a “message” with the expected value “Order found!!” and an orders array. Within the array, "expect.arrayContaining" ensures that at least one order matches the expected data, while "expect.objectContaining" verifies only the values in the specified fields of that order. Using Best Practices to Perform Assertions Best practices create stable, maintainable API automation tests by combining basic checks with flexible data matching. TypeScript test("Get Order details API test with best practice", async ({ request }) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, failOnStatusCode: true, }); const responseBody = await response.json(); expect(responseBody.message).toBe("Order found!!"); expect(responseBody.orders.length).toBeGreaterThan(0); expect(responseBody.orders).toEqual( expect.arrayContaining([ expect.objectContaining({ id: 1, product_name: "5 star 10gm Chocobar", }), ]) ); }); The test sends a GET request to fetch order details for “user_id”-“1". The use of failOnStatusCode: true ensures the test fails immediately if the API does not return a 2xx status code. The response is then parsed into a JSON object for validation. The assertions are structured in layers: TypeScript expect(responseBody.message).toBe("Order found!!"); This assertion verifies the message text, confirming that the API returns the correct message when an order is found. TypeScript expect(responseBody.orders.length).toBeGreaterThan(0); This statement ensures meaningful data is returned and avoids false positives when the array is empty. TypeScript expect(responseBody.orders).toEqual( expect.arrayContaining([ expect.objectContaining({ id: 1, product_name: "5 star 10gm Chocobar", }), ]) ); The final part of the code performs the final assertion using arrayContaining and objectContaining to verify that at least one order has the expected “id” and “product_name”, without asserting every field. These layered validations improve clarity by verifying structure, data presence, and key data values in sequence. Extracting Data From the Response Extracting data from the API response is a common and widely used pattern in API test automation. It is important in multiple ways, such as reusing the data in further tests for dynamic testing and end-to-end validation. TypeScript test('Get order details and extract the order id', async({request}) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { id: 1, }, failOnStatusCode: true, }); const responseBody = await response.json(); expect(responseBody.message).toBe("Order found!!"); expect(responseBody.orders.length).toBeGreaterThan(0); expect(responseBody.orders).toEqual( expect.arrayContaining([ expect.objectContaining({ id: 1, product_name: "5 star 10gm Chocobar", }), ]) ); const order = responseBody.orders[0]; expect(order.id).not.toBeNull(); const order_id= order.id; console.log(order_id); const product_name = order.product_name console.log(product_name) }); This test sends a GET API request and performs basic validations to ensure the API response is reliable. TypeScript const order = responseBody.orders[0]; expect(order.id).not.toBeNull(); const order_id= order.id; console.log(order_id); The code above extracts the “order_id” from the order object in the response. Before accessing it, an assertion is made to verify that the value is not null. Finally, the value of the order_id is printed in the console. TypeScript const product_name = order.product_name console.log(product_name) Similarly, other values, such as product_name, can also be extracted. Attaching the Response Body to the Playwright Report The Playwright report, by default, shows the steps executed, the number of tests run, pass/fail status, and time taken to run the tests. However, it does not attach the response body to the test report. Attaching the response body to the report improves visibility and makes the test report more informative and transparent. The following code shows how to extract the required metadata and attach it to the Playwright report. TypeScript test("Get order details API and attach the response details to the report", async ({ request, }, testInfo) => { const response = await request.get("http://localhost:3004/getOrder/", { params: { user_id: "1", }, }); expect(response.status()).toBe(200); const status = response.status(); const statusText = response.statusText(); const headers = response.headers(); const body = await response.json(); const fullResponse = { status, statusText, headers, body, }; await testInfo.attach("Full API Response", { body: JSON.stringify(fullResponse, null, 2), contentType: "application/json", }); }); The testInfo is a built-in Playwright fixture and provides utilities to manage and inspect test execution, such as attaching files to reports, updating test timeouts, and identifying the currently running test. The following lines of code extract the response metadata, such as the status code, status text, headers, and response body. TypeScript const status = response.status(); const statusText = response.statusText(); const headers = response.headers(); const body = await response.json(); Next, let’s combine all response details and create a single object containing: Status codeStatus textHeadersResponse body TypeScript const fullResponse = { status, statusText, headers, body, }; Finally, let’s attach these details to the report using the testInfo.attach() method as shown below: TypeScript await testInfo.attach("Full API Response", { body: JSON.stringify(fullResponse, null, 2), contentType: "application/json", }); The testInfo.attach() adds an attachment to the Playwright report. The attach() method has 3 parameters: Name of the attachment: The first parameter is the name, “Full API Response”, that will be shown for the attachment.Body of the attachment: The second parameter is for the body of the attachment. The JSON.stringify(fullResponse, null, 2) has 3 arguments. The first argument converts the fullResponse object into a readable, pretty-formatted JSON. The second argument is the replacer, which is null. It ensures that all properties from the fullResponse object are included as they are, without modifying anything. The third argument controls pretty-printing. Here, “2” means indent nested JSON by 2 spaces.Content type: This parameter ensures that the report treats the attachment as JSON. The following screenshot is generated after the tests are run: Test Execution Running the tests in Playwright is simple and easy. We can run the following command from the terminal: Plain Text npx playwright test To generate the report, the following command can be used: Plain Text npx playwright show-report Summary Playwright provides multiple approaches, including structure checks and matching objects and arrays for verifying response data. The right strategy should be chosen based on your project’s requirements. Based on my experience, combining response structure checks with response data validation, including the matching object and array strategy, can be used as an effective approach for validating API responses. Happy testing!

By Faisal Khatri

CORE