The Testing, Tools, and Frameworks Zone encapsulates one of the final stages of the SDLC as it ensures that your application and/or environment is ready for deployment. From walking you through the tools and frameworks tailored to your specific development needs to leveraging testing practices to evaluate and verify that your product or application does what it is required to do, this Zone covers everything you need to set yourself up for success.
The Man Who Invented Selenium Also Built Its Replacement
Diagnosing Test Failures at Scale: Correlating Execution Evidence Across Ephemeral Runners
Teams considering or committed to adopting Kotlin often plan a migration as a series of rewrites, swapping Java services for Kotlin counterparts one microservice at a time. In practice, that kind of plan tends to stall before it produces any shipped code. Kotlin's role in the JVM ecosystem is expanding because it doesn't require that kind of commitment. Companies such as Netflix, Adobe, and DoorDash have adopted it incrementally, alongside existing JVM systems, rather than through a full rewrite. The lowest-risk starting point is tests. They give you Kotlin's language benefits, including null-safety, data classes, and expressive lambdas, without touching anything that runs in production. This guide walks through adding Kotlin to an existing Java + Spring Boot project for test sources only. You'll configure Gradle, write JUnit 5 tests in Kotlin, use MockK for Kotlin-native mocking, and apply Kotest assertions to improve test readability. Your production services don't change, and your team starts experiencing Kotlin in a bounded, no-risk context. Why Testing Is the Safest Place to Introduce Kotlin Kotlin test sources are isolated from the production application artifact and don't require Kotlin in the production runtime. They can still affect CI and test infrastructure, but leave production services untouched. That separation from production makes tests the right entry point for a language your team hasn't yet committed to. Introducing Kotlin in test sources means that your existing Java services remain completely unchanged, with no refactoring, no migration risk, and no regressions in production paths. It also allows your team to experiment with real Kotlin syntax on tests they'd be writing anyway, rather than on contrived examples. Developer buy-in builds faster when engineers are writing tests they actually want to write. Kotlin tests tend to be shorter, more readable, and easier to scan at code review. Leadership and platform teams are skeptical of language changes for good reasons. The proposal "we want to use Kotlin for tests" carries almost no risk to evaluate, since it doesn't touch production or your deployment infrastructure. It's a clean, bounded experiment, and the pattern has been validated by engineering organizations across the industry. Adding Kotlin to an Existing Java + Spring Boot Project You don't need to migrate your entire build to use Kotlin in tests. If you're running a standard Spring Boot + Gradle project, you add the Kotlin JVM plugin, pull in kotlin("stdlib") and io.mockk:mockk at test scope, and IntelliJ IDEA recognizes src/test/kotlin as a source set automatically. Open the project in IntelliJ IDEA and update your build.gradle.kts, adding the Kotlin JVM plugin in the plugins block and the test-scoped dependencies: Kotlin plugins { java id("org.springframework.boot") version "4.1.0" // example version, check for latest id("io.spring.dependency-management") version "1.1.5" // example version, check for latest // Add the Kotlin JVM plugin alongside your existing Java plugin kotlin("jvm") version "2.4.10" // example version, check kotlinlang.org for latest stable } dependencies { // Your existing production dependencies remain untouched implementation("org.springframework.boot:spring-boot-starter-web") // Existing test dependency testImplementation("org.springframework.boot:spring-boot-starter-test") // Required for coroutines testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.11.0") // Kotlin stdlib (test scope only) testImplementation(kotlin("stdlib")) // MockK: Kotlin-native mocking library (check GitHub for latest stable) testImplementation("io.mockk:mockk:1.14.11") // SpringMockK: @MockkBean and @SpykBean for Spring context tests (check GitHub for latest stable) testImplementation("com.ninja-squad:springmockk:5.0.1") // Kotest assertions, no need for the full test runner (check kotest.io for latest stable) testImplementation("io.kotest:kotest-assertions-core:6.2.3") } tasks.withType<Test> { useJUnitPlatform() } Always check the Kotlin and library documentation for the latest stable releases before pinning versions in a new project. If you’re using Maven, you need to add the kotlin-maven-plugin: XML <properties> <kotlin.version>2.4.10</kotlin.version> </properties> <dependencies> <!-- Existing Spring Boot test dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <!-- Kotlin standard library, test scope --> <dependency> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-stdlib</artifactId> <version>${kotlin.version}</version> <scope>test</scope> </dependency> <!-- MockK --> <dependency> <groupId>io.mockk</groupId> <artifactId>mockk</artifactId> <version>1.14.11</version> <scope>test</scope> </dependency> <!-- Kotest assertions --> <dependency> <groupId>io.kotest</groupId> <artifactId>kotest-assertions-core</artifactId> <version>6.2.3</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-maven-plugin</artifactId> <version>${kotlin.version}</version> <executions> <execution> <id>test-compile</id> <phase>test-compile</phase> <goals> <goal>test-compile</goal> </goals> <configuration> <sourceDirs> <source>src/test/kotlin</source> <source>src/test/java</source> </sourceDirs> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> </plugin> </plugins> </build> After saving, IntelliJ IDEA displays a notification to reload the Gradle project. Use Shift+Cmd+I on macOS or Ctrl+Shift+O on Windows and Linux to trigger the sync. Once complete, IntelliJ IDEA automatically recognizes src/test/kotlin as a source set, allowing you to use Kotlin alongside the existing Java test infrastructure, like base classes, testcontainer setup, test data factories, etc. Writing JUnit 6 Tests in Kotlin Writing JUnit 6 tests in Kotlin removes a lot of the friction of test setup. The language's expressiveness lets you focus on the scenario and expected behavior instead. Consider a Spring service that manages orders. Test code often spends more time constructing test data than testing behavior. In Java, creating several variations of the same object can require repeatedly calling setters: Java Order order = new Order(); order.setId(1L); order.setCustomerId(42L); order.setTotal(new BigDecimal("99.99")); order.setStatus(OrderStatus.PENDING); Order cancelledOrder = new Order(); cancelledOrder.setId(1L); cancelledOrder.setCustomerId(42L); cancelledOrder.setTotal(new BigDecimal("99.99")); cancelledOrder.setStatus(OrderStatus.CANCELLED); The setup becomes even more repetitive as a test suite introduces additional scenarios. A test for a large order, for example, would need to repeat the same boilerplate just to change the total. Kotlin's named and default arguments can already reduce that repetition, but you can go further by creating a small test-data DSL that describes orders in terms of the scenarios your tests care about: Kotlin fun orderScenario(block: OrderScenario.() -> Unit): Order = OrderScenario().apply(block).build() class OrderScenario { var id: Long = 1L var customerId: Long = 42L var total: BigDecimal = BigDecimal("99.99") var status: OrderStatus = OrderStatus.PENDING fun pending() { status = OrderStatus.PENDING } fun confirmed() { status = OrderStatus.CONFIRMED } fun cancelled() { status = OrderStatus.CANCELLED } fun forCustomer(id: Long) { customerId = id } fun withTotal(amount: String) { total = BigDecimal(amount) } fun build() = Order(id, customerId, total, status) } Tests can now describe the data they need as scenarios: Kotlin val pendingOrder = orderScenario { forCustomer(42L) withTotal("99.99") pending() } val cancelledOrder = orderScenario { forCustomer(42L) cancelled() } val largeOrder = orderScenario { withTotal("999.99") pending() } The DSL uses Kotlin's lambdas with receivers to make OrderScenario's properties and functions available directly inside the orderScenario { ... } block. Default values provide sensible test data, while individual tests override only the attributes relevant to their scenario. Most importantly, the production Order class can remain completely unchanged in Java. This approach becomes particularly useful as a test suite grows. Instead of accumulating specialized factory methods such as cancelledOrder(), largePendingOrder(), or confirmedOrderForCustomer(), the DSL gives tests a small vocabulary for composing those scenarios themselves. The test describes *what state it needs* without repeating the mechanics of constructing the Java object. A complete JUnit 6 test can then combine the scenario DSL with MockK and Kotest assertions: Kotlin import io.kotest.matchers.shouldBe import io.mockk.every import io.mockk.mockk import io.mockk.verify import org.junit.jupiter.api.Test internal class OrderServiceTest { private val repository = mockk<OrderRepository>() private val orderService = OrderService(repository) @Test fun `returns order when found by id`() { val order = orderScenario { forCustomer(42L) withTotal("99.99") pending() } every { repository.findById(order.id) } returns order val result = orderService.getOrder(order.id) result shouldBe order verify(exactly = 1) { repository.findById(order.id) } } @Test fun `throws when order is not found`() { every { repository.findById(any()) } returns null shouldThrow<NoSuchElementException> { orderService.getOrder(99L) } } } Backtick-delimited test names (`returns order when found by id`) let you write descriptive test names directly in the function declaration, so the name appears naturally in test runner output. This can also save you from using JUnit's `@Named` annotation when you want a readable test name. When a Java repository method returns a nullable `Order?`, the Kotlin compiler forces the test code to account for the null case before using the result. That can surface mismatches between the test and the Java API contract at compile time rather than as unexpected `NullPointerException`s during test execution. A full guide is available in the docs. MockK for Kotlin-Idiomatic Mocking Mockito remains a viable choice when adding Kotlin tests to a Java application, but MockK provides an API designed around Kotlin's syntax. Its concise every { ... } returns ... and verify { ... } constructs make mocked interactions easier to read, while allowing Kotlin tests to mock existing Java classes without changing the production code. This can be particularly useful when testing Java Spring applications that use Reactor. The production application can remain entirely in Java and continue to return Mono or Flux. Kotlin is only introduced in the test suite. MockK handles the mocked dependencies, while Reactor's StepVerifier verifies the resulting publishers. Consider a Java service that returns an order reactively: Java public Mono<Order> findOrder(Long id) { return repository.findById(id); } The service can be tested from Kotlin without changing its implementation: Kotlin import io.kotest.matchers.shouldBe import io.mockk.every import io.mockk.mockk import io.mockk.verify import org.junit.jupiter.api.Test import reactor.core.publisher.Mono import reactor.test.StepVerifier internal class OrderServiceTest { private val repository = mockk<OrderRepository>() private val orderService = OrderService(repository) @Test fun `returns order when found`() { val order = Order( id = 1L, customerId = 42L, total = BigDecimal("99.99"), status = OrderStatus.CONFIRMED ) every { repository.findById(1L) } returns Mono.just(order) val result = orderService.findOrder(1L) StepVerifier.create(result) .expectNext(order) .verifyComplete() verify(exactly = 1) { repository.findById(1L) } } } The production service and repository can remain in Java. The Kotlin test uses MockK's Kotlin-friendly syntax to define the mock behavior, while StepVerifier checks that the returned Mono emits the expected value and completes successfully. The same pattern applies to Flux-returning APIs. For example, a test can verify multiple emitted values without manually subscribing to the publisher: Kotlin every { repository.findAll() } returns Flux.just(order1, order2) StepVerifier.create(orderService.findAll()) .expectNext(order1, order2) .verifyComplete() MockK doesn't replace Reactor's testing tools. Each library handles a different part of the test. MockK provides concise mocking, while StepVerifier provides assertions over Reactor publishers. Kotlin brings those APIs together in a compact test that's focused on the behavior being verified. In practice, that means you can adopt Kotlin for the test source set, use MockK to mock your existing Java dependencies, and test reactive Java code without introducing Kotlin or coroutines into the production application. Kotest Assertions for Cleaner Test Intent You don't need Kotest's full test runner to benefit from its assertion library. The kotest-assertions-core dependency adds infix matchers (shouldBe, shouldNotBe, shouldThrow, shouldContain) that layer on top of your standard JUnit 5 tests. Compare JUnit's assertion style with Kotest's: Kotlin // JUnit 5 assertions Assertions.assertEquals(OrderStatus.CONFIRMED, result.status) Assertions.assertNotNull(result.id) // Kotest matchers (same assertions, different shape) result.status shouldBe OrderStatus.CONFIRMED result.id shouldNotBe null The Kotest versions read in the same order as a sentence describing what you expect. That ordering makes the intent visible without parsing the argument order of assertEquals. shouldThrow is particularly useful for testing Spring service exception paths: Kotlin val exception = shouldThrow<NoSuchElementException> { orderService.getOrder(99L) } exception.message shouldBe "Order 99 not found" The matcher returns the thrown exception, so you can chain further assertions on its properties without additional try/catch blocks. Spring Boot Integration Tests With @MockkBean For Spring slice tests (@WebMvcTest, @DataJpaTest, or a partial @SpringBootTest), you swap Spring's @MockBean for @MockkBean from the springmockk library. It integrates with Spring's test context the same way @MockBean does, but creates MockK instances: Kotlin package com.example.demo.controller import com.example.demo.model.Order import com.example.demo.model.OrderStatus import com.example.demo.service.OrderService import io.mockk.every import org.junit.jupiter.api.Test import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest import org.springframework.test.web.servlet.MockMvc import org.springframework.test.web.servlet.get import com.ninjasquad.springmockk.MockkBean import java.math.BigDecimal import java.util.NoSuchElementException @WebMvcTest(OrderController::class) internal class OrderControllerTest { @Autowired private lateinit var mockMvc: MockMvc @MockkBean private lateinit var orderService: OrderService @Test fun `GET order returns 200 with order data`() { val order = Order( 1L, 42L, BigDecimal("99.99"), OrderStatus.CONFIRMED ) every { orderService.getOrder(1L) } returns order mockMvc.get("/orders/1") .andExpect { status { isOk() } jsonPath("$.id") { value(1) } jsonPath("$.customerId") { value(42) } jsonPath("$.total") { value(99.99) } jsonPath("$.status") { value("CONFIRMED") } } } @Test fun `GET missing order returns 500`() { every { orderService.getOrder(99L) } throws NoSuchElementException("Order 99 not found") mockMvc.get("/orders/99") .andExpect { status { is5xxServerError() } } } } The @MockkBean annotation replaces the service in the Spring context for the duration of the test. IntelliJ IDEA resolves the Kotlin DSL for MockMvc (mockMvc.get(...)) with full autocomplete support, removing the friction of looking up chained method names. Getting Started With Kotlin in Tests Introducing Kotlin through test sources, before any production code changes, is a validated incremental strategy. The tooling is mature, the Gradle setup is a handful of lines, and the first Kotlin test can ship the same day you configure the dependency. Over time, writing Kotlin tests can change how your team thinks about the language. Developers who spend a few months writing Kotlin tests start asking why they can't use data classes in production. They miss null safety in Java service code. Gradual adoption happens organically, driven by genuine preference rather than a top-down mandate. JetBrains and the Spring team have an official strategic partnership focused on improving the developer experience for backend engineers. If you want to explore what Kotlin can do for your team beyond the test suite, kotlinlang.org is the right place to start.
If you need to generate prerequisite test data, such as adding dummy records through an API, before executing your test suite, using Global Setup in Playwright is a clean, maintainable, and scalable approach. It ensures all required data is prepared upfront, keeping your test execution structured and reliable. What Is Global Setup? The Global Setup in Playwright runs once before the entire test suite begins execution. It is typically used for activities such as creating test data, seeding the database, generating authentication tokens, preparing the test environment, or cleaning up old data. Since it executes only once at the start of the run, it helps ensure efficient and consistent test initialization across the suite. How to Create a Global Setup in a Playwright TypeScript Project The following steps need to be followed to create a globalSetup in the Playwright project: Step 1: Create a global-setup.ts file in the Root Folder of the Project TypeScript import orders from './test_data/orders.json' with {type: 'json'}; async function globalSetup() { console.log('Running Global Setup...'); const apiContext = await request.newContext(); const response = await apiContext.post("http://localhost:3004/addOrder/", { data: orders, }); if(response.status() !=201) { throw new Error ("Failed to create setup test data!"); } console.log('Test data created successfully'); } export default globalSetup; The globalSetup() function will insert test data into the RESTful e-commerce demo application. The test data for creating new orders is supplied using the following orders.json file. JSON [ { "user_id": "1", "product_id": "79", "product_name": "5 star 10gm Chocobar", "product_amount": 5, "qty": 1, "tax_amt": 0.5, "total_amt": 5.5 } ] The data is required to validate the GET API tests, which will execute immediately after the Playwright test command is triggered. Step 2: Register globalSetup in playwright.config.ts TypeScript import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: './global-setup', }); The globalSetup property tells Playwright to execute the file once before the entire test suite runs. This setup file is typically used to perform pre-test activities such as creating test data or preparing the test environment, ensuring that all tests start with the required initial state. tsconfig.json: JSON { "compilerOptions": { "module": "NodeNext", "moduleResolution": "NodeNext", "resolveJsonModule": true, "types": ["node", "@playwright/test"] } } In this case, “require.resolve()” is not used because the project is configured to use ES Modules ("module": "NodeNext" and "moduleResolution": "NodeNext" in tsconfig.json). Since the project runs in ESM mode, Playwright can directly resolve the setup file using a string path ('./global-setup') instead of relying on CommonJS-based require.resolve(). Step 3: Run the Test The tests are already in place; only global-setup.ts needs to run to feed data into the system before running them. The script for running the tests is updated in the package.json file: JSON "scripts": { "test": "npx playwright test" }, Let’s run the tests by executing the following command in the terminal: Plain Text npm run test The following is a screenshot of the output after running the tests: It can be seen that the test data was created using the global setup before running the tests. How This Helps in Your API Testing Instead of creating the order in the first test and relying on test execution order, we can use a more robust approach by setting up the required data beforehand. By creating test data once during global setup, the tests remain independent, reducing flaky dependencies and unintended failures. This approach aligns with Playwright best practices and helps build a stable, maintainable automation test suite. Important Points to Note About the Global Setup It runs once per test run.It does NOT share variables directly with tests.Use files or environment variables to pass data.Assertions should not be put in the global setup, except for setup validation. Check out how to verify response data with Playwright TypeScript for a detailed understaning on assertions. Summary Global setup in Playwright runs once before the entire test suite and is primarily used for preparing the test environment before running the tests. It helps keep tests independent and avoids relying on execution order. However, since it runs only once and does not directly share variables with tests, any required data must be passed through files, environment variables, or external storage mechanisms. Happy testing!
Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned; you get soft-delete and purge protection; access is governed by RBAC or access policies; and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity/Workload Identity on AKS) with zero code changes. And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination. mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret, and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running. This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault. The azure-kv Scheme The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]: Plain Text azure-kv://my-vault/db-password # latest enabled version azure-kv://my-vault/api-key?version=abc123 # pin to a specific version id azure-kv://my-vault/creds#password # one key out of a JSON secret One Key Vault quirk to keep in mind: versions are identified by opaque hex IDs, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/azure Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order: Go import _ "github.com/xavidop/mamori/providers/azure" The identity you bind needs read access to the vault's secrets. With Key Vault's RBAC model, that's the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets): Shell az role assignment create \ --role "Key Vault Secrets User" \ --assignee "<managed-identity-object-id>" \ --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault" If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets. Load, Then Watch the Rotation Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/azure" ) type Config struct { // Tracks the latest enabled version: rotations flow in automatically. DBPassword secret.String `source:"azure-kv://my-vault/db-password"` // Pinned to an immutable version id: deliberate, reproducible. SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"` // One field out of a JSON secret. APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart log.Println("Key Vault: db-password rotated and applied") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // last VALID config, lock-free _ = cfg } Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String. What Reconciliation Guarantees When a new version of db-password becomes the latest enabled one, mamori's poll cycle detects the version change, fetches it, and then: Re-validates the entire struct – a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can't break your service.Swaps atomically – w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.Fires a diff-aware callback – ev.Fields tells you exactly which fields moved, so you only rebuild what changed. Key Vault has no push channel for "a new version was created", so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn't hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap. If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip. Structured Secrets With flatten Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct: Go type Config struct { DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"` } type DBConfig struct { Host string `mapstructure:"host"` Port int `mapstructure:"port"` Password secret.String `mapstructure:"password"` } One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself. When You Don't Need This If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you'd rather rebuild a client than recycle a pod. Wrapping Up Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen. Shell go get github.com/xavidop/mamori/providers/azure Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!
API testing is the practice of verifying that an API behaves as expected. In Agile environments, teams often perform it early in the SDLC as part of shift-left testing, enabling faster feedback and helping them identify and fix defects early. Playwright is a modern test automation framework that’s popular among testers and developers. Built by Microsoft, it makes it easy to automate both web and API testing across multiple programming languages, all within a single tool. In this tutorial, we will focus on testing APIs using Playwright TypeScript. By the end of this article, you should have a clear understanding of writing automation tests for PUT, PATCH, and DELETE API requests with Playwright TypeScript. Check out the previous articles to learn how to test GET API request and verify response data with Playwright TypeScript. What are PUT Requests? PUT requests are commonly used to update an existing resource by replacing its current data with the data provided in the request body. They are similar to POST requests; the Content-Type header plays an important role because it tells the server how to process the data being sent. When a PUT request is successful, it typically returns a 200 status along with the updated resource in the response. However, this can vary by API design; some APIs may return only a status code without a response body. How to Test PUT API Requests With Playwright TypeScript Playwright offers a request API that lets us create and manage HTTP request contexts easily. We’ll use the PUT /updateOrder API that updates the available order using its order ID. This API is a part of the RESTful e-commerce application available on GitHub. This API requires an authentication token to validate the request and authorize the update; if the token is missing or invalid, the request will fail. The order ID needs to be passed as a path parameter so the correct order can be identified, and the updated details should be included in the request body in JSON format. Since this is a PUT request, you must send the entire order object, not just the field being modified. Even for a small change, the full set of order data is required. We’ll use the following test scenario to demonstrate testing the PUT API request with Playwright TypeScript: Markdown ## Test Scenario Title: Update the existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update all the order details of order_id "1" 2. Verify that the Status Code 200 is returned in the response 3. Assert that the order details have been updated correctly To implement this test scenario, we’ll need to perform the following steps: Hit the Auth API and extract the token.Write a test to send the PUT API request and pass the token extracted in step 1. Step 1: Create a Fixture to Generate and Extract the Token We’ll need the auth token for PUT, PATCH, and DELETE API requests, so let’s create a fixture that hits the POST /auth API and extracts the token for use across other tests. Let’s create a new fixture file auth.fixture.ts, in the fixtures folder in the project’s root directory. TypeScript import { test as base, expect } from "@playwright/test"; import dotenv from "dotenv"; dotenv.config(); type MyFixtures = { token: string; }; export const test = base.extend<MyFixtures>({ token: async ({ request }, use) => { const response = await request.post("http://localhost:3004/auth", { data: JSON.stringify({ username: process.env.USERNAME, password: process.env.PASSWORD, }), headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); const responseBody = await response.json(); await use(responseBody.token); }, }); export { expect } from '@playwright/test'; A custom Playwright test setup is created by extending the base test object and adding a reusable token fixture. It uses the dotenv library to load environment variables like USERNAME and PASSWORD, ensuring sensitive data is not hardcoded, and defines a TypeScript type to make the token available with proper type safety. A .env file is created containing the credentials and placed in the project’s root folder. This ensures best practices are followed and sensitive information, such as usernames and passwords, is not exposed. As this is a demo API, the credentials used with the POST /auth endpoint are provided below. In a real-time project, the .env file should be added to the .gitignore file to ensure it is not committed to the remote repository. Plain Text #.env USERNAME=admin PASSWORD=secretPass123 Inside the fixture, an authentication API is called to generate a token; the response is validated, and the token is extracted and passed to the tests using the use() function. Finally, expect is re-exported so that both test and expect can be imported from this single fixture file, keeping test imports clean and consistent. Step 2: Updating the Order With a PUT Request and Verifying the Response Let’s create a new spec file to update the order using the PUT /updateOrder API request. TypeScript # 5_put-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Order using PUT API Request", async ({ request, token }) => { const updatedOrder = { user_id: "89", product_id: "124", product_name: "Lindt Choclate", product_amount: 50, qty: 3, tax_amt: 10, total_amt: 160, }; const id = 1; const response = await request.put( `http://localhost:3004/updateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ product_id: "124", product_name: "Lindt Choclate", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides the token as part of the test context. This token is generated before the test runs and injected via Playwright fixtures, eliminating the need to manually authenticate in each test.Request body: The updatedOrder object represents the payload sent in the request to update an existing order. It contains the required fields for the order update request body. This object is later serialized into JSON format before being sent to the API.Sending the PUT request: The request.put() function is used to send an HTTP PUT request to update the order with a specific ID. The request includes the serialized request body and headers such as Content-Type and Authorization, where the token is passed for authentication. This step performs the actual API interaction to update the order in the system.Dynamic endpoint handling: The order ID is stored in a variable and dynamically appended to the API endpoint using template literals. This makes the test more maintainable and reusable.Verifying the response: The response is first converted to JSON, and the status code is validated to be 200, indicating a successful update. Then, assertions are made using expect.ObjectContaining that partially validates the response body, ensuring that the success message and updated order details match expected values. Test Execution Before running the test, the RESTful e-commerce application should be started using Docker by running the following command from the terminal: Plain Text docker compose -f docker-compose-ecommerce.yml up -d The following command should be run from the terminal: Plain Text npx playwright test tests/5_put-test.spec.ts The following test output shows that the test was executed successfully: What Are PATCH Requests? A PATCH request is used when we want to update only part of a resource, not the whole thing. It’s similar to a PUT request, but the main difference is that PUT expects the entire payload, whereas PATCH lets us send only the fields we want to change. How to Test PATCH API Requests With Playwright TypeScript We’ll use the PATCH /partialUpdateOrder API from the RESTful e-commerce application that updates the available order using its order ID. Similar to the PUT API request discussed in the earlier section of this article, the PATCH API also needs an authentication token to update the order. As it is a PATCH request, we don’t need to send the full payload. Only the fields that require updates should be included in the request body. Test Scenario: Markdown ## Test Scenario Title: Partially update an existing orders in the system. ### Pre-condition: Valid orders are available in the system #### Test 1. Update the user_id and product_id for the order ID - 1 2. Verify that the Status Code 200 is returned in the response. 3. Assert that the order details have been updated correctly. The PATCH API request requires an authorization token. We’ll use the token from the auth.fixture.ts file that we generated in the earlier section while discussing the PUT API request. Let’s create a new spec file to partially update an order using the PATCH /partialUpdateOrder API. TypeScript # 6_patch-test.spec.ts import { test, expect } from "../fixtures/auth.fixture.js"; test("Update Partial Order using PATCH API Request", async ({ request, token, }) => { const updatedOrder = { user_id: "98", product_id: "214", }; const id = 1; const response = await request.patch( `http://localhost:3004/partialUpdateOrder/${id}`, { data: JSON.stringify(updatedOrder), headers: { "Content-Type": "application/json", Authorization: `${token}`, }, }, ); const responseBody = await response.json(); expect(response.status()).toBe(200); expect(responseBody).toEqual( expect.objectContaining({ message: "Order updated successfully!", order: expect.objectContaining({ user_id: "98", product_id: "214", }), }), ); }); Code Walkthrough Using the auth fixture for getting the token: The test uses a custom fixture that automatically provides a token, so we don’t have to call the POST /auth API every time. The token is generated before test execution and injected into the test context, ensuring secure and reusable authentication across multiple tests.Request body: The updatedOrder object contains only the fields user_id and product_id that need to be modified. As it is a PATCH request, there is no need to send the complete order payload; this makes the request lightweight and focused on partial updates.Sending the PATCH request: The request.patch() function is used to send a partial update request to the API endpoint, with the order ID dynamically added to the URL. The request includes the JSON-stringified body along with the Content-Type and Authorization headers. The token is passed in the Authorization header.Verifying the response: The response is parsed into JSON and validated to ensure a 200 status code is returned after the PATCH API is hit successfully. This verification ensures that the operation was successful. Then, expect.objectContaining is used to validate the response body, confirming that the success message and updated fields match the expected values. Test Execution To run the test, the following command should be executed from the terminal: Plain Text npx playwright test tests/6_patch-test.spec.ts The following screenshot of the test execution confirms that the test was run successfully: What Are DELETE Requests? A DELETE API request is used for removing a specific resource from the server. In most cases, it doesn’t return a response body. The resource is identified using a URI, and once the request is processed, it’s permanently deleted. Because of this, DELETE operations can have side effects, like removing records from a database. A DELETE request should be handled carefully because: Data deleted through a DELETE request is permanently removed and cannot be restored.It directly impacts the server data; hence, it is not considered a safe method.It may lead to inconsistencies if not managed appropriately.DELETE isn’t always idempotent in real-world scenarios. For example, the first DELETE request might return a 204 (No Content) indicating success, while calling it again on the same resource could return 404 (Not Found) because the resource has already been removed. How to Test DELETE API Requests With Playwright TypeScript We’ll use the DELETE /deleteOrder API endpoint from the RESTful e-commerce application for this demonstration. This API requires an order_id as a path parameter to identify the order to delete. A valid authentication token must be included in the headers. On successful execution, the order is removed, and a 204 (No Content) status is returned. Want to learn more about API Testing? Check out this tutorial to get hands-on API Testing Test Scenario: Markdown ## Test Scenario Title: Delete existing order from the system. ### Precondition - There should be some valid orders available in the system. #### Test Steps: 1. Delete the order with order_id "6" using the DELETE request. 2. Verify that 204 Status Code is returned in the response. To implement this test scenario, we need valid orders in the system, which can be created using the POST /addOrder API. Refer to the earlier tutorial on testing POST requests with Playwright TypeScript to explore different ways to generate request payloads and test the POST API request. Test Implementation Let’s create a new spec file to delete an order using the DELETE /deleteOrder API. TypeScript import { test, expect } from "../fixtures/auth.fixture.js"; test("Delete an order using DELETE API Request", async ({ request, token }) => { const id = 6; const response = await request.delete( `http://localhost:3004/deleteOrder/${id}`, { headers: { Authorization: `${token}`, }, }, ); expect(response.status()).toBe(204); }); Code Walkthrough Using the auth fixture for getting the token: The test imports a custom test from the auth fixture, which automatically provides a valid authentication token. This token is generated before the test runs and injected into the test context, so there’s no need to call the POST /auth API explicitly before the DELETE API.Sending the DELETE request: The request.delete() function is used to send a DELETE request to the API endpoint, with the ID dynamically passed in the URL to identify the order to be deleted. The request includes the Authorization header, where the token is sent to authenticate the request. No request body is required for this operation.Assertion: The test verifies that the response status is 204, which indicates that the request was successfully processed and the resource was deleted. Since DELETE APIs often don’t return a response body, validating the status code is sufficient to confirm success. Test Execution To run the test, run the following command from the terminal: Plain Text npx playwright test tests/7_delete-test.spec.ts The following screenshot confirms that the test ran successfully: A simple, practical approach is to call the GET API after running a DELETE request to confirm the resource has been removed from the system. Summary Testing PUT, PATCH, and DELETE APIs with Playwright and TypeScript helps ensure that the application handles updates and deletions correctly. While challenges such as managing test data, handling authentication, and maintaining test independence can arise, Playwright’s fixtures and built-in request handling make it easier to create clean and reusable tests. The main advantage is the ability to validate end-to-end API behavior with clear assertions and strong TypeScript support. In my experience, the tests should be isolated, avoid unnecessary dependencies between them, and validate both status codes and response data to make the API tests more robust and maintainable. Happy testing!!
AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!
Every Go service I have ever shipped ends up growing the same organ. It starts small: a couple of os.Getenv calls. Then a secret needs to come from AWS Secrets Manager, so you add the SDK. Then someone wants that secret to rotate without a deploy, so you add a time.Ticker. Then the ticker needs a mutex because two goroutines read the config while it is being replaced. Then a bad value slips in and takes down production, so you add validation. Then a secret shows up in a log line during an incident, and now you are adding a redaction wrapper too. By the end you have a bespoke ConfigManager in every repo: a loader, a fetcher, a refresh loop, a mutex, and a prayer. It is never quite the same twice, it is rarely tested well, and it is exactly the kind of code nobody wants to own. I got tired of writing it, so I built a library that does it once, properly. It is called mamori; it is open source, and this article is the full tour: what it is, why it exists, how to use it end to end, and when you should (and shouldn't) reach for it. The Restart-to-Rotate Problem Here is the assumption baked into most Go config code: configuration is static. You read it at boot, decode it into a struct, and treat that struct as immutable for the lifetime of the process. If something changes, you redeploy. That assumption was fine when "config" meant a log level and a port. It is not fine anymore. In a modern service, a meaningful amount of your config is dynamic by nature: Database passwords and API tokens rotate on a schedule (and increasingly, automatically).Vault hands you dynamic credentials with a lease that expires in an hour.Feature flags flip in the middle of the day, on purpose.TLS certificates get reissued. The industry answer to "config is dynamic" has mostly been to push the problem into the platform. On Kubernetes, the External Secrets Operator syncs your cloud secret into a Secret, and then... you still restart the pod to pick it up, or you mount it as a file and hope your app re-reads it. The dynamic part gets flattened back into a static one at the worst possible layer. mamori takes the opposite position: your process should be able to observe a change at the source and react to it, in memory, without bouncing. Rotating a password should rotate a connection pool, not recycle a pod. What Is mamori? mamori (守り, Japanese for "protection" or "safeguard") is an embedded Go library — not an operator, not a server, not a sidecar. It lives inside your process. Its tagline says it plainly: Typed, watchable config & secrets for Go. Load configuration and secrets from anywhere into validated Go structs — and keep them reconciled at runtime, without a restart. There are three concepts you actually touch, and they map cleanly onto three types: Refs. Struct fields carry a source tag containing a URL-like reference to where the value lives — env:LOG_LEVEL, aws-sm://prod/db#password, vault://secret/app#token. A ref names what you want, not how to fetch it.Providers. A provider resolves one scheme (aws-sm, vault, k8s-secret, …) into a value. Providers register themselves using the same pattern as database/sql drivers, so the core module carries zero cloud-SDK dependencies — you only pull in the providers you use.The Reconciler. This is the interesting part. Watch resolves every ref once (fail-fast at startup), then keeps watching each source. When a value changes, it re-validates the entire struct, and only if the new snapshot is valid does it atomically swap it in and hand you a diff-aware callback. Your app never sees a half-updated, invalid config. If you have used IOptionsMonitor<T> in .NET, w.Get() is the direct analogue: always the latest valid configuration, lock-free. How mamori Compares I want to be honest about prior art, because none of these primitives are new — mamori's contribution is composing them. gocloud.dev/runtimevar is the closest primitive. It watches a single variable beautifully, but it is one variable at a time, with no struct composition, no tags, no validation, no secret hygiene. mamori adds all of that on top of the same idea.Viper/koanf are excellent, but they are config-first with secrets bolted on. mamori is deliberately secrets-first, with config included: secret types redact by default, and there is a go vet analyzer that yells at you if you store a secret in a plain string.envconfig/caarlos0/env are load-once. No watching, env only.The External Secrets Operator is complementary, not competitive. mamori is for apps that want to skip the Kubernetes Secret hop entirely, or that do not run on Kubernetes at all. It keeps no persistent external state, so there is no finalizer lifecycle to babysit. And to be equally clear about what mamori is not: it is not a secrets store (no encryption at rest, no server), not a sync engine between stores, not a general feature-flag platform, and not cross-language. It is a Go-idiomatic library, and it stays in its lane. Getting Started The core module has built-in env:, file://, and dotenv: providers, so you can do something useful with zero extra dependencies: Shell go get github.com/xavidop/mamori Heads up: mamori requires Go 1.26 or newer. It leans on modern generics ergonomics, so this is a hard floor, not a suggestion. The whole API is driven by a struct and its tags. You describe what you want; mamori figures out how: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` TLSCert []byte `source:"file:///etc/tls/tls.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } // cfg is fully resolved, defaulted, and validated. } The signature is exactly what you'd hope for: Go func Load[T any](ctx context.Context, opts ...Option) (T, error) Load resolves every ref once, applies defaults, validates, and returns your typed config. It fails fast: on any resolve or validation error, it returns the zero value and a non-nil error. It never returns a partially-populated struct — you either get a complete, valid config or an error. A few things worth knowing about the tags: TagMeaningsource:"..."The ref: where the value comes from.default:"..."Used only when the ref resolves to not-found — never on an error.validate:"..."go-playground/validator/v10 syntax, run on load and on every update.flatten:"json|yaml|env"Decode one payload into a nested struct.optional:"true"Tolerate not-found with no default (field keeps its zero value). That default-only-on-not-found rule is deliberate and important: if AWS Secrets Manager is having a bad day, you want an error, not a silent fallback to a default password. Watching and Reconciliation This is why mamori exists. Swap Load for Watch and your config becomes a living thing: Go func Watch[T any](ctx context.Context, opts ...Option) (*Watcher[T], error) You read the current config with a lock-free w.Get(), and you react to changes with an OnChange callback: Go w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(30*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("%s changed: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // always the last VALID config, no lock needed The Change[T] you get in the callback carries the old value, the new value, and a per-field diff, plus a Changed(path) helper so you only do expensive work (like rebuilding a connection pool) when the field you care about actually moved. The guarantees are the whole point, so let me spell them out: Atomicity. OnChange only ever fires with a fully re-validated snapshot. If an incoming value fails validation, the update is rejected: Get() keeps returning the last good config and OnError receives a *ValidationError. Your config never transitions into a broken state mid-flight.Push where possible, poll where not. Providers whose backend can push changes (Kubernetes informers, Vault lease watchers, Consul blocking queries, fsnotify for files) push natively. Everything else is polled on WithPollInterval, jittered by ±20% by default so a fleet of pods doesn't stampede your secrets backend on the same tick. If a native watch fails to start, mamori quietly falls back to polling.Coalescing. Rapid-fire changes within a debounce window (500ms default, overridable per-field with ?debounce=) collapse into a single event.Lease-aware. If a provider reports an expiry (Vault dynamic creds), mamori schedules a refresh before the lease dies, rather than waiting for the next poll.First event. Watch resolves the initial config before it returns, so OnChange fires only on subsequent changes. Read the initial state from w.Get(). There's a nice operational escape hatch too. Transient backend outages shouldn't page anyone — but prolonged staleness should. WithStale draws that line for you: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { alert.Page("config stale", stale) return } metrics.Inc("config_transient_error") }), ) A blip retries with exponential backoff and keeps the last good value. Ten minutes of failure escalates to a hard *StaleError you can actually page on. Secrets That Behave Like Secrets Configuration and secrets are treated differently, on purpose. Any field that holds a secret should be a secret.String or secret.Bytes instead of a plain string or []byte. These types redact themselves everywhere a value normally leaks: Go s := secret.NewString("hunter2") fmt.Println(s) // [REDACTED] fmt.Sprintf("%v", s) // [REDACTED] fmt.Sprintf("%#v", s) // [REDACTED] json.Marshal(s) // "[REDACTED]" slog.Info("login", "pw", s) // pw=[REDACTED] s.Reveal() // "hunter2" <- the only way to read the plaintext It works because the type implements fmt.Stringer, fmt.GoStringer, json.Marshaler, and slog.LogValuer. The single, greppable access point is Reveal(). That is a feature: in code review or an audit, grep Reveal shows you every place a secret is actually unwrapped. (secret.Bytes.Reveal() returns []byte rather than a string, for keys and certificates.) And because "just remember to use secret.String" is exactly the kind of discipline that erodes under deadline pressure, mamori ships a go vet analyzer that enforces it: Shell go install github.com/xavidop/mamori/tools/reconcilevet/cmd/reconcilevet@latest go vet -vettool=$(which reconcilevet) ./... reconcilevet flags any field whose source points at a secret-bearing scheme (aws-sm, gcp-sm, azure-kv, vault, op, sops, k8s-secret) but is stored in a plain string or []byte. The leak becomes a build failure, not a postmortem. Providers: More Than 30 Sources The core stays dependency-free; each cloud backend is its own module you opt into. A ref's scheme picks the provider: Shell go get github.com/xavidop/mamori/providers/aws # aws-sm:// aws-ps:// go get github.com/xavidop/mamori/providers/vault # vault:// go get github.com/xavidop/mamori/providers/gcp # gcp-sm:// go get github.com/xavidop/mamori/providers/azure # azure-kv:// go get github.com/xavidop/mamori/providers/k8s # k8s-secret:// k8s-cm:// go get github.com/xavidop/mamori/providers/consul # consul:// go get github.com/xavidop/mamori/providers/doppler # doppler:// go get github.com/xavidop/mamori/providers/onepassword # op:// go get github.com/xavidop/mamori/providers/sops # sops:// That is a small slice. All in, there are more than 30 providers: secret managers (AWS Secrets Manager, AWS Parameter Store, Vault, GCP, Azure Key Vault, Doppler, 1Password, SOPS), databases (Postgres, MySQL, SQLite, MongoDB, DynamoDB, Cosmos DB, Redis), KV stores (Consul, etcd), Kubernetes Secrets and ConfigMaps, Firebase/Firestore/Remote Config, object storage (S3, GCS, Azure Blob), and a whole shelf of feature-flag platforms (LaunchDarkly, Unleash, Flagsmith, ConfigCat, Split, GrowthBook, Flipt, GO Feature Flag). Registration follows the database/sql convention — a blank import runs the provider's init(), which calls mamori.Register(...): Go import _ "github.com/xavidop/mamori/providers/vault" When you need explicit configuration (a region, a custom client), construct the provider and pass it with WithProvider. That takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) The ref grammar is compact — scheme://path[#key][?opts] — with a couple of quirks worth internalizing: The #key fragment selects one field out of a JSON payload: aws-sm://prod/db#password.AWS Parameter Store, etcd, sops, and file preserve absolute paths, so they use a triple slash: aws-ps:///myapp/log-level, file:///etc/tls/tls.crt.Doppler is the one provider where the # fragment is required: doppler://backend/prd#STRIPE_API_KEY. When a payload is a structured blob, you can decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"aws-sm://prod/redis" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } One AWS Secrets Manager call, one JSON blob, cleanly typed — with the password still redacting itself. Writing Your Own Provider If your source of truth is homegrown — an internal config service, a weird legacy database table — you can teach mamori about it. The interface is intentionally tiny: Go type Provider interface { Scheme() string Resolve(ctx context.Context, ref Ref) (Value, error) } A minimal implementation looks like this: Go func (p *Provider) Resolve(ctx context.Context, ref mamori.Ref) (mamori.Value, error) { raw, err := p.fetch(ctx, ref.Path) if isNotFound(err) { return mamori.Value{}, mamori.ErrNotFound // MUST satisfy errors.Is } if err != nil { return mamori.Value{}, err } if ref.Key != "" { raw, err = mamori.SelectKey(raw, ref.Key) // implements #key against a JSON payload if err != nil { return mamori.Value{}, err } } return mamori.Value{ Bytes: raw, Version: mamori.VersionHash(raw), // change detection; use a native revision if you have one Sensitive: true, // true for secret-bearing sources }, nil } func init() { mamori.Register(New()) } // database/sql pattern; panics on a duplicate scheme The Version field is what powers cheap change detection: if your backend exposes a revision (a VersionId, a Vault version, an mtime hash), return it; otherwise mamori.VersionHash(bytes) gives you a stable FNV hash for free. Two optional interfaces let you do better when the backend supports it: Go type WatchableProvider interface { Provider Watch(ctx context.Context, ref Ref) (<-chan Update, error) } type BatchProvider interface { Provider ResolveBatch(ctx context.Context, refs []Ref) (map[string]Value, error) } Implement Watch only if your backend can genuinely push — otherwise let mamori's poller do the work, so behavior stays consistent across providers. Implement ResolveBatch if you can fetch many refs in one round trip (the AWS provider does this automatically). Best of all, you don't have to trust yourself to get the contract right. There's a conformance kit — github.com/xavidop/mamori/providertest — that exercises resolution, not-found typing, version monotonicity, concurrency, context cancellation, native watching, goroutine hygiene, and even asserts you never accidentally log the payload. Point it at your provider, and it tells you if you've broken the rules. Middleware: Cache, Failover, Audit, Rate-Limit Because every provider is just a Provider, you can wrap one in another. The middleware package ships the decorators you'd otherwise reinvent: MiddlewareWhat it doesCache(ttl, inner)Memoize successful resolves. Errors and not-found are not cached.Failover(primary, replicas...)Try the primary, fall back to replicas on transport errors. Not-found is authoritative.RateLimit(rps, inner)Cap resolves per second so you don't trip your secrets backend's throttle.Audit(logger, inner)Log scheme, ref, latency, and outcome — never the value.Prefix(prefix, inner)Namespace every ref path — handy for multi-tenant setups. They compose by nesting, so you can read the stack top-down: Go import "github.com/xavidop/mamori/middleware" mamori.WithProvider( middleware.Cache(5*time.Minute, middleware.Audit(logger, middleware.RateLimit(10, middleware.Failover(primarySM, replicaSM), ), ), ), ) Crucially, the shipped middleware preserves the WatchableProvider and BatchProvider capabilities of whatever they wrap — decorating a provider never silently downgrades it from push to poll. Observability Out of the Box The core takes no OpenTelemetry dependency (keeping it lean), but there's a bridge module when you want metrics and traces: Shell go get github.com/xavidop/mamori/x/otel Go import ( "go.opentelemetry.io/otel" mamoriotel "github.com/xavidop/mamori/x/otel" "github.com/xavidop/mamori" ) meter, _ := mamoriotel.NewMeter(otel.Meter("mamori")) tracer := mamoriotel.NewTracer(otel.Tracer("mamori")) w, err := mamori.Watch[Config](ctx, mamori.WithMeter(meter), mamori.WithTracer(tracer), ) You get a mamori.resolve.duration histogram, mamori.refresh.count and mamori.watch.errors counters (all tagged by scheme), and a span per resolve. And because WithMeter/WithTracer take tiny internal interfaces, you can implement them against Prometheus, statsd, or a test recorder without pulling OTel into your binary at all. A Complete, Runnable Example Here is the whole loop — load, watch, react — in one file you can run today (it's the examples/basic program from the repo): Go package main import ( "context" "fmt" "log/slog" "os" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) const tokenPath = "/tmp/mamori-example-token" type Config struct { LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` APIToken secret.String `source:"file:///tmp/mamori-example-token"` } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) _ = os.WriteFile(tokenPath, []byte("initial-token"), 0o600) defer func() { _ = os.Remove(tokenPath) }() ctx, cancel := context.WithCancel(context.Background()) defer cancel() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(2*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { for _, f := range ev.Fields { logger.Info("config changed", "field", f.Path, "from", f.OldVersion, "to", f.NewVersion) } if ev.Changed("APIToken") { logger.Info("rotating clients with the new API token", "token", ev.New.APIToken) } }), mamori.OnError(func(err error) { logger.Warn("reconcile error", "err", err) }), ) if err != nil { logger.Error("watch failed", "err", err) os.Exit(1) } defer func() { _ = w.Close() }() cfg := w.Get() logger.Info("loaded config", "logLevel", cfg.LogLevel, "workers", cfg.Workers, "apiToken", cfg.APIToken) // Simulate a rotation a few seconds in. go func() { time.Sleep(3 * time.Second) _ = os.WriteFile(tokenPath, []byte("rotated-token"), 0o600) }() time.Sleep(8 * time.Second) logger.Info("final config", "apiToken", w.Get().APIToken) } Shell LOG_LEVEL=debug WORKERS=8 go run ./examples/basic Notice the token field in the log output: it prints [REDACTED] every time, even though the program is actively rotating it. That's the whole philosophy in one demo — live rotation, no restart, and nothing leaks. When to Reach for mamori (and When Not) Reach for it when: Your config or secrets genuinely change while the process runs, and you want to react in memory instead of bouncing pods.You pull from more than one source and are tired of a different SDK and refresh strategy for each.Secret hygiene matters and you want redaction and a go vet guardrail for free.You run outside Kubernetes, or you're on Kubernetes but want to skip the Secret-sync hop. Probably skip it when: Your config is a log level and a port from the environment, read once at boot. os.Getenv is fine — don't add a dependency.You need a secrets store (encryption at rest, a server component) — that's Vault's job, not mamori's.You're syncing secrets between stores — that's what ESO, vals, and teller are for. Wrapping Up mamori is the library I wish I'd had every time I typed type ConfigManager struct for the fifth time. It takes the boring, error-prone, security-sensitive plumbing — load, validate, watch, redact, reconcile — and makes it a typed struct and one function call. It's open source under the MIT license and currently at v0.1.0 — early, pre-1.0, and moving fast, so feedback and issues are genuinely welcome. Give it a spin: Shell go get github.com/xavidop/mamori Docs: mamorigo.dev/docsHomepage: mamorigo.devSource: github.com/xavidop/mamori Provider Deep Dives If you live on a particular platform, I've written focused, hands-on companions: Hot-reloading Kubernetes Secrets and ConfigMaps in Go — no pod restartsRotating AWS Secrets Manager secrets without restarting your Go serviceGCP Secret Manager, live-reloaded in GoAzure Key Vault secrets that reconcile themselves in Go If you build something with it, or write a provider for a backend I haven't covered yet, I'd love to hear about it. Happy coding!
Ever felt stuck writing Playwright tests because of messy or hardcoded test data? It usually starts simple, but as your tests grow, like testing an e-commerce or finance-related application, managing realistic data quickly becomes confusing and repetitive. More time is spent fixing test data than actually testing. That’s where the Builder Pattern, combined with Faker, can really help. It lets us create clean, flexible, and realistic test data with ease. In this article, we’ll learn how this approach can make the API tests with Playwright TypeScript easier to write, read, and maintain. What Is the Builder Design Pattern? The Builder Design Pattern is a creational design pattern that allows constructing complex objects step by step, allowing different representations of the same object. It separates object-creation logic from structure, making the code more readable and maintainable. This pattern is especially useful for objects with many optional parameters, as it avoids messy constructors and improves test data creation. What Is Faker? Faker is a library that generates realistic-looking fake data such as names, emails, addresses, numbers, and more. It is commonly used in testing to avoid hardcoding values and to simulate real-world data scenarios. By combining the Builder Pattern with Faker, we can generate realistic test data on the fly and perform testing without worrying about test data management, duplication, or maintenance. Faker can be installed using npm by running the following command in the terminal: Plain Text npm install @faker-js/faker --save-dev After installation, we can use the Faker library directly in the code by using the following import: TypeScript import {faker} from '@faker-js/faker'; How to Use the Builder Pattern With Faker in Playwright With TypeScript To use the Builder design pattern with Faker in Playwright TypeScript, the setup usually includes two main parts: ModelBuilder class The Model defines how the data should look, while the Builder class handles creating and customizing that data using Faker. Application Under Test We’ll use the POST /addOrder API of the RESTful e-commerce demo application for the demonstration. The following is the request payload of the API: JSON [ { "user_id": "string", "product_id": "string", "product_name": "string", "product_amount": 0, "qty": 0, "tax_amt": 0, "total_amt": 0 } ] Creating a Model for the Order The following model interface should be created for the Order payload: TypeScript export interface Order { user_id: string, product_id: string, product_name: string, product_amount: number, qty: number, tax_amt: number, total_amt: number } Code Walkthrough The Order interface represents the structure of an order object, where each field in the interface specifies the expected property and its data type: user_id, product_id, and product_name are strings.product_amount, qty, tax_amt, and total_amt are numbers. This interface ensures that any order object created in the code follows this exact structure, helping catch errors early and making the code more predictable and easier to maintain. Creating the Builder Class The following builder class is created to generate a new order with realistic fake values using the Faker library. TypeScript import { Order } from "../models/order.model.js"; import {faker} from '@faker-js/faker'; export class OrderBuilder { private order:Order; constructor() { const product_amount = faker.number.int({ min: 1, max: 100 }); const qty = faker.number.int({ min: 1, max: 5 }); const tax_amt = faker.number.int({ min: 2, max: 10 }); this.order = { user_id: this.generateUserId(), product_id: this.generateProductId(), product_name: faker.commerce.productName(), product_amount, qty, tax_amt, total_amt: (product_amount*qty)+tax_amt } } private generateUserId():string { return faker.number.int({min: 1, max: 50}).toString(); } private generateProductId():string { return faker.number.int({ min: 1, max: 100 }).toString(); } withUserID(user_id:string) { this.order.user_id = String(user_id); return this; } withProductId(product_id:string) { this.order.product_id=String(product_id); return this; } withProductName(product_name:string) { this.order.product_name = String(product_name); return this; } withProductAmount(product_amount:number) { this.order.product_amount= Number(product_amount); return this; } withQty(qty:number) { this.order.qty=Number(qty); return this; } withTaxAmt(tax_amt:number){ this.order.tax_amt=Number(tax_amt); return this; } withTotalAmt(total_amt:number) { this.order.total_amt= Number(total_amt); return this; } build():Order{ return this.order; } } Code Walkthrough The OrderBuilder class implements the Builder Pattern to create an Order object using Faker. The following is a detailed explanation of the code: Constructor(): The OrderBuilder class internally holds an Order object, which is based on the Order interface. In the constructor, it generates default random data using the Faker library.Private functions: The private functions, namely, generateUserId() and generateProductId(), are helper methods that generate random user IDs and product IDs, respectively.with() functions: The withX() functions, such as withUserId(), withProductId(), etc., allow overriding specific fields. Each method updates a value and “returns this,” enabling method chaining.Build functions: The build() function method finally returns the fully constructed Order object. In short, this class generates realistic test data by default while allowing selective customization, resulting in cleaner and more reusable test code. Writing the API Automation Test Using Playwright The following Playwright test demonstrates how to send a POST request using an order request payload generated through the Builder Pattern with the Faker library. TypeScript import { test, expect } from "@playwright/test"; import { OrderBuilder } from "../builders/order.builder.js"; import { Order } from "../models/order.model.js"; test("Add Order details using Builder Design Pattern", async ({ request }) => { const orderDetails = new OrderBuilder().build(); const orders: Order [] = [orderDetails]; console.log(orders); const response = await request.post("http://localhost:3004/addOrder/", { data: orders, headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); Code Walkthrough The test imports OrderBuilder to create test data, and the Order interface to ensure type safety.Inside the test, new OrderBuilder().build() generates a complete order object with realistic values using Faker.The generated order is wrapped in an array, orders: Order[] = [orderDetails], as the API expects a list of orders. A console log statement is added to display the generated order in the console.A POST request is sent to the endpoint using Playwright’s request fixture, with the order payload passed in the data field and the appropriate Content-Type header.The test validates that the response status is 201, confirming that the order was successfully created. Test Execution Before running the test, start the RESTful e-commerce application using Docker by running the following command from the terminal: Plain Text docker compose -f docker-compose-ecommerce.yml up -d The test can be executed by running the following command from the terminal: Plain Text npx playwright test tests/post-api-builder-test.spec.ts The following is the output of the test execution: The order details generated using the Builder design pattern are printed on the console with fake data and are used as the request payload in the POST /addOrder test. Writing automated tests by combining generated fake data with selectively updated static values. In the previous test scenario, all order details were generated using the Faker library. However, in real testing scenarios, certain fields may require static values. The Builder Pattern supports this by allowing specific fields to be overridden with static data, while the remaining fields continue to be generated dynamically using Faker. TypeScript test("Add Specific Order details using Builder Design Pattern", async ({ request }) => { const orderDetails = new OrderBuilder().withUserID("56").withProductName("Donuts").build(); const orders: Order [] = [orderDetails]; console.log(orderDetails); const response = await request.post("http://localhost:3004/addOrder/", { data: orders, headers: { "Content-Type": "application/json", }, }); expect(response.status()).toBe(201); }); In this test, the user_id and product_name fields are explicitly set using the withUserID() and withProductName() functions. While the Builder generates the rest of the order details using the Faker library, these two fields are overridden with the static value provided in the test. Let’s execute the test again and check the output: In the above screenshot, it can be seen that the order details are generated successfully using the Faker library; however, the value for user_id is set to "56" and product_name is set to "Donuts," as these were explicitly provided as static values for those fields. Learn how to use the Builder Design Pattern in Java with a Datafaker library to generate test data for automated testing. Summary The Builder Design Pattern is really helpful for creating clean, flexible, and maintainable test data in API automation. When combined with Faker, it enables the generation of realistic data while still allowing specific fields to be customized as needed. There are multiple ways to supply test data in API automation for POST API testing, such as using JSON files or directly generating data with the Faker library in the request payload without the Builder Pattern. However, the choice of approach depends on the application under test, the level of flexibility required, and how the test framework is designed. Happy testing!
Visual regression testing has a dirty secret: most of the failures it reports are not bugs. They are anti-aliasing differences, sub-pixel layout shifts, font-hinting changes between OS versions, a shadow that moves three pixels, or a GPU driver that rounds a gradient differently. Every one of those trips a pixel-diff, a human triages it, marks it "not a real defect," and re-baselines the golden image. Multiply that by every screen, every platform, every appearance mode, and the maintenance cost of the test suite quietly exceeds the cost of the bugs it catches. I have spent years leading rendering-quality engineering for systems that ship across iOS, macOS, visionOS, and CarPlay, and the conclusion I keep arriving at is that pixel comparison is the wrong abstraction for validating a rendered scene. The question we actually care about is not "is this frame byte-identical to the golden image?" It is "does this frame show the right thing?" Those are different questions, and the gap between them is where semantic visual validation lives. Why Pixel-Diffing Fails at Scale Pixel-level comparison — whether raw MSE or a perceptual metric like SSIM — measures signal similarity, not meaning. SSIM (Wang et al., 2004) was a real improvement over mean-squared error because it models luminance, contrast, and local structure the way the human visual system roughly does. But it is still a low-level statistic. It will happily flag a difference no user could ever perceive, and it has no concept of whether the content of the scene is correct. Two Structural Problems Make This Worse as You Scale The False-Positive Tax A rendering pipeline is nondeterministic at the pixel level across environments. Anti-aliasing, text rasterization, sub-pixel positioning, and GPU-dependent gradient rounding all vary by device and OS build without any defect being present. A strict pixel-diff treats all of this as failure. Loosen the threshold to suppress the noise, and you start missing the defects you actually care about. There is no threshold that separates "shadow rounded differently" from "the bridge failed to render" — because to a pixel-diff, both are just pixels that changed. Combinatorial Baseline Explosion A single view does not need one golden image. It needs one per platform, per resolution, per light/dark appearance, per dynamic-type size, per locale. For a 3D map surface — with a Mercator-to-globe transition, buildings, terrain, labels, and navigation overlays — the baseline matrix is enormous, and every intentional design change invalidates all of it at once. The suite spends more engineering time being re-baselined than finding bugs. The insight is that these are not tuning problems. They are a category error: we are using a similarity metric to answer a semantic question. The Architecture of Semantic Validation Semantic visual validation replaces "compare pixels to a golden image" with "extract meaning from the frame and check it against intent." A multimodal model looks at the rendered frame and answers structured questions about what it contains, and the test asserts on those answers. The pipeline has four stages: stageresponsibilityimplementation notes Capture Grab the rendered frame deterministically Native rendering-API hooks; capture at a fixed camera pose, seed, and time-of-day so the scene is reproducible even if pixels are not Normalize Prepare the frame for the model Downscale to the model’s input resolution; strip volatile overlays (clocks, FPS counters); redact any user data before it leaves the device Reason Extract semantic content and judge it A vision-language model (e.g., Qwen-VL) run with constrained/structured decoding, returning a typed verdict — not free text Decide Convert the verdict into a CI outcome Parse structured output; apply calibrated thresholds; emit pass / fail / uncertain The important architectural decision is at the Capture stage: you must make the scene deterministic (fixed pose, seed, lighting) even though the pixels are not. That is what lets a probabilistic model produce stable decisions. Implementation: Structured Output and CI Determinism The hardest objection to using an LLM in a test pipeline is nondeterminism — a test that returns different answers on the same input is worse than useless. You defeat this with four disciplines: pin the model version, decode at temperature 0, force structured (schema-constrained) output, and gate only on high-confidence verdicts while routing uncertainty to humans rather than failing the build. Python # Reference implementation - illustrative, not drop-in. API names follow the # Qwen-VL / DashScope pattern; adapt to your serving stack. What matters is the # structure (schema-constrained output, deterministic decoding, a calibrated # confidence gate, and a three-way verdict), not the specific client. from pydantic import BaseModel, Field MODEL = "qwen-vl-2.5" # pin the exact version; a model upgrade is a test change CONFIDENCE_GATE = 0.85 # derived from calibration (see below), not guessed class Verdict(BaseModel): passed: bool confidence: float = Field(ge=0.0, le=1.0) defects: list[str] # e.g. ["building geometry missing", "label overlap"] rationale: str def validate_frame(frame, expected_context: str) -> str: """Return 'pass', 'fail', or 'uncertain'. Never raises on ambiguity.""" result = client.infer( model=MODEL, image=frame, prompt=( f"You are validating a rendered map frame. Expected: {expected_context}. " "Report geometry defects, missing elements, and label collisions." ), response_schema=Verdict, # constrained decoding -> always valid, typed JSON temperature=0.0, # deterministic decoding seed=42, ) # Low confidence never hard-fails CI -> avoids a flaky pipeline. if result.confidence < CONFIDENCE_GATE: return "uncertain" # -> human review queue; pixel-diff tripwire still runs return "fail" if not result.passed else "pass" Two points readers should not miss. First, response_schema (constrained decoding) is what makes the output parseable every time — you are not regexing prose, you are getting a validated object. Second, even at temperature=0, token-level output is not guaranteed bit-identical across hardware and batching. That is fine, because you are not asserting on tokens — you are asserting on a decision (passed) that is stable well away from the confidence gate. Treat the model as a classifier, not a text generator. Model Evaluation and Calibration A confidence score the model emits is meaningless until you calibrate it. You build a labeled ground-truth set — frames you have manually classified as correct or defective — and measure the model against it: precision, recall, and a reliability diagram (does "0.9 confidence" actually mean right 90% of the time?). The confidence gate above is chosen from the precision-recall curve to hit a target false-negative rate, because in QA a missed defect costs more than a false alarm. Calibration is also environment-specific. A model tuned for standard daylight scenes will be over- or under-confident on high-contrast night rendering. Maintain per-environment operating thresholds, and re-run calibration whenever you change the model version, the prompt, or the rendering environment — each of those is a change to the test's behavior and deserves the same rigor as changing test code. Handling False Positives and False Negatives Semantic validation trades one failure mode for two. It largely eliminates the pixel-diff false-positive tax (perceptually-invisible differences stop failing the build). But it introduces false negatives (the model misses a subtle defect) and model false positives (the model hallucinates a defect that is not there). You manage this with layered defenses: Tiered validation: Keep a cheap pixel-diff as a tripwire for regions that must be exact — legal text, logos, brand color swatches — where semantic tolerance is wrong. Use the model for everything spatial and contextual.Three-way verdicts: 'uncertain' is a first-class outcome. It routes to a human review queue; it never fails the build. This is what keeps the pipeline non-flaky.Standing ground-truth audits: Periodically re-score the model against the labeled set and track precision/recall over time. A drop is your early warning that a model or prompt change regressed the validator itself. CI/CD Integration In the pipeline, semantic validation runs as a stage that only invokes the model on views whose rendering inputs actually changed — you do not pay for inference on untouched screens. A high-confidence 'fail' blocks the merge with the model's defects list attached to the check. 'uncertain' posts to a review queue and does not block. Every run logs the model version, prompt hash, confidence, and verdict, so a failing check is reproducible and explainable rather than an opaque "the AI said no." Cost, Latency, Privacy, and Observability Cost: Model inference is dollars-per-call, whereas pixel-diff is free. The tiered strategy is the control: cheap checks first, model only on changed or ambiguous views, batched where possible. Latency: Pixel-diff is milliseconds; VL inference is seconds. Run it asynchronously and in parallel across views so wall-clock stays flat, and reserve full semantic passes for pull requests rather than every commit. Privacy: Frame buffers can contain user content (a map centered on someone's home, a real address in a label). Normalize and redact on-device, and run inference inside your own trust boundary — never ship a user-bearing frame to an external API. This is a hard architectural constraint, not a policy checkbox. Observability: Track the `uncertain` rate, the false-positive/negative rate against the audit set, and confidence drift over time. These are the health metrics of the validator; when they move, you investigate before you trust another green check. A Migration Strategy From Pixel-Based Testing Do not flip the switch. Migrate in shadow mode: Run in parallel, gate on nothing. Keep pixel-diff authoritative; run semantic validation alongside it, logging verdicts without blocking.Mine the disagreements. Every case where the two disagree is a labeled example — pixel-diff-failed-but-semantically-fine is your false-positive tax made visible; semantically-failed-but-pixel-passed is a defect your old suite would have missed.Calibrate on that data. Set your confidence gate and per-environment thresholds from the real disagreement distribution.Flip gating for high-confidence defects only. Let the model block merges where it is confident; keep pixel-diff as the exact-match tripwire.Retire redundant baselines. As confidence holds, drop the golden-image matrix for everything except exact-match regions. At each step, you have a fallback and real data, so the migration is reversible and evidence-based rather than a leap of faith. Conclusion The move to semantic validation is a change in what we assert, not just which tool we run. Pixel-diffing asks "did the pixels change?" — a question whose answer is almost always "yes, and it doesn't matter." Semantic validation asks "is the scene correct?" — the question we actually have.
Most agent projects start the same way: one model, one prompt loop, three or four tools. That's the right place to start, and for a lot of workloads it's also the right place to stop. The trouble shows up later, when the same agent is asked to inspect infrastructure, parse OpenAPI specs, write docs, answer developer questions, and call operational tools. The system prompt turns into a wall of text. Tool selection gets flaky in ways you can't reproduce. Someone tunes the prompt to fix documentation quality and quietly breaks spec analysis, and nobody notices for a week. Splitting the agent is the obvious fix. But the agents were never the hard part. The hard part is the contract between them: how one agent finds another, how it hands over work, where authentication lives, and how you trace a request that crossed three runtimes before it failed. This article builds a small multi-agent system on Amazon Bedrock AgentCore Runtime using the Agent-to-Agent (A2A) protocol: an orchestrator, an OpenAPI spec analyzer, and a documentation generator. Three agents are a toy. The decisions are not. A note on versions. AgentCore and A2A are both moving fast. Everything here was checked against the AWS documentation and AgentCore CLI 0.25.x in August 2026. Pin your CLI and Python dependencies, and re-check the docs before you deploy anything you care about. Is a Second Agent Actually Worth It? Multi-agent isn't a maturity level. Every remote handoff you add buys you latency, an extra auth boundary, and a new class of failure that only shows up under load. Keep one agent when one team owns the workflow, the tools sit inside the same security boundary, and a single prompt can plausibly cover the job. Don't split because the architecture diagram looks better with more boxes in it. Split when responsibilities genuinely diverge. The signals are usually organizational before they're technical: two teams own two capabilities, the specialists want different models, the release cadences don't match, or one capability touches data the other has no business seeing. At that point the agent stops being a prompt and starts being a service, and it needs a service's contract. The arithmetic argument for a shared protocol is the familiar one. Pairwise integrations grow with the square of the number of participants, and each one tends to invent its own payload shape, auth handling, timeout, and error format. A standard protocol doesn't make any of those problems disappear. It just means every participant expresses them in the same place, which is most of the benefit. MCP and A2A Aren't Competing These two get discussed as alternatives. They aren't: they sit on different edges of the architecture. MCP connects an agent to tools and data.A2A connects an agent (or an application) to another agent. A spec analyzer might pull an OpenAPI document out of a repo over MCP, then return its analysis to an orchestrator over A2A. Both, same request. In the code below, the OpenAPI document is passed inline so the A2A flow stays readable. One caveat about "framework-agnostic," since it gets oversold: A2A is framework-agnostic at the protocol level. That doesn't mean two arbitrary SDKs will talk to each other on the first try. Both sides still need a compatible A2A implementation, transport, protocol version, and auth configuration. Once those line up, the internals genuinely don't matter; Strands on one side, LangGraph on the other, fine. What Runtime Gives You Configure an AgentCore Runtime for A2A, and it becomes a transparent proxy in front of your container. Your side of the contract is narrow: a stateless, streamable HTTP server on 0.0.0.0:9000, mounted at the root path, speaking JSON-RPC 2.0. AgentCore passes the JSON-RPC payload straight through without touching it, and adds session isolation, authentication, scaling, and observability around it. Note the port. A2A is 9000 at /, MCP is 8000 at /mcp, plain HTTP is 8080 at /invocations. Getting this wrong is the most common reason a container deploys fine and then never answers. The A2A vocabulary you need for the rest of this: A2A client: whoever is sending work. Here is the orchestrator.A2A server: the agent endpoint receiving it.Agent Card: metadata at /.well-known/agent-card.json describing identity, endpoint, skills, capabilities, and auth requirements. This is how discovery works.Message: one turn, carrying one or more parts (text, structured data).Task and artifact: a task tracks work at one remote agent; artifacts carry its output. An orchestrator fanning out to three specialists creates three separate tasks. For inbound calls, Runtime supports IAM SigV4 or OAuth 2.0 bearer tokens (JWT). A runtime version uses one of them at a time, so pick based on who's calling and where the security boundary sits. If a client authenticates with the wrong one, you get a 403 that looks nothing like an agent error. The Shape of the System The system has a hub and spoke architecture. The orchestrator is an A2A server to the user and an A2A client to the specialists, which is worth stating explicitly because it's easy to forget when you're debugging. Hub and Spoke Achitecture The orchestrator should know what each specialist advertises. It should not know how any of them work. The test: adding a fourth agent a changelog writer, say should mean one new config entry, not edits inside the agents you already shipped. Building a Specialist Prerequisites Python 3.10+Node.js 20+ (the AgentCore CLI ships as an npm package; the A2A tutorial page still says 18, but the CLI's own docs say 20)AWS CDK installed, the CLI deploys through itAWS credentials, plus model access enabled in Bedrocknpm install -g @aws/agentcore If you still have the old Python starter toolkit installed, uninstall it first. Both provide an agentcore command, and you will lose an afternoon to that. Scaffold an A2A project and pick Strands when prompted: Shell bash agentcore create --protocol A2A The generated project gives you the config, the model loader, and the A2A server wiring. The specialist below does one thing: read an OpenAPI document and pull out a compact, honest view of its operations. Python # app/SpecAnalyzer/main.py from typing import Any import yaml from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model HTTP_METHODS = { "get", "put", "post", "delete", "options", "head", "patch", "trace", } def _auth_mode(security: Any) -> str: if security is None: return "unspecified" if security == []: return "none" if isinstance(security, list) and any(item == {} for item in security): return "optional" return "required" @tool def summarize_endpoints(openapi_yaml: str) -> list[dict[str, Any]]: """Return a compact list of operations from an OpenAPI document.""" spec = yaml.safe_load(openapi_yaml) if not isinstance(spec, dict): raise ValueError("The OpenAPI document must contain a YAML object.") paths = spec.get("paths", {}) if not isinstance(paths, dict): raise ValueError("The OpenAPI 'paths' field must be an object.") global_security = spec.get("security") operations: list[dict[str, Any]] = [] for path, path_item in paths.items(): if not isinstance(path_item, dict): continue for method, operation in path_item.items(): if method.lower() not in HTTP_METHODS: continue if not isinstance(operation, dict): continue security = operation.get("security", global_security) operations.append( { "path": path, "method": method.upper(), "operation_id": operation.get("operationId"), "summary": operation.get("summary", ""), "auth": _auth_mode(security), } ) return operations agent = Agent( model=load_model(), system_prompt=( "You are an API specification analyst. " "Call summarize_endpoints before reasoning. " "Report endpoints, authentication behavior, and concrete " "specification risks. Never invent fields that aren't present." ), tools=[summarize_endpoints], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) serve_a2a is the AgentCore SDK helper that does the boring, mandatory parts: /ping, Agent Card serving, AGENTCORE_RUNTIME_URL handling, Bedrock header propagation, and binding to port 9000. The validation inside that tool is the part I'd argue for hardest. OpenAPI path items legally contain keys that aren't HTTP methods parameters, servers, $ref, orsummary. Loop over every key as though it were an operation and you'll either crash on a string or, worse, emit a confident summary containing an endpoint called PARAMETERS. Then the doc generator writes it up, and it ships. Demos never surface this because demo specs are clean. Direct dependencies for both specialists: Shell bedrock-agentcore[a2a] strands-agents[a2a] a2a-sdk PyYAML httpx The a2a extras matter. Without them, serve_a2a and the client classes aren't there. a2a-sdk is what the orchestrator's client code imports from, so it's a direct dependency even though the server side pulls it in transitively. The documentation generator uses the same wrapper with a narrower job: Python # app/DocGenerator/main.py from strands import Agent from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model agent = Agent( model=load_model(), system_prompt=( "You write developer documentation from an API request and a " "specification analysis. Produce Markdown with an overview, " "authentication notes, request examples, error handling, and a " "short Python quickstart. Never invent endpoints, fields, or " "credentials." ), ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(agent)) That one's prompt-only on purpose. If your docs have to match a house format, this is where you add output validation or template tools instead of asking a model to remember a style guide. Test It Locally Shell agentcore dev # or, without the inspector UI: python main.py Then send it a JSON-RPC message at the root path: JSON curl -X POST http://localhost:9000/ \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": "req-001", "method": "message/send", "params": { "message": { "role": "user", "parts": [ { "kind": "text", "text": "Analyze the authentication on this API." } ], "messageId": "12345678-1234-1234-1234-123456789012" } } }' | jq . Pull the Agent Card too, before you deploy anything: Shell curl http://localhost:9000/.well-known/agent-card.json | jq . Thirty seconds of curl catches a missing discovery endpoint or a card whose skills describe an agent you deleted two commits ago. Card problems are miserable to debug from the client side, because what you see is a resolver failure, not a bad card. Deploy the Specialists The CLI defaults to IAM SigV4. Configure bearer-token auth when the caller can't sign requests, the AWS tutorial walks through Cognito for this, though Cognito isn't a requirement; any OIDC provider works. Deploy each specialist as its own runtime: Shell agentcore deploy The CLI packages the code, uploads the artifact to S3, creates the runtime, and hands back an ARN: Shell arn:aws:bedrock-agentcore:us-west-2:<account-id>:runtime/SpecAnalyzer-xyz123 Separate runtimes are half the point of doing this at all. Each specialist scales, releases, and rolls back on its own, without a coordinated deploy across the whole system. The URL Nobody Warns You About Here's the step that eats the most time. The invocation URL isn't the ARN, and it isn't a friendly hostname. It's the ARN, URL-encoded, embedded in a path: https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{url-encoded-arn}/invocations/ Every colon and slash in the ARN gets escaped with quote(arn, safe='') in Python, or let the SDK do it: Python from bedrock_agentcore.runtime import build_runtime_url url = build_runtime_url( "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/SpecAnalyzer-xyz123" ) The Agent Card sits under that path, at .../invocations/.well-known/agent-card.json. Keep the trailing slash on the base URL. A missing slash or an unescaped ARN both produce a 404, which reads like the runtime doesn't exist. Calling a Remote Agent This client assumes bearer auth. For an IAM runtime, sign with SigV4 or use a client that signs for you. Python # app/Orchestrator/client.py import os from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, TextPart DEFAULT_TIMEOUT = 300 def _message(text: str) -> Message: return Message( kind="message", role=Role.user, parts=[Part(TextPart(kind="text", text=text))], message_id=uuid4().hex, ) async def call_remote_agent( runtime_url: str, text: str, session_id: str | None = None, ) -> str: headers = { "Authorization": f"Bearer {os.environ['BEARER_TOKEN']}", "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": session_id or str(uuid4()), } async with httpx.AsyncClient( timeout=DEFAULT_TIMEOUT, headers=headers, ) as http: resolver = A2ACardResolver(httpx_client=http, base_url=runtime_url) card = await resolver.get_agent_card() config = ClientConfig(httpx_client=http, streaming=False) client = ClientFactory(config).create(card) async for event in client.send_message(_message(text)): if isinstance(event, Message): return event.model_dump_json(exclude_none=True) if isinstance(event, tuple) and len(event) == 2: task, _ = event return task.model_dump_json(exclude_none=True) raise RuntimeError("The remote agent returned no A2A result.") With streaming=False, you get exactly one result, which is either a Message or a (Task, UpdateEvent) tuple. Handle both. The tuple form is what trips people up on the first run. Returning raw JSON keeps the protocol visible for a walkthrough, but don't ship it. Normalize messages and artifacts into your own response type at this boundary. Otherwise, every downstream prompt has to understand the full A2A envelope, and you end up with models reasoning about kind fields instead of about your problem. Session IDs deserve a real decision, not a uuid4() you stopped thinking about. Fresh ID for an isolated one-shot task. Same ID when you're continuing a multi-turn conversation with the same remote agent. And keep a separate correlation ID of your own, because one user request may span several runtime sessions and you'll want to stitch them back together in the traces. The Orchestrator Each remote agent becomes a tool. Endpoints come from config; the client resolves the Agent Card at the endpoint before sending work. Python # app/Orchestrator/main.py import os from strands import Agent, tool from strands.multiagent.a2a.executor import StrandsA2AExecutor from bedrock_agentcore.runtime import serve_a2a from model.load import load_model from client import call_remote_agent REMOTE_AGENTS = { "spec_analyzer": os.environ["SPEC_ANALYZER_URL"], "doc_generator": os.environ["DOC_GENERATOR_URL"], } @tool async def delegate(agent_name: str, instruction: str) -> str: """Send work to a named A2A specialist.""" runtime_url = REMOTE_AGENTS.get(agent_name) if runtime_url is None: allowed = ", ".join(sorted(REMOTE_AGENTS)) return f"Unknown agent '{agent_name}'. Available agents: {allowed}." return await call_remote_agent(runtime_url, instruction) orchestrator = Agent( model=load_model(), system_prompt=( "You coordinate two API specialists. Use spec_analyzer for OpenAPI " "parsing and risk analysis. Use doc_generator for documentation and " "SDK examples. When both are needed, analyze the specification " "first, then include that analysis in the documentation request. " "Never assign work outside an agent's advertised capability." ), tools=[delegate], ) if __name__ == "__main__": serve_a2a(StrandsA2AExecutor(orchestrator)) Two small things in there matter more than they look. The tool is async, which avoids wrapping the remote call in asyncio.run() that blows up when the agent is already running inside an event loop, and the traceback doesn't point anywhere near the cause. And returning a friendly string for an unknown agent, rather than raising, gives the model something it can recover from instead of a dead turn. The URLs belong in deployment config, a service catalog, or an agent registry. Not in source control. Let the Model Route, or Hard-Code the Path? The code above lets the orchestrator model decide who to call and in what order. That's genuinely useful when requests vary. It's also the default choice for the wrong reasons. If every documentation request must be analyzed before generation, always, no exceptions encode that sequence in Python and use the model only inside each step. Deterministic orchestration is easier to test, retry, and audit, and it doesn't drift when someone edits a prompt. Save model-driven routing for routes that are actually ambiguous. It shouldn't be there to make a fixed workflow feel more autonomous. Running It With all three runtimes deployed, send the user request to the orchestrator. Something like "Review this customer API spec and create a Python quickstart" should cause it to call the analyzer, feed that analysis to the doc generator, and return one answer. Should. That's expected behavior, not a guarantee the protocol hands you. Whether the right handoffs happen comes down to your orchestrator prompt, your tool descriptions, your eval cases, and your workflow code. A2A standardizes the conversation. It has no opinion about whether your orchestration logic is correct. Before You Call It Production Least privilege per runtime. Each runtime gets permissions for its own tools and data, nothing more. The failure mode to watch for is the orchestrator's role slowly accumulating every permission any specialist ever needed. Distinguish auth failures from agent failures. A2A application errors come back as JSON-RPC error objects, frequently with HTTP 200. Auth and authorization failures come back as native HTTP errors AccessDeniedException is a 403. AgentCore also maps runtime exceptions onto JSON-RPC codes: -32051 for not found (404), -32053 for throttling (429), -32054 for conflict (409), -32055 for a runtime client error (424). Inspect the JSON-RPC shape. A 200 is not a success. Retry deliberately. 429 and retryable-conflict 409 are worth retrying with backoff. A 424 usually means your container failed and the answer is in CloudWatch, not in a second attempt. Never blind-retry non-idempotent work, and pass a task or request identifier so a specialist can recognize a duplicate. Trace every handoff. AgentCore Observability gives you sessions, traces, and spans. Propagate your own business correlation data on top, or you'll have three sets of traces and no way to prove they belong to the same user request. Set lifecycle values on purpose. Defaults are a 15-minute idle timeout and an 8-hour max lifetime, both configurable (60 to 28,800 seconds). Long-running tasks may justify raising them; a bigger number should be a decision, not a leftover. Also watch that your A2A sessions actually go idle — a server that holds a connection open after the client is gone will happily keep a microVM warm on your bill. Cache Agent Cards carefully. Resolving on every call is simple and costs you a round trip per hop. Cache with a short TTL and invalidate when an agent's version or capabilities change. A stale card advertising a skill that no longer exists is a genuinely confusing bug. Keep protocol and SDK versions aligned. Pin dependencies, validate cards, and test a mixed-framework setup rather than assuming it works. Use private connectivity if you need it. Runtime can attach to a VPC for internal resources, and interface VPC endpoints via PrivateLink keep AgentCore API calls off the public internet. A Failure Test Matrix Before you call it done, break the handoffs rather than admiring the happy path: The specialist's Agent Card is unreachable or malformed.The bearer token expires between card resolution and invocation.The analyzer returns a valid task with no usable artifact.The doc generator times out after the analyzer succeeded.The orchestrator picks an agent that doesn't exist, or one it isn't allowed to call.A retry re-runs a task that already had an external side effect. Each of these teaches you something a successful demo can't. They also make the division of labor concrete: Runtime hosts and protects the endpoint, A2A defines the conversation, and workflow correctness is still entirely yours. Closing The value here isn't the agent count. It's that each capability gets a clear boundary, a narrow permission set, and its own release cycle. Runtime provides the managed execution boundary; A2A provides a common way to discover agents and pass work between them. So, start with one agent. Split it when responsibilities, ownership, or security boundaries justify a network hop, and not before. Where the sequence is fixed, keep the orchestration deterministic. Where routing is genuinely variable, let the orchestrator reason over well-described capabilities, and instrument every handoff you add. References AWS: Deploy A2A servers in AgentCore RuntimeAWS: A2A protocol contractAWS: AgentCore Runtime authentication and authorizationAWS: LifecycleConfiguration API referenceAWS: Configure AgentCore Runtime for VPC accessAgentCore CLIAgent2Agent protocol specification
Conventional unit tests remain essential around an LLM application, but they do not answer the most important production question about whether a change preserved useful behavior. A parser can still return valid JSON while the answer becomes less grounded. A tool call can satisfy its schema while selecting the wrong tool. A retrieval pipeline can return documents successfully while omitting the evidence required for the final answer. This gap exists because an LLM application is not a deterministic function whose correctness can always be represented as actual == expected. Empirical work on code generation has shown substantial output variation across repeated calls, including at temperature zero, so single-run assertions can mistake sampling noise for either success or failure. Regression testing therefore has to evaluate behavior statistically and semantically, not only execution paths. Unit Tests Prove Contracts, Not Behavior The first layer should still look familiar. Deterministic properties deserve deterministic tests like JSON must parse, required fields must exist, tool arguments must conform to a schema, forbidden operations must remain blocked, and latency or token budgets can be checked numerically. Anthropic’s evaluation guidance explicitly separates success criteria from the mechanisms used to measure them and includes exact-match, code-based, human, and model-based grading as different options rather than treating one metric as universal. A contract test for a support-answering endpoint can remain intentionally narrow: Python def validate_contract(result): assert result["status"] in {"answered", "abstained"} assert isinstance(result["answer"], str) assert len(result["answer"]) <= 2_000 assert all(c["source_id"] for c in result["citations"]) Passing this function proves that the response is consumable by downstream software. It does not prove that the answer is correct, that every material claim is supported, or that an abstention occurred when evidence was missing. Those are behavioral properties and need evaluators matched to the application’s actual failure modes. Research on LLM-based judging also shows why a single generic "quality" score is insufficient as judges can exhibit position, verbosity, and self-enhancement biases, even though strong judges can correlate well with human preferences under controlled evaluation. Build the Regression Corpus From Failures A useful regression suite begins with representative tasks rather than prompts invented solely for testing. Production traces, manually verified examples, support escalations, retrieval misses, tool selection failures, malformed outputs, and previously fixed incidents should become durable evaluation cases. LangSmith’s current evaluation documentation describes this feedback pattern where production traces can be added to datasets so that a failure observed in live traffic becomes a repeatable offline test, while experiments compare application versions on the same dataset. Each case should preserve enough context to reproduce the behavior that matters. A question alone is often insufficient for RAG or agent systems because the retrieved documents, tool state, permissions, conversation history, and expected policy can affect the result. The expected value also should not always be a canonical sentence. A stronger case stores assertions about required facts, forbidden claims, acceptable citations, expected tool choices, and whether abstention is mandatory. Python case = { "input": "Can an expired license be renewed online?", "required_facts": {"renewal_window": "30 days"}, "forbidden_claims": {"automatic_extension"}, "must_cite": {"policy-17"}, "expected_action": "answer" } The corpus also needs slices. A global average can improve while an important category regresses. Long queries, multilingual requests, ambiguous requests, high-risk tool actions, low-retrieval-confidence cases, and specific product domains should retain explicit tags so candidate performance can be compared within those populations. Google’s production ML guidance similarly recommends monitoring real-world metrics and inspecting data slices because aggregate quality can conceal skew or deterioration in subgroups. Score Properties Instead of Strings Exact matching works for classifications, IDs, tool names, and other canonical outputs. Open-ended language requires property-based evaluation. For RAG, retrieval and generation should be measured separately. Ragas formalized this decomposition with metrics targeting retrieval quality, answer relevance, and faithfulness rather than collapsing the entire pipeline into one score. That separation matters operationally because an unsupported answer and a retrieval miss require different fixes. A production-like evaluator can combine deterministic checks with semantic scoring: Python def evaluate(case, result): return { "contract": contract_score(result), "groundedness": groundedness_score( result["answer"], result["evidence"] ), "task_quality": rubric_score( case["input"], result["answer"], case ), "tool_correctness": tool_score(case, result), } The evaluator should expose dimensions rather than immediately averaging them. A groundedness score of zero must not be hidden by excellent style. A wrong financial action must not pass because the explanation is fluent. Hard safety and contract requirements should act as vetoes, and softer qualities such as completeness or tone can be aggregated only after those constraints pass. LLM as a judge is useful when the desired property cannot be expressed with code, but the judge itself becomes part of the test infrastructure. G-Eval demonstrated that rubric-driven LLM evaluation can align better with human judgments than older reference-based metrics for some generation tasks, while later judge research documented systematic biases and sensitivity to evaluation setup. The practical consequence is straightforward with the judge model, rubric, prompt, sampling settings, and parser should be versioned like any other dependency. Borderline cases should be sampled repeatedly or routed to human review rather than converted into false precision by a single score. Gate Changes Against a Baseline, Then Close the Production Loop Regression testing is most informative when a candidate is compared with a pinned baseline on identical cases. Absolute thresholds such as "quality must exceed 0.85" can hide a meaningful drop from 0.94 to 0.86. A gate should therefore combine non-negotiable case failures with relative movement in quality, cost, and latency. Python def release_allowed(baseline, candidate): if candidate["critical_failures"] > 0: return False quality_delta = candidate["quality"] - baseline["quality"] latency_ratio = candidate["p95_ms"] / baseline["p95_ms"] return quality_delta >= -0.01 and latency_ratio <= 1.10 The tolerances in this example are application policy, not universal constants. More important is the comparison model where the baseline and candidate run against the same versioned corpus, results remain inspectable by case and slice, and repeated samples are used when output variance is material. Recent research on judge reproducibility reinforces this requirement, finding that temperature control can reduce variability without eliminating it and arguing that grader disagreement should be treated as an evaluation signal rather than ignored. CI can then apply evaluation at different depths. Pull requests can execute a compact, high-signal corpus covering critical behaviors, while scheduled or pre-release runs can execute broader datasets and repeated samples. Official LangSmith guidance supports offline evaluation for comparing versions before deployment and provides CI/CD integration patterns around evaluation runs. The important design principle is not the specific platform, as evaluation results must be release evidence rather than a dashboard consulted only after a failure. Offline evaluation still cannot fully represent production traffic. Query distributions change, documents change, tools return new states, and upstream services evolve. Production monitoring therefore completes the regression loop. Google’s production ML guidance emphasizes live quality measurement, drift detection, and ongoing monitoring rather than treating validation as a one-time pre-deployment event. Sampling real traces, reviewing low-confidence or high-impact outcomes, and promoting confirmed failures back into the regression corpus turns operational incidents into permanent test coverage. AI regression testing becomes reliable when deterministic software tests and behavioral evaluations are treated as complementary rather than interchangeable. Unit tests should enforce contracts and invariants, curated datasets should preserve real failure modes, evaluators should measure task-specific properties, baseline comparisons should detect meaningful degradation, and production traces should continuously refresh the suite. An LLM application is ready for release not when every generated sentence matches a fixture, but when measured behavior remains within explicit quality, safety, cost, and latency tolerances under representative conditions. That shift turns evaluation from an ad hoc prompt-checking exercise into a disciplined software engineering control for probabilistic systems.
Sr. QA Lead Manager,
3Pillar
Lead Engineer,
Technical University of Crete
Blogger, QA, Mentor, Trainer,
Freelancer