Testing, Tools, and Frameworks

Most JMeter test plans I’ve inherited share a common shape. Two hundred threads, one ramp-up, a flat plateau, and a results table that says “p95 was 480ms.” Somebody declares the system performant, the test plan goes into a Confluence page, and nobody runs it again until the next major release. The problem is that the test doesn’t model anything. The traffic shape is wrong, the user behavior is wrong, the data volumes are wrong, and the security controls aren’t being exercised. The system passes the test and then fails in production at peak load because production traffic doesn’t look like the test. This article is about the difference. How to design realistic load profiles, run distributed load that actually scales, and use the test data to find specific security-related bottlenecks (auth latency, encryption overhead, audit logging contention) that the simple test plan never surfaces. Why the Canned Test Plan Misses The default JMeter test plan does three things wrong: It uses constant load. Real traffic has spikes, valleys, and bursts. Constant load only tells you about steady-state behavior. The interesting failures happen during transitions.It uses uniform users. Every thread does the same thing. Real users have a mix of behaviors. Some browse, some search, some submit, some upload. A constant ratio is wrong; the ratio shifts by time of day.It tests on cached data. The first test run hits cold caches. The second run hits warm caches. By the time you’re looking at the results, everything is warm, and you’re measuring cache performance, not application performance. For a clinical system, these issues compound. The peak isn’t a steady 200 users; it’s a Monday-morning admission rush where every clinic is opening simultaneously, plus the lab batch results coming in from overnight, plus the medication reconciliation jobs running. The simple test doesn’t capture any of this. Designing a Realistic Profile The first move: instrument production. Look at actual traffic for 30 days. Pull out the patterns. What I look for: Request distribution by endpoint. What percent of traffic is GET /patients/{id}? What percent is POST /orders? The distribution is rarely uniform.Daily and weekly patterns. Healthcare systems have strong weekday patterns. Morning admission peaks, midday discharge peaks, evening lulls. Weekend patterns are different from weekday patterns.Burst characteristics. What’s the largest 5-minute spike in the last 30 days? How does p99 behavior change during the spike?User session shape. A user logs in, performs some actions, logs out. The actions aren’t random. There’s a typical sequence. From that, the JMeter test plan starts looking different. Instead of one thread group doing one thing, you have multiple thread groups with different behaviors: Thread group 1: Clinician users, doing chart review (heavy reads, light writes).Thread group 2: Admission staff, doing patient registration (medium writes, audit-heavy).Thread group 3: Lab system, posting results (high write volume, batch-shaped).Thread group 4: Reporting, doing aggregation queries (low frequency, expensive). Each thread group has its own ramp-up, plateau, and think times. The combined load looks more like production. XML <ThreadGroup> <stringProp name="ThreadGroup.num_threads">120</stringProp> <stringProp name="ThreadGroup.ramp_time">300</stringProp> <stringProp name="ThreadGroup.scheduler">true</stringProp> <stringProp name="ThreadGroup.duration">3600</stringProp> <!-- Clinician chart review pattern --> <ThroughputController> <stringProp name="throughput">85.0</stringProp> <!-- 85% of these threads do chart review --> </ThroughputController> </ThreadGroup> The Throughput Controller is what lets you mix behaviors within a thread group with realistic ratios. The assumption that burned us wasn’t volume — it was arrival shape. We had an inbound integration with an external system, and capacity planning assumed requests would arrive as they were submitted on the other side: a steady trickle across the day, the same shape as our own front-end traffic. The external system didn’t work that way. It accumulated submissions on its side and dumped the entire batch at once. The capacity model said 5,000 requests per hour; reality was that same 5,000 arriving in a burst measured in minutes. Nobody suspected the integration, because the daily totals matched the model exactly — steady-state capacity was fine, burst capacity wasn’t, and the constant-load test plan we’d been running had never exercised a burst at all. The fix was twofold: decouple arrival rate from processing rate by putting a message bus in front of the integration endpoint, so the batch lands in the queue at whatever rate it arrives and the system drains it at a sustainable pace; and rebuild the test plan to match — the integration thread group now fires its full daily volume in a short window, because that’s what production actually does. On the next run, the burst cleared without touching the rest of the system. The lesson: instrument the arrival pattern before designing the test, or production will run the experiment for you. Distributed Load A single JMeter instance maxes out somewhere around 1,000–2,000 threads depending on the test complexity. Above that, you need distributed load: multiple JMeter slaves driven by a master. The setup: Shell # On each slave node: jmeter-server -Djava.rmi.server.hostname=10.0.1.50 # On the master: jmeter -n -t test-plan.jmx -R 10.0.1.50,10.0.1.51,10.0.1.52 -l results.jtl The slaves run the test; the master aggregates results. The thing that breaks first in distributed mode is the aggregation. If your slaves are generating tens of thousands of samples per second and shipping them to the master, the master becomes the bottleneck, and your test results lag reality. Three settings that matter: mode=StrippedBatch in jmeter.properties. This compresses sample data before shipping to the master. Without it, the network between slaves and master saturates first.summariser.interval=30. Batches the summary updates rather than streaming every sample.Disable graphical listeners during the test. They consume memory and add overhead. Run with -n (non-GUI) and analyze the results file afterward. The other thing distributed mode breaks: tests that share state. If your test plan uses CSV Data Set Config to read user accounts, each slave needs its own copy of the CSV, and you need to make sure two slaves aren’t both using the same user concurrently. Either split the CSV across slaves or use a different uniqueness mechanism (UUID-based usernames, for example). Modeling Auth and Session Correctly Most simple test plans get authentication wrong. They either log in once at the start of the test (which lets the server cache too aggressively) or they log in on every request (which makes the test mostly about login throughput). Real users log in at the start of a session, perform many actions over 30+ minutes, and then log out. The test should match. Plain Text HTTP Request: POST /auth/login → captures access_token via Regex Extractor HTTP Header Manager → Authorization: Bearer ${access_token} Loop Controller (50 iterations of mixed actions) HTTP Request: GET /api/patients/{id} HTTP Request: GET /api/encounters HTTP Request: POST /api/notes Think Time: random 5-15 seconds HTTP Request: POST /auth/logout This shape exercises the actual session lifecycle. It also surfaces token-refresh issues if your access tokens expire mid-session. Most production auth bugs only show up in tests that have realistic session durations. For OAuth flows specifically, the JMeter HTTP Request can do the password grant or client credentials grant directly. For authorization code flows, you usually need a BeanShell sampler or JSR223 sampler to handle the redirect chain. Identifying Security Bottlenecks Here’s where realistic load testing earns its keep. Several of the bottlenecks I’ve found in production load tests are security-related, and they only show up under load: Authentication latency. Every request validates the access token. If the token validation calls back to an identity provider over the network, that’s a hop on every request. Under load, the IdP becomes the bottleneck. The fix is local token validation (JWT signature check rather than introspection) for the hot path.Authorization decision latency. ABAC policy evaluation can be expensive. If the authorization service is calling out to a database for attributes on every request, that’s database load proportional to traffic. Caching the policy decision at a session level (with appropriate TTL) is a meaningful win.Audit log contention. Every PHI access generates an audit event. If the audit log is a synchronous database write, the audit log table becomes a hot spot. The fix is asynchronous audit (write to a queue, batch insert from the queue) or partitioning the audit table by time.Encryption overhead. TLS handshake cost matters at scale. If your load balancer terminates TLS and connection reuse is poor, you’re paying a handshake on every request. Connection keepalive on the client side and sufficient backend connection pool size on the server side are the relevant levers.Rate limiter contention. Rate limiters that use a centralized store (Redis is common) can themselves become a bottleneck. Every request reads and writes the limiter state. Under high load, the Redis instance becomes the gating factor. A pattern I’ve seen play out: the audit log turns out to be the bottleneck. During a ramp test, throughput plateaued at roughly 60% of projected peak, and latency started climbing on read endpoints that should be cheap. Nobody suspected audit, because audit is “just an insert.” But every PHI access wrote a synchronous insert to a single audit table, and the table didn’t have the right indexes for how it was being used. The compliance queries that ran against it — who accessed which patient, over what date range — had no covering index, so each one scanned an enormous and constantly growing table, holding locks and I/O while thousands of inserts per minute queued up behind it. Every request in the system paid for that contention, because every request carried a synchronous audit write in its path. The database wasn’t saturated overall; one table was. The fix was twofold: index the audit table for its real query patterns and partition it by time so scans and index maintenance stayed bounded; and move the audit write itself out of the request path — inserts go to a queue and land in batches, so a slow audit table can no longer slow down a chart view. On the next ramp, the same load cleared projected peak with margin. The audit requirement didn’t change — every PHI access was still fully logged — but the logging stopped competing with the requests it was logging. Test Data That Doesn’t Lie A test that runs against a database with 500 patient records doesn’t tell you anything about a system that will run against 5 million. Database performance is non-linear: index efficiency, query plan choice, and table scan behavior all change at scale. The test environment should have: Production-scale data volumes. Not real PHI; synthetic data at production scale.Production-shape data distribution. If 10% of patients have more than 100 encounters and 1% have more than 1,000, the test data needs that shape.Realistic relationships. Patients have encounters, encounters have orders, orders have results. The relational density affects query performance. The synthetic data generation is its own engineering problem. Tools like Synthea (an open-source synthetic patient generator) produce realistic enough data for most testing. For specific use cases, you may need to generate your own. Don’t use de-identified production data. De-identification is harder than it sounds, and a flawed de-identification means PHI is now in a non-PHI environment. Synthetic is the safer answer. Reading the Results The metrics that matter, in priority order: Error rate. If the test is producing 5xx responses, that’s the first thing to fix. Performance numbers from a test where 10% of requests are erroring don’t represent anything.p95 and p99 latency, by endpoint. Average latency is misleading. The user experience is shaped by the tail. p99 latency that’s 10x p50 latency tells you there’s contention somewhere; it just doesn’t tell you where.Throughput per endpoint. If you ramp load from 100 to 1,000 RPS and throughput plateaus at 600 RPS, that’s the system’s actual capacity. Latency above that point goes vertical.Resource utilization on each tier. CPU, memory, network, disk on the application servers, database, cache. The bottleneck is whichever resource saturates first. If application CPU is at 95% but database CPU is at 30%, you scale the application tier. If it’s the other way around, the application tier scaling won’t help. A common misread of these numbers: application-server CPU pinned at 90% while database CPU sits comfortably around 40%, so the team does the obvious thing — scales the application tier horizontally and re-runs the test. Same throughput ceiling, except now more application servers are pinned. The application CPU isn’t doing application work; it’s churning on connection-pool waits, timeouts, and retries, because the database is the actual constraint. The misleading part is that database CPU looks healthy. The real problem is contention — sessions stack up waiting on locks and I/O for a handful of hot rows, and waiting doesn’t burn CPU. Utilization tells you which tier is busy; it can’t tell you why it’s busy, and busy-waiting on a downstream constraint looks identical to real work on a CPU graph. The wait-event statistics on the database tell the true story in about five minutes, once someone finally looks. The fix isn’t more application servers — it’s resolving the row contention, after which the original server count clears the target load. The lesson generalizes: utilization identifies the bottleneck only when the bottleneck is throughput-bound. Contention hides behind moderate utilization, and you find it in wait statistics, not CPU graphs. Don’t call a bottleneck until you’ve seen what the busy tier is actually busy doing. What to Do With the Results The output of a load test should produce one of three actions: No action. The system handles projected peak load with margin. Document the capacity, archive the test plan, set a calendar reminder to re-run before the next major release.Tuning. A specific bottleneck is identified, the fix is in configuration or code, and the next test run validates the improvement.Architectural change. The bottleneck is structural, and the fix is significant. The load test produces the case for the work; without the test data, the architectural change is hard to prioritize. The mistake I see most: load tests that run, produce numbers, and then sit in a Confluence page with no action. The test isn’t valuable for its own sake. It’s valuable for the decisions it enables. If no decisions came out of the last test, the test was probably not asking a useful question. What I’d Do Differently If I were standing up a load testing program from scratch: Run the simple test first to validate the harness, then throw it away. The first useful test is the one with realistic profiles. Run the test against a production-scale environment, even if that’s expensive. Tests against under-scaled environments produce misleading results. Include the security stack in the test path. Don’t bypass authentication, authorization, or audit logging to “isolate the application.” The security stack is part of the application’s performance. Set explicit pass/fail criteria before the test, not after. “Acceptable” is what you said before you saw the results, not what you negotiated after. Run the test on a regular cadence, not just before releases. Capacity changes as the system evolves. The test that passed six months ago doesn’t necessarily reflect today’s system. The version of JMeter testing I’d put in front of any production system is the one where the results actually inform decisions. Most JMeter setups don’t get there. The ones that do are the ones where the test was designed to model production, not to produce a number for a release checklist.

Software testing has always been binary at its core. A test passes, or it fails. The build is green, or it is red. The release goes out, or it gets blocked. This binary model has served software teams well for decades because the systems being tested were deterministic — the same input reliably produced the same output, every time. AI systems are not deterministic. And yet most teams are still testing them with a binary framework that was never designed to handle probabilistic behavior. This is one of the most significant gaps in enterprise AI quality engineering right now — and it is quietly producing false confidence across organizations deploying AI at scale. The Problem With Binary Testing for AI Systems When you test a traditional function, a pass means the function behaved correctly for that input. When you test an AI model with a binary pass/fail framework, a pass means the model produced an acceptable output for that particular input at that particular moment. It tells you almost nothing about how the model will behave across the full distribution of real-world inputs it will encounter in production. Consider a practical example. You build a test suite of 500 cases for an AI-powered fraud detection system. The model passes 487 of them — a 97.4% pass rate. Your pipeline shows green. Confidence is high. What your test suite does not tell you: How confident was the model on each of those 487 passes? Was it 99% confident or 51% confident?How does the model perform on inputs that fall outside your 500 test cases?Are the 13 failures clustered in a specific transaction type that happens to represent 40% of your production volume?Is the model's confidence degrading over time as data distribution shifts? Binary pass/fail answers none of these questions. Confidence scores do. What Confidence Scores Actually Tell You A confidence score is the model's self-reported probability that its output is correct. A model that classifies a transaction as fraudulent with 98% confidence is telling you something very different from a model that makes the same classification with 54% confidence — even if both outputs look identical from a binary perspective. For enterprise teams, confidence scores unlock four dimensions of AI quality that binary testing simply cannot surface. 1. Uncertainty Mapping When you aggregate confidence scores across your test suite, you can map where your model is uncertain. Consistently low confidence scores on a particular input pattern signal a coverage gap — the model is operating outside its reliable domain. This is actionable information. Binary results just tell you the model passed. 2. Threshold Calibration Confidence scores allow you to define actionable thresholds. A model that is less than 70% confident should route to human review. A model that is less than 40% confident should reject the action entirely. You cannot build these guardrails without confidence data — you are just guessing at where the risk lies. 3. Distribution Shift Detection As your production data changes over time, confidence scores will drift before accuracy degrades. This makes confidence monitoring an early warning system for distribution shift. By the time your binary tests start failing, the model has already been making low-confidence decisions in production for weeks or months. 4. Risk Stratification Not all AI decisions carry the same consequence. A low-confidence recommendation in a product suggestion engine is recoverable. A low-confidence decision in a payment routing or medical triage system is not. Confidence scores let you stratify AI decisions by risk and apply proportional oversight — something binary results make impossible. Implementing Confidence-Aware Testing in Practice Shifting to confidence-aware testing does not require replacing your existing test infrastructure. It requires extending it. Add Confidence Capture to Your Test Assertions Instead of just asserting that the model output matches an expected value, capture the confidence score alongside every assertion. Your test output should include the confidence distribution across your test suite, not just the pass/fail count. Python def test_fraud_classification(model, test_input, expected_label): result = model.predict(test_input) confidence = result.confidence_score assert result.label == expected_label, f"Label mismatch: {result.label}" assert confidence >= MINIMUM_CONFIDENCE_THRESHOLD, \ f"Low confidence prediction: {confidence:.2%} on input type {test_input.category}" # Log for distribution analysis log_test_result( input_category=test_input.category, expected=expected_label, predicted=result.label, confidence=confidence, passed=(result.label == expected_label) ) Define Confidence Thresholds By Risk Tier Work with your domain experts to define what confidence level is acceptable for each category of AI decision. These thresholds should be part of your test specifications, not afterthoughts. YAML confidence_thresholds: high_risk_decisions: minimum: 0.85 human_review_below: 0.90 standard_decisions: minimum: 0.70 human_review_below: 0.75 low_risk_decisions: minimum: 0.60 Test the Distribution, Not Just Individual Cases A model can pass every test case individually while still having a problematic confidence distribution. Add aggregate assertions to your test suite that validate the shape of confidence across your full test set. Python def test_confidence_distribution(model, test_suite): results = [model.predict(case) for case in test_suite] confidence_scores = [r.confidence_score for r in results] mean_confidence = sum(confidence_scores) / len(confidence_scores) low_confidence_count = sum(1 for c in confidence_scores if c < 0.70) low_confidence_rate = low_confidence_count / len(confidence_scores) assert mean_confidence >= 0.80, \ f"Mean confidence too low: {mean_confidence:.2%}" assert low_confidence_rate <= 0.05, \ f"Too many low-confidence predictions: {low_confidence_rate:.1%} of test cases" Monitor Confidence in Production, Not Just in Testing Confidence-aware testing must extend beyond your test suite into production monitoring. Set up dashboards that track confidence score distributions on live traffic, alert on confidence degradation, and trigger retraining or review workflows when confidence drops below defined thresholds. What This Looks Like in Practice A retail enterprise I worked with deployed an AI model for inventory replenishment decisions. Their initial test suite had a 96% pass rate. The team was comfortable with the release. After introducing confidence-aware testing, the picture looked different. The model was consistently making replenishment decisions with confidence scores between 55-65% for seasonal products — a category that represented a significant portion of their inventory value. Binary testing had masked this entirely because the model's outputs happened to align with expected values in the test data, even though the model was operating with low certainty. After setting a confidence threshold of 80% for high-value inventory decisions and routing lower-confidence predictions to a human reviewer, the team caught a systematic miscalibration in the seasonal product segment before it reached production. The binary tests had given them a false green. The confidence scores gave them the truth. The Governance Case for Confidence Scores Beyond the technical benefits, there is a governance argument for confidence-aware testing that is becoming increasingly difficult to ignore. Regulatory frameworks and enterprise AI governance standards are beginning to require explainability and documented uncertainty bounds for AI systems making consequential decisions. A binary pass/fail test result does not satisfy an auditor asking how certain your AI system was when it made a particular decision. A confidence score does. If your organization is operating AI systems in regulated domains — finance, healthcare, retail payment processing — building confidence measurement into your testing and monitoring infrastructure is not just good engineering practice. It is the foundation of a defensible governance posture. Conclusion Binary pass/fail testing was built for deterministic systems. AI systems are probabilistic by nature, and testing them as if they are deterministic produces false confidence at exactly the moments when you need accurate confidence most. Confidence scores do not replace binary testing. They complete it. They answer the questions that pass/fail cannot: how certain was the model, where is it uncertain, and is that uncertainty clustered in ways that create production risk? The teams that get AI quality engineering right in the next few years will not be the ones with the greenest dashboards. They will be the ones who understood that green does not mean confident — and built their testing infrastructure accordingly.

My first attempt to deploy a Spring Boot microservice on AWS Fargate didn’t fail loudly. It failed quietly — in a loop. ECS kept launching tasks, the Application Load Balancer kept marking them unhealthy, and the service never stabilized. The logs looked fine, the container looked fine, but the ALB replaced every task within seconds. The root cause was painfully simple: Spring Boot needed 45 seconds to start, and my ALB health‑check timeout was 5 seconds. The tasks never had a chance. That night changed how I build and deploy microservices. It forced me to rethink startup behavior, JVM sizing, networking, task definitions, and the entire CI/CD pipeline. This article is the guide I wish I had before that incident — a practitioner’s walkthrough of deploying a production‑ready Spring Boot service on AWS Fargate, with real artifacts and the details that matter when things go wrong. The Architecture That Finally Worked Once the health‑check issue was fixed, the architecture settled into a predictable, cloud‑native flow: Developers push code to GitHubGitHub Actions builds the JARDocker image is built and pushed to Amazon ECRECS service runs AWS Fargate tasksTraffic enters through an Application Load BalancerTasks run in private subnetsConfiguration comes from Parameter Store and Secrets ManagerLogs and metrics flow to CloudWatch It’s the standard modern microservice pipeline — but the difference between “standard” and “production‑ready” is in the details. The Spring Boot Service The microservice itself was simple — a REST API with a few endpoints. The real complexity wasn’t the controller logic; it was everything around it: startup time, health checks, configuration management, and container behavior under load. A Dockerfile Built for Production My first Dockerfile looked like the one many tutorials start with: a single‑stage build running as root with no JVM tuning. It worked locally but failed under real load. Fargate tasks with default JVM heap sizing inside a 2GB container are a classic OOM story. Here’s the hardened version that finally stabilized deployments: Dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN useradd -u 1001 springuser WORKDIR /app # Layer extraction for faster builds COPY target/*.jar app.jar # JVM tuning for Fargate ENV JAVA_OPTS="\ -XX:MaxRAMPercentage=75 \ -XX:+UseContainerSupport \ -XX:+ExitOnOutOfMemoryError \ " USER springuser ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"] This eliminated the OOMKilled events I saw on 2GB tasks and made startup time predictable. Pushing to Amazon ECR With Real Commands The first time I wrote down my ECR commands, they were placeholders. In production, they need to be exact: C aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 docker push \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 Immutable semantic version tags make rollbacks predictable and prevent “latest‑tag roulette.” The ECS Task Definition That Actually Runs in Production A real Fargate deployment lives or dies by its task definition. Here’s the JSON I use today — including secrets pulled from Parameter Store and Secrets Manager: JSON { "family": "employee-service", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "512", "memory": "1024", "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/employeeServiceRole", "containerDefinitions": [ { "name": "employee-service", "image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:us-east-1:<ACCOUNT_ID>:parameter/db/password" }, { "name": "API_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:thirdparty/api" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/employee-service", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } The ALB Health Check That Stopped the Outage My outage happened because the ALB was impatient. Here’s the configuration that finally stabilized deployments: settingvalue Path /actuator/health Interval 20 seconds Timeout 10 seconds Healthy threshold 3 Unhealthy threshold 3 Spring Boot startup time + ALB patience = stable deployments. Why Fargate Tasks Belong in Private Subnets Early on, I deployed tasks in public subnets because it felt simpler. It wasn’t. Public IPs meant the containers were directly reachable from the internet — port scans, bot traffic, and noisy logs. Moving tasks to private subnets solved several problems at once: Reduced Attack Surface No public IPs. No direct inbound traffic. Only the ALB can reach the tasks. A Single Secure Entry Point The ALB handles TLS termination, redirects HTTP→HTTPS, performs health checks, and integrates with WAF. Clients never bypass it. Cleaner Security Groups ALB SG: inbound 443 from the internetTask SG: inbound only from ALB SG Nothing else touches the containers. Compliance Alignment PCI, SOC 2, HIPAA — all prefer minimizing public exposure. Controlled Outbound Access Tasks use a NAT Gateway for outbound calls (updates, third‑party APIs) without exposing themselves. Better Scalability ALB target groups automatically track tasks across AZs as ECS scales. The architecture becomes simple and predictable: Internet → ALB (public subnets) → Fargate tasks (private subnets) It’s quieter, safer, and easier to operate. The GitHub Actions Workflow That Deploys Automatically Here’s the pipeline that builds, tests, pushes, and deploys the service: YAML name: Deploy to Fargate on: push: branches: ["main"] jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v4 with: java-version: "21" - name: Build JAR run: mvn -B clean package - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 - name: Build and Push Image run: | docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 ${{ env.ECR_REGISTRY }/employee-service:1.0.3 docker push ${{ env.ECR_REGISTRY }/employee-service:1.0.3 - name: Deploy ECS Service uses: aws-actions/amazon-ecs-deploy-task-definition@v2 with: task-definition: ecs-task.json service: employee-service cluster: prod-cluster Auto Scaling With Real Target Tracking JSON Target tracking is the simplest and most reliable scaling strategy for Fargate: JSON { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 60 } I use 50% as the target because it balances cost and responsiveness. What I Learned Every failure taught me something: ALB timeouts taught me to respect startup timeOOMKilled tasks taught me to tune the JVMPublic subnets taught me to isolate workloadsManual deployments taught me to automate everything AWS Fargate really does deliver on its promise — no servers to manage, automatic scaling, and clean integration with ECS — but only after you learn the hard parts. If you’re deploying Spring Boot on Fargate, I hope you learn those lessons from this article instead of from your own outage.

In Part 1, we solved one direction of the multi-cloud connectivity problem: a workload running in Google Cloud interacting with an AWS cloud resource. A GKE pod read a Google-issued OIDC token from the metadata server, handed it to AWS STS via AssumeRoleWithWebIdentity, and received short-lived AWS credentials, with no static access keys stored anywhere. MultiCloudJ wrapped the token dance behind a portable client so the application code never touched a provider SDK directly. This article covers the return trip: a workload running in AWS calling into Google Cloud — specifically, an Amazon EKS pod reading and writing a Google Cloud Storage (GCS) bucket — again with zero long-lived credentials. The zero-trust principle is identical. The mechanism is a bit different. And that asymmetry is the single most important thing to understand before you build it. Authentication Flow from AWS to GCP 1. Build a SigV4-signed GetCallerIdentity request (signed with STS credentials): It's assumed that the EKS pod already holds temporary AWS credentials. 2. Call sts.googleapis.com for token exchange: The pod sends that signed request to Google Cloud as the input to an OAuth 2.0 token exchange. It is asking Google, "Here is proof of who I am on AWS - please give me a Google token to access cloud resources." 3. Replay the GetCallerIdentity signed request: Google does not trust the request blindly. It runs the signed request against AWS STS on the caller's behalf. 4. Response with ARN: AWS checks the signature and replies with the caller's ARN (the AWS role identity) as part of the GetCallerIdentity response. Now Google knows exactly which AWS identity is asking - proven by the signature, with no shared secret. 5. Validate the ARN with the pool: Google checks that ARN against the Workload Identity Pool rules - which AWS account and which role are allowed in, and how the ARN maps to a Google identity. 6. Access token: Once the ARN passes, Google returns a short-lived access token to the EKS pod. 7. Access the resource with the access token: The pod uses that token to read and write Cloud Storage. When the token expires (usually within an hour), the flow repeats. Nothing long-lived is ever stored. Summary: AWS proves the pod's identity by answering Google's replayed request, and Google issues a short-lived token based on that proof. No access keys, no service-account key files - just a signed request and a temporary token crossing the trust boundary. Please note that this authentication flow can be used for any cloud service and is not specifically for cloud storage. Direct Pool Access vs. Service Account Impersonation Once Google has verified the caller's AWS identity through the signed request, it still has to map that AWS identity to something that actually holds permissions on the bucket. There are two ways to do this mapping, and you should pick one before you grant any IAM role. Option 1: Direct Pool Access You grant the Cloud Storage role straight to the federated identity. In IAM, the member looks like this: principalSet://iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/* The permission sits on this pool principal, not on the AWS role. The AWS role never holds any GCP permission. Its only job is to prove identity: it answers Google's replayed GetCallerIdentity request so Google knows which AWS identity is asking. Google then checks that identity against the pool rules and, if it is allowed in, treats the caller as this pool principal. The bucket role, such as roles/storage.objectAdmin, is bound to that principal, so that is where the actual access comes from. No service account sits in the middle. In your code, the value you pass is the pool provider resource name (the audience), and Google issues a token that represents the pool identity directly. Option 2: Service Account Impersonation You create a GCP service account, grant that service account the bucket role, and then let the federated identity impersonate it. The federated identity needs roles/iam.serviceAccountTokenCreator on that service account, and the exchange gets a second hop: first a pool token, then an impersonated service-account token. In your code, the value you pass is the service account email. Which to Choose For a straight AWS EKS to GCS case like this one, direct pool access is the better default: Fewer moving parts. No service account to create, no token-creator grant to manage, and no second token hop.Tighter blast radius. The bucket permission is tied to identities coming through this specific pool, not to a service account that other workloads might also be able to impersonate. You can narrow it further to a single AWS role with an attribute condition on the principal.Less to audit. One IAM binding on the bucket tells the whole story. Reach for impersonation only when you actually need what a service account gives you: You must reuse an existing service account that already carries permissions across many GCP resources.A downstream Google API or tool only understands service-account identities and cannot evaluate a principalSet:// member.Your organization standardizes on service accounts as the single unit of access, to stay consistent with other human and machine grants. In short, direct pool access is simpler and safer, so use it unless a concrete requirement forces impersonation. Set Up Workload Identity Pool on GCP Before any code runs, you configure the trust relationship on Google Cloud once. Three things: a pool, an AWS provider inside it, and an IAM grant on the bucket. Create the Workload Identity Pool: The pool is the identity container that your AWS workloads will be represented as.gcloud iam workload-identity-pools create aws-pool --location="global" --display-name="AWS workloads"Create the AWS provider inside the pool: The provider is the entry gate. It tells Google to trust GetCallerIdentity results from a specific AWS account, how to map the caller's ARN into a Google attribute, and which callers are allowed in.Two important parts here: The attribute mapping turns the caller's raw ARN into a stable attribute.aws_role value with the session name stripped, so grants survive session rotation.The attribute condition is the first gate: only callers from your AWS account are admitted, before any IAM binding is even checked. Shell gcloud iam workload-identity-pools providers create-aws aws-provider \ --location="global" \ --workload-identity-pool="aws-pool" \ --account-id="123456789012" \ --attribute-mapping="google.subject=assertion.arn,attribute.aws_role=assertion.arn.contains('assumed-role') ? assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn,attribute.account=assertion.account" \ --attribute-condition="assertion.account == '123456789012'" Grant the bucket role to the pool principal: This is the direct pool access model. The permission binds to the AWS role (via the mapped attribute), not to a service account. Shell gcloud storage buckets add-iam-policy-binding gs://my-archive-bucket \ --role="roles/storage.objectAdmin" \ --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/aws-pool/attribute.aws_role/arn:aws:sts::123456789012:assumed-role/my-eks-role" After this, the EKS pod's role can federate into the pool and read/write the bucket, and the application code in the next section never touches any of this setup again. Implementation With MultiCloudJ MultiCloudJ exposes the same BucketClient abstraction you saw in Part 1; you build it for the "gcp" provider and attach a CredentialsOverrider that carries the federated identity. The library handles the SigV4 signing, the STS token exchange, and (on the impersonation path) the generateAccessToken call internally; your code just does blob operations (full example). Java private static final String REGION = "us-west-2"; // The audience is the full Workload Identity Pool provider resource name. // We grant the bucket role directly to this pool principal (direct pool // access), so no service account sits in the middle. private static final String AUDIENCE = "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider"; // The supplier runs on every GCP token refresh. Each time it signs a fresh // GetCallerIdentity request with the pod's AWS role (IRSA, picked up from the // ambient AWS credential chain) and returns the subject token GCP expects. Supplier<String> webIdentityTokenSupplier = GcsFromAws::buildSubjectToken; CredentialsOverrider overrider = new CredentialsOverrider.Builder(CredentialsType.ASSUME_ROLE_WEB_IDENTITY) .withRole(AUDIENCE) .withWebIdentityTokenSupplier(webIdentityTokenSupplier) .build(); // Portable client: same API as the AWS side in Part 1, // only the provider string changes. BucketClient bucketClient = BucketClient.builder("gcp") .withBucket("my-archive-bucket") .withCredentialsOverrider(overrider) .build(); ListBlobsPageResponse page = bucketClient.listPage(ListBlobsPageRequest.builder().withMaxResults(10).build()); page.getBlobs().forEach(b -> System.out.println(b.getName())); // Signs a GetCallerIdentity request with the pod's AWS role, then shapes the // signed request into the URL-encoded JSON envelope that Google STS expects // as an AWS4 subject token. private static String buildSubjectToken() { // Google requires the audience to travel inside the signed headers, so it is // bound to the signature and the request cannot be replayed against any other // target. SignOptions options = SignOptions.builder() .withCustomHeader("x-goog-cloud-target-resource", AUDIENCE) .build(); StsUtilities stsUtil = StsUtilities.builder("aws").withRegion(REGION).build(); // Passing null means "just sign a GetCallerIdentity request, there is no // service payload to hash." The library fills in Action=GetCallerIdentity. SignedAuthRequest signed = stsUtil.newCloudNativeAuthSignedRequest(null, options); JsonObject envelope = .. // construct json object from signed request uri return URLEncoder.encode(envelope.toString(), StandardCharsets.UTF_8); } Conclusion Part 1 showed GCP calling AWS, and Part 2 completes the picture with AWS calling GCP. Both use the same idea: federation, no static keys, and only short-lived credentials. They differ only in how identity is proven. GCP to AWS presents a Google OAuth identity token, while AWS to GCP sends a signed request that GCP verifies with AWS. This is exactly where MultiCloudJ earns its place. All of these provider-specific differences, such as the bearer token here, the signed request and replay there, the STS token exchange, the service-account impersonation, and the token refresh, are abstracted away inside the library. You build one portable client, attach a credentials overrider, and call the API. Your application code never learns which cloud it is talking to or which way the call is going, so it stays clean, portable, and free of long-lived secrets in both directions.

Six months ago, building a RAG pipeline meant a full week of plumbing: an embedding job here, a vector store there, a retriever glued on with duct tape, and an orchestration layer that broke every time you touched it. I've built enough of these the hard way — hand-rolled vector search, custom chunking scripts, the works — to know exactly how much pain that "week" usually hides. Last week, I rebuilt the same thing on Azure AI Foundry. It took an afternoon. Not because the underlying problem got easier — grounding an LLM in your own data is still genuinely hard — but because Microsoft finally killed most of the integration tax that used to eat the first sprint of every RAG project. Here's what actually happened, warts included. The Old Way Was a Trap If you've built RAG before, you know the pattern: you don't fail at RAG, you fail at the seams between the pieces. Your chunking strategy doesn't match your embedding model's context window. Your retriever returns great results in a notebook and garbage in production because nobody wired up hybrid search. Your "agent" is really just a for-loop that stuffs retrieved text into a prompt and hopes. Foundry's whole pitch is that it owns those seams instead of leaving them to you. I was skeptical. I'm less skeptical now. What I Actually Did Step one: spin up a Foundry project. Not a hub-based one — those are legacy at this point, and if a tutorial has you creating one, skip it. The newer Foundry project type is the one to use. Step two: deploy two models. A chat model and an embedding model. Click, click, done. Both show up with their own endpoints. This part genuinely takes five minutes, and it's the first sign you're not building infrastructure anymore — you're configuring it. Step three: point Foundry at my documents. Blob storage in, Azure AI Search out. Foundry handles the chunking and embedding generation itself. I turned on hybrid search (keyword plus vector) because pure vector search on enterprise docs tends to miss exact terms people actually search for — product names, error codes, that sort of thing. If your content has a lot of that, don't skip this. Step four — and this is the part that's different from every tutorial I read two years ago. I didn't write a retrieval pipeline. I registered the search index as a tool on the agent and let the agent decide when to call it. Here's the whole thing: Python from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient.from_connection_string( credential=DefaultAzureCredential(), conn_str=os.environ["AIPROJECT_CONNECTION_STRING"], ) agent = project.agents.create_agent( model="gpt-4o-mini", name="docs-assistant", instructions=( "Answer only using retrieved context. " "Cite the source document for every claim. " "If the answer isn't in the retrieved content, say so." ), tools=[{ "type": "azure_ai_search", "index_connection_id": search_connection_id, "index_name": "example-index", }], ) thread = project.agents.create_thread() project.agents.create_message(thread.id, role="user", content="What's our refund policy for enterprise plans?") run = project.agents.create_and_process_run(thread.id, agent.id) No manual embedding calls at query time. No hand-written "retrieve top-k, stuff into prompt" logic. The agent framework does that internally, and it does it well enough that I stopped fighting it after the first try. Step five: For anything beyond simple lookups, I turned on agentic retrieval in Azure AI Search. Classic RAG fires one query per user turn, which quietly falls apart the moment someone asks a compound question — "compare our Q3 and Q4 policy and tell me what changed for renewals" is two questions wearing a trench coat. Agentic retrieval breaks that into sub-queries, runs them in parallel, and merges the results before generation. If your users ask messy, multi-part questions — and they do — turn this on from day one. Retrofitting it later is more annoying than it should be. Step six: Tested in the playground, then deployed the same agent behind a REST endpoint. Nothing about the agent changed between prototype and production. That alone would've saved me a full day on past projects. Now, the Part Everyone Skips I'm not going to pretend this is magic, because it isn't, and the tutorials that pretend otherwise are setting people up to get burned in a security review. Access control is on you. Foundry doesn't look at your documents and infer that HR files shouldn't be visible to the sales team. You configure document-level security filters in Azure AI Search yourself, and if you skip this, you've built a very articulate way to leak sensitive data. API keys are a prototype crutch, not a production plan. Move to Microsoft Entra ID before anything customer-facing goes live. This migration is a real afternoon of work, not a checkbox — budget for it. Retrieved documents are untrusted input. Prompt injection through a poisoned PDF is a real attack surface in every RAG system, Foundry included. Your system instructions need to assume the retrieved content might be trying to manipulate the model, because eventually it will. The costs stack. Embedding generation, index storage, and the extra tokens from stuffing retrieved passages into every call — none of this is free, and it compounds faster than people expect once you're past a demo and into real traffic. Model it before you commit to a chunking strategy at scale, not after. Was It Actually Worth It? Yes — but not for the reason most "look how easy this is" posts claim. The value isn't that RAG got simple. Grounding a model in the right data, with the right access controls, still takes real thought. The value is that Foundry took the boring week — the SDK wrangling, the manual retrieval loops, the glue code nobody wants to own — and turned it into an afternoon of configuration. That frees up the time you actually need for the parts that matter: is your data any good, is it chunked sensibly, and can you trust what comes back? If you've been putting off a RAG project because the infrastructure felt like too much, this is the moment to try again. Just don't skip the access control step to save time. That's the part that actually bites.

Throughout my career, I’ve held many roles in the QA conversation: As a developer, waiting on test teams to validate features before a release could shipAs a tech lead, watching sprint capacity dwindle while we converted Jira stories into test cases by handAs an architect, auditing a test repository with 4,000 cases where no one knew which ones mattered So when “AI test generation” started appearing as part of every testing product, I was both interested…and skeptical. After spending time with several of these tools, I’ve decided that the phrase “AI test generation” covers two fundamentally different architectures. The first is a large language model sitting behind a prompt.The second is an actual testing agent that examines your requirements, attachments, and existing test library before writing any tests. The industry has started calling that second approach agentic test creation, and the gap between that and AI test generation is much wider than the names suggest. In this article, I’ll give examples of both, showing the differences along the way, so that you can understand which one a vendor is trying to sell you. The Problem With Generic AI Test Case Generation The first wave of AI test case generation tools followed a simple pattern. You paste in a user story, the tool wraps it in a prompt, sends it to a general-purpose LLM, and gives you back the response as test cases. It’s basically ChatGPT with a QA skin. If you’ve ever pasted a Jira ticket into ChatGPT and asked for test cases, you’ve already used this architecture. The vendor version adds a nice UI and an export button. As with much LLM output, the results here might look impressive at first glance. For example, give it this story: As a returning customer, I want to apply a promo code at checkout so that my discount is reflected in the order total. And you might get a dozen plausible test cases in just seconds, testing valid code, invalid code, expired code, empty field, case sensitivity, etc. I ran exactly this exercise against a mature e-commerce regression suite. The results? Seven of the 12 generated cases already existed in the checkout suite, some nearly word-for-word in intent.Two referenced an “Apply Discount” button that simply doesn’t exist.None of the test cases were linked back to a requirement, so traceability remained manual. The model did exactly what it was asked to do. The problem was that it didn’t have context. As you can imagine, over time this approach produces some pretty bad effects. You’ll start seeing problems like duplicated coverage, rising regression time, near-copies of the same test that drift apart, and more. And you’ll end up with a repository you just don’t trust. What Is Agentic Test Creation? Agentic test creation, on the other hand, is a workflow where an AI agent, given a requirement, plans and executes a multi-step process that: Gathers context from the requirement and its attachmentsAnalyzes the existing test libraryFigures out what’s already coveredGenerates test cases that fill the gaps, each one linked back to the requirement. The word “agentic” is important here. A single LLM call is a stateless function: prompt goes in, text comes out. An agent, on the other hand, is a loop. The model reasons about a goal, calls tools to gather information, observes the results, and revises its plan before producing output. The ReAct paper formalized this “reasoning-plus-action” loop, and Anthropic’s Building Effective Agents is the clearest practitioner-level treatment I’ve read. Here’s an overview of the difference in the two approaches: Generic ai test generationagentic test creation Input The text you paste in as a prompt Requirement, acceptance criteria, attachments and images, existing test cases, SDLC history Awareness of existing coverage None Checks the test cases linked to the requirement before generating Duplication Frequent; every run starts from zero Existing cases are reused instead of regenerated Traceability Manual, after the fact Each generated case links to its requirement Typical failure mode Redundant, or references UI that doesn’t exist Gaps in context How an Agentic Test Creation Workflow Runs Let’s go back to the promo code story; this time we’ll run it through an agentic pipeline: The agent parses the story, its acceptance criteria, and an attached checkout mockup.It queries the existing test library and finds 40 checkout-related test cases.It maps the story’s scenarios against those 40 and identifies that seven are already covered. It reuses those cases rather than regenerating them.It creates six new cases targeting real gaps: promo code combined with a gift card, currency rounding on percentage discounts, an expired code entered against a saved payment method, etc.Each new case has a link back to its requirement.A QA engineer reviews the batch, edits two cases, rejects one, and commits the rest. As you can see, this agentic pattern is a serious improvement over vanilla AI and is becoming best practice. Implementing Agentic Test Creation There are basically two ways to implement agentic test creation: buy a commercial product or roll your own. Commercially, you can see an example of agentic test creation with Tricentis’ qTest. Here, an agent runs inside the test management platform itself, where it follows the above loop: it analyzes a requirement, considers the attachments, reuses test cases linked to that requirement, and stamps each case with a marker for the reviewer.To assemble your own version of the loop, you can wire agent frameworks to your test infrastructure through tools like the open-source Playwright MCP server and Selenium or Playwright projects. As is typically the case with build vs. buy, build costs money and time, but can create value using your context plumbing. Where Agentic Test Creation Falls Short What are some downsides to agentic test creation? There are a few, though they are minor. First, the context loop adds cost and latency to every generation. You’ll pay for library queries and multiple model calls per requirement instead of just one. Second, the coverage mapping is only as good as your repository. A messy library means messy output. And third, a review gate only works if reviewers stay engaged. Approving a 30-case batch on a Friday afternoon? That can fail just as it always has. What Changes Day to Day for QA Engineers? The engineers I know are skeptical of AI tooling. And they have good reasons. Agentic test creation changes their daily job (though I believe the change is less than the marketing suggests). The role doesn’t go away; it just…shifts. QA engineers move from authors to reviewers. Instead of hand-writing the fifteenth variation of a checkout test from a Jira ticket, they now evaluate a proposed batch, check the coverage map, and approve or reject the outputs. Accountability still stays with the QA team: a human sign-off gates everything. But the hours previously spent transcribing requirements into test steps are now spent on the quality assurance testing methods that models handle poorly: exploratory testing, risk analysis, and deciding what should be tested in the first place. Transcribing was the boring part anyway, right? Practical First Steps Finally, here are a few suggestions on agentic testing based on my experiences: Run a duplication audit first. An agentic tool builds on whatever it finds in your repository, including your duplicates. Clean input makes for better output.Pilot one project. Demos hide failures. Use a real project with real Jira tickets. Challenge your solution with missing acceptance criteria and stale attachments! Find your holes quickly.Keep the human review. Treat every generated case as a draft. And track your reviewer rejection rate. It’s a great signal of whether your approach is working.Measure. Keep track of three numbers: duplication rate, reviewer rejection rate, and elapsed time from requirement to test.Ask vendors: “What does your tool read before it generates?” If the answer is “your prompt,” you’re looking at a ChatGPT wrapper. If the answer includes existing tests, attachments, and requirements history, you’re looking at an agentic solution. Conclusion My readers may recall my personal mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” — J. Vester Hand-transcribing user stories into test steps has never extended the value of anyone’s intellectual property. Agentic test creation, whether you go commercial or build your own, with a review gate in place, sends that transcription task to a machine where it belongs, leaving you to be the all-important human in the loop. Have a really great day!

By John Vester CORE

Building a single AI agent is not usually the hard part. You send a prompt to a model, get a response back, and wire it into your app. Done. The hard part starts when that agent becomes one step in a larger system. A real AI workflow might need to ingest a file, extract text, chunk it, generate embeddings, call an LLM, write results to a database, sync to an external API, and notify a user. Those steps do not behave the same. Text extraction might finish in seconds. An LLM call might take minutes. A sync job might fail because some external API is having a bad day. That is where a lot of "agent" systems stop looking magical and start looking like regular distributed systems. I have seen this fail in boring ways: The same job gets processed twice.A worker writes to the database, then crashes before marking the job complete.A model call runs longer than expected and the message gets picked up again.A retried tool call creates duplicate external writes.Failed jobs sit in processing until someone manually checks the database. None of this is new. AI agents do not magically avoid old infrastructure problems. They still need queues, retries, idempotency, durable state, and monitoring. AWS SQS is a good fit for that middle layer. It is not a full workflow engine. I would not use it for every orchestration problem. But if you need a durable queue between independent agent stages, SQS is simple, reliable, and usually enough. The Coordination Problem A basic multi-stage AI workflow often looks like this: Plain Text Input source -> ingestion -> processing -> generation -> sync The first version is usually a database table with a status column. That works for a while. Then concurrency shows up. Two workers read the same pending row. A process crashes and leaves a job stuck in processing. Someone adds sleep(30) because the previous step "usually finishes by then." That last one is the kind of fix that works just long enough to become a production bug. A queue gives each stage a cleaner boundary. One stage publishes work. Another stage consumes it. If the next stage slows down, the queue absorbs the backlog instead of forcing the whole pipeline to wait. Plain Text Input Source -> ingest_queue -> Ingestion Worker -> chunk_queue -> Chunking Worker -> embedding_queue -> Embedding Worker -> summary_queue -> Summary Worker -> sync_queue -> Sync Worker Now ingestion can scale separately from summarization. If LLM generation is slow, messages pile up in summary_queue. That is not automatically a failure. That is what the queue is there for. A failed summary worker does not corrupt the whole workflow. The message can be retried. If it keeps failing, it moves to a dead letter queue. Standard Queues vs. FIFO Queues SQS gives you two main queue types: standard queues and FIFO queues. Standard Queues Standard queues give at-least-once delivery and best-effort ordering. A message can be delivered more than once. Messages may not arrive in the exact order sent. That sounds scary, but most background AI work should already handle this. Use standard queues for work like document processing, embedding generation, batch classification, independent user requests, and webhook processing. For these jobs, throughput matters more than strict ordering. FIFO Queues FIFO queues preserve ordering within a MessageGroupId and support deduplication. Use when sequence actually matters: conversation turns, per-user workflows, ordered state transitions. Python response = sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps(payload), MessageGroupId=payload["user_id"], MessageDeduplicationId=payload["task_id"] ) Be careful with the group ID. If every message uses the same MessageGroupId, you have serialized the whole queue by accident. Give each conversation, user, or workflow its own group ID so you preserve ordering per entity while allowing parallelism across different ones. My default rule: start with standard queues unless ordering is clearly required. Then make the handler idempotent. That matters more than the queue type. Ensuring Idempotency in Your Agent Flow Idempotency means the same task can run more than once without creating duplicate or incorrect side effects. This is the part I would not skip. SQS standard queues use at-least-once delivery, so duplicates are part of the contract. But this matters even more with AI workloads because model calls are expensive and outputs can be non-deterministic. Retrying the same prompt may cost money and return a different answer. Retrying the same tool call may send a duplicate email or write a second database row. The basic pseudo workflow: Plain Text receive message check if task already completed if completed, delete message and exit if not completed, process task store result delete message Simple version: Python def handle_message(message, store, sqs, queue_url): payload = json.loads(message["Body"]) task_id = payload["task_id"] if store.already_completed(task_id): sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "skipped", "task_id": task_id} result = run_agent_logic(payload) store.mark_completed(task_id, result) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "completed", "task_id": task_id} The store can be Postgres, DynamoDB, Redis, or anything durable with atomic writes. For Postgres, a unique constraint saves you: SQL CREATE TABLE agent_task_results ( task_id TEXT PRIMARY KEY, status TEXT NOT NULL, result JSONB ); INSERT INTO agent_task_results (task_id, status) VALUES ($1, 'processing') ON CONFLICT (task_id) DO NOTHING; If the insert succeeds, this worker owns the task. If it does nothing, another worker already claimed or completed it. The Failure Case I Designed Around Plain Text summary_queue -> Summary Worker -> Postgres -> sync_queue The summary worker receives a message, calls an LLM, writes the summary to Postgres, then deletes the SQS message. Now suppose the worker writes to Postgres but crashes before deleting the SQS message. From SQS's point of view, the job never finished. After the visibility timeout expires, another worker receives the same message and runs the task again. Without idempotency, that retry may call the LLM again, generate a slightly different summary, and write a second result. A safer handler checks whether model output already exists before calling the model: Python def summary_handler(payload, store): task_id = payload["task_id"] existing = store.get(task_id) if existing and existing.get("model_output"): summary = existing["model_output"] else: text = load_text(payload["input"]["text_uri"]) summary = call_llm(text) store.save_model_output(task_id, summary) store.save_final_result(task_id, {"summary": summary}) return {"next_stage": "sync", "next_input": {"summary_task_id": task_id} That avoids repeating the expensive part if the first attempt already got that far. Visibility Timeout When a worker receives a message, SQS hides it from other workers for the visibility timeout. If the worker finishes, it deletes the message. If the worker crashes, the message becomes visible again after the timeout expires. Too short: another worker receives the same message while the first is still running. Duplicate execution. Too long: failed jobs take too long to retry. Plain Text visibility_timeout = 2x to 5x expected processing time Reference: Metadata validation: 30-60 secondsEmbedding generation: 1-5 minutesLLM-heavy summary: 5-15 minutesLong document analysis: 15+ minutes with heartbeat For long-running tasks, extend visibility: Python sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=receipt_handle, VisibilityTimeout=extension_seconds ) The message should describe the work, not carry the workload. Bad: JSON {"task_id": "123", "full_pdf_text": "... thousands of lines ..."} Better: JSON { "task_id": "123", "stage": "summarize", "input": {"document_uri": "s3://bucket/docs/input.pdf"}, "metadata": {"user_id": "789", "priority": "normal"} } Store large files in S3. Send references through SQS. Do not let the queue become your storage layer. Dead Letter Queues A DLQ captures messages that fail repeatedly. Without one, poison messages cycle forever. Python sqs.set_queue_attributes( QueueUrl=main_queue_url, Attributes={ "RedrivePolicy": json.dumps({ "deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5 }) } ) Use 3-5 as a starting point. A DLQ is not a trash bin - it's an alert. AI-Agent-Specific Failure Modes Duplicate LLM calls: Bigger bill, possibly different answer. Use task_id as idempotency key.Non-deterministic outputs: Store first successful output.Tool-call side effects: Make idempotent.Long-running inference: Use visibility heartbeat. What to Monitor MetricWhyApproximateAgeOfOldestMessageUser-facing delayApproximateNumberOfMessagesVisibleBacklogDLQ message countRepeated failures Two alerts: Oldest message exceeds latency targetDLQ has messages When SQS Is Not the Right Tool RequirementBetter fitSimple async tasksSQSVisual multi-step workflowStep FunctionsComplex event routingEventBridgeHuman approvalsStep Functions I have seen teams burn hours building multi-agent systems with database polling and sleep timers. It works at demo scale. It usually does not survive production traffic. SQS gives you durable message delivery primitives. But the app still needs idempotent handlers, visibility timeout tuning, and DLQ monitoring. Default architecture: One queue between major stagesStandard queues unless ordering requiredEvery handler idempotentLarge payloads outside the queueVisibility timeouts based on real processing timeDead letter queues for failures The difference between an AI demo and a reliable AI system is rarely the prompt. It is the infrastructure around the prompt. Build that layer intentionally.

An AI agent ingests a customer ticket, queries three internal APIs, looks up a record in a database, drafts a response, and hands it off to a human reviewer. Every individual call returns a 200. The unit tests pass. The integration tests pass. The response gets approved. A week later, the support team notices the agent has been quietly recommending the wrong refund tier for one product category — for nine days. Nothing crashed. Nothing logged an error. The system did exactly what its tests said it should do. Standard test automation cannot catch this class of failure. Functional tests verify that an endpoint responds. API contract tests verify the schema. Unit tests verify that a function returns the expected output for fixed inputs. None of them verify that the agent's reasoning was sound, that the tools it chose were the right tools, or that the response it produced was actually correct for the user's problem. The behavior space is too wide, and the failure surface is non-deterministic. An LLM-driven agent can produce different outputs for the same input on consecutive runs, choose a different tool path depending on subtle prompt drift, or hallucinate a parameter that happens to type-check. Quality engineering for agentic systems is not the same problem as quality engineering for deterministic services. It needs a different toolkit. The Three Failure Modes Standard Tests Do Not See Three classes of failure dominate production incidents in agent systems, and none of them show up in conventional pipelines. Tool-Selection Drift The agent has a catalog of tools: database queries, API calls, and retrieval indexes. A prompt change one sprint up the chain causes the agent to start preferring Tool A over Tool B for an edge-case input. Both calls succeed. The new tool returns slightly stale data. The user never notices the source changed — only that the answer is wrong. Standard tests assert that the tool wrapper works, not that the agent picks the right tool for the right scenario. Semantic Regression Without API Regression The agent's output is grammatically correct, contextually plausible, and factually wrong. A summarization agent drops a critical clause. A code-generation agent produces a function that compiles and passes its tests but uses a deprecated API. A retrieval-augmented agent surfaces an answer from a stale document because the freshness filter changed weight. The response passes every structural assertion and fails the user's actual need. Compound Errors Across Multi-Turn Flows Each turn looks reasonable in isolation. The end-to-end outcome is wrong because errors accumulated across steps. This is the AI equivalent of distributed-system drift: each component is healthy, and the contract between them is broken. Detecting Failure With Evaluation Harnesses, Not Assertions The right primitive for agent quality is not the unit test. It is the evaluation set: a curated corpus of inputs paired with reference outputs — or reference judgments — that is run continuously against the agent and scored on multiple dimensions. An evaluation harness treats the agent as a black box and asks the only question that matters: for these inputs, did the agent do the right thing? Three-Signal Evaluation The following Python sketch runs a regression evaluation against an agent endpoint each time the underlying prompt, model, or tool catalog changes. It captures three signals: structural validity, semantic similarity to a reference, and an LLM-as-judge score for cases where exact-match comparison is too brittle. Python # evaluation/agent_regression.py from dataclasses import dataclass from agent_client import call_agent from judge_client import score_with_judge from sentence_transformers import util, SentenceTransformer embedder = SentenceTransformer("all-MiniLM-L6-v2") @dataclass class EvalCase: case_id: str input_prompt: str expected_tool_path: list reference_answer: str pass_threshold: float = 0.80 def run_eval(cases: list[EvalCase]) -> list[dict]: results = [] for c in cases: response = call_agent(c.input_prompt) # 1. Tool-path check — deterministic tool_path_match = response.tool_calls == c.expected_tool_path # 2. Semantic similarity to reference — cheap and fast emb_ref = embedder.encode(c.reference_answer, convert_to_tensor=True) emb_out = embedder.encode(response.text, convert_to_tensor=True) similarity = float(util.cos_sim(emb_ref, emb_out)) # 3. LLM-as-judge — slower, used for nuance and subtle factual errors judge_score = score_with_judge( question=c.input_prompt, reference=c.reference_answer, candidate=response.text, ) results.append({ "case_id": c.case_id, "tool_path_match": tool_path_match, "similarity": similarity, "judge_score": judge_score, "passed": ( tool_path_match and similarity >= c.pass_threshold and judge_score >= 4 # on a 1–5 scale ), }) return results Three signals matter because no single one is reliable. Embedding similarity catches gross drift but not subtle factual errors. LLM judges catch subtle errors but are themselves non-deterministic and need calibration. Tool-path checks are deterministic but only verify mechanism, not outcome. The combination is what produces a credible quality signal. SignalCatchesBlind SpotTool-path checkMechanism-level deviationsDoes not verify outcome qualityEmbedding similarityGross semantic driftSubtle factual errorsLLM-as-judgeNuanced reasoning failuresNon-deterministic; needs calibration Production Replay Diffing Pair the harness with a snapshot of production traffic. Sample 1% to 5% of real requests, replay them against a candidate version of the agent, and diff the responses against the production baseline. The diff itself is the regression signal — not "did the agent answer correctly," but "did the new version answer differently from the trusted version, and is that difference an improvement or a regression?" Python # evaluation/replay_diff.py def replay_against_baseline(samples, baseline_agent, candidate_agent) -> list[tuple]: flagged = [] for s in samples: baseline = baseline_agent.respond(s.input) candidate = candidate_agent.respond(s.input) if baseline.tool_calls != candidate.tool_calls: flagged.append((s.id, "tool_path_diverged")) continue sim = cosine_similarity(baseline.text, candidate.text) if sim < 0.85: flagged.append((s.id, f"semantic_diff:{sim:.2f}")) return flagged A 15% divergence rate between baseline and candidate on a 1,000-sample replay is a red flag whether the absolute scores are passing or not. Behavior stability matters as much as behavior quality. Structural Fixes: Deterministic Shells, Bounded Tools, and Decision Logs Detection alone does not solve the problem. Three architectural patterns make agent quality tractable in production. Wrap Non-Deterministic Core Logic in a Deterministic Shell The LLM call is the only non-deterministic component; everything around it can be made repeatable. Cache prompts and responses by hash, freeze the model version per release, and treat the prompt template as a versioned artifact in the same way you treat a schema migration. A bumped prompt is a release event with its own evaluation gate, not a config tweak. Python # agent/release_gates.py PROMPT_VERSION = "v2025.11.03" MODEL_VERSION = "model-prod-v3.2" def validate_release(eval_results: list, baseline_results: list) -> None: pass_rate = sum(r["passed"] for r in eval_results) / len(eval_results) baseline_rate = sum(r["passed"] for r in baseline_results) / len(baseline_results) if pass_rate < 0.90: raise ReleaseBlocked(f"Pass rate {pass_rate:.2%} below 90% gate") if pass_rate < baseline_rate - 0.02: raise ReleaseBlocked( f"Regression: candidate {pass_rate:.2%} vs baseline {baseline_rate:.2%}" ) Constrain Tool Selection An agent with twenty tools and full freedom to chain them produces a combinatorial behavior space that no evaluation set can fully cover. Group tools by capability domain, expose a small surface to the agent, and use a router — which can itself be deterministic, rules-based, or a lightweight classifier — to decide which capability group is in scope for a given input. This shrinks the search space the agent operates in and makes its behavior testable. Log Every Decision, Not Just the Final Response Capture the prompt, the tool calls, the intermediate outputs, the retrieved context, and the final answer, all with the same correlation ID. When a user reports a wrong answer two weeks later, you need to reconstruct exactly what the agent saw and did. Without a decision log, every incident becomes archaeology. Python # agent/trace.py def execute_with_trace(agent, request) -> str: trace = { "request_id": request.id, "prompt_version": PROMPT_VERSION, "model_version": MODEL_VERSION, "input": request.input, "steps": [], } for step in agent.iter_steps(request): trace["steps"].append({ "tool": step.tool_name, "input": step.tool_input, "output_hash": hash_output(step.output), # hash only — not raw output }) trace["final_output"] = agent.final_output write_trace(trace) return agent.final_output The trace store becomes the substrate for everything downstream: regression evaluation, incident response, drift detection, and the production samples that feed the next replay run. Trade-Offs to Plan For Agent evaluation is not free, and teams that ignore the costs end up absorbing them silently into platform budgets until a budget review forces a conversation nobody wants. ConcernRealityMitigationInference costAn LLM-as-judge call costs roughly what a production agent call costs; a 1,000-case eval per PR doubles inference spend for that PRRun the full eval nightly; use a 100-case smoke set on every code changeLatencyPer-step tracing adds milliseconds per call; tool I/O logging raises storage costsBudget explicitly; don't absorb silentlyInference footprintReplay testing requires a live baseline alongside the candidate, doubling the footprint during a release windowTime-box release windows; tear down the baseline after the gate passesNovel failuresHarnesses catch known failure modes, not new onesBuild a feedback channel from production users into the eval set The third trade-off is the most underestimated. A meaningful share of agent regressions are discovered first by users reporting that something "feels off" — a tone shift, a drop in helpfulness, or a new category of mistake. Every confirmed regression becomes a permanent test case, and the eval set grows over time into a living artifact of what the agent has been wrong about. A Pre-Ship Checklist Before shipping any significant change to a prompt, model version, or tool catalog, validate each item: Baseline eval set contains at least 100 curated cases covering known edge scenarios Three-signal harness runs in CI: tool-path match, semantic similarity, and LLM-as-judge Prompt templates are versioned in source control with semantic versioning Model version is pinned per release; a version bump triggers the full eval gate Release gate enforces ≥90% pass rate AND ≤2% regression versus baseline Production replay pipeline samples 1%–5% of live traffic against every candidate Decision log with correlation ID is wired into every agent execution path Feedback channel routes user-reported regressions into the eval set as new test cases Evaluation cost appears as an explicit line item in the platform budget The Cost of Skipping This Teams that ship agent systems without these practices pay in trust. A handful of confidently wrong answers reach the right user at the right time, and the perception shifts from "useful assistant" to "unreliable tool." Recovering that perception is far more expensive than the engineering work to prevent it. The traditional QA toolkit — unit tests, integration tests, contract tests, and end-to-end suites — is necessary but not sufficient for agentic systems. The new layer is evaluation infrastructure: curated eval sets that grow with production learnings, replay pipelines that compare candidate versions against trusted baselines, decision logs that make every agent action reconstructible, and release gates that block regressions before they ship. Don't wait for the first user-reported regression. Version your prompts like schemas, log every step the agent takes, and treat behavior stability as a first-class quality signal. The alternative is explaining to a customer why an answer that used to be right is now confidently wrong — a conversation no quality engineer wants to have twice.

Every test suite starts as an asset. The first hundred tests are almost universally useful. They cover the system's core behaviors, and they catch regressions. They give the team justified confidence that the software is behaving as intended. The team is proud of them. The CI pipeline is fast. The signal-to-noise ratio is high. Then the suite grows. It grows because the system grows and the coverage targets demand it. It grows because new engineers arrive and write tests in new styles, because incidents trigger the addition of regression tests that reproduce specific bugs, because automated test generation tools produce tests at scale. By the time the suite reaches several thousand tests, the team's relationship with it has changed in ways they may not have fully realized. The CI pipeline takes forty minutes. Flaky tests fail intermittently and are ignored. Large portions of the suite were written for features that have since been refactored or removed. Yet nobody is sure which portions, so nothing is deleted. The suite still passes, technically, but the team has quietly learned not to trust it. At this point, the suite has become a liability. It is consuming maintenance time without generating proportionate evidence about system quality. The team knows this, dimly, but the path to addressing it is unclear. How do you prune a test suite you do not fully understand? How do you govern a test asset that has accumulated without design? This article answers such questions. I borrow from investment portfolio management because test suites and investment portfolios share the same fundamental problem: assets that once generated returns can become liabilities through neglect, accumulation, and failure to rebalance. The principles that make a portfolio manageable over time are the same principles that make a test suite manageable over time. Disciplines like classification, return measurement, periodic rebalancing, and governance can translate directly into test suite practices. The Core Argument Volume is not quality. A test suite of 10,000 tests that are poorly designed, poorly maintained, and weakly assertive generates less confidence than a suite of 1,000 tests that are precisely targeted, well-maintained, and behaviourally specific. Test suites become liabilities through three mechanisms: accumulation without governance, flakiness without remediation, and maintenance cost that exceeds evidence value. The solution is portfolio thinking: classify tests by their return on investment, measure their actual contribution to quality evidence, prune ruthlessly, and govern continuously. Architectural decisions in test design determine the long-term cost structure of the suite. The wrong architecture, applied at scale, makes the liability irreversible without a full rewrite. How Test Suites Become Liabilities The transition from asset to liability is gradual and never announced. It happens through recognizable mechanisms that compound over time. Understanding them is the prerequisite for preventing them. Mechanism 1: Accumulation Without Governance Test suites grow continuously and shrink rarely. Code is deleted or refactored constantly — engineers remove features, rename functions, restructure modules. Tests, however, are treated as permanent. The instinct is conservation: who knows what a test is protecting, even if it seems redundant? Deleting a test looks risky. Deleting code does not. This asymmetry produces a suite that expands indefinitely relative to the codebase it is meant to cover. The result is a suite with substantial dead weight. There are tests for deleted features that now test nothing meaningful. Other tests duplicate each other's coverage region. Many tests' original purpose has been lost. Another important category is the tests whose assertions were already weak when written and have become weaker as the system evolved. Dead weight tests do not merely occupy space. They consume CI time, they produce noise in failure reports, and they create cognitive load when engineers try to understand what the suite is covering. This is a form of testing technical debt: Accumulated through individually reasonable decisions, expensive in aggregate, and resistant to removal because nobody can confidently identify them without significant analysis. Mechanism 2: Flakiness Without Remediation A flaky test is one that passes and fails non-deterministically for the same code. It is not a test that catches intermittent bugs; it's an unreliable test. It depends on timing, on network calls, on shared state, on randomness, or on other environmental conditions that the test does not control. Flaky tests are one of the most corrosive forces in a test suite, and they act through a specific psychological mechanism. When a test fails intermittently, engineers learn that its failure is not meaningful. They re-run the suite. The test passes. They merge. Over time, the team develops a learned response to failure that generalizes from the specific flaky test to the suite as a whole: failures are probably noise. Re-run and proceed. This generalization is catastrophic. A team that has learned to distrust its test suite has no test suite, in any meaningful sense. The suite still runs, still produces a pass/fail result, still appears in the CI pipeline. But the team does not act on its failures with the urgency that meaningful test failures require. The suite has become wallpaper. # A flaky test: the canonical pattern def test_user_session_expires(): user = create_test_user() session = login(user) # Wait for session to expire (30 second timeout in test config) import time time.sleep(31) result = validate_session(session.token) assert result.valid == False # This test fails when: # - The CI runner is under load and sleep(31) completes in 28s of wall clock # - The session timeout is configured differently in different environments # - The test database is shared and another test refreshed this session # - Clock drift between the test process and the session service # Each failure produces a re-run. Each re-run that passes reinforces the team's belief that the failure was noise. # Meanwhile: the session expiry logic may be genuinely broken. The flaky test is providing cover for a real defect. The flaky test's danger is not that it fails. It is that it teaches the team not to care when tests fail. Mechanism 3: Maintenance Cost That Exceeds Evidence Value Every test has a maintenance cost. When the code it tests changes, the test must change with it. When the test infrastructure changes, the test may break. When the team's understanding of the system evolves, the test's assertions may become outdated, wrong, or irrelevant. The maintenance cost of a test is a function of its coupling to implementation details, its reliance on shared state and infrastructure, and the specificity of its assertions. A test with high maintenance cost and high evidence value is a reasonable investment — it is expensive to maintain, but it catches important issues. A test with high maintenance cost and low evidence value is a pure liability — it is expensive to maintain and contributes little to the team's understanding of the system's behavior. The problem is that most test suites contain a large proportion of the latter category, and nobody has ever measured it. # High maintenance cost, low evidence value # This test breaks every time the internal implementation changes, # but it verifies almost nothing about behaviour. def test_user_service_calls_repository(): mock_repo = MagicMock() service = UserService(repository=mock_repo) service.get_user(user_id=42) # Assert that the internal implementation called the repository mock_repo.find_by_id.assert_called_once_with(42) # This test verifies HOW the code works (it calls the repository), not WHAT it does (it returns the correct user for a given ID). # When UserService is refactored to use a cache before the repository, this test breaks. But nothing about the user-facing behavior changed. The test is coupled to implementation, not to behavior. # Low maintenance cost, high evidence value # This test survives implementation changes as long as behavior is preserved. def test_get_user_returns_correct_user(): user_id = create_test_user(name='Alice', email='[email protected]') result = user_service.get_user(user_id) assert result.name == 'Alice' assert result.email == '[email protected]' # This test survives: adding a cache, changing the repository implementation, refactoring the service class, changing the internal call structure. It fails only when the behavior changes: when get_user returns wrong data. # That is exactly when a test should fail. The difference between testing implementation and testing behavior is the single most important architectural decision in test design, and it determines the long-term maintenance cost of every test in the suite. Your Test Suite Is An Investment Portfolio Investment portfolio management rests on a small number of principles that have proven durable across a century of application. Not all assets generate equal returns. Returns degrade over time without active management. Diversification reduces risk. Concentration in low-return assets is a silent tax on the portfolio. Rebalancing is not a one-time activity but an ongoing discipline. Each of these principles translates directly into test suite management, and this is not a metaphor. It’s a lens that makes obvious the not-so-obvious. Investment PrincipleTest Suite Equivalent Not all assets generate equal returns Not all tests generate equal evidence. A unit test that verifies a critical financial calculation returns more quality evidence per unit of maintenance cost than an assertion-free smoke test of the same code. Returns degrade over time without active management Tests written for one version of the system may generate less evidence as the system evolves. A test written when a feature was new may be testing a behavioral region that is now low-risk. New high-risk regions have appeared without corresponding tests. Diversification reduces risk A test suite that covers only the happy path is concentrated in a low-risk region. Coverage across success paths, error conditions, boundary cases, and failure modes provides a diversification that catches more defects that matter. Concentration in low-return assets is a silent tax A suite dominated by implementation-coupled unit tests with weak assertions imposes a maintenance cost that exceeds its evidence value. The tax is paid in CI time, in maintenance effort, and in the false confidence the suite provides. Rebalancing is an ongoing discipline Test suites must be actively pruned, reclassified, and restructured as the system and its risk profile evolve. A suite that is not actively managed drifts toward a portfolio of low-return legacy tests surrounding a shrinking core of high-value evidence generators. Know what you hold and why Every test in the suite should have a reason to exist that can be stated in terms of the behavior it verifies and the risk it mitigates. Tests whose purpose cannot be stated are candidates for removal. Viewing your test suite as an investment portfolio is very practical. It provides a vocabulary for conversations about test suite investment that many organizations avoid. A VP of Engineering who would never accept an investment portfolio managed without return measurement, diversification analysis, or rebalancing should apply the same standard to a test suite that consumes significant engineering resources. Classifying Tests by Return on Investment The first practical step in portfolio management is classification. Before you can manage a test suite for return, you need to understand what your current suite contains. The following classification system provides the vocabulary for that analysis. The 4 Test Asset Classes Borrowing from financial asset classification, test assets can be organized into four classes based on their evidence value and maintenance cost. The classification is not precise — tests exist on a spectrum — but it provides sufficient resolution to make management decisions. ClassEvidence ValueMaintenance CostManagement Action Core Holdings High Low-Medium Protect, expand. These are the suite's most valuable assets. They verify critical behavior, survive implementation changes, fail for the right reasons, and are cheap to maintain relative to the evidence they generate. Speculative Assets High High Monitor and restructure. These tests verify important behavior but are expensive to maintain — typically because they are coupled to implementation, depend on complex infrastructure, or are end-to-end tests with significant environmental dependencies. Legacy Positions Low-Medium Medium Review and prune. These tests were once valuable but have drifted from the system. They may be testing deleted features, duplicating other tests, or asserting against behavior that is now low-risk. Liabilities Low High Remove immediately. These tests generate little evidence and consume significant maintenance resources. Flaky tests, assertion-free tests, and tests that verify implementation rather than behavior at high maintenance cost all fall here. Measuring Evidence Value Evidence value is a function of three parameters: the criticality of the behavior the test verifies, the quality of the test's assertions, and the degree to which the test covers a region of the behavior space that is not covered by other tests. It cannot be measured with a single metric, but it can be assessed with a structured set of questions. # Evidence value assessment questions for any test # 1. BEHAVIOUR CRITICALITY # What behavior does this test verify? # If this behavior were wrong in production, what would the consequence be? # [Critical / Important / Standard / Trivial] # 2. ASSERTION QUALITY # What does this test assert? # Would this test fail if the behavior were wrong? # Does it assert the outcome, or the implementation? # [Strong: verifies correct output / Weak: verifies execution / None: assertion-free] # 3. COVERAGE UNIQUENESS # Is this the only test that covers this behavior? # Or do 5 other tests cover the same region? # [Unique / Partially duplicated / Fully duplicated] # 4. ORACLE RELIABILITY # Is the expected value in the assertion correct? # Was it calculated carefully, or copied from the implementation? # [Independent calculation / Copied from output / Unknown] # Evidence value score (illustrative, not prescriptive): # Critical behaviour + Strong assertion + Unique coverage + Independent oracle # -> Core Holding # Trivial behaviour + Weak assertion + Fully duplicated + Unknown oracle # -> Liability Evidence value cannot be automated entirely — the criticality assessment requires human judgment about the system's risk profile. Everything else can be partially automated. Measuring Maintenance Cost Maintenance cost is more tractable to measure than evidence value because it leaves traces in version control and CI data. The following signals are reliably available in any modern engineering environment and correlate strongly with actual maintenance burden. # Maintenance cost signals — extractable from git and CI data # 1. CHANGE FREQUENCY # How often has this test file changed in the last 6 months? # High change frequency = high coupling to implementation details # # git log --follow --format='%ad' --date=short -- tests/test_user_service.py # | sort | uniq -c -> count of changes per date # 2. FAILURE RATE IN CI # What proportion of CI runs did this test fail? # High failure rate with passing re-runs = flaky # High failure rate correlated with code changes = potentially useful # # Extractable from CI API (GitHub Actions, Jenkins, CircleCI all expose this) # 3. TIME TO EXECUTE # How long does this test take to run? # Tests that take >5s in a unit test suite are almost always # inappropriately coupled to I/O or network. # # pytest --durations=0 | sort -k2 -rn | head -20 # 4. DEPENDENCY COMPLEXITY # How many mocks, fixtures, and setup steps does this test require? # High setup complexity = high coupling = high maintenance cost # # Count fixture dependencies and mock patches in test file # 5. LAST MEANINGFUL FAILURE # When did this test last fail, and was the failure meaningful? # A test that has not had a meaningful failure in 2 years either covers low-risk behavior or covers behavior # that is also covered by other tests. All five signals are extractable without reading test code — they come from CI data and version control history. A maintenance cost dashboard can be built from these signals in a day. Architectural Decisions That Determine Long-Term ROI The decisions made when writing tests determine whether the suite will remain manageable at scale or will compound into a liability faster than it can be pruned. The following architectural decisions have the highest long-term impact on test suite ROI. Architecture Decision 1: Test at the Right Level The most consequential architectural decision is which level of the system to test at. Tests at different levels have different evidence value, different maintenance costs, and different execution speeds. These characteristics are not fixed — they are determined by the system's architecture and by what the test is trying to verify. The classical framework — the test pyramid — prescribes many unit tests, fewer integration tests, and fewer still end-to-end tests. This is a sensible heuristic for many systems, but it is applied too dogmatically in most organizations. The right distribution depends on what the system does, where its risk concentrates, and what each level of test is capable of verifying. # The test level decision and its consequences # Level: Unit test # Verifies: individual function behavior in isolation # Evidence value: high for logic-heavy code; low for glue code and I/O # Maintenance cost: low if behavior-coupled; high if implementation-coupled # Execution speed: milliseconds # What it cannot verify: integration behavior, configuration, infrastructure # Level: Integration test # Verifies: behavior of components working together # Evidence value: high for interface contracts and data flow # Maintenance cost: medium (depends on infrastructure complexity) # Execution speed: seconds # What it cannot verify: full user journey, performance, environment-specific behavior # Level: End-to-end test # Verifies: user-facing behavior through the full stack # Evidence value: high for critical user journeys; low for edge cases # Maintenance cost: high (UI changes, environment dependency, slow feedback) # Execution speed: seconds to minutes # What it cannot verify: the root cause of a failure (too much indirection) # The portfolio implication: # Unit tests should dominate in volume because they are cheapest to run and maintain. # Integration tests should cover interface contracts and data boundaries. # End-to-end tests should cover critical user journeys ONLY. # End-to-end tests that cover what unit tests already verify are Liabilities. Architecture Decision 2: Control Your Dependencies Tests that depend on external systems — databases, APIs, message queues, file systems — are more expensive to run. They are more prone to flakiness, and more difficult to maintain than tests that control their own dependencies. Dependency architecture in tests is as important as dependency architecture in production code. # Uncontrolled dependency: test behavior depends on external database state def test_user_count_is_correct(): # Assumes the test database is in a known state. # Other tests may have added or deleted users. # Test order matters. Parallelization breaks this test. count = user_repository.count() assert count == 5 # This number means nothing outside a controlled context # Controlled dependency: test owns its state def test_user_count_reflects_created_users(db_session): # db_session fixture creates a clean, isolated database for this test # and tears it down afterward. user_repository.create(db_session, email='[email protected]') user_repository.create(db_session, email='[email protected]') user_repository.create(db_session, email='[email protected]') count = user_repository.count(db_session) assert count == 3 # This number is meaningful: we created exactly 3. # The second test can run in parallel, in any order, in any environment. # It is deterministic because it controls its own data. # This is the architectural property that prevents flakiness at scale. Test isolation is an architectural property, not a testing technique. It must be designed into the test infrastructure, not retrofitted to individual tests. Architecture Decision 3: Make Failures Diagnostic A test that fails is only useful if its failure tells you something specific about what went wrong. A test whose failure message is "AssertionError" on a complex object comparison tells you that something is wrong somewhere. A test whose failure message is "Expected discount for GOLD tier to be 10%, got 8% for input price=100.00" tells you exactly where to look. Diagnostic failures are an architectural choice, made when the test is written, that determines the return on investment of every future failure. A test that fails usefully is worth more than a test that fails mysteriously. This is because the cost of investigating a mysterious failure is substantial and is paid every time the test fails. # Failure that tells you nothing def test_order_totals(): orders = [order_service.create(item, qty) for item, qty in test_data] totals = [o.total for o in orders] assert totals == [10.00, 25.50, 8.99, 150.00, 3.50] # On failure: AssertionError: [10.00, 25.50, 9.99, 150.00, 3.50] # Which order failed? What was the input? What was expected? Unknown. # Failure that tells you exactly what happened import pytest @pytest.mark.parametrize('item,qty,expected_total', [ ('widget', 1, 10.00), ('gadget', 3, 25.50), ('doohickey', 1, 8.99), ('thingamajig', 2, 150.00), ('gizmo', 1, 3.50), ]) def test_order_total(item, qty, expected_total): order = order_service.create(item, qty) assert order.total == expected_total, ( f'Order total for {qty}x {item}: ' f'expected {expected_total}, got {order.total}' ) # On failure: AssertionError: Order total for 1x doohickey: # expected 8.99, got 9.99 # Immediate diagnosis. Zero investigation required. Diagnostic failure messages are not cosmetic. They are the mechanism by which a failing test pays back its maintenance cost. Invest in them. Architecture Decision 4: Design for Speed A test suite that takes forty minutes to run in CI is a suite that will be circumvented. Engineers will merge without waiting for results. Test results will be checked retrospectively, if at all. The feedback loop that makes continuous testing valuable will be broken by the latency of the pipeline. Speed is an architectural property of the test suite, determined by decisions made when tests are written. The primary driver of suite slowness is not the number of tests — it is the presence of slow tests that could be fast with different architectural choices. # Speed anti-patterns and their architectural fixes # Anti-pattern 1: sleeping in tests # time.sleep(5) # Wait for async operation # Fix: use explicit synchronisation or event-driven waiting # result = wait_for(lambda: cache.has_key('result'), timeout=5) # Anti-pattern 2: hitting real external services # response = requests.get('https://api.payment-gateway.com/charge') # Fix: stub at the HTTP boundary, not at the function boundary # with responses.activate(): # responses.add(POST, 'https://api.payment-gateway.com/charge', # json={'status': 'success'}, status=200) # result = payment_service.charge(amount=100) # Anti-pattern 3: starting a full application server for unit tests # app = create_app() # Full Django/Flask app with all middleware # client = TestClient(app) # Fix: test the function directly; use the test client only for # integration tests that specifically need the HTTP layer # Anti-pattern 4: sequential tests that could run in parallel # Fix: pytest-xdist for parallelization # pytest -n auto tests/unit/ # runs in parallel across CPU cores # Prerequisite: tests must be isolated (Architecture Decision 2) # Speed target: # Unit test suite: < 2 minutes # Integration test suite: < 10 minutes # Full suite including E2E: < 20 minutes # If any suite exceeds these targets, it needs architectural attention, not a faster CI machine. Speed targets are architectural constraints, not optimization goals. A suite that exceeds them should be redesigned, not scaled up on more powerful hardware. Governing the Suite: The Ongoing Discipline Portfolio rebalancing is a continuous process. A test suite that is audited and pruned once will drift back toward liability-heavy within a few months if governance is not ongoing. A continuous discipline that prevents accumulation from reoccurring involves at least the following basic ingredients. Test Addition Every new test added to the suite should pass a brief but explicit review against the classification criteria. This is the test equivalent of a code review. It should take as long as reading the test carefully and asking the four classification questions: what behavior does this verify, how strong are the assertions, does it duplicate existing coverage, and is the expected value independently calculated? # Test review checklist (applied during code review, not separately) # 1. WHAT BEHAVIOUR DOES THIS TEST VERIFY? # If you cannot answer this in one sentence, the test needs revision. # 2. WHAT IS THE CONSEQUENCE OF THIS BEHAVIOUR BEING WRONG? # Critical / Important / Standard / Trivial # If Trivial: question whether the test belongs in the suite at all. # 3. DOES THIS TEST DUPLICATE EXISTING COVERAGE? # Search for existing tests that cover the same behavior. # If duplicate: consolidate rather than add. # 4. ARE THE ASSERTIONS BEHAVIOUR-COUPLED OR IMPLEMENTATION-COUPLED? # If implementation-coupled: request revision before merging. # 5. COULD THIS TEST BECOME FLAKY? # Does it use time, randomness, network, or shared state? # If yes: what is the isolation strategy? # 6. HOW LONG DOES THIS TEST TAKE TO RUN? # Measure it. If >1 second in a unit test: investigate why. These six questions add approximately two minutes to a code review. They prevent months of maintenance cost. The Quarterly Portfolio Review Once per quarter, the team should conduct a structured review of the test suite's health using the metrics available from CI data and version control. This review is not a full audit — it is a health check that identifies emerging problems. Quarterly Portfolio Review: Key Metrics Flakiness rate: proportion of test runs that contain at least one flaky failure. Target: <1%. Action threshold: >3%. Suite execution time trend: is the suite getting faster or slower? A suite that grows by >10% in execution time per quarter without a proportional increase in behavior coverage is accumulating liability. Test-to-code ratio by module: modules with very high test counts relative to code size may be over-tested in low-risk areas; modules with very low test counts may be under-tested in high-risk areas. Change-failure correlation: when tests fail in CI, how often is a real defect found? A high false-failure rate indicates that tests are flaky or poorly specified. Defect escape analysis: defects that reached production in the last quarter — were they in covered code? Were the covering tests assertive enough to have caught them? This is the ground-truth question for suite effectiveness. Quarantine queue length: the number of tests currently in the flaky quarantine. This number should decrease each quarter. If it is increasing, flakiness is being generated faster than it is being resolved. The Deletion Culture The most important governance practice is one that runs against engineering instinct: the normalization of test deletion. In most engineering cultures, adding tests is virtuous and deleting tests is suspicious. This asymmetry is the primary driver of suite accumulation. It must be explicitly reversed. Test deletion should be treated as a sign of quality maturity, not of quality regression. A team that deletes tests is a team that understands its suite well enough to identify what is not earning its place. A team that never deletes tests is a team that has lost track of what it holds. Signs of a Healthy Test Portfolio The suite execution time is stable or decreasing despite the codebase growing. Flaky test rate is below 1% and trending downward. When the suite fails in CI, the team investigates immediately rather than re-running. Every test in the suite can be described, by any team member, in one sentence: what behavior it verifies and why that behavior matters. The team deletes tests regularly, without controversy, as part of routine refactoring. The last production incident was not in code with high test count — and when analyzed, the failure was either in genuinely untested behavior or in a known, accepted risk region. Wrapping Up More tests do not produce more confidence. They produce more volume. Whether that volume translates into confidence depends on the quality of the tests. It depends on factors like their assertion strength, their coverage of meaningful behavior, their maintenance cost, and their diagnostic clarity when they fail. Volume without quality produces a test suite that is expensive to maintain and unreliable as a quality signal. It is a liability that grows faster than it can be managed. The portfolio model provides a framework for managing test suites as assets over time: classify by evidence value and maintenance cost, prune the liabilities, restructure the speculative assets, protect the core holdings, and govern continuously. The architectural decisions that determine long-term ROI must be made when tests are written, because they are difficult and expensive to retrofit. The discipline this requires is not primarily technical. It is cultural: the normalization of test deletion, the institutionalization of test review alongside code review, and the replacement of volume metrics with evidence quality metrics. Teams that develop this discipline produce smaller, faster, more reliable test suites. They generate higher-quality evidence than the large, slow, noisy suites they replaced.

By Stelios Manioudakis CORE

Most articles on Page Object Model are written by people who maintain twelve tests. This is one written by somebody who has lived inside a 2,400-test web automation suite for three years and watched it ossify, get rebuilt, and ossify again. I don’t think POM is wrong. I think the version of POM that gets taught — one class per page, methods that wrap WebElement clicks — falls apart somewhere around 300 tests. The version we run today still calls itself POM, and the page classes look like the original ones, but underneath there are three or four patterns layered on that nobody told me about when I started. This article is about those patterns. The stack: Java 17, Selenium 4.14, TestNG 7.8, Maven, running locally and on a Selenium Grid 4 in Kubernetes. About 2,400 tests, ~38 minutes wall time on 24 parallel nodes, ~6% flake rate that we are continuously fighting to keep under 8%. What the Textbook POM Gives You, and Where It Stops The textbook is fine for one page. You write a LoginPage, it has a loginAs(user, pass) method, your test calls it. You feel good. Now you have 40 pages, half of them inherit a header and footer, three have modal dialogs, one is a wizard with seven steps, and you’ve got a single BasePage class that’s 1,100 lines long and includes a method called Wait-For-Thingie-To-Be-Ready-But-Only-If-FlagX-Is-Set. The pain points I hit, in the order I hit them: Pages that have shared regions (the global header, the side nav, a footer that’s actually loaded async). If you put header methods on every page class, you get duplication. If you put them on BasePage, you get a 1,100-line god class. Pages that are really states. A “shopping cart” isn’t one page; it’s empty-cart, populated-cart, and during-checkout. The same URL, three behaviors. Wait strategies that need to be page-specific. The dashboard takes 4-7 seconds to load because it’s running 11 GraphQL queries; the settings page loads instantly. A single global Thread.sleep(5000) in BasePage is how you get a 90-minute test suite. Tests that need to set up state without going through the UI. We have a checkout test that needs the user to already have three items in their cart. Going through the UI to add three items is 14 seconds per test, times the 200 tests that need a populated cart = a lot of compute. Cross-browser differences. Chrome and Firefox behave differently around shadow DOM. A click that works in Chrome might silently no-op in Firefox 119. POM by itself doesn’t tell you where to put the workaround. The hybrid framework is the answer to those five problems. There are four patterns layered on top of textbook POM. Pattern 1: Component Classes for Shared Regions The first move is to stop pretending the header and footer belong to the page. They don’t. They are components that happen to render on the page. Java public class GlobalHeader { private final WebDriver driver; private final WebDriverWait wait; @FindBy(css = "[data-test='header-search']") private WebElement searchInput; @FindBy(css = "[data-test='header-cart-icon']") private WebElement cartIcon; @FindBy(css = "[data-test='header-cart-count']") private WebElement cartCount; public GlobalHeader(WebDriver driver) { this.driver = driver; this.wait = new WebDriverWait(driver, Duration.ofSeconds(10)); PageFactory.initElements(driver, this); } public CartPage openCart() { cartIcon.click(); return new CartPage(driver); } public int getCartItemCount() { wait.until(ExpectedConditions.visibilityOf(cartCount)); return Integer.parseInt(cartCount.getText().trim()); } public SearchResultsPage search(String query) { searchInput.clear(); searchInput.sendKeys(query); searchInput.sendKeys(Keys.ENTER); return new SearchResultsPage(driver); } } Then in any page that has the header (which is every page after login), the header is a field, not inherited behavior: Java public class HomePage { private final WebDriver driver; public final GlobalHeader header; public final GlobalFooter footer; public final SideNav nav; @FindBy(css = "[data-test='homepage-hero']") private WebElement hero; public HomePage(WebDriver driver) { this.driver = driver; this.header = new GlobalHeader(driver); this.footer = new GlobalFooter(driver); this.nav = new SideNav(driver); PageFactory.initElements(driver, this); } public boolean isHeroVisible() { return hero.isDisplayed(); } } In tests: HomePage home = new HomePage(driver); int cartCount = home.header.getCartItemCount(); The reason this scales: when the header changes, and headers always change, you fix one class. Not 40. We did the migration from BasePage containing the header methods to dedicated component classes in early 2023. It took two engineers about four days. Worth every hour. Sangeeta, who was new to the team at the time, did most of it; she said later it was the only refactor where she ended up writing fewer lines of code than she deleted. Components nest. Our CheckoutPage has a CheckoutSidebar which has a PromoCodeWidget which has its own apply/clear methods. Test code reads down the tree: checkout.sidebar.promoCode.apply("HOLIDAY20"); It reads like the actual product. That’s the test of a good POM split: does the test code map to how a user would describe what they’re doing? Pattern 2: Loadable Component for State-Aware Pages Borrowed shamelessly from Selenium’s LoadableComponent, but we ended up writing our own because Selenium’s version assumes you can get() a URL to load, which doesn’t work for SPAs and modal dialogs. Java public abstract class LoadablePage<T extends LoadablePage<T>> { protected final WebDriver driver; protected final WebDriverWait wait; protected LoadablePage(WebDriver driver) { this.driver = driver; this.wait = new WebDriverWait(driver, Duration.ofSeconds(15)); } /** Returns true once the page is loaded enough to interact with. */ protected abstract boolean isLoaded(); /** Override to return a useful error if isLoaded() times out. */ protected String loadError() { return getClass().getSimpleName() + " did not load within timeout."; } @SuppressWarnings("unchecked") public T waitUntilLoaded() { try { wait.until(d -> isLoaded()); } catch (TimeoutException e) { throw new RuntimeException(loadError() + " Current URL: " + driver.getCurrentUrl(), e); } return (T) this; } } A page extends it: Java public class DashboardPage extends LoadablePage<DashboardPage> { @FindBy(css = "[data-test='dashboard-widgets-loaded']") private WebElement loadedSentinel; public DashboardPage(WebDriver driver) { super(driver); PageFactory.initElements(driver, this); } @Override protected boolean isLoaded() { try { return loadedSentinel.isDisplayed(); } catch (NoSuchElementException | StaleElementReferenceException e) { return false; } } } The trick is the [data-test='dashboard-widgets-loaded'] element. It’s a hidden div that the application renders when all 11 GraphQL queries on the dashboard have resolved. We had to ask the frontend team to add it. They pushed back at first (“you should test the user-visible state, not internal state”) and then I showed them the 27 tests that were flaking because we were waiting on the wrong element. They added the div. This is the negotiation move that nobody writes about: getting data-test attributes added to the application is half of POM in practice. Aman on the frontend team and I had a recurring 15-minute Tuesday standup for about three months in 2023 where we went through “tests flaked, here are the elements I need stable selectors for,” and he’d merge the PR by Wednesday. That meeting did more for our flake rate than any wait-strategy refactor. Pattern 3: Test Data Builders, Not UI Setup Tests that need preconditions should not click their way through the UI to set up. They should call APIs. We have a TestDataBuilder that sits next to the page objects. It uses the same authenticated session as the test: Java public class TestDataBuilder { private final ApiClient api; private final String userId; public TestDataBuilder(ApiClient api, String userId) { this.api = api; this.userId = userId; } public CartBuilder withCart() { return new CartBuilder(); } public class CartBuilder { private final List<String> productSkus = new ArrayList<>(); private String promoCode; public CartBuilder withProduct(String sku) { productSkus.add(sku); return this; } public CartBuilder withProducts(String... skus) { productSkus.addAll(Arrays.asList(skus)); return this; } public CartBuilder withPromoCode(String code) { this.promoCode = code; return this; } public Cart build() { // POST /api/cart with the user's auth token CartResponse resp = api.post("/cart", Map.of("userId", userId, "skus", productSkus, "promo", promoCode), CartResponse.class); return new Cart(resp.cartId); } } } In the test: Java @Test public void checkoutWithFullCart() { Cart cart = data.withCart() .withProducts("SKU-1029", "SKU-3344", "SKU-4101") .withPromoCode("HOLIDAY20") .build(); CartPage page = new CartPage(driver).waitUntilLoaded(); assertEquals(3, page.getItemCount()); CheckoutPage checkout = page.proceedToCheckout(); // ... rest of test } That builder skipped the 14 seconds of UI clicks. Multiplied across 200 tests on every CI run, it cut about 47 minutes off our suite wall time. The other thing it did, which I didn’t expect, was reduce flake, because the test wasn’t fighting the UI for setup; the actual assertion ran cleaner. The pushback I got on this approach was philosophical: “you’re not testing the cart-add flow if you skip it.” Correct. We test the cart-add flow in one dedicated test. We don’t re-test it 200 times across other suites. This is the same argument as “don’t test the framework”; it just hits earlier than people expect. Pattern 4: Browser-aware action helpers When you find a Chrome-Firefox-Safari difference, you want exactly one place to put the workaround. We have an Actions helper class that wraps the most common interactions and dispatches per browser: Java public class SmartActions { private final WebDriver driver; private final BrowserType browser; public SmartActions(WebDriver driver) { this.driver = driver; this.browser = detectBrowser(driver); } public void click(WebElement el) { switch (browser) { case FIREFOX -> firefoxClick(el); case SAFARI -> safariClick(el); default -> el.click(); } } private void firefoxClick(WebElement el) { // Firefox 119 has a bug where clicks on elements with // pointer-events: none parents silently fail. JS click bypasses. if (isInsideShadowDom(el)) { ((JavascriptExecutor) driver).executeScript("arguments[0].click();", el); } else { el.click(); } } private void safariClick(WebElement el) { // Safari needs a scroll-into-view before click on long pages ((JavascriptExecutor) driver).executeScript( "arguments[0].scrollIntoView({block: 'center'});", el); try { Thread.sleep(150); } catch (InterruptedException e) {} el.click(); } } That Thread.sleep(150) in SafariClick offends every clean-code instinct in your body. It’s there because it works and the documented WebDriverWait alternatives don’t. The Safari driver has its own race condition between scroll and click that I tracked through 60 hours of debugging and ended up logging as a bug against safaridriver. They acknowledged it, and I haven’t seen a fix. Page objects use SmartActions instead of calling .click() directly: public CartPage proceedToCheckout() { actions.click(checkoutButton); return new CartPage(driver); } When a new browser quirk shows up, you fix it in one place and 2,400 tests inherit the fix. TestNG configuration that actually parallelizes Selenium’s parallelization story is fine. TestNG’s parallelization story is fine. Getting them to play nice with a remote grid took longer than I want to admit. The key insight: parallel="methods" plus thread-count in your testng.xml is necessary but not sufficient. You also need a WebDriver factory that creates a new driver per thread, and a BeforeMethod that doesn’t accidentally leak drivers across threads. Java public class DriverFactory { private static final ThreadLocal<WebDriver> DRIVER = new ThreadLocal<>(); public static WebDriver get() { if (DRIVER.get() == null) { DRIVER.set(create()); } return DRIVER.get(); } private static WebDriver create() { String browser = System.getProperty("browser", "chrome"); String gridUrl = System.getProperty("grid.url", "http://selenium-hub:4444/wd/hub"); DesiredCapabilities caps = new DesiredCapabilities(); caps.setBrowserName(browser); try { return new RemoteWebDriver(new URL(gridUrl), caps); } catch (MalformedURLException e) { throw new RuntimeException(e); } } public static void quit() { WebDriver d = DRIVER.get(); if (d != null) { d.quit(); DRIVER.remove(); } } } BaseTest: public abstract class BaseTest { protected WebDriver driver; @BeforeMethod(alwaysRun = true) public void setUp() { driver = DriverFactory.get(); driver.manage().window().setSize(new Dimension(1440, 900)); } @AfterMethod(alwaysRun = true) public void tearDown() { DriverFactory.quit(); } } The ThreadLocal matters. Without it, two parallel test threads will share a driver and corrupt each other. The first time we deployed to the grid, we had a 22% flake rate that was almost entirely shared-driver corruption. Adding ThreadLocal fixed it in an afternoon. testng.xml for the parallel run: XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd"> <suite name="full-regression" parallel="methods" thread-count="24"> <listeners> <listener class-name="com.example.framework.RetryListener"/> <listener class-name="com.example.framework.ScreenshotOnFailureListener"/> </listeners> <test name="regression"> <packages> <package name="com.example.tests.regression"/> </packages> </test> </suite> The retry listener handles transient failures (network blip, grid node death). One retry, no more. We had a phase where engineers were setting it to retry 3-5 times, and the suite would “pass” but actually be hiding real bugs. One retry. That’s the rule. If a test needs three retries to pass, it’s a flaky test, and we file it as a bug, not a feature. What I’d tell you to skip Two things I tried and reverted on. We spent a quarter trying to use Cucumber as the test runner because some product stakeholder wanted “BDD.” We wrote ~80 step definitions, and they slowed everything down. The page objects were now wrapped in step definitions, which made the test code less expressive, not more. The product stakeholder lost interest by Q2, and we ripped Cucumber out. If you’re considering Cucumber: be very sure the product side will actually read the .feature files. They usually don’t. We also tried to auto-generate page objects from the application’s component library. Some early @FindBy attempts used the React component names as selectors. It worked for trivial pages and broke on anything dynamic. The lesson, six months in: page objects encode test intent, not application structure. Generating them from the app’s components produces page classes that mirror the implementation, which is the opposite of what you want; a page object should outlive the implementation that backs it. What we’re working on now The current evolution is moving the heavier setup builders behind a service that the test suite calls. Right now, TestDataBuilder calls our app’s API directly; we’re abstracting that into a “test fixture service” that can also seed the database directly when the API doesn’t expose what we need. The argument for it: there are still 30 or so tests that go through the UI to set up state because the API doesn’t have an endpoint. We could either add the endpoints (the right answer; the frontend team has been saying yes for nine months) or seed via SQL (the wrong answer; the fast answer). We’re doing both, in parallel, depending on which is faster for the specific test. Anyway, that’s the framework. POM at the bottom, components above it, loadable pages on top of those, builders for setup, smart actions for browser quirks, ThreadLocal for grid parallelism. Four patterns, all of which I learned by getting them wrong first.