Using Graph RAG and Specialized Agents to Repair Playwright Tests

End-to-end tests are crucial components of modern CI/CD pipelines, helping to ensure that changes do not cause regressions before deployment. Playwright offers network inspection, automatic waiting, browser isolation, tracing and cross-browser execution. While useful for improving automation, these features cannot replace the engineering work needed to develop, identify and monitor effective tests.

Requirements, source code, page structure, APIs, selectors, authentication rules, test data can all change at the same time in a sprint. The reason for a Playwright test failure can include any of the following: regression in the application, the application's outdated locator, outdated test data, inability to reach the application, or instability of the timing.

This test-case checkout sprint demonstrates the addressing of that problem by specialized agents, graph retrieval-augmented generation, execution evidence, and bounded repair, without obscuring real regressions.

The Test-Case Sprint

An e-commerce team plans a checkout update with three features:

• Promotional-code support
• Automatic refresh of expired authentication tokens
• A redesigned order summary

The sprint defines seven acceptance criteria:

1. Customers can apply valid promotional codes.
2. Discounts appear in the order summary.
3. Invalid or expired codes produce clear errors.
4. Totals include discounts, tax, and shipping.
5. Authentication refresh preserves the shopping cart.
6. An order can be submitted only once.
7. Successful submission displays an order identifier.

The implementation changes several artifacts:

• src/pages/CheckoutPage.tsx
• src/components/PromoCodeForm.tsx
• src/components/OrderSummary.tsx
• src/services/auth.ts
• src/services/checkout.ts
• tests/checkout.spec.ts
• playwright.config.ts

The interface team also changes the test identifier of "place-order-button" to "submit-order."

Post-change tests fail in 14 locations in Chromium, Firefox, and WebKit. With a conventional pipeline, failures are reported, but it is not possible to reliably identify which tests are to be repaired and which failures are product defects.

Why One General-Purpose Agent Is Not Enough

In the testing workflow, there are multiple tasks that call for various types of reasoning.

Requirement analysis identifies the behavior(s) that need to be validated. Test planning maps behavior to preconditions, data, actions, and assertions. To generate code, one needs to know Playwright. Controlled browsers and artifact collection for execution. Traces, screenshots, network events, and repository changes are used for diagnosis [1]. Conservative decisions for repair must be made, and the original requirement must be retained.

Assigning every responsibility to one unrestricted agent introduces several risks:

• Irrelevant context moves between stages.
• Large prompts increase cost.
• Failed actions may be repeated.
• Obsolete selectors may be regenerated.
• Application regressions may be misclassified as test defects.
• Repairs may weaken assertions merely to produce passing tests.

This research separates these responsibilities into requirement, planning, generation, execution, diagnosis, and repair experts.

Each expert operates under a specific tool policy, input schema, output schema, and validation contract [2]. The architecture does not require six different foundation models. Multiple roles can use specialized configurations of the same model.

Building a Software Knowledge Graph

Classical RAG usually retrieves independent text chunks through semantic similarity. Software evidence is relational.

A uniform requirement can be implemented across several files. A page renders multiple components, and these components are comprised of interface elements. A test covers singular or multiple criteria of acceptance

This study represents these relationships in a project knowledge graph. Possible node types include:

• User stories
• Acceptance criteria
• Commits
• Source files
• Classes and functions
• Pages
• Components
• Interface elements
• Tests
• Execution traces
• Defects

Representative relationships include:

• implements
• contains
• renders
• depends_on
• changed_by
• covers
• failed_in
• fixed_by

A useful retrieval path for promotional-code testing could be:

• Apply valid promotional code
• CheckoutPage
• PromoCodeForm
• POST /api/promotions/validate
• Promotional-code input
• Existing checkout test

Another path could connect order submission with a historical defect:

• Checkout service
• POST /api/orders
• Submit-order button
• Checkout submission test
• Duplicate-order defect

Semantic retrieval finds the artifacts whose language is related to the selected language [3]. Dependency paths are the basis for a structured retrieval. The recency feature avoids stale results from stale selectors and provenance is used to point to the repository revision and source of each artifact.

Graph traversal does not gather irrelevant evidence, thanks to depth limits and token budgets.

Routing the Task to Specialists

Calling every expert for every task would increase latency and token cost. This study uses sparse routing to activate only the required specialists.

The router considers:

• Requirement complexity
• Changed-file distribution
• Historical defects
• Execution risk
• Available evidence
• Remaining pipeline budget

Generation and execution experts may only be needed for minor text changes [4]. The checkout sprint impacts the following items: authentication, payment behavior, order totals and duplicate submission protection. Therefore, all six roles are activated in the router and the human approval for the repair if a high risk is involved is necessary.

Each criterion must be broken down into observable behaviors by the requirements expert. Often, for security reasons and better planning, the planner creates scenarios of positive, negative and of retrieval.

A valid promotional-code scenario might contain:

Precondition: Authenticated customer with products in the cart

Test data: Consider a promotional code, ‘SAVE10’

Actions:

1. Open checkout
2. Enter SAVE10
3. Apply the code

Expected result:

• The order summary displays the discount
• The final total includes the discount, tax, and shipping

Cleanup: Remove the test order and release the promotional code

Generating the Playwright Test

For the Playwright code to be created, the generation expert uses the validated plan and retrieved graph evidence [5].

Locator selection follows a stability hierarchy:

1. Accessible role
2. Label
3. Stable visible text
4. Approved test identifier
5. CSS selector when stronger options are unavailable

A generated test could look like this:

import { test, expect } from "@playwright/test";

test("applies a valid promotional code", async ({ page }) => {
  await page.goto("/checkout");

  await page.getByLabel("Promotional code").fill("SAVE10");

  await page.getByRole("button", { name: "Apply code" }).click();

  await expect(page.getByTestId("discount-line"))
    .toContainText("SAVE10");

  await expect(page.getByTestId("discount-amount"))
    .toHaveText("-$10.00");

  await expect(page.getByTestId("order-total"))
    .toHaveText("$102.40");
});

The test validates a business outcome instead of checking only that the page remains visible.

Static validation rejects:

• Fixed delays
• Unsupported selectors
• Hidden execution-order dependencies
• Missing imports
• Weak assertions
• Tests without acceptance-criterion mappings

Accepted tests run in isolated browser contexts with deterministic setup and cleanup.

Collecting Execution Evidence

The execution expert runs the tests and collects:

• Playwright traces
• Screenshots
• Videos
• Console messages
• Network requests and responses
• Browser errors
• Timing data
• Retry outcomes

Clerical failure is a way of separating between determinate failures and fragile and unpredictable behavior of the clerical. In like manner, cross browser execution will determine browser-specific parity but won't consider each browser difference to be an application defect. The 14 test-case failures form three groups.

Case 1: Obsolete Test Identifier

Seven tests fail with this error:

Timeout waiting for [data-testid="place-order-button"]

Based on the traces, it is established that that the checkout page loads successfully. Graph retrieval connects the button component and acceptance criterion to the new `submit-order` identifier.

The diagnosis expert classifies the failures as test defects. The repair expert proposes the smallest supported patch:

// Previous locator

page.getByTestId("place-order-button");

// Repaired locator

page.getByTestId("submit-order");

The patch changes no actions or assertions. It passes static validation, targeted execution, and the relevant checkout regression subset.

Case 2: Authentication Regression

Four tests fail after an authentication token expires. Network evidence shows:

POST /api/auth/refresh -> 200

POST /api/orders -> 401

Refresh request is successful; order request still uses the expired token. This is possible because there is a correlation between the failure and the changes to the auth.ts and checkout.ts.

An application defect is diagnosed by the diagnosis expert. No permission is given to the repair agent to change the tests.

Relaxing the test or adding retries to it would hide the regression. On the other hand, the pipeline produces a defect report including the requirement, the trace, the events of the network, the files affected, as well as the version of the repository.

Case 3: WebKit Timing Instability

Some WebKit runs crash sporadically as the order summary is recalculating. No code changes for repeated runs yields passing and failing result.

The original test checks the total immediately:

await applyButton.click();

await expect(orderTotal).toHaveText("$102.40");

The trace shows a visible recalculation state. The diagnosis expert classifies the failure as flaky synchronization.

The repair waits for an observable state transition:

await applyButton.click();

await expect(page.getByTestId("summary-status"))

   .toHaveText("Updated");

await expect(page.getByTestId("order-total"))

   .toHaveText("$102.40");

This repair avoids a fixed timeout. Five repeated WebKit runs and the relevant regression subset must pass before acceptance.

Bounded Repair and Governance

Autonomous repair should never operate without limits.

A patch is eligible only when:

• Diagnostic confidence exceeds the configured threshold.
• Evidence identifies a specific test defect.
• The acceptance-criterion mapping remains unchanged.
• Static validation passes.
• The targeted test passes.
• A relevant regression subset passes.
• The repair-attempt limit has not been exceeded.

Application defects, ambiguous failures, payment changes, authorization rules, and security-sensitive workflows require human review.

Every decision should record:

• Selected experts
• Retrieved graph nodes
• Model outputs
• Tool calls
• Execution evidence
• Failure classification
• Proposed patch
• Validation outcome
• Human approval status

This audit trail makes the workflow reproducible and reviewable.

Test-Case Outcome

After diagnosis:

• Seven failures are classified as locator-related test defects.
• Four failures are classified as application regressions.
• Three failures are classified as flaky synchronization problems.
• Ten tests receive validated repairs.
• Four application failures remain visible for developers.
• No assertions are removed or weakened.
• Every repair remains connected to its acceptance criterion.

These values are illustrative and are not production measurements.

Practical CI/CD Considerations

A production implementation should:

• Build the graph incrementally.
• Cache reusable retrieval results.
• Pin browser, repository, model, prompt, and policy versions.
• Restrict expert tool access.
• Define token, execution-time, and repair budgets.
• Limit browser concurrency.
• Redact secrets before model access.
• Store generated tests and traces as reviewable artifacts.
• Require approval for high-risk repairs.
• Report assisted and autonomous outcomes separately.

Throughout the procedure, the quality of the graph persists as a major dependency. A primary factor that misleads retrieval is the presence of non-uniform and obsolete relationships. Model variability, routing errors, test-data instability, integration cost, and privacy requirements also limit adoption.

Conclusion

Refresh request fails, but order request continues to use the previous (expired) token. This works because the failure is related to changes in the `auth.ts' and checkout.ts' files.

An application defect is diagnosed by the diagnosis expert. Permits repair agent to change the tests (no).

It would make the assertion less strict and add retries, thus burying the regression. On the other hand, the pipeline will create a defect report, which will contain the requirement of the trace, the events in the network, the affected files, and the repository revision.

References

1. J. Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” NeurIPS, 2024.  
2. C. S. Xia et al., “Agentless: Demystifying LLM-Based Software Engineering Agents,” 2024.  
3. D. Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” 2024.  
4. J. A. Pizzorno and E. D. Berger, “CoverUp: Effective High-Coverage Test Generation for Python,” 2025.  
5. S. Gu et al., “TestART: Improving LLM-Based Unit Testing via Co-Evolution of Automated Generation and Repair Iteration,” 2024.