AI-Assisted Testing: Real-Life Use Cases vs. Myths

There’s a lot of hype and conflicting information surrounding AI in software development and testing. Are there any real productivity gains? Are those impressive stats real, or just part of a polished pitch for VC investors? Can we really improve our release cadence with AI? One thing is clear: AI has permeated all aspects of the software development life cycle (SDLC), including quality assurance (QA).

In this article, I’ll share insights from real-life commercial projects on what’s possible with AI in QA and which aspects of software testing can be really enhanced by this groundbreaking technology.

Where Does AI in QA Stand Today?

The interest in AI-assisted testing is real: there’s hardly a business that hasn’t struggled with various QA challenges, be it a lack of expertise, modest headcount, or frequent release cycles dictated by the pressures of a competitive SaaS landscape. So it’s only natural to turn to AI for some cost optimization and efficiency improvements.

The demand for AI-assisted testing is further backed by global market predictions: AI-enabled QA is forecast to grow from $1.01 billion in 2025 to $4.64 billion by 2034. But beyond the impressive market forecasts, what can generative AI actually do for a QA team right now? Here are some use cases we’ve tried and adopted in our QA practice.

Test Scenario Generation

We are seeing a definitive shift away from manual test authoring toward AI-augmented design. Instead of spending hours writing repetitive documentation, testers now use AI to generate baseline coverage. They can finally focus on high-value tasks like risk analysis, edge cases, and system behavior. Whether it’s ChatGPT, Gemini, or specialized tools like Qase.io or AI Test Case Generator, the market is currently teeming with options to help teams brainstorm scenarios and broaden their coverage.

Test Data Generation

Manual data preparation is often the most tedious part of QA. We use AI-driven tools to create realistic, high-volume datasets that mimic production data without compromising user privacy.

AI-generated test data works well for functional testing, but high-stakes tasks still require the specific quirks of real-world data. We recommend a hybrid approach: mask a small portion of actual production data for compliance purposes, and rely on AI-generated synthetic datasets for everything else.

AI-Improved Bug Reports

With a single prompt, testers can quickly polish their bug reports, eliminating the need for tedious back-and-forth with developers. AI can flag areas for improvement, such as unclear titles, and rewrite them to be user-friendly and actionable. It can also identify missing context in reproduction steps or help evaluate bug severity and business impact.

Defect Prediction

Generative AI is also helpful in pinpointing high-risk areas where bugs are most likely to occur. To increase the accuracy of the results, you’ll need to provide the AI with the features in scope, recent changes, known risk factors, and any gaps in test coverage. For security reasons, remove specific company names from your user stories. Instead, use a generic description of the business, such as “a brokerage firm” or “an e-commerce platform”.

You can expect an output along the lines of:

• A ranked list of high-risk modules (from highest to lowest probability of defects)
• Primary risk drivers for each (e.g., “Recently refactored,” “Failed in previous release,” “Low coverage”)
• Strategic testing suggestions (e.g., prioritize exploratory testing, add negative tests, or automate specific flows)
• Recommended historical metrics or tools to sharpen prediction accuracy over time

AI for Localization Testing

Localization testing can be partially automated and significantly improved with tools such as Spling and Applitools. We leverage Spling for high-velocity automated spell-checking and grammar proofing. For layout issues, Applitools’ Visual AI is ideal for detecting UI breaks. For example, localized German text is often 30% longer than the English original, and Applitools helps us catch text that overflows buttons or overlaps other design elements.

AI-Powered Accessibility Testing

Ensuring software is usable for everyone is both a legal and ethical necessity. AI tools like Axe and AccessiBe automate application scans against WCAG (Web Content Accessibility Guidelines) much faster than a human ever could. They are an integral part of a professional web accessibility audit. These tools also help uncover various usability issues, the resolution of which benefits users both with and without disabilities.

Test Code Generation

Test code generation speeds up writing test automation scripts, where automation engineers act more like editors than writers. However, there’s a caveat: the end result largely depends on how well the prompt is built. If you simply prompt “create a login test,” the AI will give you a generic, brittle script. But if you provide context — “create a Playwright script in TypeScript for a login page with two-factor authentication using the Page Object Model” — the AI provides a sophisticated, maintainable framework.

Common Myths: Where AI Still Falls Short

While AI solutions are evolving at breakneck speed, it is important to distinguish between marketing promises and technical reality. Many of the limitations we see today may eventually be solved, but for now, several key myths persist.

Myth 1: AI Testing Agents Are Fully Autonomous

AI testing agents promise autonomous exploration of websites or apps without continuous intervention. Unlike traditional test automation, which requires creating and maintaining test scripts, agentic automation aims to simplify this time-consuming, labor-intensive process.

In a perfect world, an AI testing agent can explore a website or app without constant supervision. Unlike traditional automation, which requires manually writing and maintaining scripts, a perfect autonomous agent would independently identify core user journeys, generate manual test cases, and execute them by simulating real user behavior. It would then provide a comprehensive report, complete with screenshots and logs.

While some off-the-shelf solutions claim to do this, true autonomy remains out of reach for several reasons:

• Lack of Exhaustive Coverage: AI agents are excellent at following common paths but often miss the subtle, complex scenarios that a human tester would prioritize.
• The Context Gap: Agents lack a big picture understanding of the business logic. It is incredibly difficult to provide an AI with the full system context it needs to understand why a certain behavior is a bug rather than a feature.
• Test Flakiness: Agentic automation is still prone to flaky results (where a test fails or passes inconsistently despite no changes to the code), making it difficult to trust the output without human verification.
• The Configuration Tax: Out-of-the-box agents rarely provide meaningful results. To get the most out of these tools, engineers must spend significant time configuring and customizing them, which often negates the time savings from autonomy.

Myth 2: AI Automatically Makes Testing Faster

There are always two sides to the coin. Yes, AI drives efficiency in use cases we described above; there’s no denying that. In fact, McKinsey reports that a global insurer accelerated coding tasks and testing efficiency by 50% thanks to adopting generative AI.

We believe AI is only faster once you’ve invested the time to understand how it fits your specific project and business needs. Blindly adopting the latest testing agent making waves online is unlikely to yield the expected ROI or tangible improvements in product quality.

For example, when testing the capabilities of testers.ai, we concluded that the tool is not a good fit for projects where automation is already in place, as regression testing with Playwright is much faster: Playwright autotests handle the same task in an average of 1-1.5 minutes whereas the Gemini 2.5 Flash model powering testers.ai takes 7–10 minutes to perform the same tests.

Myth 3: AI Will Eventually Replace Human Testers

AI testing tools and agents offer some truly powerful features and automation capabilities. At the same time, each of them comes with its own limitations and risks. Only a human tester with years of experience can properly vet these tools and select the right one to meet specific business goals.

For example, when exploring testers.ai’s capabilities, we concluded that the tool is best suited for smoke testing, as it quickly detects UI issues, broken buttons, and other surface-level problems. However, it’s not a good fit for projects where automation is already in place, as regression testing with Playwright is significantly faster. Furthermore, manual regression testing performed by specialists familiar with the project and documentation remains much more accurate. We also encountered false positives: in one instance, the AI agent tried to enter an email address into a name field, and then flagged the resulting validation error as a bug.

Another issue is that if business requirements contain gaps or mistakes and are blindly fed to an AI, the result will be flawed test cases. We still need QA engineers to critically evaluate both the input provided to the AI and the output it produces. Uncovering usability issues, possessing deep domain knowledge, and having a real-world understanding of pitfalls in QA workflows — these and many other aspects of a QA engineer’s job cannot be automated.

Needless to mention the hard limits of LLMs: context window constraints, a lack of genuine business intuition, and the high cost of inference.

These examples perfectly illustrate the necessity of a human-in-the-loop approach: you need expert guidance before investing in AI-assisted QA to truly benefit from breakthroughs in the field.

Is AI Making QA Better or Just More Complex?

The answer depends entirely on your approach. For QA leaders, the path forward requires a blend of bold adoption and disciplined skepticism. Those who blindly automate will find themselves managing more complexity, not less. Success will not come from chasing every viral tool, but from targeted, sandboxed experimentation focused on long-term ROI. The winners in this new era will be those who master the tech without ever losing sight of the engineering fundamentals.