Will ChatGPT and Generative AI “Replace” Testing?

There is a lot of buzz within the software testing and development communities about ChatGPT and the role of generative AI in testing.

Some of the opinion pieces, webinars, and videos focus on the potentially beneficial applications of generative AI for testing speed and quality. However, many focus on how ChatGPT might impact a tester's future job prospects and security. Some go as far as questioning whether ChatGPT will replace the role of manual testers and SDETs.

The popularity of this type of content is natural, given the anxiety associated with wavering job security in tech and the natural hunger of testers to learn the latest tools and skills. Meanwhile, some organizations still view testing as a problematic cost center. The appeal of automation and AI for these managers, in turn, becomes cost avoidance while still going through the motions of "QA."

How well-founded is the speculation that ChatGPT might "replace" testing or, at least, replace testers? And, for senior management, how should you think about the role of AI-based automation in your organization's future QA investment?

A Worry as Old as Automation Itself

The view that newer technologies and newly skilled workers will render existing jobs redundant is not new. The term "luddite," for instance, stems from a movement of textile workers in 19th Century England who opposed textile machinery as a threat to the value of their highly-skilled work.

Within software testing, these same concerns emerged some 10+ years ago when marketers presented test execution automation as a "silver bullet" for testing efficiencies and bloated testing budgets.

And now, today, ChatGPT and generative AI brings the promise that it will automate the automation itself, automating the labor performed by SDETs and engineers. These toilsome tasks might include writing test scripts and Gherkin specifications or sourcing test data and analyzing requirements.

Let's assume that ChatGPT and generative AI can automate these tasks. For the sake of this article, let's park the sizeable challenges surrounding the data used to train ML/AI models, questions of test coverage, and issues associated with governance, bias, and more.

What room will be left for the testers and SDETs who have spent years crafting their skills and expertise if intelligent automation can produce test automation at a fraction of the time and cost?

The Testing Discipline Is Going Nowhere

Yet, automating test execution did not automate away testing or QA. In fact, it created a raft of new and complex processes, requiring new skills, tools, and dedicated roles like the SDET.

No matter how much testing has been "automated," there has always been more to automate and maintain. Testing exhaustively is more of an impossible goal today than it was a decade ago.

The average test team has automated just 15-20% of all tests, reflecting this ever-growing testing requirement. The total number of possible test scenarios that could be automated just keeps growing.

To understand this ever-growing "to-do" list, we must consider the relationship between speed and complexity. We must consider how doing individual tasks faster tends to create more tasks to complete, however paradoxical this might at first appear.

Moore's Law and More to Test

One appeal of automation is the time and cost efficiency by which it executes high volumes of tasks. In testing, automation has been well suited to repetitive and recurring tasks like regression testing. ChatGPT and generative AI promise to expand the scope beyond scenarios pre-defined by humans, further reducing the labor associated with creating and running tests.

Yet, the efficiency gained through automation is not isolated to testing. If it were, testing could hope to close the gap in exhaustive test coverage. Instead, the testing requirement continues to grow in size and complexity.

As QA becomes faster in its individual processes, so does software design, development, and every constituent part of the SDLC. These individual efficiencies enable faster software delivery, making changes of greater functional size and logical complexity. This leaves less time to test.

The number of logical combinations to test will continue to grow, as will the need to maintain and refactor ever-growing volumes of historical code and tests. No matter how much you optimize, your delivery processes will also grow in complexity:

As Moore's law reminds us, this growing complexity of software systems will not slow down.

One benefit of automation is that we can offload this complexity and the associated workload to machines that are constantly growing in processing power. As a result, they can execute ever more tasks ever faster. Yet, this efficiency, in turn, brings more complexity and, with it, more work to do.

The requirement to optimize and go faster will never go away, while the sum total of work in front of us will never in itself decrease.

How Will ChatGPT Sit Alongside Your Testing?

It stands to reason, then, that ChatGPT and generative AI will not "replace" testing or remove the need to invest in QA. Instead, like test execution automation before it, generative AI will provide a useful tool for moving faster.

Yet, there will always be a need for more work, and at least a constant (if not greater) need for human input. Testers' time might be applied less to repetitive tasks like scripting, but new processes will fill the void. Meanwhile, the creativity and critical thinking offered by testers will not diminish in value as these repetitive processes are automated; such creativity should be given greater freedom.

At the same time, your testers will have vital insight into how generative AI should be used in your organization. Nothing is adopted overnight, and identifying the optimal applications of tools like ChatGPT will be an ongoing conversation, just as the testing community has continually explored and improved practices for getting the most out of test automation frameworks.

Lastly, as the volume of possible test scenarios grows, automation and AI will need a human steer in knowing where to target its efforts, even as we can increasingly use data to target test generation.

How Can You Use ChatGPT in Your Testing?

So, what can you do to get the most out of ChatGPT in your testing?

If you are a tester or SDET, approach generative AI like you approach test automation tools and frameworks. Skill up and learn new technologies, considering which are the best fit for your organization. There is already a range of resources to learn how to leverage ChatGPT in testing, even if many are still quite speculative.

These new skills will not only help you remain competitive within an increasingly crowded job market but also help you become a leader of beneficial change at your organization.

If you are a QA manager or director, give your teams the time, space, and investment needed to experiment and explore these new tools and techniques.

Change and improvement in testing practices are often blocked because teams are too busy pumping out deliverables using outdated techniques. Rather than hope that generative AI will provide a silver bullet to these growing inefficiencies, sacrifice a fraction of your teams' available testing time to learn about new tools, and consider how they can best be applied. Remember that your solutions will lie in "engineering augmented by AI," just as test automation offered an accelerator – but not a replacement – to Quality Engineering.