Test automation is an essential subset of software testing. By using automated testing, we can expedite the process of software validation and increase testing coverage. However, there are a lot of challenges in applying test automation for applications under test (AUT).
Without overcoming these challenges, testers might face countless nightmares that can cause software automated testing failure. This topic aims to outline the top five challenges that have the highest impact on the overall automation test effort and project success. Hopefully, the earlier these challenges are understood, the better solutions are prepared to deal with them.
1. Effective Communication and Collaboration
This is perhaps a challenge not just in test automation but also in manual testing teams. However, it is more complicated in test automation than in manual testing because it requires more communication and collaboration by the automation team. Indeed, test automation is an investment. Therefore, like any other investments, to get the whole team members involved in identifying test automation objectives and setting targets, we need to spend significant efforts on communication and provide huge evidence, historical data, and we even do a proof of concept.
In addition, to have clear purposes and goals, we necessarily keep the entire team on the same page. Unlike manual testers, automation testers not only talk with developers, business analysts, and project manager about plan, scope, and timeframe but also discuss what should and shouldn’t be automated with manual testers, developers, and technical architects. Moreover, we have to present the cost and benefit analysis along with the Return on Investment (ROI) analysis to the higher management team. Without management team support, the whole test automation effort will be put at risk. Hence, how we communicate and collaborate among these teams and others effectively is a big challenge. Clearly, ineffective communication and collaboration can easily turn test automation experiences into a nightmare.
2. Selecting the Right Tool
Nowadays, there are a variety of testing tools, ranging from free and open-source tools like Selenium and Katalon Studio to commercial tools like TestComplete that support different testing types and technologies. Each tool tends to support particular situations. Vendors of testing products have a tendency to exaggerate the ability of their products. Vendors often assume that they have a “secret sauce” for all automation tastes. This causes misconceptions and confusions for us to select an appropriate testing tool satisfying our needs. Plus, many of us do not do enough research before making a decision about tool selection, and we tend to buy popular commercial tools quickly based on an inadequate evaluation.
Remember that a sufficient assessment includes defining a set of tool requirements criteria based on the AUT and the experience of experts who have already used the tools considerably. Unfortunately, people do not have enough resources to fulfill this requirement. No matter what kind of process and testing methodology we have, if a tool does not match our technical and business expectations, we will give up using it. Eventually, test automation will fail and not be applied in testing activities any longer. In my point of view, choosing a test tool is as complicated as getting married to a person. If you marry an inappropriate person, you'll probably break up sooner or later. Similarly, without a suitable test tool, we will deadly end up with failed test automation effort.
3. Demanding Skilled Resources
Some people claim that test automation can be just handled by manual testers or any technical testers because many test tools already support recording test scripts and playing back them so easily and quickly. This is a huge myth. In fact, test automation requires necessary technical skills to accurately design and maintain test automation framework and test scripts, build solutions, and resolve technical issues. Automated testing resources need to have strong knowledge of framework’s design and implementation. To fulfill these job requirements, obviously, these resources need to have both strong programming skills and solid test automation tools.
On the other hand, others believe that developers can entirely manage and take the test automation responsibilities. However, how developers can write correct test scripts at testers’ views and end-users’ needs is a big concern although they easily develop a piece of codes in accordance with the test automation framework. Certainly, we can utilize our resources within our test automation process to be more effective. However, skilled resources are always of importance in test automation effort.
4. Selecting a Proper Testing Approach
Automation tests not simply require a right tool to create scripts but also need a correct testing approach. This is one of the biggest challenges for test automation engineers. Technically, it’s vital for testers to find an appropriate test automation approach. In order to do so, they have to answer several important questions: How do you reduce effort in both implementation and maintenance of test script and test suite? Will automation test suites be having a long lifetime? How do you generate useful test reports and metrics? With adopting the agile development in recent years, the application under test often changes through development cycles.
Therefore, it's important to know how to design and implement automation test suites to correctly identify these changes and keep up-to-date quickly with reasonable maintenance effort. It is ideal to have a test automation solution that can detect these issues to automatically update and re-validate the test without any human intervention. It’s certainly not easy to address these difficult questions.
5. High Upfront Investment Cost
Talking about test automation, most of us agree that automated regression testing is crucial and useful in most agile contexts. However, when turning into the cost, we have many concerns. As a matter of fact, the initial phase of test automation is usually expensive. It’s necessary to analyze, design, and build a test automation framework, library, reusable function, etc. In some cases, it is required to take into account licensing costs, facilitating costs, and operating costs such as hardware and software costs.
Moreover, even though we can use free open-source tools to reduce the licensing costs, we might spend significant efforts on learning, training, and maintaining them. Furthermore, we also take hidden costs into consideration. How to account for hidden costs such as meeting, communicating, and collaborating. As a result, how we can ensure that these things cannot affect our decisions. Although there is a huge payoff in the long run after running some regression testing cycles, convincing the stakeholders to have a consensus about this investment is a big challenge. Actually, just due to budget constraints, many people tend to give up test automation even though they agree with an executable goal and high ROI.
These challenges are not the only ones found in test automation, but they certainly are common. If we don’t have solutions to overcome them, the test automation project can easily result in failure.
Test automation is nowadays dominating in agile development contexts. Many test automation projects, to some extent, have been brought out tremendous benefits and successes when people are aware and control test automation challenges effectively. There are still controversies about whether we should promote test automation in testing activities or not. From where I stand, test automation might have a huge payback, contributing to valuable investments.