Software automation tools and frameworks are crucial for the success of test automation projects. But it is a difficult task to select the right set of test automation tools for a new software project. There are many tools with each having several unique good features. One single tool may not satisfy all project requirements and constraints, resulting in multiple automation tools to be used. Chosen tools seem to be working at the beginning, but troubles occur later as hidden problems are overlooked at the time of tool selection.

Understanding the priorities in determining automation tools is part of our survey which questioned more than 2,000 software test professionals about their experience with test automation. The survey results show that more than 60% of respondents with 3+ years of experience in test automation agreed that licensing & support costs and good test reports are the most important priorities in deciding a test automation tool. Moreover, respondents with less experience in test automation prioritize training and support materials over other criteria.

A tool vendor often provides a list of key features and characteristics when comparing their tool with others in the market. They tend to highlight their tool’s strengths while ignoring important features that may pinpoint its weaknesses. This practice confuses not only new test automation adopters but also experienced ones in making a decision about which tools and frameworks for new test automation projects.

The survey respondents suggested some of their favorite tools, including:

Open-source tools (coding experience required):

• Selenium

• Appium

• Robot Framework

• Cucumber

Free tools (limited community support):

• Katalon Studio

Commercial tools (high support and license costs):

• TestComplete

• UFT

• Ranorex

Tool Selection Priorities by Respondents

The top four attributes chosen by at least 50% of respondents as their most important priorities in tool selection include:

1. Training, documentation, tutorials, guidelines

2. Good test reports

3. Licensing and support costs

4. Level of programming skills required

Particularly, we considered two respondent groups, one with at least 3 years of experience in test automation and the other with 1 year or less. It is intuitive to believe that the feedback on the top priorities for selecting tools by experienced test automation professionals is more reasonable than that by less experienced ones.

The top six priorities when selecting automated testing tools considered by more than 50% of respondents with 3+ years of experience are:

1. Licensing and support costs

2. Good test reports

3. Training, documentation, tutorials, guidelines

4. CI, DevOps support

5. Level of programming skills required

6. Level of skills and experience required

Instead of presenting expert opinions, this article reports several of the findings from this study surveying 2,000+ software professionals about their perspectives on automation tool selection priorities. These findings serve as a reference for checking expert opinions and making decision about which tools to be used in test automation projects. This survey suggests several important implications:

Cost versus technical expertise. This survey shows that licensing and support costs is the top tool selection priority. This is a reason why most respondents have used open-source tools and frameworks for their projects such as Selenium, JMeter and Appium. However, most of these tools, require technical expertise (programming skills and experience) to build, integrate and deploy before utilizing them effectively. Fortunately, recently emerging automation solutions like Katalon Studio provide an integrated and ready-to-use environment by leveraging popular open-source frameworks and libraries such as Selenium and Appium.

Good test reports are important. The basic report of a test execution is whether a test case is passed, which is supported by most, if not all, automation tools. However, the automation community needs informative and insightful test reports to help diagnose and analyze defects and root causes, test coverage, test effectiveness, and other analysis. Such reports become necessary for managers to make informed decisions about the quality of their products.

It is essential for an automation tool to support CI and DevOps tool integration. As test automation is the cornerstone of CI and DevOps practices, a serious automation tool must be able to integrate with CI and DevOps toolchains. The lack of CI and DevOps support of an automation tool can significantly reduce its effectiveness and benefits in agile projects where speed is a priority. As the trend of increasing DevOps adoption continues, the integration capability becomes an essential feature for automation tools.