Top Priorities in Selecting Automated Testing Tools
Top Priorities in Selecting Automated Testing Tools
Surveys of software testers show that these are the most important criteria to keep in mind when choosing a test automation tool.
Join the DZone community and get the full member experience.
Join For FreeLearn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market.
Software automation tools and frameworks are crucial for the success of test automation projects. But it is a difficult task to select the right set of test automation tools for a new software project. There are many tools with each having several unique good features. One single tool may not satisfy all project requirements and constraints, resulting in multiple automation tools to be used. Chosen tools seem to be working at the beginning, but troubles occur later as hidden problems are overlooked at the time of tool selection.
Understanding the priorities in determining automation tools is part of our survey which questioned more than 2,000 software test professionals about their experience with test automation. The survey results show that more than 60% of respondents with 3+ years of experience in test automation agreed that licensing & support costs and good test reports are the most important priorities in deciding a test automation tool. Moreover, respondents with less experience in test automation prioritize training and support materials over other criteria.
A tool vendor often provides a list of key features and characteristics when comparing their tool with others in the market. They tend to highlight their tool’s strengths while ignoring important features that may pinpoint its weaknesses. This practice confuses not only new test automation adopters but also experienced ones in making a decision about which tools and frameworks for new test automation projects.
The survey respondents suggested some of their favorite tools, including:
Open-source tools (coding experience required):
Free tools (limited community support):
Commercial tools (high support and license costs):
Tool Selection Priorities by Respondents
The top four attributes chosen by at least 50% of respondents as their most important priorities in tool selection include:
Training, documentation, tutorials, guidelines
Good test reports
Licensing and support costs
Level of programming skills required
Particularly, we considered two respondent groups, one with at least 3 years of experience in test automation and the other with 1 year or less. It is intuitive to believe that the feedback on the top priorities for selecting tools by experienced test automation professionals is more reasonable than that by less experienced ones.
The top six priorities when selecting automated testing tools considered by more than 50% of respondents with 3+ years of experience are:
Licensing and support costs
Good test reports
Training, documentation, tutorials, guidelines
CI, DevOps support
Level of programming skills required
Level of skills and experience required
Instead of presenting expert opinions, this article reports several of the findings from this study surveying 2,000+ software professionals about their perspectives on automation tool selection priorities. These findings serve as a reference for checking expert opinions and making decision about which tools to be used in test automation projects. This survey suggests several important implications:
Cost versus technical expertise. This survey shows that licensing and support costs is the top tool selection priority. This is a reason why most respondents have used open-source tools and frameworks for their projects such as Selenium, JMeter and Appium. However, most of these tools, require technical expertise (programming skills and experience) to build, integrate and deploy before utilizing them effectively. Fortunately, recently emerging automation solutions like Katalon Studio provide an integrated and ready-to-use environment by leveraging popular open-source frameworks and libraries such as Selenium and Appium.
Good test reports are important. The basic report of a test execution is whether a test case is passed, which is supported by most, if not all, automation tools. However, the automation community needs informative and insightful test reports to help diagnose and analyze defects and root causes, test coverage, test effectiveness, and other analysis. Such reports become necessary for managers to make informed decisions about the quality of their products.
It is essential for an automation tool to support CI and DevOps tool integration. As test automation is the cornerstone of CI and DevOps practices, a serious automation tool must be able to integrate with CI and DevOps toolchains. The lack of CI and DevOps support of an automation tool can significantly reduce its effectiveness and benefits in agile projects where speed is a priority. As the trend of increasing DevOps adoption continues, the integration capability becomes an essential feature for automation tools.
Find out more about how Scalyr built a proprietary database that does not use text indexing for their log management tool.
Like This Article? Read More From DZone
Published at DZone with permission of Vu Nguyen . See the original article here.
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}