DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Deployment
  4. Top Priorities in Selecting Automated Testing Tools

Top Priorities in Selecting Automated Testing Tools

Surveys of software testers show that these are the most important criteria to keep in mind when choosing a test automation tool.

Vu Nguyen user avatar by
Vu Nguyen
·
Jun. 28, 18 · Opinion
Like (1)
Save
Tweet
Share
7.30K Views

Join the DZone community and get the full member experience.

Join For Free

Software automation tools and frameworks are crucial for the success of test automation projects. But it is a difficult task to select the right set of test automation tools for a new software project. There are many tools with each having several unique good features. One single tool may not satisfy all project requirements and constraints, resulting in multiple automation tools to be used. Chosen tools seem to be working at the beginning, but troubles occur later as hidden problems are overlooked at the time of tool selection.

Understanding the priorities in determining automation tools is part of our survey which questioned more than 2,000 software test professionals about their experience with test automation. The survey results show that more than 60% of respondents with 3+ years of experience in test automation agreed that licensing & support costs and good test reports are the most important priorities in deciding a test automation tool. Moreover, respondents with less experience in test automation prioritize training and support materials over other criteria.

A tool vendor often provides a list of key features and characteristics when comparing their tool with others in the market. They tend to highlight their tool’s strengths while ignoring important features that may pinpoint its weaknesses. This practice confuses not only new test automation adopters but also experienced ones in making a decision about which tools and frameworks for new test automation projects.

The survey respondents suggested some of their favorite tools, including:

Open-source tools (coding experience required):

  • Selenium

  • Appium

  • Robot Framework

  • Cucumber

Free tools (limited community support):

  • Katalon Studio

Commercial tools (high support and license costs):

  • TestComplete

  • UFT

  • Ranorex

Tool Selection Priorities by Respondents

The top four attributes chosen by at least 50% of respondents as their most important priorities in tool selection include:

  1. Training, documentation, tutorials, guidelines

  2. Good test reports

  3. Licensing and support costs

  4. Level of programming skills required

Particularly, we considered two respondent groups, one with at least 3 years of experience in test automation and the other with 1 year or less. It is intuitive to believe that the feedback on the top priorities for selecting tools by experienced test automation professionals is more reasonable than that by less experienced ones.

The top six priorities when selecting automated testing tools considered by more than 50% of respondents with 3+ years of experience are:

  1. Licensing and support costs

  2. Good test reports

  3. Training, documentation, tutorials, guidelines

  4. CI, DevOps support

  5. Level of programming skills required

  6. Level of skills and experience required

Instead of presenting expert opinions, this article reports several of the findings from this study surveying 2,000+ software professionals about their perspectives on automation tool selection priorities. These findings serve as a reference for checking expert opinions and making decision about which tools to be used in test automation projects. This survey suggests several important implications:

Cost versus technical expertise. This survey shows that licensing and support costs is the top tool selection priority. This is a reason why most respondents have used open-source tools and frameworks for their projects such as Selenium, JMeter and Appium. However, most of these tools, require technical expertise (programming skills and experience) to build, integrate and deploy before utilizing them effectively. Fortunately, recently emerging automation solutions like Katalon Studio provide an integrated and ready-to-use environment by leveraging popular open-source frameworks and libraries such as Selenium and Appium.

Good test reports are important. The basic report of a test execution is whether a test case is passed, which is supported by most, if not all, automation tools. However, the automation community needs informative and insightful test reports to help diagnose and analyze defects and root causes, test coverage, test effectiveness, and other analysis. Such reports become necessary for managers to make informed decisions about the quality of their products.

It is essential for an automation tool to support CI and DevOps tool integration. As test automation is the cornerstone of CI and DevOps practices, a serious automation tool must be able to integrate with CI and DevOps toolchains. The lack of CI and DevOps support of an automation tool can significantly reduce its effectiveness and benefits in agile projects where speed is a priority. As the trend of increasing DevOps adoption continues, the integration capability becomes an essential feature for automation tools.

Test automation Testing Open source Continuous Integration/Deployment

Published at DZone with permission of Vu Nguyen. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How Observability Is Redefining Developer Roles
  • Fraud Detection With Apache Kafka, KSQL, and Apache Flink
  • How To Create and Edit Excel XLSX Documents in Java
  • Mr. Over, the Engineer [Comic]

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: