Gartner's Voice of the Customer Report: 3 Key Takeaways

Gartner is a well-known name among prestigious research companies and has been a reputable source of reference for customers to evaluate products from various industries. This article will give you a recap of the three latest findings of the software industry, providing you with more insight into this business field, as well as its product offerings.

Best Test Automation Tools as of Q1 2019

Having recently announced the results of the annual Gartner Peer Insights "Voice of the Customer": Software Test Automation publication, Gartner seeks to give insights into the software test automation industry’s latest products. The goal of the report is to help product users gain better insights into different types of product offerings based on end-users’ reviews and Gartner’s validation criteria.

All Gartner reviews go through a strict validation process to ensure they are qualified, valuable, and authentic. To be selected as a Customer’s Choice, vendors are required to have a product aligned with the relevant market, receive an overall rating of 4.2 or higher out of 5 stars, and have at least 50 approved customer reviews.

Without further ado, the top choices of the Gartner Peer Insights ‘Voice of the Customer’ in the Software Test Automation industry are as follows:

Katalon Studio

Katalon Studio, developed by Katalon, is a powerful automation solution with a comprehensive and integrated set of features for Web, API, and mobile test automation that users can implement into their CI/CD pipeline. Katalon's most popular product – Katalon Studio – is considered a tool that is easy to use, rich with features, and user-friendly for both novice and experts. Katalon Studio also supports a wide range of software methodologies such as data-driven testing, behavior-driven integration (BDD), CI – DevOps practices, and more.

“Easy To Use, Good UI And The BEST In The Industry Of Automation.”— Product Engineer, QA in the Manufacturing Industry

Visual Studio

Developed by Microsoft, Visual Studio IDE is commonly recognized as a quality integrated development environment (IDE) that is perfectly compatible with .NET technology. It is even considered a great alternative to existing lightweight IDE, and a complete tool to develop applications that are based on Web, cloud, system, and data.

“Visual Studio IDE is the only IDE you need.”— Software Developer in the Services Industry

Selenium WebDriver

Selenium WebDriver is known for being the most popular free and open-source test automation framework. According to the peer reviews, Selenium WebDriver is commended for its flexible support of multiple system environments, browsers, and programming languages including Java, Python, C#, and PHP.

NeoLoad

NeoLoad, developed by Neotys, is a testing platform for enterprises to automate tests from APIs, mobile, to web applications. Peer reviewers consider NeoLoad an “excellent and great tool” with a strong feature set to analyze test results. The software is also commended as “a very efficient and up-to-date-interfaced platform.”

Parasoft

Parasoft provides innovative tools that automate time-consuming testing tasks and provide management with intelligent analytics necessary to focus on what matters. Parasoft products, including Parasoft Development Testing Platform, Parasoft Virtualize, Parasoft SOATest are recommended for their flexibility, comprehensiveness, and rich features such as integration with Selenium WebDriver, XML messages, functional API testing, and so on.

Unified Functional Testing

Developed by Micro Focus, Unified Functional Testing (UFT) is praised as a top-notch tool for web-based and API testing. The paid software is considered a value automation solution that is also easy to use and implement. UFT is also packed with versatile features, as well as the technical support services.

Tricentis

The Austrian software company is known for the two most popular testing products: Tricentis Tosca and qTest, which provides solutions to optimize Agile and DevOps. Peer Insights reviewers recognize these products as a “scriptless automation solution” and a great tool for ERP testing as well as many other technologies.

Apply Shift-Left Testing to Deliver Higher Quality Software Faster

“Application leaders responsible for developing digital business applications are under increasing pressure to maximize quality and velocity of delivery,” according to QASymphony, a software automation enterprise. Adopting a “shift left” testing approach is no longer an option, but a requirement for development teams to release products with high quality faster.

There are three major challenges that software development teams are facing throughout the software development life cycle. First, conventional software development teams tend to focus on coding – which is the first phase of a development cycle – instead of testing these features. This likely causes code instability and decreasing developer efficiency.

Second, it is costly and time-consuming for teams to manually test and integrate with other tools, let alone perform these tasks early and often. And third, user-interface-driven testing is also slow, fragile, and does not provide full test coverage.

Adopting a “shift left” approach to testing is the key to tackle these challenges. The “shift left” model means teams should start testing features at the beginning stage and throughout the entire development cycle. In other words, “test early, test often.”

According to the report, application leaders in digital business should, first, combine coding and testing into one activity; and perform this activity at the beginning of the development cycle. Second, team leaders and testers should make it easier for developers to receive feedback about the quality of changes by implementing continuous integration (CI) and automated testing.

Finally, the report proposed a restructuring of the Test Automation Pyramid (Figure 1) to allocate automation efforts and yield maximum benefits.

“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.”

Critical Capabilities for Automation Software Testing in 2019

While the software automation industry is waiting for Gartner to release the list of critical capabilities for 2019, application leaders can still refer to the 2018 report to evaluate which functional software test automation offerings are best for their testing purposes.

These critical capabilities are also evaluated based on five common testing use cases: Continuous Testing, Open-source-based Testing Acceleration, Progressive Web Apps/Responsive Web/Native Apps, API/Web Service Testing, and Enterprise End-to-end Testing.

1. Match to nontechnical role skills

This criterion measures how suitable and user-friendly the product is, not only to test automation experts, but also to other roles with limited to no testing experience in the team.

2. Match to technical role skills

This criterion evaluates the product’s ability to help test automation experts – including testers and developers with proficient coding skills – enhance their productivity.

3. Templates and accelerators

Whether and how well the product supports industry verticals and/or vertical applications through templates and accelerators.

4. Change impact analysis
This measures how effectively the product helps identify sets of relevant test cases that need to be executed when there are changes in requirements and/or functionalities. This criterion also considers other factors such as risk, criticality, or business priority.

5. Cross-platform and cross-browser support

This criterion addresses the range of platforms, browsers, and environments on which the product can operate, including Windows, macOS, Linux, Android, and iOS.

6. Intelligent automation

Measures how well the implementation of artificial intelligence (AI), machine learning, self-healing heuristics, bots, and automatic (re)generation of tests the product offers.

7. Breadth of technology support

This criterion evaluates the product on its coverage of different technologies, including UI frameworks and widgets, API formats and protocols to allow end-to-end testing.

8. Integration Capabilities

How well and how much the automation product integrates with other tools and environments, including IDE, CI/CD, defect management, and ADLM.

9. Open-source support

Whether and how well the product supports well-known open-source tools and frameworks such as Selenium, Appium, JUnit, and so on.

10. Dashboards and reporting

This final capability assesses the product’s ability and effectiveness in providing tracking tools after the tests have been executed.

References

• Gartner Peer Insights Customer’s Choice

• Best Automation Testing Tools for 2019 (Top 10 reviews)

• 4 New Gartner Research Reports on Software Testing — Tricentis blog