Definitive Guide to Selecting Your Continuous Testing Tools
See what considerations and criteria you and your team should have for your next testing tools.
Join the DZone community and get the full member experience.Join For Free
As a software developer and testing expert with over 20 years in this industry, I've seen tools and solutions grow and disappear from the market. That's OK, and that's the lifecycle and "price" of innovation. Unfortunately, I have also seen some wrong practices when it comes to choosing such tools, especially around the testing domain.
You may also enjoy: Ask Yourself These Questions Before Selecting A Testing Tool
In this post, I would like to make some order in what needs to be top of mind for managers and practitioners when trying to either change and existing solution or pick a new one.
Focus on the Process
I cannot stress enough the importance of the existing processes that are being followed within the organization that is evaluating a new tool. "Process" includes the cadence of releases, as well as tester, developer, and business collaboration. Process also means Agile vs. Waterfall vs. other methods like ATDD/BDD, as well as the quality and the definition of done.
If the teams aim to release on a weekly basis, and their test automation suite is either unstable enough, and if the lab won't be able to cope with the size of the tests and platforms, or the time it takes to analyze a regression test report is too long, this needs to impact the decision process as well as the evaluation criteria process.
In most organizations that I've met, the teams are distributed and share different skills in both development and testing. This is also a driver to the way test development and execution is processed.
Consider Overall Organizational Objectives
Teams need to ask themselves what good looks like. Do they aim towards a "right" test automation coverage of over 80%? What is the maximum time that they have to test within each iteration or sprint? Before asking the questions around what the tool can do for you, teams need to feel solid about their objectives and where they plan to grow their teams.
Just doing a checklist of tools functionalities isn't relevant, and cannot stand over time since the domain is changing constantly; the tools sometimes become obsolete and your own objectives might change as well.
Bottom line: "Don't let the tool functionality drive your selection criteria, but rather let your own objectives and process fit do it."
In addition, teams ought to be focusing as part of the continuous improvement agenda on measurements, metrics, and visibility into how they operate over time. For this, the organization needs to set objective metrics that match where the teams are aiming to be in 6 months, 12 months, and 2 years.
There are various market reports like the recent Google DORA DevOps report that was showing variance between Elite DevOps organizations and lower maturity organizations. These kinds of market objective metrics can serve as a baseline for setting the overall goals.
What Can the Tool Do for Me?
Only after making sure that the above questions are checked, and you have a good understanding of where you wanna be and what your team needs to be successful and meet their goals, then go and create the right list of selection criteria for your testing tools.
Such a list from my experience ought to include the 6 characteristics in the graphic below and be able to cover the functionalities of test creation, test analysis, and test environment (lab) for proper coverage and execution.
Some tools come with AI and ML capabilities, some have record and playback, some require advance coding skills. You should consider all and in many cases, probably combine a few tools to get to the goal you've set.
For each of the above categories, whether you're evaluating an open-source or a commercial one, you should focus on the test creation and analysis of each. Not all frameworks and tools were created equally.
- Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber)
- Provide assertion functions (Chai, Jasmine, Jest, Unexpected)
- Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma)
- Generate code coverage reports (Istanbul, Jest, Blanket)
- Provide a browser or browser-like environment with control of scenario execution, UI testing, and more (Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe)
- Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble)
Even these tools constantly shift and change as the market matures.
As highlighted in this post, it is about you and your team, not the tool you're evaluating. Try focusing on what your process needs to look like, see what objectives the business and management have for your projects, and then see how tools can fit into your processes. Don't follow other mistakes that list features and functionalities of their tools and make it your decision criteria, but rather come prepared with yours and see how tools help meet your goals. Don't get me wrong here, features are critical for the success and enable your test automation coverage, but not every tool also fits your entire processes that starts from the creation of tests through analysis and maintenance.
Published at DZone with permission of Eran Kinsbruner, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.