One of the challenges in applying software test automation successfully in your projects is to select the appropriate automated testing tool or framework. It is a challenging decision, as you have many tools to choose from and a number of requirements to satisfy, and automated testing tools may have hidden problems that you overlook at the time of making the decision. Making the right tool choice is crucial to avoiding problems related to tools that haunt your project execution.
In selecting a tool, you have a set of selection constraints that can be classified into hard and soft constraints. Hard constraints are fixed and must be satisfied in any case. These constraints include platforms, executing environments, programming languages, and frameworks used to develop the target application under test (AUT). Soft constraints can be compromised or partially met. Such constraints include schedule, cost, maintainability, usability, the level of programming, and technical skills required. As these constraints are adjustable, you may find it challenging to balance them in order to select the best tool, given that most of the tools do not satisfy all of these constraints perfectly.
Tool Selection Criteria
To objectively evaluate a set of tools, it is recommended that you compare them based on the same set of criteria. You can define a set of criteria using your tool selection constraints and common characteristics of automation testing tools. Below are common characteristics that you may consider when weighing tools.
On which platforms does the tool being evaluated run? Platforms can be Windows, MacOS, Unix, Linux, Web, Mobile, etc. Some tools, such as Selenium and Maveryx support multiple platforms, while others like Ranorex run mainly on Windows.
AUT Programming Languages
Many tools support only a certain programming language used to write the AUT. For example, Abbot and Maveryx are used for testing Java applications. Many other tools like TestComplete and Ranorex can work with any languages used in the AUT provided that it can run on supported platforms.
Which languages does the tool support for writing test scripts? Many automation testing tools provide flexible scripting options, allowing testing teams to write test scripts in their most favorable languages.
It is important to know whether the tool is actively maintained, upgraded, and supported by the original vendor and communities of users and developers. You most likely need help from the vendor or communities during your project.
If the tool has not been released recently or the documentation is not up to date, it is possible that the vendor and the user community are no longer supporting the tool, or at least the interest in the tool has waned.
This criterion refers to how easy to learn and use the tool. It also indicates whether the tool is stable, robust, and efficient. Most software testing tools are easy to learn and use their basic features at first, but they require much more time to master advanced features, especially programming and maintaining scripts efficiently.
One of the most challenging problems in test automation is to maintain test scripts to reflect changes in the requirements of AUTs. This is a time-consuming activity in projects where requirements change frequently, which is the nature of agile projects. Some tools on the market such as Maveryx and QTP have capabilities to automatically, albeit not fully, update scripts when software is changed. At the present, however, most of the script maintenance is done manually.
Required Programming Skills
One selling point for many automated testing tools is to support non-programmers, allowing testers who have limited programming skills to do automation test effectively. Tools like Selenium, Katalon Studio, Ranorex, and TestComplete provide the record and playback features with scripts automatically generated. Still, some programming knowledge is needed to enhance and maintain test scripts, especially when you have complex test cases which have sophisticated scenarios and require many loops and checkpoints.
Automated testing approaches: Various approaches are introduced in existing automated testing tools on the market. Available tools follow one or many of the following:
Linear: This involves procedural scripts executing step-by-step from the start to the end of a test case. The linear scripts are typically generated by the tool’s record feature.
Record and playback: Tools have capabilities to record tests’ actions, generate test scripts, and playback test scripts automatically.
Structured: Unlike linear, this approach allows scripts to include control structures such as “if-else” and loop (“for,” “while”).
Data-driven: The test execution flow is driven by data stored externally, in a database, spreadsheets, or files.
Keyword-driven: The test execution flow is driven by keywords typically stored in tables mapping keywords and input data needed for the execution. Objects captured from the record and playback capabilities are used as keywords in many tools such as Katalon Studio, Selenium, Ranorex, and TestComplete.
Model-based: Test procedures or scripts are automatically generated using requirements and behaviors model.
Hybrid: Supporting two or more of the above. Most existing tools provide this hybrid approach.
Of course, cost is an important factor in deciding which tool to use. You have to take into account both licensing and support costs. Some tools require no or nominal cost to acquire, but they incur a large sum of money spent to call for external support. Additional cost comes from training and spending extra effort for understanding and solving problems related to the tool.
Quantifying Criteria’s Value
It is sometimes difficult to determine a winner when facing a number of choices with many similar descriptive or qualitative characteristics. Some criteria are more precise to quantify using a numerical scale than using descriptions. For example, the level of programming and technical skills required can be easily quantify using a rating scale between zero (none or no programming or technical skills required) to 10 (highest level or advanced programming and technical skills required).
Many of the criteria above can be quantified using the same range, such as, from 0 (none) to 10 (highest). For criteria with a definitive value no or yes, using zero for no and 10 for yes.
Putting Things Together
The table below provides a brief summary of the evaluation of seven popular testing tools using the criteria discussed above. Much of the ratings for support, usability, script maintainability, and required programming skills are based on my subjective judgment. You may have a different assessment, resulting in other rating values. Due to space limitation, the information on the tools shown in the table is not complete, and the costs of commercial tools can vary dependent on types of license and maintenance needs.
|Criteria||Maveryx||Selenium||Cucumber||Katalon Studio||TestComplete||Ranorex||UFT (QTP)|
|Platform||Windows, Linux, and Mac||Cross-platform||Cross-platform||Windows, Mac, Android, iOS||Windows (mainly), Android, iOS||Windows (mainly), Android, iOS||Windows (mainly)|
|AUT programming languages||Java||Web-based languages||Ruby, Java, .NET, Flex or web applications||Many||Many||Many||Many|
|Scripting languages||Java||Many (Java, C#, Perl, Python, etc.)||Ruby, Java, C#||Java/Groovy||VBScript, JScript, DelphiScript, C++Script, C#Script||C#, VB.NET||VBScript|
|Support||7 (very active)||9||7||7||9||8||9|
|Usability||8 (easy to use)||7||6||8||9||8||9|
|Script maintainability||6 (smart object detection)||3 (tool provides little support)||1 (almost no support by tool)||6 (object repository, Script editor, object re- identification)||7||6||7 (smart object detection and correction)|
|Required programming skills||7 (high level)||6 (support with record/ playback)||5||4 (record and playback, intuitive testing steps)||7 (strong record/playback capabilities)||6||8|
|Automated testing approaches||Data-driven, keyword-driven||Record/ playback, keyword-driven, data-driven||Structured, keyword-driven, data-driven||Data- and keyword- driven||Record/ playback, keyword-driven, data-driven||Record/ playback, keyword-driven, data-driven||Record/ playback, keyword-driven, data-driven|
|Cost||Free (community edition), from €940/license (professional edition)||Free||Free||Free (community edition)||From $999/license||From €609/ license, one year maintenance||From $8000/ license|
There are other criteria that you can include in your evaluation in order to meet your particular tool requirements. They may include performance, stress testing, reporting, and other certain capabilities. With these specifics, you can narrow down the most suitable tool that satisfies your needs.