Exploratory Testing Tool Comparison: The Ultimate Guide

Exploratory testing is nothing new. In fact, industry leaders believe that all software testing is actually what we’ve been calling “exploratory testing” since Cem Kaner coined the term in 1984. Moreover, the concept of exploratory testing is now so mature that many of its core evangelists are retiring the term altogether.

Nevertheless, the recent trend of organizations adopting and scaling Agile development processes has driven a sudden surge of interest in exploratory testing. More and more teams are now looking to exploratory testing as a collaborative approach to Agile testing: a way to uncover more defects, faster.

This renewed interest in exploratory testing has set off a wave of innovation in exploratory testing tools. Sure, you could plan and track your exploratory testing with Excel spreadsheets, Word documents, and manual screen captures. But why would you, now that you can take advantage of technologies that make exploratory testing planning, documentation, and reporting a breeze?

If you’re looking to adopt or scale exploratory testing, here’s a comprehensive comparison of the top exploratory testing tools that you might be considering. We’ve broken down the comparison criteria into four categories—planning, execution, reporting, and compatibility—and defined each comparison criteria following the tables.

* DZone Editor's Note: The below tables are screenshots of the interactive tables available within the original article on the Tricentis blog. Visit the blog post to see the tables more clearly and interact with them.

Planning

Execution

Reporting

Compatibility

Exploratory Testing Tool Planning Criteria Definitions

Utilize Session-Based Test Management: The ability to use session-based test management to structure exploratory testing.

Manage Multiple Testers / Session: The ability to add more than one tester to a session to make exploratory testing applicable for large-scale implementations.

Invite Testers: The ability to invite testers to a session; the invite informs them about the session and provides a link that lets them join the session.

Reject Session Invitation: The ability to reject an invitation to a session; the rejection informs the session owner that a tester cannot join a session.

Write Session Charter: The ability to write a mission statement (testing goals, testing scope) for all testers of a session.

Write Tester Charter: The ability to assign additional test instructions to a specific tester in a session; this supplements the session charter.

Define Test Objectives: The ability to define test objectives (e.g. user stories) to set the scope of a session in an unambiguous fashion; this prevents testers from drifting into unfocused exploration.

Assign Test Objectives to Testers: The ability to assign test objectives across the team to distribute the testing workload.

Time-Box Session: The ability to allocate a fixed period of time to a session in order to prevent testers from overcommitting to testing tasks.

Attach Files to Sessions: The ability to attach any kind of files (e.g. specifications) to a session and enrich what is stated in the (session, tester) charters.

Configure E-Mail Notifications: The ability to configure e-mail notifications and specify who should be informed about certain session changes (e.g. a tester completed a test).

Configure Permissions: The ability to configure permissions to define who should have the right to perform certain tasks (e.g. delete a session).

Exploratory Testing Tool Execution Criteria Definitions

Capture Videos: The ability to capture videos during testing and link them to certain test actions; this helps document issues (e.g. bugs).

Trim Videos: The ability to trim recorded videos to increase focus and reduce the data volume.

Annotate Videos: The ability to add certain annotations (e.g. text, shapes) to videos; this helps document issues (e.g. bugs).

Record Test Actions Automatically: The ability to automatically record the user's mouse and keyboard interactions; this enables the testers to focus on testing, not on manual documentation.

Record Browser App: The ability to record the user's interactions with apps running in the browsers.

Record Windows Apps: The ability to record the user's interactions with any application on a Windows platform.

Record MacOS Apps: The ability record the user's interactions with any application on a MacOS platform.

Record Mobile Apps: The ability to record the user's interactions with a mobile (Android, iOS) application.

Define Interactions Limit: The ability to define the how many test actions should be saved; this helps reduce the amount of storage space needed.

Document Test Actions Manually: The ability to manually capture test actions; this lets testers enrich what the automated recordings.

Record Selected Applications: The ability to select which applications are recorded; this prevents recording the interactions with unwanted applications (e.g. someone checking Outlook in the middle of a session).

Edit Recorded Test Actions: The ability to edit all the test actions that were recorded automatically; this provides the tester more control over the resulting test documentation.

Merge Test Actions: The ability to manually merge test; this provides the tester more control over the resulting test documentation.

Record Audio: The ability to capture the tester's voice while recording a video.

Capture & Annotate Individual Screenshots: The ability to capture and annotate individual screenshots, and link them to certain test actions; this allows the testers to enhance the test documentation according to their needs.

Define Screenshot Quality: The ability to define the quality (high, medium, low) of screenshots; this helps reduce the amount of storage space needed.

Continue Scenarios: The ability to continue the documentation of a certain test scenario; this allows the testers to add/remove information to their test ideas.

Test Against Selected Test Objectives: The ability to let the tester specify which test objectives (e.g. requirements) his or her test ideas should be associated with.

Link Session to Raised Issues: The ability to link issues (e.g. bugs, feature requests) raised during testing to a session; this helps measure the effectiveness of a session.

Link Session to Test Objectives: The ability to link test objectives (e.g. user stories) to a session; this helps measure the overall session coverage.

Link Scenarios to Raised Issues: The ability to link one or more scenarios to issues (e.g. bugs) raised during testing; this allows other team members (e.g. developers) to see how these issues were found.

Link Scenarios to Test Objectives: The ability to link one or more scenarios to test objectives (e.g. user stories); this allows teams to measure the coverage of each test objective in a certain session.

Attach Files to Scenarios: The ability to attach any kind of files (e.g. design documentation) to scenarios so that the source of the corresponding test idea is stored alongside the associated scenario.

Attach Files to Individual Test Actions: The ability to attach any kind of files (e.g. specification) to an individual (failed) test action; this helps team members understand the expected result of that particular test action.

Flag Test Actions: The ability to flag test actions (e.g. question, note, concern) to allow the testers to more easily find these test actions during the debriefing with the session owner.

Exploratory Testing Tool Reporting Criteria Definitions

Flag Sessions: The ability to flag sessions (e.g. sprint acceptance test) so they are easier to find and retrieve later.

Playback Test Actions: The ability to playback all scenario test actions in the browser to allow others (e.g. developers) to reproduce an issue (e.g. bug) linked to that scenario.

Export Individual Media Files: The ability to export individual annotated screenshots and videos in order to immediately share specific results with others (e.g. session owner, developer).

Export Test Documentation: The ability to export the entire scenario documentation to a single file (e.g. pdf) to share these findings with people who don't have access to the related exploratory testing tool.

Browse Sessions (Search, Filter, Sort): The ability to search, filter and sort sessions so testers can more easily find the sessions they are associated with.

Monitor Session Status & Results: The ability to continuously monitor the overall session status and its results for real-time insight.

Write Test Summaries: The ability to write test summaries lets testers to provide an execution summary that the session owner can review during the session debriefing.

Close & Reopen Session: The ability to close a session lets testers define a clear end-state to a session.

Reschedule Session: The ability to reschedule a certain session creates a follow-up session with the same session setup (e.g. same testers, same test objectives).

Export Results for Test Automation Tools: The ability to export the results of an entire session, an individual testers or scenario; the exported results can be used as the basis for test automation.

Exploratory Testing Tool Compatibility Criteria Definitions

Application Type: How the application is packaged (as a standalone application or a browser plugin).

Supported Technologies: What types of applications can be tested. Windows includes any Windows-supported technology, including HTML and all supported browsers).

Enterprise Suitability: How well the application meets the process and technical requirements of enterprise teams (e.g., Global 2000 organizations).