10 Best Open Source Test Automation Frameworks for Every Purpose

At the beginning of 2018, we decided to bring together the best open source testing frameworks out there. This rating had to be diverse – we chose not to focus on one specific type of tools, but to include a variety – for mobile apps and load testing, for iOS and Android, keyword driven and behavior driven. The only common feature they share is that they are all highly reliable.

1. Selenium

Selenium is probably the most popular open source test automation framework for web applications. Selenium is a base for a vast amount of other testing tools.

• Cross-platform and cross-browser;
• Supports a wide range of languages such as C#, Java, Python, PHP, Ruby, etc.;
• Highly extendable through a wide rage of libraries and APIs to meet everyone’s needs and requirements;
• Flexible and maintainable;
• Possibility to write more advanced test scripts to meet various levels of complexity;
• One of the largest support networks existing;
• Provides a playback tool (Selenium IDE) for test authoring without the need to learn a specific scripting language.

2. Carina

Carina is a Java-based test automation framework built on top of the most popular open-source solutions (Selenium, Appium, TestNG) which allows to reduce dependence on a specific technology stack.

• Unites all testing layers: mobile applications (web, native, hybrid), WEB applications, REST services, databases;
• Supports all popular browsers (IE, Safari, Chrome, Firefox) and mobile devices (iOS/Android) – it reuses test automation code between IOS/Android up to 70-80%;
• As far as this framework is built in Java, it is cross-platform. Tests may be easily executed both on Unix or Windows OS;
• Framework supports different types of databases – both relational and nonrelational (MySQL, SQL Server, Oracle, PostgreSQL), providing amazing experience of DAO layer implementation using MyBatis ORM framework;
• API testing is based on Freemarker template engine. It enables great flexibility in generating REST requests and responses are dynamically changed by incoming arguments.

3. Google EarlGrey

EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests.

• Powerful built-in synchronization with the UI, network requests, animations, various queues, etc.;
• Tests are easier to write and simple to maintain;
• Synchronization ensures that the UI is steady before any actions are performed;
• all interactions occur only on visible elements;
• Test stability and high repeatability;
• Allows to manually implement customized timings if needed;
• Works in conjunction with the XCTest framework and integrates with Xcode’s Test Navigator to run tests directly from Xcode or the command line;
• Flexible design.

4. Cucumber

Cucumber is a Behavior Driven Development (BDD) tool which is used to write acceptance tests for the web applications.

• Quick and easy set up an execution;
• Allows reusing code in the tests;
• Both specification and test documentation are uploaded in a single up-to-date document;
• Can be helpful to involve in the process business stakeholders or other people not familiar with testing who can’t easily read the code as test reports are written in business readable English;
• Automation of functional validation in easily readable and understandable format;
• Cross-platform;
• Initially implemented in Ruby, extended to Java framework;
• Code can be used together with Selenium, Watir, Capybara, etc.;
• Supports languages like Python, PHP, Perl, .NET, Scala, Groovy, etc.;
• Focuses on the end-user experience.

5. Watir

Watir is a powerful tool for web automation testing based on a family of Ruby libraries.

• Tests are easy to read and maintain;
• Mimics the behavior of a user interacting with a website;
• Supports cross-browser testing;
• Supports data-driven testing;
• Integrates with BBD tools like Cucumber, Test/Unit, etc.;
• Supports your web app no matter what language it’s developed in;
• Extendable through a set of different APIs;
• Manual browsing is combinable with commands in Ruby;
• Ruby is a concise and flexible scripting language making it an ideal choice for automated scripts;
• As long as it’s written in Ruby, it gives you the power to connect to databases, read data files and spreadsheets, export XML, and structure your code as reusable libraries.

6. Appium

Appium is designed to test mobile applications. It’s built with the idea that you should not be recompiling your app or modifying it in any way in order to test it. You also shouldn’t be locked into a specific language or framework to run your tests.

• Easy setup process;
• Cross-platform – allows to test against multiple platforms (iOS, Android, Windows, Mac) using the same API;
• Reuses code between iOS, Android, and Windows test suites;
• Supports simulators (iOS) and emulators (Android);
• Supports native, hybrid and mobile web applications and desktop apps;
• Based on client/server architecture;
• By using vendor-provided automation frameworks under the hood there’s no need to compile in any Appium-specific or third-party code or frameworks to the app. This means you’re testing the same app you’re shipping;
• User is free to use whatever test runner and test framework you want; the client libraries are simply HTTP clients and can be mixed into your code giving the possibility to manage your test environment any way you like;
• You can create elements repositories and manage them after changes.

7. RobotFramework

RobotFramework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). Its core framework is written in Python, but also supports IronPython (.NET), Jython (JVM) and PyPy as well.

• Keyword driven approach to simplify tests and make them readable;
• Reusable higher-level keywords can be created from existing ones;
• Easy-to-use tabular test data syntax;
• Cross-platform;
• Rich ecosystem including variety of APIs consisting of generic test libraries and tools that can be developed as separate projects;
• Highly extensible natively – using Python and Java libraries, and by the means of different API’s;
• Can be extended natively using Python or Java;
• Other languages supported via a remote interface;
• Clear testing reports;
• Detailed logs;
• Easy integration.

8. Apache JMeter

Apache JMeter is designed specifically for load testing and can be used to test performance both on static and dynamic resources, Web dynamic applications. It simulates a heavy load on a server (group of servers), network, or object to test its strength or to analyze and measure overall performance under different load types.

• Ability to load and performance test many different applications/server/protocol types: Web, SOAP, FTP, LDAP, TCP, mail protocols, shell scripts, java objects;
• Cross-platform;
• Dynamic HTML testing reports that are ready to present;
• Complete portability and 100% Java purity;
• Caching and offline analysis/replaying of test results;
• Highly Extensible core (pluggable, scriptable samplers; timers to set several load statistics);
• Personalization is achieved through data analysis and visualization plugins;
• Functions can be used to provide dynamic input to a test or provide data manipulation;
• Integration through libraries for Maven, Gradle, and Jenkins.

9. Gauge

Gauge is an advanced lightweight cross-platform test automation tool. Gauge is built with a plugin architecture so that you can use it with any language, IDE, and ecosystem you want.

• Easy to get started – installed and initialized with a single command;
• Simple, flexible and rich syntax;
• Test specification can be written in Markdown – no specific structure is needed – documentation is in the format of your choice;
• Data driven execution (supports external data sources);
• Allows requirements to be written in a way that will be understood by everyone involved in a project because it provides the ability to author test cases in business language;
• Consistent language support (C#, Java, Ruby, Intellij);
• A modular architecture with plugins that is highly extensible;
• Test cases that are easily maintainable and understandable.

10. Robotium

Robotium is a test automation framework made to simplify the task of writing powerful and solid automatic black-box UI tests for Android. With the help of this framework, developers are able to write function, system, and user acceptance test scenarios covering numerous Android activities.

• Tests both native and hybrid Android apps;
• Can be used both for testing applications with the source code available and applications with unknown implementation details;
• Faster test case execution;
• Requires minimal knowledge of the application under test;
• Handles multiple Android activities automatically;
• Minimal time needed to write solid test cases;
• Readability of test cases is greatly improved, compared to standard instrumentation tests;
• Test cases are more robust due to the run-time binding to UI components;
• Integrates smoothly with Maven, Gradle or Ant to run tests as part of continuous integration.

These were the top 10 of test automation frameworks for the beginning of 2018. Some of them are brand new, some are well-known and have earned a good reputation through time. It’s always nice to automate the testing process to a certain scale in order to save time, effort and reduce the number of testing errors.

Do you have something to add to this list? What is your favorite automation testing framework from your personal testing experience? We would be happy to know.