Selenium vs. JMeter: Which to Choose and When

Apache JMeter™ and the Selenium framework are two different tools with different scopes in the modern testing context. In this article, we will focus on the pros and cons of each tool and determine which of them is a key tool for which scenario. The title of this article seems to focus on a sort of conflict between these tools, but instead, I will analyze how these tools can work together.

With the growing adoption of testing and test automation in development projects, choosing the right testing tool will influence the entire project and results positively. Therefore, it's not uncommon to read testing tools' assessments and draw comparisons between them.

When reading tool comparisons you can draw relevant conclusions for your development process. For example:

• Which tool fits my project goal better?
• Which tool requires less training for my team?
• Which tool will have less of an impact on my organization's policies?

This blog post will compare JMeter and Selenium, and hopefully, you will reach the conclusion that both can be useful for you, for different use cases.

Both JMeter and Selenium are tools that simplify the testing process. They are both

• Created for web application testing
• Are relatively easy to learn and well documented
• Are developed in Java
• Are released through open source with the Apache License

JMeter is an open source load and functional testing tool for various network protocols, run as a Java desktop application with a graphical interface. Some of the supported protocols are: JDBC database connections, FTP, LDAP, web services, websocket, JMS, HTTP, RTE, and generic TCP connections. JMeter is developed as a pluggable and extensible tool suite with JMeter plugins, which enable advanced sampling and validation.

JMeter's functionalities are organized in components. Each component belongs to a specific category that has a role and is used hierarchically in the test script. JMeter scripts are developed in the form of a components tree in the JMeter GUI (you can also use non-GUI mode if you prefer scripting).

As you can see in the diagram below, the same type of element can have different scopes and can appear in different places throughout the tree.

The JMeter components fit into the JMeter GUI like this:

Selenium is an open source functional testing framework that automates browsers. Selenium provides multiple facilities for web application testing across different browsers and platforms. Selenium is not just a single tool but a suite of software. The two most important parts of this suite are Selenium WebDriver and Selenium IDE. The first one is a set of language-specific APIs for browser automation. The second is a Firefox/Chrome extension that provides a "record & replay" feature. Selenium scripts can also be recorded with the free Chrome Extension.

Selenium scripts are mainly developed as code, to obtain maximum advantage of WebDriver's API. This API has bindings on various programming languages. Each web browser & operating system couple require a different driver implementation. WebDriver is an abstraction layer that requires correct driver installation on the execution machine.

The user can write in any of the supported languages to Selenium WebDriver's API, which automates any of the supported drivers.

Now, let's see when we should use each tool. Let's look at when we have the following hypothetical web application architecture:

• User Desk - a generic personal computer or mobile device
• Web Browser - an application used for accessing information on the world wide web
• Front End - the layer seen in the world wide web, which is contacted by the browser
• Back End - the layer that supplies the services that execute the business vision of the application
• RDBMS - a data layer that returns data in SQL

(This schema is an oversimplification that takes into account technologies and not implementations).

This application needs testing and validation. Let's see which tool we should use in each case.

Web Browser testing is a validation of how the web application is presented to a user on web browsers. These testing phases are normally performed on more than one browser type, and therefore can be also called cross-browser testing.

Web browser testing verifies:

• How web pages are rendered
• How the user session is handled
• The Quality of user experience

Selenium is de facto the standard for these types of tests.

Suppose that we have following test requirement:

1. Open the browser at https://web.whatsapp.com/
2. Verify the scan code changes every 30 seconds

The smartest way to test this feature is Selenium. This is because we need to open the web page and monitor the scan code tag to ensure it has renewed the display code.

The code should fetch a tag with the scan code and monitor it via a Selenium object called WebElement. This object periodically fetches a new scan code every 30 seconds and compares it with previous one saved.

@Test
public void testScanCodeRenew() throws InterruptedException {

driver.get("https://web.whatsapp.com/");

By xpath = By.xpath("//div[contains(@class,'_2EZ_m')]");
WebElement scanCode = new WebDriverWait(driver, 10)
.until(ExpectedConditions.presenceOfElementLocated(xpath));

Supplier<String> getScanCode = 
           () -> scanCode.getAttribute("data-ref");

String oldCode = getScanCode.get();

for(int index = 0 ; index < 3 ; index++) {

Thread.sleep(30_000);

String newCode = getScanCode.get();

Assert.assertNotEquals(oldCode, newCode);

oldCode = newCode;
}
}

JMeter shows its best capabilities when involved in testing machine-to-machine interfaces. For example, WebSocket. WebSocket web applications can improve browser functionalities, by adding an Enhanced Application Client (typically in Javascript), which can setup a communication channel based on the custom protocol.

Even though WebSocket might be encapsulated into a web session and affect the browser, the user/Selenium will not realize it, so we will use JMeter for testing WebSockets.

Suppose that we have the following test requirement:

1. Contact the WebSocket endpoint at .
2. Send/receive the text request/response of unchanged messages.

The JMeter script will use one of JMeter's elements: the Peter Doornbosch WebSocket Sampler. The resulting script performs the following steps:

• Establishes a communication channel based on Secure WebSocket technology
• Holds the constant request/response rate
• Formats the request message dynamically
• Verifies the response message to match the request

This example describes quite an easy protocol (that sends back the message unchanged), but it can be a starting point to a more complex scenario:

• By modifying the "Format Message" step it is possible to customize message generation (e.g. using Groovy code)
• By modifying the "Message Assertion" and "Header Assertion" steps it is possible to customize message verification.

Modern web applications put a lot of emphasis on producing a user interface that provides a good user experience: it has to be visually appealing and clear enough to convey the message in just a few seconds. For example, interacting with draggable components that can be overlapped to obtain different groupings (e.g. a cart for e-commerce site).

Suppose we have the following test requirement:

1. Open a browser at https://the-internet.herokuapp.com/drag_and_drop
2. Drag and drop one of the two displayed elements over the other.

The smartest approach to test this scenario is Selenium because this feature is related to how components are rendered on the browser. JMeter cannot help because actions related to rendering components cannot be monitored.

The code will verify the position of draggable components before and after the "drag and drop" operation. The test will pass if the component positions in the resulting data structure switched when the test ended. For the "drag & drop" actions, you can use Selenium's capability to execute Javascript code directly into a WebDriver object, by using WebElement objects as arguments.

@Test
public void testDragAndDropWithCheck() throws InterruptedException {

driver.get("https://the-internet.herokuapp.com/drag_and_drop");

By css = By.cssSelector("div[id^=\"column-\"]");
WebDriverWait wait = new WebDriverWait(driver, 10);

Supplier<List<WebElement>> fetchComponents = () -> wait
.until(ExpectedConditions.presenceOfAllElementsLocatedBy(css));
/**
 * Starting check for element position
 */
List<WebElement> startingCheck = fetchComponents.get();

Assert.assertEquals("Starting - Draggable number does not match!", 2, startingCheck.size());
Assert.assertEquals("Starting - A position does not match!", "A", startingCheck.get(0).getText());
Assert.assertEquals("Starting - B position does not match!", "B", startingCheck.get(1).getText());

int index = ThreadLocalRandom.current().nextInt(startingCheck.size());
WebElement from = startingCheck.get(index);
WebElement to = startingCheck.get(1 - index);

JavascriptExecutor jse = (JavascriptExecutor) driver;

jse.executeScript(
        dndScript() + "simulateDragAndDrop(arguments[0], arguments[1])",
        from,
        to);
/**
 * Ending check for element position
 */
List<WebElement> endingCheck = fetchComponents.get();

Assert.assertEquals("Ending - Draggable number does not match!", 2, endingCheck.size());
Assert.assertEquals("Ending - A position does not match!", "A", endingCheck.get(1).getText());
Assert.assertEquals("Ending - B position does not match!", "B", endingCheck.get(0).getText());
}

Back-end systems export business logic via the Web Service. The same back-end can perform this operation to various front-ends (e.g. Web and Mobile). This layer is typically an example of machine-to-machine interface, therefore JMeter is perfect for this type of testing.

Suppose that we have the following test requirement:

1. Send a JSON payload to https://reqres.in/api/users to create a new user
2. Verify that the response payload now contains a correctly created new user

The JMeter script will simulate a user querying with a REST message. The script uses a standard HTTP Sampler to send a POST message with a JSON payload.

JSON

The JMeter script can use JMeter's template feature with the following steps:

• Replacing the SOAP request with REST, so the XML payload is replaced with a JSON payload
• Using a JSON Assertion to verify the response status, instead of an XPath assertion, to validate the JSON payload fields

A tip when developing REST test cases: when defining assertion steps but there is no precise value to assert, try using a smart regular expression, and match a string instead. In the example above I used the following regex, which you too can use in your scripts:

• An id attribute with integer value: the regex is [0-9]+
• A createAt attribute with a date field formatted as ISO 8601: the regex is [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}Z

Now that we've seen in which cases you should use each tool, let's compare JMeter and Selenium in two parameters:

• Technological knowledge prerequisites
• The ability to create testing suites

The number of developers working on a test automation project can change, depending on the activity level of the project. Therefore, it's important to understand how complex it is to learn to script in each of these tools and to maintain their scripts. Of course, the more we automate, the less we need to train people.

Selenium Required Knowledge

Selenium-based testing requires knowledge of WebDriver API. Moreover, it's necessary to be proficient in the chosen developing language (e.g. Java, Python, C#, etc), developing a toolchain (e.g. Maven, Ant, MS Visual Studio, etc) and operating a testing framework (JUnit, TestNG, unittest, NUnit, etc). It's possible to have a "Test Library" module for simplifying the procedures, but that also requires training on. Selenium is also suitable for BDD.

JMeter Required Knowledge

JMeter-based testing requires knowledge of the JMeter component organization, or at least ten basic components. New developers must be trained on environment setup and the required plugins according to their test project goal. However, because JMeter has a GUI and doesn't require scripting, it's easier to run and learn.

When writing tests, test cases are not enough. Rather, developers and teams should have a test suite that aggregates test scripts to guarantee test coverage. This is also helpful for test automation.

Automated test suites:

• Aggregate multiple test scripts
• Simplify test script selection by using custom languages (e.g. XML, YAML, etc)
• Use logic to execute tests in a certain order
• Use logic to conditionally execute (or not) test scripts
• Can execute a test script using custom parameter values
• Produce a human-readable report for test execution.

Not JMeter nor Selenium have built-in test aggregation abilities. However, there are solutions:

The Selenium test suite can be organized in a TestNG XML suite. As the name suggests, it's an XML file that describes an entire Selenium test suite/scenario to be executed. The resulting file can be shared among test machines, granting they have the same test coverage logic. To obtain this it's necessary to use TestNG as a testing framework and a Selenium test as a TestNG test. This is an easy operation but it can become expensive if there is a large number of test cases to be migrated/adapted.

The JMeter test suite is a group of JMX scripts. A better way to aggregate these scripts is to use Taurus, which can handle them with YAML file. The JMX script files do not need to be refactored and they are ready to be aggregated into a YAML file. It's possible to customize parameter values passed to each JMX script in the suite. Moreover, it's easy to split activities:

• Develop a JMX script as usual (e.g. JMeter GUI)
• Aggregate JMX scripts with a YAML file in Taurus

Let's see.

No. Selenium can be used for some performance testing (e.g. limited bandwidth testing), but it cannot be used to load test with accuracy like JMeter. For example, Selenium cannot generate a controlled number of request for seconds because it does have an equivalent of JMeter's Constant Throughput Timer. Moreover, using a target request per second in JMeter requires fewer resources from the execution machine, because Selenium tries to instantiate many web browser processes.

No. JMeter can record detailed web sessions originated by web browser applications (see Test Script Recorder and BlazeMeter Chrome Extension) and replay the web session as the server sees them. However, each replay does not reproduce client side elaboration (e.g. Javascript logic in web browser) or web page rendering. So if our test is focused on how a web page is presented to a user, Selenium is a better choice.

Yes. There are various situations where both tools can sync on the same testing application. We will describe some of them briefly.

The WebDriver Sampler plugin is a reference plugin to add a Selenium test case in a JMeter script. It works with some limitations related to resource consumption, but if there are only a few iterations, it has good practical trade-offs, including easy configuration.

Another case could be executing data-driven testing from JMeter on multiple Selenium WebDriver instances. Didn't we say it was resource consuming? Yes, but with certain requirements and hardware it's an achievable goal. Of course, it's not so easy and it requires developing to obtain a testing layer between JMeter and Selenium, to handle multiple instances setup and them concurrent usage.

It's also possible to convert a Selenium test into a JMeter test using the Test Script Recorder. Practically speaking is required to execute a Selenium test through a proxy and this proxy is the Test Script Recorder.

BlazeMeter also offers recording and creating a synced Selenium and JMeter script, which can be run in parallel. In addition, BlazeMeter has a new end-user experience feature, which enables you to put in your URL and see what users see when your site is under load.

Regardless of your role in test automation and development, both JMeter and Selenium are important and useful tools to be used during test project planning and execution. I hope this blog post helps you choose the right tool for you and your team, according to your project goals. You can also integrate and sync these tools, with open source frameworks like Taurus. So start testing now!

To get the free recorder that records and creates JMeter and Selenium scripts, click here.

To learn about Taurus, click here.

To get a BlazeMeter demo, click here.