Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Are You Using the Right Load Test Tool for the Job?

DZone's Guide to

Are You Using the Right Load Test Tool for the Job?

An in-depth look at Jmeter, Gatling, and Selenium to help you choose the right one for the job. Check out the pros and cons of each for your next test.

· Performance Zone ·
Free Resource

xMatters delivers integration-driven collaboration that relays data between systems, while engaging the right people to proactively resolve issues. Read the Monitoring in a Connected Enterprise whitepaper and learn about 3 tools for resolving incidents quickly.

The Flood platform prides itself on being tool agnostic and supports a number of leading open-source tools as well as it's own simple Test Builder tool.

JMeter and Gatling are purpose-built performance tools which generate load on the HTTP protocol level and then we have Selenium — which works differently in the way that it generates load on the Browser UI level. Both types of tools are very useful for performance testing but each one (like anything in life) has trade-offs. That trade-off is complexity vs. concurrency.

A lot of today's web applications are significantly complex — try visiting amazon.com with the Chrome Developer Tools add-in enabled on the network tab. You can easily view every request that the browser makes on the front page for images, dynamic content (XHR), javascript libraries etc. and as of the time of this writing contains 349 individual requests.

If we were using JMeter or Gatling to simulate simply going to the front page, we'd have to script every single request and worry about correlating session ID's etc. There's more bad news than good news with this. The bad news is it will take even the most expert scripting wizard a significant amount of time to script this. Additionally, if anything changes — e.g. code is written and functionality on the page changes — guess what? You'll need to update your script!

The good news in Selenium, you would focus on interacting with the page objects themselves and not worry about the individual requests — which makes it a significantly easier scripting task. The bad news is with this method - concurrency suffers due to the number of resources a single browser level user needs to have available. At the present time, a single Flood node can only support 5 concurrent Selenium users as opposed to approximately 1,000 Jmeter or Gatling users per node, a significant difference.

Now, different technologies aside, let's take a look at each tool's advantages and disadvantages in terms of fit for purpose.

The Test Builder built into the Flood user interface is a great tool for very simple HTTP based load tests. Its aim is to get up and running very quickly for new users without needing the knowledge to script a test in JMeter, Gatling or Selenium.


Pros

  • You can very easily add the URLs you would like to test.
  • Easily configure the load test parameters such as number of users, AWS region, ramp-up time and test scenario duration.
  • Run a simple test within minutes.
  • Can build simple tests and then export to Jmeter for further advanced scripting techniques

Cons

  • Not suited for advanced scenarios such as using parameterization or using dynamic data sets.
  • No support for Browser Level Users

JMeter is arguably the most feature-rich and powerful open-source load testing tool available at the moment. It allows you to build upon basic tests created with our very own Test Builder with more advanced techniques such as:

  • Full parameterization of variables
  • Capturing and verifying a wide variety of responses for manipulation & re-use
  • Enables the development of quite complex scenarios
  • And many more features

Pros

  • Open-source.
  • Platform independent (Java based)
  • Proven in Enterprise-level testing
  • Support for a wide variety of standard and 3rd party add-ins which enables you to cover almost all possible types of web performance testing scenarios.
  • Ability to create very complex load tests (if needed).
  • Ability to use a Ruby DSL code rather than the default tree view. This is accomplished by using the Ruby-jmeter gem which also allows you to run tests in Flood.

Cons

  • No support for Browser Level Users
  • Does not run client-side Javascript

Gatling is a well-proven and powerful open-source load testing tool. Using the intuitive Scala scripting language, it helps even first-time users get up and running fairly quickly.

It allows you to build upon basic tests created with our very own Test Builder with more advanced techniques such as:

  • Full parameterization of variables
  • Capturing and verifying a wide variety of responses for manipulation & re-use
  • Enables the development of complex scenarios
  • And many more features

Pros

  • Open-source.
  • Easy to use and descriptive Scala scripting language used to create tests
  • Proven in Enterprise-level testing
  • Ability to create very complex load tests (if needed).

Cons

  • No support for Browser Level Users
  • Does not run client side Javascript

Selenium is a very powerful open-source testing tool mainly used for automated functional testing via interacting with browser level objects. It can be very useful even for load testing as it allows users to re-use existing functional tests and run them with virtual concurrent users.

One of the main attractions in choosing Selenium over a protocol level load testing tool such as Jmeter or Gatling - is the ability to create your scripts in a user based functional flows rather than looking at how to simulate users on a protocol level. Protocol Level User's are often much harder for complex applications.

Pros

  • Open-source.
  • Supports a number of languages to develop scripts; including Java which is supported by Flood.
  • Ability to create tests using a 'functional flows' instead of diving into the protocol and coding every single request.
  • Ability for virtual users to execute client-side javascript.

Cons

  • As threads are GUI/Browser level - which uses up more resources - concurrency is impacted to 5-10 users per Flood load injection node.

Discovering, responding to, and resolving incidents is a complex endeavor. Read this narrative to learn how you can do it quickly and effectively by connecting AppDynamics, Moogsoft and xMatters to create a monitoring toolchain.

Topics:
performance ,comparison ,selenium ,jmeter ,gatling

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}