Selenium Grid Performance Problems

You can run your Selenium-based test automation scripts on the browser you want and distribute it with the Selenium Grid project. This is actually a very simple and easy solution.

According to the architecture, thanks to the Selenium grid project, you raise a hub and register different machines to this hub as a node. In this way, the hub directs you according to which browser or operating system you will run. If that node is used by a different test, it will queue you up and continue running the next test when it is finished.

I can hear what the experienced test engineers are saying. That job won’t be that easy. In teams with more than 100 test scenarios, Node management on the Grid project is unfortunately not as capable as it seems.

I have listed the main problems encountered as follows:

• Faulty tests hanging on nodes. 
• If too many nodes are connected to a hub, experiencing interruptions. 
• If there are too many tests in the queue, it will timeout after a while or cannot be forwarded to the nodes. Severe slowness with too many test runs.

What I see most fundamentally is that the current Grid project is not designed properly for heavy load. Besides, it does not have a stateful structure. So if the hub crashes, everything starts over.

Several projects were developed that solved this. One of the most popular is the Zalenium project developed by Zalando, but the main problem here is that the distributed structure has limits. In addition, if it experiences similar problems in real instances due to its Docker-based operation, the access will be completely lost again.

One of the alternatives you can use for Selenium Grid management is Testinium. Although Testinium is a cloud-based product, you can use private cloud and on-premise versions. One of its most beautiful features is that it manages the queue structure on itself instead of Selenium Grid. Due to the Selenium Grid structure, it constantly checks for all tests waiting in the test queue and becomes unresponsive after a while.

Testinium, on the other hand, directs the tests that can only run on the Selenium Grid according to the number of parallel runs you specified, and loads only the test run load to the Grid. Thanks to its internal reservation structure, requests waiting for busy nodes are kept in the queue.

There are test limits that you can put in the test queue for products that provide the Selenium Grid service as SaaS. When you send tests triggered by a CI tool like Jenkins to a product without queue management, your tests that exceed the limit will be rejected. Testinium does not set a limit for the tests you can send to the queue. You can send your 10,000 scenarios to the test queue and run these tests on nearly 500 nodes.

In addition, there are mechanisms that perform timeout management and healthcheck controls on nodes. This way, even if a node is broken, if there is no test on it, it can be reconnected.

For example, with a bank we work with, we can run and complete nearly 200 nodes and more than 10,000 tests with this system.

Different features in Testinium are as follows:

• Queue management 
• Real mobile device management and management against device disconnections 
• Wipe feature to delete private information on mobile devices 
• Ability to take screenshots automatically 
• Video runs of all tests 
• Docker support 
• Jenkins plugin 
• Comparing in process test runs with past versions