Speed Up Mobile Testing Cycles With Test Sharding

It’s obvious that companies nowadays are investing more in automation solutions to streamline the workflow of build, test, deploy, and monitor mobile apps. As developing high-quality apps requires lots of intensive testing, companies will most likely highlight speed as one of the cornerstones when evaluating test automation tools and services.

With speed, you can either get the same amount of tests done in less time or cover more test cases within the same amount of time. It depends on your use case which one you want to achieve. When you parallelize your tests across hundreds of real devices, more devices and tests are covered in one single test run. But how to verify a lot of tests, say 100 or even 1000 tests, in less time? Test sharding is the answer.

Why Do Test Sharding?

Imagine that you have 1000 tests taking 100 minutes in total to complete on a single device. Wouldn’t this take too much time, keeping every team in the pipeline waiting for testing feedback? Now, what if we divide these 1000 tests to get them run on 10 devices (100 tests for each device)? It will theoretically take 10 minutes rather than 100 minutes to finish the test session and get the test reports.

In the end, the purpose of doing test sharding is to spread tests across devices and ease the testing workload so that you can execute automated tests faster, increase throughput, and scale continuous delivery with a short feedback loop.

How to Shard Tests on Bitbar Cloud

Bitbar Cloud comes with the ultimate scalability so that allows you to run tests across any number of real devices concurrently. As to test sharding, we have the out-of-the-box support for it. You can try it out with a free trial. 

With our Key/Value pair argument capability available from the Advanced Options when setting up a test run, you can use 2 new Key/Value pairs, numShards and shardIndex. And we will address handling these within the testdroid.properties -config file (available for editing by system administrator).

The test runner supports splitting a single test suite into multiple shards, so you can easily run tests belonging to the same shard together as a group, under the same instrumentation instance. Each shard is identified by an index number. When running tests, use the -e numShards option to specify the number of separate shards to create and the -e shardIndex option to specify which shard to run.

For example, to split the test suite into 10 shards and run only the tests grouped in the second shard, you would give the following value to key/value pairs in test options. This can be done either in our Cloud UI or through API.

numShards=10
shardIndex=2

P.S. If you are not in the scope of doing test sharding but would love to run a specific test case rather than the entire test case, we have the option as well. Simple define which test package or class you want to execute in the Advanced Option.

Device Availability

To get shorter testing cycles, there are two pillars. As you’ve seen, one is splitting tests and the other one comes to the devices. More specifically, device availability.

By the time you split tests across a few devices, you want to make sure every device is available so that all the shards can be executed simultaneously. If the tests are executed at a different time, then you are basically doing it sequentially and your so-called ‘test sharding’ will bring minimal benefits to you.

Bitbar Cloud hosts a public cloud that offers the highest number of unique device models with different SW and HW combinations. Beyond that, we are well prepared for the increasing demand of a single device model by providing several copies in our shared testing environment.

That being said, setting up a private infrastructure with a few dedicated devices is the best solution to maximize the device availability in terms of doing split testing. With access to this kind of private testing environment, you can distribute your tests to all of the connected devices to ensure the fastest possible test execution time and allow you to forward feedback to developers instantly.

To Sum Up

The nature of automation is to reduce the manual work that needs to be repetitively done and decrease lead time with expected outcomes. By using advanced techniques like test sharding, you can attain a more tightly closed feedback loop among DevOps teams.