Mobile Load Testing: The Next Phase in Mobile Testing

I was at the O’Reilly Velocity and STPCon performance/testing conferences recently, where I spoke about a process you can follow to develop a testing plan for your mobile app. This process is based on gathering information that you need in order to:

1. Know your users
2. Know your app
3. Know your matrix
4. Know your devices
5. Know your performance
6. Know your automation
7. Know your edge

Mobile Testing Requires More Than Just Functional Testing

When I got to the “know your performance” section, I presented new insights about mobile app performance, including when the app is under load.

We’ve all used apps that performed poorly due to some back-end issue, leaving you staring at a screen that is either blank, partially updated, or showing just a spinning icon to hypnotize you so you don’t realize how long you’re waiting.

Mobile testing requires more than just functional testing. It’s not enough to say that all of the functions in the app work. They have to work quickly, especially when there may be hundreds of thousands to millions of people using the app at the same time.

Maybe your app is going to be used during the busy holiday shopping season. Or on April 15th, when everyone is filing taxes at the last minute in the USA. Or during the Olympics to get the latest stats on the current game and the winners and losers. Or when millions of kids are trying to catch Pokemon. Whenever your app is going to be used, it has to be ready, along with any back-end services, to deliver content quickly and reliably.

Read: 22 mobile stats everyone should know

How to Test in the Third Dimension

After building out your test matrix of tests to run on specific devices, add another dimension to the matrix for the different levels of load under which the app will be tested, to compare performance characteristics. That’s the z-axis on the cube.

You want to know how the app will perform not just when there is no load on the back-end systems, but also when those might be slow, or delayed, or even unavailable.

Some of the load levels you might use for testing include:

• No-load: How does the app perform functionally with no back-end load?
• 10 virtual users: Same test as above, but now with minimal load. This is a smoke test for performance.
• 100 virtual users: Is the back-end system able to support this minimal load with no impact?
• 1000 virtual users: Does the app perform as well or are there now app waits for connections, data, response?
• Max-virtual users: Growing to whatever limit is needed to meet expected demand at peak use.
• 2x or 3x or 10x Max-virtual users: Even better to go beyond that level to find the system stress-point and test what might be possible if demand was under-forecast and response is greater than expected. Remember: It’s much better to have that as a best-case scenario than worst-case.

Takeaway

Functional testing for mobile is not enough to ensure that your mobile visitors get the same fast, reliable user experience your desktop users do. Apply the seven steps to pragmatic mobile testing to improve your functional test automation, as well as adding load testing appropriately to test the impact on performance and functionality.