How to Build a Cross-Platform Matrix for Usability Testing

When developing an app, it’s vital that you build a testing strategy for it.

Some developers think in terms of Android or iOS only, but there is a myriad of other variables that should be considered and tested for. Especially with apps for mobile devices.

You may also like: Top 8 Usability Testing Methods

It Isn’t Only the OS

When you get your first beta testers, there is a good chance that they have a wide variety of devices at their disposal.

Developers should use all of them. Not only to see if the user experience works with the desired operating system but also to ensure that you’re getting the same experience on different types of devices.

Cross-platform testing is when you not only test across various devices but also operating systems. In the case of a web app, you’ll also need to test across multiple browsers to ensure that you’re getting the user experience that you’d like to see.

How Do I Test Everything?

In a word; don’t. You won’t have enough time or personnel to test everything. When you look at designing a testing model, it’s very important to look at a few factors:

• Who are your users?
• Which test platforms do you want to use?
• Limits on test cases.

It’s extremely important to know your users.

There’s a new trend in mobile app development that is iOS first. This makes sense for some apps because Apple users tend to be early-adopters and usually have a higher disposable income, so they are more likely to try out a new app and pay for additional features.

The thinking goes that you can get your app “out into the wild” and get a big enough user-base to highlight the major “show-stopper” bugs in your app. That should lead to a better experience for everyone. The problem is that most business users – and I’d argue the vast majority of devices – are run on Android. So you’re limiting a large segment of users at the start.

If you know the vast majority of your users use Android, you may skip the iOS from the start. On the other hand, if your users tend to work between a laptop and a mobile device, you may choose to design testing to cover web-browsers too. Here you’ll want to develop specific use cases to test how your users will be interacting with your app, and on which devices.

Testing Platforms

When selecting which platforms to test, it’s important to know that you will not be able to test them all. If it’s a browser app, there are a lot of different browsers on a lot of different machines running a lot of different operating systems.

The same is true for mobile devices.

Android and iOS are the big two, but those devices run on a wide variety of devices with a wide variety of screen sizes and that isn’t even taking into account that various versions of the OS that may be running on these devices. Don’t beat yourself up if a few users end up on a machine running some form of windows from the late 90s that you didn’t test for.

You will not be able to test for all variables. That being said, you can use the test cases that you’ve developed for your users to get a feeling of which platforms to test, and on which devices. You’ll also need to decide if you want to test on actual devices or if you can use an emulator to get the idea of the UX.

Limits on Testing

When designing your tests, it’s very important to define the scope of the tests. Which devices will be used. Which browsers or Operating Systems should be tested, and which versions of each should be supported. This will keep you from scope-creep and make sure that you can complete testing and roll-out in a timely manner. It’s really easy to get into a test-everything-mode. That’s not only expensive but as I’ve pointed out, nearly impossible from a practical standpoint.

You Know What to Do

A few other things that could save you both time and money:

• Clearly define your audience.
• Limit platform testing.
• Automate testing where possible.
• Use an emulator to save yourself from buying devices.

Platform coverage and testing design aren’t much different than any other type of testing work. You need to clearly define who your customers are, and dedicate resources to testing on the devices and platforms that they use in your defined use-cases.

Further Reading

What Is Usability Testing and Why Do You Need It?

13 Common Mistakes That Happen During Usability Testing