How to Test Mobile App Performance: 3 Key Components

You’ve probably interacted with an app on your phone or tablet that’s slow, takes a long time to load, freezes, or even crashes on you altogether. 

Frustrating, right?

On the flip side, you can probably think of an app that you love to use because from day one, it’s never given you any trouble.

Or maybe you never paid any mind to an app that works quickly, because isn’t that how it’s supposed to be?

So, what causes one app to be crash-prone and another, fast, and reliable? 

Whether an app has good or bad performance depends on three factors: the backend, the network, and the app itself running on the device.

A developer or mobile tester can measure the performance of an application in different scenarios.

For example, they can test for when there’s a concurrency of users on the app at the same time, on different devices (which vary in hardware resources and screen sizes), and multiple networks such as 3G, 4G, Wifi, and more.

The reality is that many variables affect the performance of a mobile application. Moreover, a user may have a very bad experience with your app and the cause might not even have anything to do with the code or its implementation.

But, by running performance tests for each of these three factors, you’ll be able to identify problems and optimize your app for the best user experience possible. 

Keep reading as we’ll cover the different types of tests for each factor, what to measure, and what tools are available to help you along the way. 

1st Mobile Performance Factor: The Backend

A mobile app’s backend architecture is generally based on an application server, a web server, and a database. 

When it comes to the backend, the things related to performance that are important to know when an app is under load are the server’s response times, database queries times, and the server’s resource usage. 

Using this information, it’s easier to detect issues such as: 

• High server response times
• Bottlenecks or breakpoints in the database and application server resources
• Poor implementation of escalation policies 

So what kind of tests are normally run to check the app’s backend performance? Load tests.

This is when you simulate load on the backend in different ways, whether it be through stress testing, peak testing, endurance testing, load testing, etc. 

In general, the objective of these tests is to understand how the backend systems of an app behave and handle a certain volume of concurrent users.

Several tools allow you to load test your mobile app. The most commonly used ones include:

Apache JMeter – the number one open-source load testing tool

Gatling – a developer-friendly, open-source load testing tool with scripts written in Scala

BlazeMeter – a cloud performance testing platform that scales your JMeter or Gatling tests for a greater amount of concurrent users

2nd Mobile Performance Factor: The Network

With regards to the network that the device is connected to, there are two key things to measure: latency and bandwidth.

• Latency is the time that elapses when information is sent on the network (measured in milliseconds).
• Bandwidth is the maximum capacity (the amount of data) that can be transmitted through the network (measured in bits per second). 

An app’s performance can vary depending on, for example, whether it’s connected to a 3G network or a 4G network, and unfortunately, this is beyond an app developer or tester’s control. 

But, it is possible to incorporate the network during the mobile app performance testing process, simulating the different types of networks and measuring their impact on the response times, both on the server-side and the client-side.

Luckily, the third factor is something that’s been overlooked for years and we now have ways to incorporate it into our test cycles and gain greater control over it.

3rd Mobile Performance Factor: The Client

For a mobile app, the client is the software that runs on the device, the app itself and it’s very important to know how it will behave there, as it is sharing resources with the other apps installed on someone’s phone and is a huge factor in the user experience.

At Apptim, we’ve seen firsthand, companies lose money due to poor client-side performance.

“We track everything, and we know that each year we are losing $100M a year due to app crashes and screen freezes that are not being caught by our test automation suite…”

-Mobile Team Lead at Major Rideshare company

This means that even if you’re doing functional testing, incorporate automation, or even have CI/CD in place, these bugs could still slip into production that are performance related.

It becomes really challenging thing for businesses to start thinking, “How can we improve this? What type of tests do we need to do early on in our development and testing cycles, so we can eliminate this regression earlier?”

So, enter client-side performance testing.

But, it’s tricky… Why?

With client-side testing, the variety of devices, operating systems and screen sizes of a device that an app is running on comes into play.

Today there are thousands of mobile devices on the market that vary by brand, hardware, and operating system. 

And each year, it becomes more and more difficult to ensure that an app works well on most of the devices on the market because the amount only continues to grow. 

This image below shows the different Android versions in use over the past seven years: 

And here are the different versions that were used in just 2019: 

Testing client-side performance allows you to understand how the app behaves on a device and how it uses the resources that it shares with other apps.

Running tests for client side performance, you’ll also be able to find areas to improve that you do have control over (unlike the network), resulting in a better user experience.  

At Apptim, we consider it critical that any mobile app development team integrate client-side performance testing into its test cycle.

There are several areas with different metrics you can gather to understand client side mobile app performance:

Device resource usage (How the app uses the resources of the device it’s running on):

• % CPU 
• Memory (Mb)
• Data sent and received by the application (Kb)
• % Battery usage

Rendering (Measures how the app is drawn on the device):

• Render time (ms)
• Frames per second (fps)
• Render lag (janks)

To give you an idea of what would be acceptable or at least a recommendation, in order to achieve a fluid rendering, which is set at around 60 frames per second, each frame of an app needs to be completed in less than 16 milliseconds.

If that’s not the case, the app can create a disruption in the animation and end up freezing.

Of course, it depends on what type of app.

When it comes to gaming apps, the requirements are even higher, but for other apps, you should actually measure how your app rendering times are; how many frames per second it’s drawing. With this information, you can identify if there’s any disruption in the animation and how the experience of the user is.

On the other hand, if it’s excessively drawing the screen, it will require high CPU or GPU which leads to a battery drain. This is why it’s important to be careful with managing the rendering of your app in the device itself.

Errors:

• Exceptions
• ANR (Application Not Responding) in Android

For Android specifically, if an application stops responding, typically a message will appear on the screen with the option to either wait or close the app.

This is generally due to the fact that certain events may not execute in a specific period of time. For example, if you click on an input and there’s no response after five seconds and the app takes it as an ANR, it will show you the message.

Transaction response time (how long it takes to complete an action):

• Server response times 
• Database response time 
• Network latency
• Screen drawing time

The transaction response time can be measured from the user perspective, but it’s not only what’s happening on the device.

It takes things like the server response time, which can be measured with a proxy, for example. It will also include, if there’s any database call, it will include the SQL time as well. It will also include any network latency and of course the rendering times.

If you think about the time that a transaction takes to be completed, you should consider that there are different parts that add to the transaction time. If you want to improve it, you need to measure each of them and see where the opportunities for optimization are.

For many teams, one key area that they focus on regarding this is the app startup time, or how long it takes to open when clicked on. This is something that you can measure improve over time.

Once you’ve got all of this information, you can uncover critical problems such as:

• Memory leaks: when an app doesn’t free up memory well, it increases until it runs out of available memory and then crashes 
• Screen freezes: due to very high rendering times
• Crashes: exceptions that jump in the app and cause it to suddenly close… bummer!

Should we use emulators or real devices?

I always recommend testing mobile app performance on real devices. Why?

Because that’s when you get the actual performance of your app, if you think about emulators, you’re going to be using emulated hardware, so it’s not true to what will happen on the real devices of your users. That’s why its always recommended to use real devices for client-side mobile app performance testing.

This leads to the next question, what tools can we use for the client-side?

For native apps, there are profiling tools such as Android Studio and Instruments (for Android and iOS respectively). 

These tools are of great help when debugging an app and finding the root of certain kinds of performance problems.

In addition to Android Studio and Instruments, I recommend using Apptim to give entire development teams access to mobile app performance information and KPIs, where anyone can quickly and easily run tests.

With Apptim, you can test all of your native apps’ performance with just one tool, instead of one for Android and one for iOS and also compare performance over time. 

Test mobile app performance as early as possible

Now, you may be wondering, “At what point in the development cycle should I start running client-side performance testing?”

If you ask me… start as soon as possible. 

If you start testing early on, from the first version of your app, you’ll give yourself more time to understand how it behaves, get familiarized with client side metrics, and run these tests alongside functional ones in each sprint. 

Another advantage of testing early is you can test on different devices at your disposal as well as understand how these metrics evolve over time. 

It’s important to understand how the metrics change over time so you can identify what changes to the app alter its performance (find performance regressions) and have more time to remedy degradations.

This helps a developer understand how the decisions they make affect performance and ultimately, avoid disappointing users when they find that the app is running slower than usual.

I hope this post provided you an overview on how to test mobile app performance! 

You can also watch this webinar, where I discussed all of the above and gave a demo of Apptim ��

Have any questions? Leave us a comment!