The Top Five KPIs for Mobile App Businesses

Somehow “crash” entered the mobile user’s vernacular. Any time an app abruptly closes or they are forced to terminate the app, they exclaim, “crash” in a negative review or submitted complain. However, a “crash” is a code exception and a user has no way of knowing whether the app error is a code exception. What if the error is something else?

Understandably users hate it when apps crash, or even if they merely slow down or freeze for a few seconds. According to a survey by Dimensional Research, if an app crashes, freezes, or has errors, 53 percent of users will uninstall it.

With the focus on crashes, all mobile app developers use a crash reporting solution to monitor app performance and surface code exceptions when things go wrong.

What’s surprising is that app crashes represent only a minimal amount of user problems — about 0.3 percent for a moderately tuned app. Other types of errors, like Out-of-Memory (OOMs), account for more than four times as many app shutdowns than crashes. Performance issues, which can lead users to abandon your app, come into play about 10 times as often.

Concentrating only on actual code exception crashes means that you’re focusing your efforts on something that affects a relatively small amount of your users. As a result, you are wasting valuable development time and ignoring issues that lead to user churn.

Traditional crash reporting software lacks visibility into other issues. This is why it’s critical to have a wider view of how your app is performing.

Here are the most important performance indicators to monitor and the benchmarks metrics to strive for:

KPIs to Monitor and Benchmark Metrics:

1. Crash-free Session Rate

Crash-free session rate is currently the only stability KPI and is by far the most known KPI to monitor app health. The crash-free rate connotes the percentage of sessions that do not end in a code exception. A low-quality app typically has a crash-free rate of ~99 percent. The benchmark for most verticals is 99.5 percent.

To improve the rate beyond 99.5 percent, forward-thinking apps realize that the stacktrace produced by a code exception is not always useful. Instead, they look at the events and conditions leading to the crash. The most common culprit is a network call error that occurred earlier in the session, upon which the code relied. Think about client-side errors, like a timeout, or a malformed JSON response; few apps handle these and other common conditions correctly.

While less than one percent of sessions should not be the focus of a mobile engineering team when other errors affect users at much higher rates, crashes are often the easiest to detect and simplest to solve.

2. Out Of Memory (OOM) — Low Memory Sessions and CPU Pegging

Out-of-Memory abrupt closes or OOMs occur 4 to 10 times more often than crashes. Like “crashes”, OOMs have become a term of art in development and are defined as any abrupt close of an app that is not a code exception. (Please note that this definition of OOMs only applies to iOS. The somewhat equivalent for Android is ANRs and covered in the next section.)

The general coding definition of an “OOM” is when the app is out of memory. However, an app can close for several reasons and Apple makes this definition a black-box for iOS.

The benchmark is a 99 percent of OOM-free session rate. This number can vary widely between app types based on heavy image use, a multitude of concurrent network calls, or general poor coding practices resulting in memory leaks. The worst performing apps are generally in Education or Social media — both of which make sense considering the skew towards low-end devices and requirements for many image assets.

Embrace has determined that iOS may close an app, e.g. an “OOM” because of an out-of-memory condition or CPU pegging.

• Out-of-Memory — in a typical mobile app, over four percent of sessions contain a memory warning. These are indicators of a high-likelihood that the OS will close the app.
• CPU Pegging — over 0.7 percent of sessions are pegged at 90 percent or more for over a second. While a developer should peg the CPU, Embrace has determined that pegging the CPU for a prolonged period of time highly correlates to the OS closing the app.

3. App-Not-Responsive (ANRs)

Similar to Out-of Memory, ANRs are when apps "hangs" and fail to recover. On Android, the OS eventually prompts the user to terminate the app. A typical app has an ANR-Free rate of 98 percent and the highest-quality apps are over 99.5 percent.

While Google Play reports ANRs, their methodology is faulty:

• The key to solving ANRs is understanding when they start and not when they conclude. An ANR may occur for over five to 10 seconds before the user terminates.
• Many ANRs are not reported by Google, as the user left before prompted to terminate the app.

4. Force Terminated Sessions

Users terminate over six percent of sessions on average. Unlike the aforementioned KPIs, force terminations are invaluable when compared across versions and builds.

When a feature is released, a new SDK is added, or an abnormal users complain or leave the app. The best indicator is force terminations. Users will close an app when its frozen or behaves improperly. Feature analytic tools measure the impact of these non-crash and non-OOM or ANR errors in terms of retention or revenue, both of which are delayed metrics that tell a mobile team of an issue when it's already too late.