Going Beyond the Chest-Thumping Metrics

I recently wrote about tracking unique active users the right way for mobile and web apps on DZone and promised to write about loyalty, attrition, and metrics that matter.

Vanity metrics — the “feel-good” metrics that neither reflect nor necessarily correlate well to the key drivers of a business — continue to be used widely to market and report on growth and success. Examples of vanity metrics include:

• App downloads or installs.
• Messages sent over platform.
• Registered users.
• Page views.
• Number of tweets.
• Number of events processed.
• Number of followers.
• Daily, weekly, or monthly active users.

It takes infrastructure, energy, and resources to be able to deliver billions of messages a day or process billions of events a day to makes lives of customers easy and to provide value. Marketers love amplifying the achievements by announcing growth numbers:

• 140 billion apps cumulatively downloaded from Apple’s App Store.

• Facebook averaged 1.18 billion daily active users in September 2016.

• Pyze is processing more than 1 Billion events daily.

• Google delivered over 3 billion app downloads driven by ads.

Eric Ries of The Lean Startup says vanity metrics are not actionable and entrepreneurs should invest energy in collecting and analyzing metrics that help them make decisions. Apple does not use “Cumulative downloads” for managing their business, but the company has used it to report growth of the app store. Facebook, Google, and we at Pyze don't use the metrics listed above to report growth for managing our businesses.

So, some metrics are great for chest-thumping, reporting and marketing, but not for hinging your business on.

The Vanity Metrics

Let’s look at two vanity metrics: app downloads or installs; and daily, weekly, and monthly active users. Let's see how these, and their derivatives, are of value to product managers, growth marketers, and data scientists.

App Downloads or Installs

App downloads or installs do not reflect success and are not directly related to revenue. Hence, those stats are labeled as a “vanity metric”.

High Acquisition Costs

Installs can be easily engineered by paying for app downloads indirectly via social marketing, search engine marketing, or app store search ads. Although a large number of installs is beneficial to any app publisher, if the cost of app user acquisition via expensive advertising campaigns is significantly higher than the average revenue per user (ARPU), then gaining more users could eventually lead to losses.

Paying for Downloads

App downloads or installs can also be manipulated by paying for downloads directly via Fiverr and numerous other services. It is important to note that app stores detect script-based installs that usually show up as downloaded from “desktop” versus device, and paying for downloads has zero positive effect on app store rankings.

App stores look at engagement and uninstalls for rankings. Most purchased downloads eventually have a negative affect on rankings.

Daily, Weekly, and Monthly Active Users

Daily, weekly, and monthly active users are used for tracking and reporting growth by most app companies. Many SaaS businesses charge based on monthly active users. But a case can be made why they are vanity metrics.

Not Actionable

Daily, weekly, and monthly active users are not directly actionable and cannot be used to drive change.

Miscalculated and Inflated

The active user counts are often miscalculated to inflate the numbers. For example,

• MAU is not an aggregation of a month’s DAUs, and
• WAU is not an aggregation of a week’s DAUs.

Large False Positives

• Even if a user of an app launches the app for a second or two, they are counted toward the active user counts.
• A user who has forgotten his or her password and does not bother to recover the password is counted toward the active user count.
• An app launched and closed immediately as a result of a click on a deep-, app- or universal-link is counted towards the active user counts.

DAU Does Not Correlate With Revenue

Users, based on in-app purchases are in games, are typically grouped as:

• Minnows (i.e. users who spend the least amount).
• Dolphins (i.e. users who spend a moderate amount).
• Whales (users who spend the most), and...
• ...Freeloaders

For a specific game that used in-app purchases, we found:

• 4% Minnows
• 2% Dolphins
• 6% Whales
• The rest 88% Freeloaders.

Others have reported similar numbers: only 5.2% of users are spenders.

Clearly, a large number of DAUs does not mean a large number of paying customers.

Derived Vanity Metrics

Any metric based on the DAU vanity metric is also a vanity metric. Microsoft closed the acquisition of LinkedIn this week. The most recently published LinkedIn numbers are $933 million in revenue and 450 Million registered users, implying an ARPU of $2.07.

• Revenue per active user, or average revenue per daily active user (ARPUDAU), includes users who use the app for a second or cannot get past the login screen. In the LinkedIn example above, the ARPUDAU was $8.30
• Stickiness Factor is a ratio of DAU and MAU and can be calculated incorrectly. See here.

Going Beyond the Vanity Metrics

Vanity metrics, installs, and DAU/MAU/WAU may not be directly actionable, but we continue to use them for reporting and find value.

You Need Installs

An install is an important step in the funnel. It is preceded by acquisition and awareness marketing (channels: search engine, social, and App Store search ads) and followed by retention (channels: push notifications, MMS, and SMS) and engagement marketing (in-app messages). Without an installed user base, you have no users to retain and engage.

Stickiness vs. Stickiness Factor

Stickiness is a ratio of DAU and MAU and indicates the depth of engagement. The closer the ratio of DAU and MAU are to 1, the better the stickiness, and it means that more of your users are returning to the site every day.  See why the 7-day stickiness factor and 30-day stickiness factor are much better metrics than of DAU/MAU.

Periodic Trends

If you compare this week’s installs with last week’s by overlaying them on top of each other, you get an idea of how your installs are doing week-over-week.

Personalize On-Boarding

A significantly large number of users drop off after installing the app and using the app for the first time. App publishers should personalize the on-boarding process for users individually based on what you already know about the users and the users’ devices. We have written about personalizing the on-boarding process for mobile apps and for web apps.

Activations

Activations, as a metric, are a much better alternative than installs — and they are an actionable metric. They represent a relevant growth metric. It is important to keep track of important milestones in the lifecycle, and activations help app publishers do exactly that. Examples to track include:

• App is launched or resumed Nth time.
• App used for M days in last N-day period.
• App used on K unique days,
• App usage exceeds X minutes.
• App usage exceeds X minutes over Q previous days.
• App user took 50 photos in a photography app.
• App user shared 2 photos in a social photography app.
• App user invited others to join the app.
• App user provided email address and signed up for the weekly brief email.

Loyalty

Loyalty is a much better alternative to daily active users, as it indicates how loyal users are. Examples include:

• Number of times a user uses an app in last 7 days.
• Number of times user used app in last 30 days.
• Number of sessions an app user had longer than 15 minutes for a creative app.
• Number of searches app user performed in a search app in the last 7 days.

A number of metrics get written off and ignored as “vanity metrics.” The key is to be able to differentiate vanity metrics from actionable metrics. Some metrics are relevant for reporting and marketing, and others can be used for growing your business.

Unfortunately, a majority of analytics packages provide reports only on vanity metrics and calculate most metrics inaccurately or without giving much thought.