Antipattern of the Month: Vanity Metrics

You've got to accentuate the positive, e liminate the negative, l atch on to the affirmative. D on't mess with Mister In-Between. —  Bing Crosby

No one likes to be associated with negativity or failure. That goes for organizations as surely as it does for individual people, and hence development teams are often pressured into casting their effort in the best possible light. Data might be cherry-picked by them, or by senior managers, to please stakeholders and to secure ongoing engagement. Meanwhile, any evidence that things are going less than swimmingly might be suppressed. Poorly informed decisions will then be encouraged, based on a flawed appreciation of the status of an initiative.

In other words, when a team inspects the metrics they have gathered, they can be tempted to filter the data so as to create the most positive spin to stakeholders. Vanity metrics are thereby chosen to strike a posture, rather than to build objective and timely understanding. Actionable metrics, which would better inform observers as to the most prudent course of action, are thus "conveniently" ignored in favor of vanity measures.

Vanity metrics are often used or abused within organizations to justify continued investment despite immediate warning signs of danger. For example, the hit rates on an e-commerce landing page may be showcased while a perilously low conversion rate into completed transactions might be ignored. Forecasts which are derived from such biased data are unlikely to be fit for purpose.

Vanity Metrics is Antipattern of the Month at agilepatterns.org

Vanity Metrics

Intent: Use only the most favorable metrics in order to strike a posture

Proverbs: "Fine words butter no parsnips."

Motivation: Team members can be tempted to show their efforts in the best light rather than in a more objective one. They may cherry-pick data in support of this position so as to encourage the further commitment of stakeholders.

Structure: A team reviews the progress of a project they are working on by inspecting the metrics they have gathered. They then filter these metrics down to a reduced data set which gives the most positive interpretation of project status to stakeholders.

Applicability: Vanity metrics can be used to entice third-party investors. They are also often used within organizations to justify projects, and to protect and extend budgets. 

Consequences: If vanity metrics are used to influence decision making, then those decisions will be taken with an unrepresentative view of the real status of a project. Any forecasts based on this information may be unfit for purpose, and the opportunity to secure advance warning of any problems may be lost. 

Implementation: Vanity Metrics are explicitly dealt with in the Lean Startup approach and the use of actionable metrics is favored.