Apdex is Fatally Flawed

i’ve been looking into a lot of different statistical methods and algorithms lately, and one particularly interesting model is apdex. if you haven’t heard of it yet, apdex has been adopted by many companies that sell monitoring software. the latest apdex specification (v1.1) was released on january 22, 2007 and can be read here .

here is the purpose of apdex as quoted from the specification document referenced above: “ apdex is a numerical measure of user satisfaction with the performance of enterprise applications, intended to reflect the effectiveness of it investments in contributing to business objectives. ” while this is a noble goal, apdex can lead you to believe your application is working fine when users are really frustrated. if apdex is all you have to analyze your application performance, then it is better than nothing, but i’m going to show you the drawbacks and a better way to manage your application performance.

at the heart of apdex is a static threshold (defined as t). the t threshold is set as a measure of the ideal response time for the application. this t threshold is set for an entire application. if you’ve been around application performance for a while you should immediately realize that all application functions perform at very different levels comparatively. for example, my “search for flights” function response time will be much greater than my “view cart” response time. these 2 distinctly different functionalities should not be subjected to the same base threshold (t) for analysis and alerting purposes.

another thing that application performance veterans should pick up on here is that static thresholds stink. you either set them too high or too low and have to keep adjusting them to a level you are comfortable with. static thresholds are notorious for causing alert storms (set too low) or for missing problems (set too high). this manual adjustment philosophy leads to a lack of consistency and ultimately makes historical apdex charts useless if there have been any manual modifications of “t”.

so let’s take a look at the apdex formula now that we understand “t” and its multiple drawbacks:

• satisfied count = the number of transactions with a response time between zero and t
• tolerating count = the number of transactions with a response time between t and f (f is defined next)
• f = four times t (if t=3 seconds than f=12 seconds)
• frustrated count (not shown in the formula but represented in total samples) = the number of transactions with a response time greater than f

when my colleague ian withrow looked at the formula above, he had some interesting thoughts on apdex:

“when was the last time you "tolerated" a site loading four times slower than you expected it to? if t=1 i’ll tolerate four seconds, but if t=3 no way am i waiting 12 seconds on a fast connection … which brings me to another point. the notion of a universal threshold for satisfied/tolerating is bunk even for the same page (let alone different pages). my tolerance for how a website loads at work on our fast connection is paper thin. it better be one or two seconds or i’m reloading or going elsewhere. on the train to work where i know signal is spotty on my iphone, i’ll load something and then wait for a few seconds.”

getting back to the apdex formula itself; assuming every transaction completes in less than the static threshold (t) the apdex will have a value of 1.00, which is perfect, except the real world is never perfect, so let’s explore a more plausible scenario.

you run an e-commerce website. your customers perform 15 clicks while shopping for every one click of the checkout functionality. during your last agile code release, the response time of the check-out function went from four seconds to over 40 seconds due to an issue with your production configuration. users are abandoning your website and shopping elsewhere since their check-outs are lost in oblivion. what does apdex say about this scenario? let’s see …

t = four seconds
total samples = 1500
satisfied count = 1300 (assuming most were under four seconds with a few between four and 16 seconds (f=4t))
tolerating count = 100 (the assumed number of transactions between four and 16 seconds)

apdex = .90


according to the apdex specification, anything from a .85 - .93 is considered “good”. we know there is a horrible issue with checkout that is crushing our revenue stream, but apdex doesn’t care. apdex is not a good way to tell if your application is servicing your business properly. instead, it is a high-level reporting mechanism that shows overall application trends via an index. the usefulness of this index is questionable based on the fact that apdex is completely capable of manipulation based on manually changing the t threshold. apdex, like most other forms of analytics, must be understood for what it is and applied to the appropriate problem.

a better way: dynamic baselines and analytics

in my experience (and i have a lot of enterprise production monitoring experience), static thresholds and high-level overviews of application health have limited value when you want your application to be fast and stable regardless of the functionality being used. the highest value methodology i have seen in a production environment is automatic baselining with alerts based upon deviation from normal behavior. with this method, you never have to set a static threshold unless you have a really good reason to (like ensuring you don’t breach a static sla). the monitoring system automatically figures out normal response time, compares each transaction's actual response time to the baseline and classifies each transaction as normal, slow, or very slow. alerts are based on how far each transaction has deviated from normal behavior instead of an index value based on a rigid four times the global static threshold, like with apdex.

if you want to make yourself look great to your bosses (momentarily), then set your t threshold to 10 seconds and let the apdex algorithm make your application look awesome. if you want to be a hero to the business, then show them the problems with apdex and take a free trial of appdynamics pro today . appdynamics will show you how every individual business transaction is performing and can even tell you if your application changes are making or costing you money.