Breaking Down Apdex

this post comes from bill kayser at the new relic blog.

how well is your web application servicing your customers right now?

that’s a common starting point when using an apm tool like new relic. to put it more succinctly:

how are you doing?

there’s plenty of evidence and opinion to suggest that watching your average response time won’t give you anything near a complete answer to this question, which leaves many customers choosing to rely on the apdex measurement instead. so what does apdex really represent? let’s take a closer look.

start by considering what it really is you want to know from looking at a chart when you ask the “how am i doing?” question. let’s assume you are focused on the user experience and, in particular, page load times as seen from the user. so you could start with a chart showing the page load time of every request processed by your server over some representative sample period.  here is scatter plot of response times for one of our key transactions in new relic:

this does give us all the information we want: every page request for a 20 minute period. clearly the typical experience is maybe 1.0 - 1.2 seconds, where the center of mass is. but there are a lot of outliers and probably even many that are off the chart that we are missing completely. is it really helpful? how are you doing?

let’s try a histogram. this will show us a count of page load times within discrete buckets, giving us a better sense of how many outliers there are relative to the center of mass. here’s the same data as above:

this feels a little more digestible. you can see the range of response times starting at around 100 ms and going up to eight seconds. you can see that 3.6 percent of requests are completely off the chart. does it really tell you how you are doing? seems like you are getting there, but let’s try to add a little more information. below is the same histogram but with markers to indicate the mean response time (green line), 95th percentile response time (dashed line) and the median or 50th percentile (red line). also indicated are the middle two quartiles, the red region, representing the response times that fall between the 25th and 75th percentiles.

so now we’re getting somewhere. there’s a little more shape to our data. we have a much better sense of how significant those slowest loading pages are, and how prevalent they are. most of our users are in the 1.2 - 2.4 second range, even though our average response time is 2.5 seconds. about 25 percent of our users have completely intolerable response times, more than seven seconds.

how are you doing?  not very well.

at least we have a much more informed position from the data on this chart than we would if we erased everything but the green line. in fact, we could probably use even more information. to understand our user experience we’d probably want to know if this distribution is consistent across all our users, or if there are vast differences among different browser versions or regions. unfortunately the data we have  at hand is often limited in the number of annotated attributes, so it’s hard to correlate.

worse yet, we may not even have access to the event level needed to build charts like the ones above. histograms and scatter plots require storing a lot of data. even just a couple of percentile measurements like median and 95 percent can be tricky when you have to aggregate and resample data. so, what you are left with often is just average response time.

here’s where apdex can fill the gap. apdex is like a histogram with just three buckets: satisfy, tolerate and fail. the buckets represent requests that are satisfying the user, requests that users just tolerate , and requests that fail to meet the users’ expectations completely. the bucket intervals are 0, t, and 4t, where t is a parameter you choose in advance. here’s what that looks like if you shade the apdex buckets in our histogram:

i chose 1500 ms as a t value. the yellow region are the pages under 1500 ms. the red region goes from 1500 ms to 4t, or six seconds. the black region are the failed pages, those that took longer than six seconds. the apdex score is a formula based on the count of pages falling into each of these three regions. you take the number of satisfying requests plus half the number of requests in the middle area, and divide by the total count. for this t value, our score is 0.7.

if you move the t value up or down, the regions adjust and so does your score. if i move the t value down to one second, the regions all shift left and my apdex score goes from 0.7 to 0.49.

with t at one second, we are basically treating every request above four seconds as a failure. it doesn’t matter if it’s four seconds or 40 seconds–the result is the same: we failed. we are graded harshly for requests longer than one second, and anything under one second is viewed as serving our users. the difference between 800 ms and 1,000 ms is not important to us because it’s probably not noticeable by our users.

so when you ask "how well are you serving your customers right now?" this isn’t a bad place to start. pick a t value that characterizes your expectations for your site. we are pretty comfortable with description based on a value of t= one second so we’ll use that. now we can use the score to answer the question how well we are serving our customers on a scale from 0 to 100. you’ll be in a much better place than you would be with the answer that simply states the average response time is 2 seconds.

how are we doing? we are at a 49 on a scale of zero to 100. not very good at all.

the other nice thing about an apdex score is it allows you to answer the question quickly for an array of web transactions or applications. you can set a t value individually for each key transaction or application and see a list of scores and quickly identify where you need to focus on for improvement. looking at a column of response times isn’t really going to help since those times might mean different things for each individual transaction or application. is a response time of two seconds good or bad? it probably depends on a number of different assumptions. many of those assumptions can be encapsulated with a well considered t value selection.

but the most important thing about an apdex score is that unlike histograms and percentiles it can easily and inexpensively be collected and re-sampled.

note: the charts and data used in this post are available for browsing using a tool i developed for experimentation with different visualizations called marlowe, available on github .

further reading:

the problem with averages , david heinemeier hansson

what the mean really means , brendan gregg

what do you mean: revisiting statistics for web response time measurements by david m ciemiewicz (2001)

marlowe data exploration tool