I was buying a used book through Amazon this evening. Three resellers offered the book at essentially the same price. Here were their ratings:

- 94% positive out of 85,193 reviews
- 98% positive out of 20,785 reviews
- 99% positive out of 840 reviews

Which reseller is likely to give the best service? Before you assume it’s the seller with the highest percentage of positive reviews, consider the following simpler scenario.

Suppose one reseller has 90 positive reviews out of 100. The other reseller has two reviews, both positive. You could say one has 90% approval and the other has 100% approval, so the one with 100% approval is better. But this doesn’t take into consideration that there’s much more data on one than the other. You can have some confidence that 90% of the first reseller’s customers are satisfied. You don’t really know about the other because you have only two data points.

A Bayesian view of the problem naturally incorporates the amount of data as well as its average. Let θ_{A} be the probability of a customer being satisfied with company A’s service. Let θ_{B}
be the corresponding probability for company B. Suppose before we see
any reviews we think all ratings are equally likely. That is, we start
with a uniform prior distribution θ_{A} and θ_{B}. A uniform distribution is the same as a beta(1, 1) distribution.

After observing 90 positive reviews and 10 negative reviews, our posterior estimate on θ_{A} has a beta(91, 11) distribution. After observing 2 positive reviews, our posterior estimate on θ_{B} has a beta(3, 1) distribution. The probability that a sample from θ_{A} is bigger than a sample from θ_{B} is 0.713. That is, there’s a good chance you’d get better service from the reseller with the lower average approval rating.

Now back to our original question. Which of the three resellers is most likely to satisfy a customer?

Assume a uniform prior on θ_{X}, θ_{Y}, and θ_{Z},
the probabilities of good service for each reseller. The posterior
distributions on these variables have distributions beta(80082, 5113),
beta(20370, 417), and beta(833, 9).

These beta distributions have such large parameters that we can
approximate them by normal distributions with the same mean and
variance. (A beta(*a*, *b*) random variable has mean *a*/(*a*+*b*) and variance *ab*/((*a*+*b*)^{2}(*a*+*b*+1)).) The variable with the most variance, θ_{Z},
has standard deviation 0.003. The other variables have even smaller
standard deviation. So the three distributions are highly concentrated
at their mean values with practically non-overlapping support. And so a
sample from θ_{X} or θ_{Y} is unlikely to be higher than a sample from θ_{Z}.

In general, going by averages alone works when you have a lot of customer reviews. But when you have a small number of reviews, going by averages alone could be misleading.

Thanks to Charles McCreary for suggesting the xkcd comic.*Source: http://www.johndcook.com/blog/2011/09/27/bayesian-amazon/*

## {{ parent.title || parent.header.title}}

## {{ parent.tldr }}

## {{ parent.linkDescription }}

{{ parent.urlSource.name }}