This post is meant to summarize that paper and I definitely encourage you to read the full report over here: http://www.iqcloud.net/wp-content/uploads/2013/07/Cloud-Computing-Performance-A-Comparative-Analysis-of-5-Large-Cloud-IaaS….pdf
The Objective of the Study
The objective of the paper is to determine the price-performance value of the cloud providers. It provides some valuable insight for customers when selecting their preferred cloud vendor.
Figure 1: Principle of value proposition [figure from the paper]
Who is Being Compared
The study (done in June 2013) compared five large IaaS providers in the industry:
The tests were run three times in five consecutive days: May 25, 2013 to May 29, 2013.
The most common size for cloud server, medium size (or an equivalent or similar setup) was chosen from the five cloud vendors:
Figure 2: Medium VM spec [figure from the paper]
The tests used Unixbench 5.1.3 to benchmark the performance of the Linux OPS running on virtualized infrastructure, producing ratings out of 10 stars. Details of Unixbench can be found here: https://code.google.com/p/byte-unixbench/
Two important pieces of information were collected:
- Performance: how well the provider scores on Unixbench, and how consistent the scores are
- Price-Performance: after performance scores are established, cost is factored in to help understand how much performance a user can expect in return for their money, i.e., the value.
The performance results show that Windows Azure provides the best performance, notably three times higher than AWS EC2 on average!
Figure 3: Performance only results [figure from the paper]
Figure 4: Average Unixbench score, derived from Figure 3 [figure from the paper]
Performance / Price = Value
Retail hourly price of the cloud providers were captured (on a pay-as-you-go basis) on the date of the experiment.
Figure 5: Pay-per-hour price [figure from the paper]
By taking each score and dividing by the price, we can get a relative price-to-performance score for each provider. And here are the scores (the higher the score, the better):
Figure 6: Price-performance results [figure from the paper]
The CloudSpecs score is a further normalized value from Figure 6, taking the highest value to 100. Here are the scores:
With the CloudSpecs score, the ratio of each of the providers are formed as following
While acknowledging that Unixbench is just one test, customers may always consider other factors when selecting their cloud vendor.
To conclude, Amazon EC2 and Windows Azure offer the lowest price at $0.12 per hour. However, Windows Azure performs much better than EC2 in this experiment (by approximately three times). The experiment also shows that Rackspace scores worst in term of price-performance.