Recently, I found an analysis paper about cloud server performance conducted by an independent cloud performance metrics company, Cloud Spectator.

This post is meant to summarize that paper and I definitely encourage you to read the full report over here: http://www.iqcloud.net/wp-content/uploads/2013/07/Cloud-Computing-Performance-A-Comparative-Analysis-of-5-Large-Cloud-IaaS….pdf

The Objective of the Study

The objective of the paper is to determine the price-performance value of the cloud providers. It provides some valuable insight for customers when selecting their preferred cloud vendor.

Figure 1: Principle of value proposition [figure from the paper]

Who is Being Compared

The study (done in June 2013) compared five large IaaS providers in the industry:

• Amazon AWS EC2
• Rackspace OpenStack Cloud
• HP Cloud
• SoftLayer CloudLayer Compute
• Windows Azure

Methodology

Timeframe

The tests were run three times in five consecutive days: May 25, 2013 to May 29, 2013.

VM Size

The most common size for cloud server, medium size (or an equivalent or similar setup) was chosen from the five cloud vendors:

Figure 2: Medium VM spec [figure from the paper]

Benchmark

The tests used Unixbench 5.1.3 to benchmark the performance of the Linux OPS running on virtualized infrastructure, producing ratings out of 10 stars. Details of Unixbench can be found here: https://code.google.com/p/byte-unixbench/

Information

Two important pieces of information were collected:

• Performance: how well the provider scores on Unixbench, and how consistent the scores are
• Price-Performance: after performance scores are established, cost is factored in to help understand how much performance a user can expect in return for their money, i.e., the value.

The Results

Performance Only

The performance results show that Windows Azure provides the best performance, notably three times higher than AWS EC2 on average!

Figure 3: Performance only results [figure from the paper]

Figure 4: Average Unixbench score, derived from Figure 3 [figure from the paper]

Performance / Price = Value

Retail hourly price of the cloud providers were captured (on a pay-as-you-go basis) on the date of the experiment.

Figure 5: Pay-per-hour price [figure from the paper]

By taking each score and dividing by the price, we can get a relative price-to-performance score for each provider. And here are the scores (the higher the score, the better):

Figure 6: Price-performance results [figure from the paper]

CloudSpecs Score

The CloudSpecs score is a further normalized value from Figure 6, taking the highest value to 100. Here are the scores:

With the CloudSpecs score, the ratio of each of the providers are formed as following

Conclusion

While acknowledging that Unixbench is just one test, customers may always consider other factors when selecting their cloud vendor.

To conclude, Amazon EC2 and Windows Azure offer the lowest price at $0.12 per hour. However, Windows Azure performs much better than EC2 in this experiment (by approximately three times). The experiment also shows that Rackspace scores worst in term of price-performance.