A December 18, 2015 article from the Journal of Cloud Computing points out the lack of empirical data and research gathered about the performance of cloud service providers’ virtual machine instances. Last year they finally took the time to build out a full array of categories and benchmarks that they could use to compare the performance of top cloud vendors (IaaS compute services specifically – i.e. EC2, GCC, etc). Their findings were published just last month and are very interesting and readable for non-academics. There are graphs for each of their benchmarks, measuring the CPU performance, memory performance, disk I/O, Mean Response Time (MRT), availability, variability, and provisioning time.

The cloud service providers measured include Amazon, Microsoft Azure, Aruba, Digital Ocean, Google Compute Cloud, Joyent, Microsoft Azure, Rackspace, and IBM Softlayer. This article will simply run down the list of 1st place winners in each performance category, but definitely read the details of how they did the benchmarking for any potential biases you might see (they pointed out a few themselves), and because many of the 2nd or 3rd place performers in these categories were so close to the leaders that you’d probably want to consider their services as well.  In fact, their research may be more useful for identifying which CSP to watch out for.

Here were the first place winners:

Virtual CPU Performance

Benchmark: Sysbench

• Fastest small instances: Rackspace

• Fastest medium, large, and x-large instances: Google Compute Cloud

Benchmark: Pystone

• Fastest small instances: Google Compute Cloud

• Fastest medium and large instances: TIE between Google Compute Cloud and Amazon EC2

• Fastest x-large instances: Google Compute Cloud

Example of one of the journal’s graphs for CPU performance with the Pystone benchmark. (Higher is Better)

Zant and Gagnaire Journal of Cloud Computing 2015 4:24   doi:10.1186/s13677-015-0049-1

Memory Performance

Benchmark: Stream

Fastest Copy, Add, Scale, and Triad operations: Google Compute Cloud

Disk I/O Performance

Benchmark: Bonnie++

Sequential input and output Per-Character, Block, and Rewrite: Digital Ocean (by a lot)

Mean Response Time (MRT)

Benchmark: Round Trip Time pinging one of the CSP’s virtual machine’s addresses from France.

Aruba has a data center in France, so it showed the fastest. Although it’s not certain who else this test may have favored, Rackspace had the fastest MRT (by a lot) without having a datacenter in France.

Availability

Benchmark: Cedexis

• Highest percentage uptime: Rackspace

Variability in Performance Measurements

Benchmark: Relative Standard Deviation – Pystone, Sysbench, Triad

• Lowest Variability in CPU Performance: Amazon

• Lowest Variability in Memory Performance: Softlayer

• Lowest Variability in Disk Storage I/O: Softlayer

Provisioning Time

Benchmark: Manual provisioning

All CSPs took under a minute.

Fastest Provisioning Time: Rackspace

Value for the Price

The value for the price was a bit hard to read, but I assume that you use the center of each circle (of varying diameters) to determine where it falls on the price and performance axis.  Small circles closest to the upper right corner are the lowest priced, highest performing CSPs with the least amount of performance variability.  Here’s the value chart for large instances.

Zant and Gagnaire Journal of Cloud Computing 2015 4:24   doi:10.1186/s13677-015-0049-1

Check out the entire report for the graphs and descriptions that I didn’t cover in this article!

Did you think this was a good series of benchmarks for objectively comparing the performance of CSPs?