Google, Digital Ocean, & Rackspace Gain 1st Place Rankings in Several Empirical Cloud Performance Benchmarks
The Journal of Cloud Computing has attempted to make a series of benchmarks that fairly compares the various performance aspects of the top cloud compute service providers.
Join the DZone community and get the full member experience.
Join For FreeA December 18, 2015 article from the Journal of Cloud Computing points out the lack of empirical data and research gathered about the performance of cloud service providers’ virtual machine instances. Last year they finally took the time to build out a full array of categories and benchmarks that they could use to compare the performance of top cloud vendors (IaaS compute services specifically – i.e. EC2, GCC, etc). Their findings were published just last month and are very interesting and readable for non-academics. There are graphs for each of their benchmarks, measuring the CPU performance, memory performance, disk I/O, Mean Response Time (MRT), availability, variability, and provisioning time.
The cloud service providers measured include Amazon, Microsoft Azure, Aruba, Digital Ocean, Google Compute Cloud, Joyent, Microsoft Azure, Rackspace, and IBM Softlayer. This article will simply run down the list of 1st place winners in each performance category, but definitely read the details of how they did the benchmarking for any potential biases you might see (they pointed out a few themselves), and because many of the 2nd or 3rd place performers in these categories were so close to the leaders that you’d probably want to consider their services as well. In fact, their research may be more useful for identifying which CSP to watch out for.
Here were the first place winners:
Virtual CPU Performance
Benchmark: Sysbench
Fastest small instances: Rackspace
Fastest medium, large, and x-large instances: Google Compute Cloud
Benchmark: Pystone
Fastest small instances: Google Compute Cloud
Fastest medium and large instances: TIE between Google Compute Cloud and Amazon EC2
Fastest x-large instances: Google Compute Cloud
Example of one of the journal’s graphs for CPU performance with the Pystone benchmark. (Higher is Better)
Zant and Gagnaire Journal of Cloud Computing 2015 4:24 doi:10.1186/s13677-015-0049-1
Memory Performance
Benchmark: Stream
Fastest Copy, Add, Scale, and Triad operations: Google Compute Cloud
Disk I/O Performance
Benchmark: Bonnie++
Sequential input and output Per-Character, Block, and Rewrite: Digital Ocean (by a lot)
Mean Response Time (MRT)
Benchmark: Round Trip Time pinging one of the CSP’s virtual machine’s addresses from France.
Aruba has a data center in France, so it showed the fastest. Although it’s not certain who else this test may have favored, Rackspace had the fastest MRT (by a lot) without having a datacenter in France.
Availability
Benchmark: Cedexis
Highest percentage uptime: Rackspace
Variability in Performance Measurements
Benchmark: Relative Standard Deviation – Pystone, Sysbench, Triad
Lowest Variability in CPU Performance: Amazon
Lowest Variability in Memory Performance: Softlayer
Lowest Variability in Disk Storage I/O: Softlayer
Provisioning Time
Benchmark: Manual provisioning
All CSPs took under a minute.
Fastest Provisioning Time: Rackspace
Value for the Price
The value for the price was a bit hard to read, but I assume that you use the center of each circle (of varying diameters) to determine where it falls on the price and performance axis. Small circles closest to the upper right corner are the lowest priced, highest performing CSPs with the least amount of performance variability. Here’s the value chart for large instances.
Zant and Gagnaire Journal of Cloud Computing 2015 4:24 doi:10.1186/s13677-015-0049-1
Check out the entire report for the graphs and descriptions that I didn’t cover in this article!
Did you think this was a good series of benchmarks for objectively comparing the performance of CSPs?
Opinions expressed by DZone contributors are their own.
Comments