DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Google, Digital Ocean, & Rackspace Gain 1st Place Rankings in Several Empirical Cloud Performance Benchmarks

Google, Digital Ocean, & Rackspace Gain 1st Place Rankings in Several Empirical Cloud Performance Benchmarks

The Journal of Cloud Computing has attempted to make a series of benchmarks that fairly compares the various performance aspects of the top cloud compute service providers.

Mitch Pronschinske user avatar by
Mitch Pronschinske
·
Jan. 19, 16 · Analysis
Like (2)
Save
Tweet
Share
4.63K Views

Join the DZone community and get the full member experience.

Join For Free

A December 18, 2015 article from the Journal of Cloud Computing points out the lack of empirical data and research gathered about the performance of cloud service providers’ virtual machine instances. Last year they finally took the time to build out a full array of categories and benchmarks that they could use to compare the performance of top cloud vendors (IaaS compute services specifically – i.e. EC2, GCC, etc). Their findings were published just last month and are very interesting and readable for non-academics. There are graphs for each of their benchmarks, measuring the CPU performance, memory performance, disk I/O, Mean Response Time (MRT), availability, variability, and provisioning time.

The cloud service providers measured include Amazon, Microsoft Azure, Aruba, Digital Ocean, Google Compute Cloud, Joyent, Microsoft Azure, Rackspace, and IBM Softlayer. This article will simply run down the list of 1st place winners in each performance category, but definitely read the details of how they did the benchmarking for any potential biases you might see (they pointed out a few themselves), and because many of the 2nd or 3rd place performers in these categories were so close to the leaders that you’d probably want to consider their services as well.  In fact, their research may be more useful for identifying which CSP to watch out for.

Here were the first place winners:


Virtual CPU Performance

Benchmark: Sysbench

  • Fastest small instances: Rackspace

  • Fastest medium, large, and x-large instances: Google Compute Cloud

Benchmark: Pystone

  • Fastest small instances: Google Compute Cloud

  • Fastest medium and large instances: TIE between Google Compute Cloud and Amazon EC2

  • Fastest x-large instances: Google Compute Cloud

Image title

Example of one of the journal’s graphs for CPU performance with the Pystone benchmark. (Higher is Better)
Zant and Gagnaire Journal of Cloud Computing 2015 4:24   doi:10.1186/s13677-015-0049-1


Memory Performance

Benchmark: Stream

Fastest Copy, Add, Scale, and Triad operations: Google Compute Cloud


Disk I/O Performance

Benchmark: Bonnie++

Sequential input and output Per-Character, Block, and Rewrite: Digital Ocean (by a lot)


Mean Response Time (MRT)

Benchmark: Round Trip Time pinging one of the CSP’s virtual machine’s addresses from France.

Aruba has a data center in France, so it showed the fastest. Although it’s not certain who else this test may have favored, Rackspace had the fastest MRT (by a lot) without having a datacenter in France.


Availability

Benchmark: Cedexis

  • Highest percentage uptime: Rackspace


Variability in Performance Measurements

Benchmark: Relative Standard Deviation – Pystone, Sysbench, Triad

  • Lowest Variability in CPU Performance: Amazon

  • Lowest Variability in Memory Performance: Softlayer

  • Lowest Variability in Disk Storage I/O: Softlayer


Provisioning Time

Benchmark: Manual provisioning

All CSPs took under a minute.

Fastest Provisioning Time: Rackspace


Value for the Price

The value for the price was a bit hard to read, but I assume that you use the center of each circle (of varying diameters) to determine where it falls on the price and performance axis.  Small circles closest to the upper right corner are the lowest priced, highest performing CSPs with the least amount of performance variability.  Here’s the value chart for large instances.

Image title

Zant and Gagnaire Journal of Cloud Computing 2015 4:24   doi:10.1186/s13677-015-0049-1


Check out the entire report for the graphs and descriptions that I didn’t cover in this article!

Did you think this was a good series of benchmarks for objectively comparing the performance of CSPs?

Cloud computing Google (verb)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Writing a Modern HTTP(S) Tunnel in Rust
  • How To Generate Code Coverage Report Using JaCoCo-Maven Plugin
  • SAST: How Code Analysis Tools Look for Security Flaws
  • PHP vs React

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: