With test-driven development, continuous integration/continuous deployment, and DevOps practices now the norm, most organizations understand the importance of testing their applications.
But what about the cloud those applications are going to live on? Too many companies miss this critical step, leading to gaps in their operations, which can lead to production issues, API outages, inability to upgrade, problems when trying to upgrade and general instability of the cloud.
It all begs the question: “Do you even test?”
At Rackspace, our support teams use a proactive approach to operations, and that begins with detailed and comprehensive testing so that not only your applications but your cloud is ready for your production workload.
For Rackspace Private Cloud Powered by Red Hat, we collaborate closely with Red Hat; we test the upstream OpenStack code as well as the open-sourced projects we leverage for our deployment, such as Ceph and Red Hat OpenStack Platform Director. This is done in a variety of ways, like sharing test cases upstream with the community via Tempest, creating and tracking bugs, and creating bug fixes upstream.
The Rackspace and Red Hat team also work together on larger scale and performance tests at the OpenStack Innovation Center, which we launched last year in conjunction with Intel to advance the capabilities of OpenStack.
Recent tests have included performance improvements in relation to offloading VXLAN onto network cards, scaled upgrade testing from Red Hat OpenStack Platform version 7 to version 8, and testing of scaled-out Ceph deployments. Data from this testing will be made available to the community as the detailed results are analyzed.
Building on the upstream testing, the Rackspace operations team leverages Rally and Tempest to execute 1,300 test cases prior to handing the cloud over to the customer. This testing serves as a "1,300-point inspection" of the cloud to give you the confidence that your cloud is production ready and a report of this testing is handed over to you with a guide to help you get started with your new cloud. These test cases serve to validate and demonstrate the functionality of the OpenStack APIs, with specific scripts testing things such as (just to name a few):
- administration functions
- creating instances and cinder volumes
- creating software defined networks
- testing keystone functions and user management
Upgrades Made Easy
One of the key requirements for enterprises is the ability to upgrade software without impacting the business.
These upgrades have been challenging in OpenStack in the past, but thanks to the Rackspace/Red Hat collaboration, we can now make those upgrades.
To deliver this, the Rackspace team runs the latest version of OpenStack code through our lab and executes the 1,300-point inspection. When we are satisfied with that, we test upgrading our lab to the latest version and execute our 1,300-point test again, thus confirming that the new version of OpenStack meets requirements and that the code is safe for your environment.
So that the code deploys properly to your cloud, our operations team executes a 500-script regression test at the start of a scheduled upgrade window. Then, our team upgrades your cloud and executes the regression test again. The final step in the scheduled upgrade window is to compare our pre- and post-regression test results to validate that the upgrade was successful.
Since launch, our teams have been working to refine that process by incorporating Red Hat’s Distributed Continuous Integration project.
With Distributed Continuous Integration, Red Hat extends the testing process related to building Red Hat OpenStack Platform to Rackspace's in-house environment. Instead of waiting for a general availability release to start testing Rackspace scenarios, pre-release versions are delivered and tested following a CI process. Test results are automatically shared with Red Hat's experts and, along with Rackspace, new features are debugged and improved taking into consideration new scenarios.
Using DCI to test pre-released versions of Red Hat OpenStack Platform helps ensure we're ready for the new general release just after launch. Why? Because we have been running incremental changes of the software in preparation for general availability.
DCI also helps existing Rackspace private cloud customers, by allowing the Rackspace operations team to test code changes from Red Hat while they're being developed, allowing us a shorten the feedback loop back to Red Hat engineering, and giving us a supported CI/CD environment for your cloud at a scale not possible without a considerable investment in talent and resources.