Continuous Deployment and Continuous Integration
production (2-3 weeks if all goes well) everything else slows down...
We all know about continuous integration (CI). We (xsights) are seeing a lot of ROI on the investments we made into signing up to CI esp. when we augmented by automated testing. Everyday we can and do actually release several versions and run load tests on them on different server seeking the best release to "freeze". Where "freezing" means the system that will be used in production. That's the way it is done, right? You work on a release for x months (sometimes years..) and after an integration period at the client's site you go live and forget about them (save for show-stopping bugs) until the next major update several months later. For instance, setting up our "methodology" I figured a 6 months release cycles for major versions plus 1 month releases of minor version would suffice.
These days, however the more I think about it, The more I believe this approach is not the right one for our situation. We are going to go with Continous Deployment. There are many reasons for that primarily:
- The requirements stream (new reqs and changes) is still pouring in - and in increased rate.
- We are building a service platform not an installable product - This means we get to control where how it is run so there's less overhead in streamlining updates
- It would be our first public installation - were going to roll out bug fixes anyway (I personally never write bugs... but you know ;) ). We'll need a proper procedure for updates anyway
- We are doing this anyway - every new demo/internal run we have is using the "stablest" latest and greatest :)
What does Continuous Deployment means? It means we'll be rolling out new production versions on a daily basis (or even more frequently!)
How that's going to work? Well, First there are the prerequisites:
- Version control - something that can track and tag versions...
- Continous integration which is set up and running. You cannot release continously if every new development offer breaks the system. An automated smoke test is
- Automates tests that provide a good approximation of real usage (.e.g. usage patterns, loads etc.) - You need to be able to have a high onfidence level that whatever you have.
- Staging environment - An environment as close as possible to the production system (just like you'd have for a web-site)
- Monitoring and alert system - You need to be able to identify problems quickly - the running
Then there's the process:
- identify a baseline Stable release -The "fallback".
- pick the most promising build of the day
- run it through more extensive tests (at least burn-down and load - which means a release maybe 1-2 days old by the time it hits the production system)
- if everyhting is cool deploy to staging - else fix bugs and repeat from 2
- repeat tests in staging environment
- Update system (This is a point that still needs work so we can support a version hot-swap i.e. without a shutdown)..
- If the monitoring system alerts unforeseen bugs (i.e. problems with the release) - roll back to stable release, try to add test for the problematic behavior and repeat from 2
- if the release worked good enough - mark it as the next Stable release
That's the plan. However as we all know, talk is cheap so be sure to check in here in 2-3 months to hear how did we fare :)