On July 11th, Gartner released its 2013 Magic Quadrant for Integrated Software Quality Suites. I would normally think that research did not apply to me, but I received an email from Gartner letting me know that UrbanCode was mentioned. Curious.
As it turns out, Gartner rates IBM as a Leader’s in the Leader’s quadrant** (where positioning is based on completeness of vision and ability to execute). UrbanCode was described as contributing to IBM’s capabilities around continuous test.*
I normally think in terms of Continuous Integration and Continuous Delivery. Where does “Continuous Test” fit in? Paul Duvall, a co-author of the great Continuous Integration book, describes continuous testing as running as many tests as possible with every code change in an article on DeveloperWorks. That’s completely in line with modern thinking about CI and CD. We want to run rigorous tests as frequently as possible to maximize feedback.
Do we need the term “Continuous Test”?
If continuous test is just a subset of continuous delivery, do we need to call it out as its own thing? Unfortunately, I think we do. The emphasis that CI & CD place on testing is too frequently ignored.
We have been frustrated by this for a long time. In 2008, I wrote a somewhat trollishly titled piece “Continuous Integration: Was Fowler Wrong?“. That article focuses on how early versions of Martin’s definitive article described CI as centered on an automated build. However, CI is centered on rapid testing. Most tests require an accurate build to be present. The build itself is an input to those tests and answers the “does it compile?” test. Languages that do not compile still need CI, but they may not need a build. Builds are merely ancillary to CI.
Similarly, deployment automation is a necessary aspect of continuous delivery, but not sufficient. Deployments serve to deliver the application into environments where they may be tested quickly and correctly. They also provide for testing of the production release process as part of normal development and testing activities.
Because so many in the industry focus on the build and deployment mechanics rather than the tests they are for, promoting the term Continuous Test may be helpful in reminding people to drive feedback back.
Maciej Zawadzki, an Urbancode co-founder, speaks to the feedback aspects of CI and CD at length in the webinar “Risking Above the Noise: CI, CD, DevOps and What They Mean to You“.
* “Magic Quadrant for Integrated Software Quality Suites” Thomas Murphy and Nathan Wilson.
** Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.