Over a million developers have joined DZone.

Continuous Testing – Where does it fit next to Integration and Delivery?

DZone's Guide to

Continuous Testing – Where does it fit next to Integration and Delivery?

· DevOps Zone ·
Free Resource

Do you need to strengthen the security of the mobile apps you build? Discover more than 50 secure mobile development coding practices to make your apps more secure.

On July 11th, Gartner released its 2013 Magic Quadrant for Integrated Software Quality Suites. I would normally think that research did not apply to me, but I received an email from Gartner letting me know that UrbanCode was mentioned. Curious.

As it turns out, Gartner rates IBM as a Leader’s in the Leader’s quadrant** (where positioning is based on completeness of vision and ability to execute). UrbanCode was described as contributing to IBM’s capabilities around continuous test.*

I normally think in terms of Continuous Integration and Continuous Delivery. Where does “Continuous Test” fit in? Paul Duvall, a co-author of the great Continuous Integration book, describes continuous testing as running as many tests as possible with every code change in an article on DeveloperWorks. That’s completely in line with modern thinking about CI and CD. We want to run rigorous tests as frequently as possible to maximize feedback.

Do we need the term “Continuous Test”?

If continuous test is just a subset of continuous delivery, do we need to call it out as its own thing? Unfortunately, I think we do. The emphasis that CI & CD place on testing is too frequently ignored.

We have been frustrated by this for a long time. In 2008, I wrote a somewhat trollishly titled piece “Continuous Integration: Was Fowler Wrong?“. That article focuses on how early versions of Martin’s definitive article described CI as centered on an automated build. However, CI is centered on rapid testing. Most tests require an accurate build to be present. The build itself is an input to those tests and answers the “does it compile?” test. Languages that do not compile still need CI, but they may not need a build. Builds are merely ancillary to CI.

Similarly, deployment automation is a necessary aspect of continuous delivery, but not sufficient. Deployments serve to deliver the application into environments where they may be tested quickly and correctly. They also provide for testing of the production release process as part of normal development and testing activities.

Because so many in the industry focus on the build and deployment mechanics rather than the tests they are for, promoting the term Continuous Test may be helpful in reminding people to drive feedback back.

Maciej Zawadzki, an Urbancode co-founder, speaks to the feedback aspects of CI and CD at length in the webinar “Risking Above the Noise: CI, CD, DevOps and What They Mean to You“.

* “Magic Quadrant for Integrated Software Quality Suites”  Thomas Murphy and Nathan Wilson.

** Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Check out tips for blazing the way from agile to DevSecOps with security built into your mobile app toolchain.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}