Continuous Integration Refcard Released: Meet The Author

This weeks refcard covers Continuous Integration: Patterns and Anti-Patterns with tips on how to get started using continuous integration techniques in your project. I spoke with Paul Duvall, author of the refcard and CI expert about the benefits of continuous integration. Paul also outlines a set of tools that he recommends to get going with CI.

DZone: Could you introduce yourself to the readers please?

Paul Duvall:  I run a company called Stelligent and we provide products and services for large-scale build and deployment automation. I also wrote the book Continuous Integration: Improving Software Quality and Reducing Risk which won the Jolt award a couple of years ago. My background is in software development of enterprise systems and developed an early fondness for all things automated because it enabled technologists to focus on novel problems rather than doing than the same thing over and over again. I've always found it ironic that as software/systems professionals we don't automate as much for ourselves as we do for our users. That is, we might develop a software system to automate portions of our customer's processes, yet we spend needless time copying files between machines or configuring a development environment or whatever.

At Stelligent, we're providing products that automate much of the software delivery process by leveraging cloud computing and other technologies. In the next month, we will be releasing a beta version of our Continuous Integration as a Service (AKA "CI as a Service") product in which users can simply click a button and get a fully functional Continuous Integration instance and tech stack using a web application. We're really excited about how this will help our customers be more productive. Our goal is to help our customers literally click one button (or type a single command) to create their software from source at any point in time. By implementing many of the patterns I describe in this refcard, teams are able to achieve this goal.

DZone: What is the main principle behind CI? When did it really start to take off?

Paul: To integrate changes between developers on a team whenever a change is applied. Like most technical practices, it has evolved along the way. Back in the early to mid 90s, Steve McConnell advocated the "daily build and smoke test" and it was lauded as a key practice used at Microsoft. Continuous Integration took the concept of the daily build to another level and not just perform these builds once a day, but when any team member commits code to the repository. This concept came out of the eXtreme Programming methodology and touted by Kent Beck and Martin Fowler. In the early 2000s, Fowler wrote the seminal article on Continuous Integration as a key practice on successful Agile projects. When I was writing the book on CI, it wasn't a common practice for teams. At that time, only 27% were even using a daily build and smoke test.

These days, it seems  to be a much more common practice. However, the next part of this evolution is starting to take place and that is more "Continually working production-ready software". Some are referring to this as "Continuous Deployment" but it's much more than this. This is where CI and release management are starting to converge and means that teams are beginning to automate more and more of the software delivery process, including provisioning, database assembly, deployment, etc. out of business necessity as companies are finding it critical to deliver software updates early and often.

DZone: Can you give us some of the key advantages to continuous integration?

Paul: A key advantage of CI is to reduce the time between when a defect is introduced into a codebase and when it's fixed. This is because CI provides instant visibility on whether the software can be built. When effectively used, CI can also tell you whether your software is functionally working or not. Furthermore, because changes are applied and problems are discovered often, changes are often much less complex than when teams wait days or weeks to learn of a particular problem. This lessened complexity reduces the costs of development and maintenance.

DZone: Are there any downfalls to CI?

Paul: It's all in how you apply it. One of the downsides to CI is that development teams can no longer hide behind the "we're almost done" excuse. The software is either working or it isn't. If your build is only performing an anti-pattern I like to call "Continuous Compilation" and not running tests, or automating deployment, etc. then you can get a false sense of security that everything is working when, in reality, you're a long way away from having working software. Again, most of CI's subpractices come down to how you apply them. Finally, some complain about the time it takes to setup, configure and maintain an effective CI system. That's where using products that have automated much of this process can help speed adoption of CI, so that you're focusing on the practice, not the management of all of the CI tools.

DZone: When you speak of patterns in this card what do you mean?

Paul: A pattern is an approach used in the software industry - borrowed from building architecture patterns - to provide a common vernacular so that we can better communicate particular techniques. A pattern is a solution to a problem in a particular context. Context is the important part of the definition and is why a pattern differs from a "best practice". A "best practice" implies that it doesn't matter what you're doing, the practice should be followed. A pattern takes the particular context into consideration. The other key feature of a pattern is the easy-to-remember name that accompanies it. I chose to use two-word pattern names in describing CI patterns. I also reference several SCM patterns that have been previously published by Berczuk and Appleton as the SCM and CI practices are tightly coupled and you can't perform effective CI without effective SCM.

DZone: What is the most useful CI pattern?

Paul: Expeditious Fixes - If team members aren't fixing build errors as soon as they occur, you will not succeed at CI. Timing is crucial to effective CI. Otherwise, teams will quickly develop complacency and not fix their builds because the defects become overly complex.

DZone: And what is your favourite CI pattern?#

Paul: I can't just name one! Over the past few years, we've been solving complex deployment and provisioning problems for our customers, so my favorites are the more esoteric patterns such as Externalize Configuration, Disposable Container, Continuous Deployment and Single-Command Provisioning. On the build side, I think Independent Build - not requiring an IDE - is critical to many of the other build and deployment patterns so if our customer can't create their software without their IDE, we know this is one of the first problems to solve.

DZone: Can you give an example of an anti-pattern that those new to CI may fall into?

Paul: The anti-pattern I see many teams fall into is machine-specific build and deployments. An easy test is to go to a new machine, perform a checkout from your version-control system. From only the files from this checkout (the Repository), you should be able to type a single command and get working software. If you can't, it means your build/deployment requires machine-specific configuration and it increases the maintenance and risk of introducing a configuration error. All of this configuration is repetitive, error-prone and can be automated. This anti-pattern is covered in various contexts in the Minimal Dependencies, Single Command, Unified Deployment patterns.

DZone: What is your top CI tip?

Paul: Get started now! Don't try to make everything perfect before rolling out a CI system and processes; just get going and adapt from there. Along the same lines, start small and build up the process. Some get concerned that they don't have any tests or they haven't automated their deployment, etc. and don't get started because of fear. Also, some are concerned that their build process is so complex that it can't be automated. This is the perfect reason to automate it. As Fowler mentions in the foreword of my book "Oddly enough, it seems that when you run into a painful activity, a good tip is to do it more often.

DZone: Can you recommend a CI toolbox for Java developers to get started with CI?

Paul Duvall: This can be a touchy subject to some, so I want to say up front that I'm not advocating the use of any of these tools, just that these are some of the tools that I've used when creating CI systems. The precise toolset a team may choose to use depends upon numerous factors including project, cost and customer constraints - to name a few. Therefore, I suggest that readers focus more on the type of tool and determine which one meets their particular needs for their CI ecosystem. Just because I'm not mentioning a particular tool doesn't mean I'm not using it or that I don't think it's a good tool; these are meant to be illustrative. I try to focus more on freely-available tools because people can download and use them quickly. There are good reasons to choose commercial tools. As I implied before, you don't need to be using all of these tools to get significant benefit from CI. Start small and build it up. Ok, there's my disclaimer. On with the list:

Java Platform

SCM
Subversion

Build Tool
Ant
Maven

Dependency Management
Ivy
Archiva

Continuous Integration Server
Hudson

Automated Testing
JUnit (Unit)
DbUnit (Component with JUnit)
Selenium (Functional)
JMeter (Load)
SoapUI (Web Services)
AntUnit (Build testing)

Database Upgrades
Liquibase

Deployment
Java Secure Channel

Developer Documentation
Grand (Build Diagrams)
UMLGraph (UML)
SchemaSpy (ERD)

Code Quality Analysis
CheckStyle (Coding Standards)
FindBugs (Code Errors)
Cobertura (Code Coverage)
JavaNCSS (Cyclomatic Complexity)
Simian (Code Duplication)
JDepend (Dependency Analysis)
Sonar (Code Quality Reporting & Aggregation)
ratproxy (security)

GUI Installer
IzPack

Feedback

Email
X10
SMS
Jabber (IM)