This post comes from Matt Sneeden at the New Relic Blog.
Delivering quality software is something that each and every one of us at New Relic cares about. A lot of effort goes into the code that ships with each release. Recently, we took a hard look at how we get new code into our customers’ hands. After completing a release a few months back, the .NET agent team sat down and looked at what it takes to get a tested release candidate out into the wild. We soon realized that our delivery process was almost entirely manual. It required over 20 complicated steps and literally took about a half a day for one person to complete.
Given that we try to target releases at roughly one month intervals, you may be thinking “Four hours once a month, is that really so bad?” Well, let’s think about that some more. If you strictly look at the time requirements, a four-hour process isn’t so bad. But the real issues lie elsewhere. Lengthy, manual processes are very error prone. It’s extremely easy to lose focus, get distracted and make a mistake, or even skip a step or two, or worse. And if you get back to the release intervals, one month is more than enough time to forget subtle details or items that may or may not be documented.
Automated processes, however, are much more reliable. When designed correctly, they can ensure consistent results and save lots of time. We on the .NET team have spent the time and mental capital to transform our delivery process into a streamlined automated process. What was once accomplished in 20 steps now can be done in a mere two actions that trigger a swath of automated workflows. Now our releases take five minutes instead of four hours. And that gives us an extra three hours and 55 minutes to focus on other ways to make New Relic even better!
How Did We Do It?
Looking at our manual process at a high level, we were able to break up our process into three main blocks and tackle those one at a time. The first involved more of the preparation work, like configuring release jobs in Jenkins to build the code from the correct branch, tag or commit in GitHub. The second block makes the physical push of the version to our production servers. The third and last section, handled the post-release items such as sending emails, updating release notes, etc. Approaching the project this way allowed us to do a couple of things, tackle the effort in smaller, more manageable pieces and also roll out pieces one at a time to prove out each block as we went along.
The way we chunked the pieces up also coincided with the general technologies used in each block. The majority of the phase one work in Jenkins utilized Groovy to interact directly with the Jenkins API. Bash scripts handled the majority of the heavy lifting in phase two to do things like curl the release candidate from Jenkins and then scp it out to our servers. PowerShell brought up the rear in phase three by handling the generation of internal emails, parsing release notes and making HttpWebRequests to update a few other areas not serviceable by APIs.
What’s In It For Our Customers?
As developers, any time we can be more efficient it’s a win. Let’s say there’s a bug in our software. (And yes it does happen.) By improving our internal processes, we can respond and get a fix out the door as quickly as possible. With a quick and easy process, scheduling becomes less of an issue. Previously, our releases took several hours. If the process didn’t start by early afternoon, the chances were a release wouldn’t happen until the next day. Now, that process takes minutes – in the morning, mid-day or late afternoon – and this gives us the ability to push whenever. All of this adds up to customers getting new code in their hands quickly.
How to Succeed
Take your time when automating a manual process. Start from the beginning and take things one-step at a time. As developers, we may want to do it all straight out of the gate. By working slowly, you can focus on each step while keeping the overall goal in the back of your mind. The chances are this process isn’t going to be an overnight thing. Take your wins where you find them. And remember, even if your first efforts only automate five steps, that’s five fewer steps you’ll have to endure the next time around.
Overall, the biggest piece of advice I can give you when attempting to automate a release or other complex process, is to treat the code that handles your release like it is, well, code. Keep it version controlled and maintain it. After all, this could very well be the code that puts the application in the hands of your customers.
Part of the Bigger Picture
These efforts play a major part in our ongoing plans for Continuous Integration (CI). Delivering software efficiently is just one piece of the puzzle in our longer-term goals for improving our agent team processes. We’re already hand at work focusing on improving our functional automated testing and it will soon spider out to other areas as well. Stay tuned …