Git Flow and Immutable Build Artifacts

We love Git Flow. It’s awesome to have a standard release process where everybody is using the same terminology and tooling. It’s also great to have out-of-the-box answers to the same questions that get asked at the start of every project. For example:

• “How are we going to develop features in isolation?”
• “How are we going to separate release candidates from ongoing development?”
• “How are we going to deal with hotfixes?”

Now it’s enough to just say ‘We use Git Flow’, and everybody’s on the same page.

Well, mostly. Whilst Git Flow is terrific for managing features and separating release candidates from ongoing development, things get a little hazier when it comes time to actually release to production. This is because of a mismatch between the way that Git Flow works and another practice that is common in large development projects: immutable build artifacts.

Immutable what?

On most enterprise projects I work on these days, it’s considered a pretty good idea to progress exactly the same build artifact through testing, pre-production and into production. It doesn’t matter whether it’s a JAR file, WAR file, tar.gz file, or something more exotic – the key point is that you build it once, then deploy the same thing to each downstream environment. If you find a bug during this journey, you fix the bug, build a whole new artifact, and start the process again.

In the absence of any established terminology, let’s just call these things immutable build artifacts. (Note that you’ll still need a separate, external file for those things that have to be different between environments, but by keeping the amount of stuff in that file to an absolute minimum, you’ll minimise your potential exposure to variances between environments.)

Without immutable build artifacts, many development managers will start to sweat nervously. This is usually because at some stage in their careers they’ve been up at 2am in the morning prising apart build artifacts and trying to understand what changed between the pre-prod build (which worked just fine) and the current prod build (which is inexplicably failing).

A bunch of things can cause this sort of problem. For example, it could be that somebody managed to sneak some change in between the two builds, or that some downstream build dependency changed between the two builds, or even that somebody tweaked the build process between the two builds.

‘That’ll never happen to me’, I hear you say, ‘if everybody follows our development methodology correctly’. And therein lies the problem:  it’s difficult to absolutely guarantee that a developer won’t at the last minute do something stupid, like slip something into the wrong branch, upload an incorrectly-versioned downstream dependency, or mess with your buildbox configuration.

My point is this: why open yourself up to the risk at all, when having a single artifact will eliminate a whole class of potential defects?

Now for the actual problem

A while back, I was working on a project using both Git Flow and immutable builds.

A junior developer came to me looking confused. He was trying to understand when it would be OK for him to finish the current Git Flow release. The testers had just signed off on the current build. But if he finished the release, Git Flow was going to merge the release branch into master – and strictly speaking, he should create a new build from that. But if he created a new build, the testers were going to want to test it.

If they found a problem, the fix was going to have to happen in a new release branch. But then when he closed that release, he was going to have to do a new build from master, which the testers would want to test again, right? And then, if they found a bug in that…

I admired his highly conscientious approach to his work, but worried that his mind was going to disintegrate as it spun around this build-management paradox.

I also had to acknowledge that he had stumbled head first into something that I had been wilfully ignoring in the hope that it would resolve itself; namely, the fundamental mismatch between Git Flow and the concept of immutable builds.

Put simply, Git Flow works on the premise that production builds will only be done from the master branch. In contrast, immutable builds work on the premise that production builds will be done from either a release branch or a hotfix branch. There is no perfect way to work around this mismatch.

The pragmatic solution

Put bluntly, if you want truly immutable builds, then you should do them from the release or hotfix branch rather than master. However, because this runs contrary to how Git Flow works, there are a couple of important consequences to keep in mind.

1. Concurrent hotfixes and releases require special handling

If you start and then finish a hotfix whilst a release branch is in process, then that hotfix’s changes won’t automatically be brought into the release branch by Git Flow. This means that a subsequent build from the release branch which goes into production won’t include the hotfix changes. Here’s a diagram illustrating the problem:

Note that the inverse problem would apply if you tried to start a release whilst a hotfix was in progress. Thankfully, Git Flow will not let you have two release branches in progress at the same time, so we don’t have to worry about that particular possibility.

(If you’re wondering how Git Flow avoids these scenarios in its regular usage, remember that it always merges back into master when a release or hotfix is finished, and that you’re supposed to always build from master. Consequently, if you’re building from master post-release or post-hotfix, you’ll always be getting the latest changes into the build.)

The problem of concurrent hotfixes and releases can be solved by merging the hotfix branch into the release branch just before the hotfix branch gets finished (or vice-versa if you started a release whilst a hotfix was in progress). Here’s what it looks like:

However, you will need to remember to do this merge manually because Git Flow won’t do it for you.

2. Version tags will be incorrect by default

When you finish a release or hotfix, Git Flow merges the corresponding branch back into master, and then tags the resultant merge commit on master with the version number. However, because your immutable artifact will have been built from the release/hotfix branch, the commit SHA for the version tag in Git will be different from the SHA that the artifact was actually built from. Continuing on from our previous example:

You may or may not care about this, for a number of reasons.

Firstly, from the perspective of Git Flow, the merge commit on master should never have any changes in it. The only scenario that might lead to it having changes is if you have run concurrent hotfix and release branches (as described in the previous section) and forgotten to merge them prior to finishing. And you’llnever forget to do that now will you? :)

Secondly, in the likely event that you are using some sort of continuous integration server to produce your builds, that server can probably associate its own number with each build, and stamp the resultant artifact with that number. The CI server will probably also have recorded the SHA that each build was done from. So from the artifact you could probably work backwards to the SHA that was used to produce it.

If you’re nevertheless wary of the confusion that this backtracking process might introduce when trying to debug a production issue at 2am in the morning, you might still insist on having correct version tags.

In that case the best thing I can think of is to manually create the tag on the release/hotfix branch yourself before finishing it and then, when finishing the release, run git flow [release/hotfix] finish  with the -n option to stop it from trying to create the tag again (which would fail because the tag now already exists).

I’m on a roll with my diagrams, so what the heck, let’s do one more:

Wrapping Up

On balance, I think the benefits of having immutable builds outweigh the costs of deviating from the Git Flow way of doing things. However, you do have to be aware of a couple of problematic scenarios.

We’ll be experimenting in coming months with using Git Flow with our own immutable builds, and will let you know if anything else weird pops up. Who knows, perhaps we’ll even end up with our own version of Git Flow. Either way, I’d be interested in hearing from anybody else who has had the same problem.