Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Branching Considered Harmful! (for Continuous Integration)

DZone's Guide to

Branching Considered Harmful! (for Continuous Integration)

Branch-based development can be a good trade-off in some scenarios but should be considered harmful for true CI. Branches can also be useful in trunk-based environments.

· DevOps Zone
Free Resource

Download “The DevOps Journey - From Waterfall to Continuous Delivery” to learn learn about the importance of integrating automated testing into the DevOps workflow, brought to you in partnership with Sauce Labs.

This article is featured in the new DZone Guide to DevOps: Continuous Delivery and Automation, Volume IV. Get your free copy for more insightful articles, industry statistics, and more!

That’s all – a pretty straightforward setup. But what about style checks or architectural reviews? What about all those work-in-progress commits? Won’t we be breaking production all the time with problems the tests don’t catch, or with incompatibilities that we’re introducing?

Yes, in most current setups committing straight to trunk and thence to production would indeed end up breaking things
a fair amount of the time. And that is precisely the point: by allowing ourselves to develop “off trunk” for long periods of time, we allow ourselves to ignore the work necessary to protect trunk and production from bad code.

Avoiding work may not sound like a bad thing, but it almost always comes at the cost of a steadily worsening review bottleneck that can see even simple changes delayed for days or more. And if we are truly aiming for a continuous deployment setup, we will eventually need to put in the effort to protect trunk anyway.

Furthermore, the steps we need to take in a trunk-based development environment in order to prevent production incidents are exactly the kinds of best practice steps we generally talk about striving for.

If you’re working in what you might consider a reasonably modern software development environment, chances are you’re doing Agile, and are aspiring to build out a Continuous Integration and Deployment pipeline to push code to production quickly. Chances are that you’re also doing some form of branch-based development — whether straightforward feature branches, something more formalized such as git-flow, or a pull request-based process using GitHub or similar.

Why the branches? How and when did the perception arise that branching is A Good Thing? A branch-based development strategy generally implies delayed merging, and delayed merging is, almost literally, the exact opposite of the “continuous integration” most teams consider themselves to be practicing. Branch-based development usually means either largely more complex and risky merges, or a manual code review bottleneck, or both — things we almost never want in our process if the aim to deliver code to production quickly.

I should brie y clarify that what I mean by “branch-based development strategy” here are approaches that allow for, or even encourage, the existence of branches that accumulate changes that are not on trunk (or “master”, or “HEAD”, or whatever the mainline branch in your environment is called) for days, weeks, or longer. These changes are only incorporated into trunk after an explicit, non-automated approval. There are also development strategies using branches that in practice look much more like trunk-based development — more on those later.

But isn’t branch-based development so well supported by leading source control management systems such as Git because it’s A Good Thing? Aren’t contributions via pull requests, and hence branches, the way many, many well-known and successful open-source projects work?

Yes, many open-source projects, including the Linux kernel community that gave rise to Git, use branches in their development strategy — and for good reason. If you are working in a highly distributed environment where there is no centralized server, there can be no notion of “merging to master” or “committing to trunk,” because no such thing as trunk exists. Determining the “current global state of trunk” requires merging multiple tracks of development from separate repositories.

Even if you have a centralized server which is the canonical reference for a particular codebase, branches and pull request reviews make sense if you are in an environment where many contributions are being proposed by developers who are not familiar with the code, and who in addition are not in a position to ask for advice from more expert developers on an ongoing basis. They may be geographically separated, with few or no overlapping working hours to allow for discussions, or they may work for different companies and simply do not have the opportunity to contact the experts often.

In such cases, a development strategy using branches and delayed integration — usually after manual code review — makes sense as an acceptable compromise given the various constraints. However, projects in these situations are very rarely examples of code that gets shipped to production frequently. So if that is our goal, they hardly seem like models that we should be seeking to emulate.

More importantly, the constraints under which these projects operate are simply not applicable to most commercial software development settings. We almost always have a central repository whose mainline is the global reference point so that the idea of “merging to trunk” on an ongoing basis is indeed meaningful. And we usually have sufficiently frequent access to someone with enough experience of the codebase that we could arrange for reasonably regular feedback if we so chose.

“So what exactly are you proposing?” I hear you ask. I am proposing something we’ll refer to here as “trunk-based development.” It’s not a new concept by any means, having been around at least as long as the idea of continuous integration, to which it is closely related. But I think that, if we are serious in our desire to ship code quickly, it’s definitely a concept worth revisiting. It goes something like this:

  1. When you have written an amount of code locally that you think should work, commit it to trunk.
  2. A commit to trunk triggers a series of tests and verification steps that are intended to ensure that the system will not be negatively impacted by the change.
  3. If all the tests and verification steps succeed, the code change is deployed to production.
  4. Otherwise, the commit could be rolled back automatically, or the “offending” developer is notified that they have broken trunk.

That’s all — a pretty straightforward setup. But what about style checks or architectural reviews? What about all those work-in-progress commits? Won’t we be breaking production all the time with problems the tests don’t catch, or with incompatibilities that we’re introducing?

Yes, in most current setups committing straight to trunk and thence to production would indeed end up breaking things a fair amount of the time. And that is precisely the point: by allowing ourselves to develop “off trunk” for long periods of time, we allow ourselves to ignore the work necessary to protect trunk and production from bad code.

Avoiding work may not sound like a bad thing, but it almost always comes at the cost of a steadily worsening review bottleneck that can see even simple changes delayed for days or more. And if we are truly aiming for a continuous deployment setup, we will eventually need to put in the effort to protect trunk anyway.

Furthermore, the steps we need to take in a trunk-based development environment in order to prevent production incidents are exactly the kinds of best practice steps we generally talk about striving for.

Want more architectural and code style input? Introduce regular “check-ins” with a more experienced colleague, or do more pair programming! Concerned about checking in code that compiles but doesn’t work correctly when it makes it to production? Write more tests to ensure your code does what you expect! Working on a feature that shouldn’t be active until changes to other systems are deployed? Use feature flags! Aware that your system is too complicated to be tested reliably with unit and integration tests? Implement automated monitoring of system health, customer metrics, etc., and alert or roll back if behavior is unexpectedly poor! And so on.

These are all things we know about and generally agree that we should have — we just never get round to them because we allow ourselves to work in a manner that doesn’t compel us to implement them to keep our systems running.

But if we do force ourselves to go down this path, the bene ts in terms of speed can be enormous. A colleague who tried to see how far he could take this ended up with what he called “ludicrous mode:” every time a file was saved in his IDE, the code would be compiled and, if successful, the unit tests for that project were run. If those succeeded, the change would automatically be committed to trunk and, if no stages in the subsequent delivery pipeline failed, would make its way all the way to production.

At his peak, he was deploying to production fifty times an hour. But he said that the first thing the experiment taught him is that it was essential to write failing unit tests first before starting on the actual implementation, because that was required to ensure bad code didn’t make it to production. Good practice not as a nice-to-have, but out of necessity!

Having said all that, in practical terms, there are still good use cases for branches, of course: for example, they are often the easiest way of “throwing a code diff out there,” especially during a discussion with remote participants. It’s also easier in many current continuous integration and deployment tools to configure a “create pull request, trigger tests, automatically merge to trunk, continue pipeline” flow than the “merge to trunk, trigger tests, continue pipeline or automatically revert” process sketched above.

The key to both of these cases, though, is that they involve, respectively, throw-away and short-lived branches that are automatically merged into trunk. There is no long-running “off trunk” development followed by a large merge or manual review.

Of course, for most teams (including the ones I am part of!) trunk-based development isn’t something that could be implemented today: a lot of the work required to protect trunk from bad commits is still needed, and that work isn’t easily or quickly done. But we can embrace trunk-based development as a desirable goal and reconsider our use of branch-based approaches that build delay and manual bottlenecks into our flow to production. That way, we can get back to focusing on what it would take to build a system and process that will allow us to deliver code as quickly and safely as we can.

So go on, ask yourself: what would your team need to do to be able to develop safely on trunk?

More DevOps Goodness

For more insights on implementing unambiguous code requirements, Continuous Delivery anti-patterns, best practices for microservices and containers, and more, get your free copy of the new DZone Guide to DevOps!

If you'd like to see other articles in this guide, be sure to check out:

Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure, brought to you in partnership with Sauce Labs

Topics:
devops ,trunking ,branching ,continuous integration

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}