Long-Running Branches Considered Harmful

originally posted by jade rubick

i work on the core app team at new relic. we’re responsible for running the product website and data collectors that collect all the juicy data you see on our charts and graphs. and recently we hit a milestone. last week, we deployed to our site every weekday. this velocity allows us to be a more responsive organization. but it also makes the pain points that much more obvious. it forces us to continually improve our engineering practices and to have a lot of conversations that we wouldn’t have otherwise.

one of these conversations is about how to deploy more than once a day. we know that this is going to require a lot of changes, so our team is arguing the merits and tradeoffs of such changes. culturally, new relic is very fond of experimentation. if we’re unsure of a course of action, we’ll try something and see what the result is. so this week, we’re experimenting with pull requests.

although pull requests seem like they’re always a great idea, they do have some drawbacks. they make your git history less linear and more confusing to navigate. this is a section of our git history before we use pull requests. (see how nice and clean looking it is to have a linear git history!)

and this is just a small portion of our git history from last week, when we started using pull requests. (looks like the information superhighway!)

there are some things we really like about pull requests, though. they give us a nice mechanism for code review and agile qa before code is merged to master.

but another disadvantage of pull requests can be that this workflow promotes long-lived feature branches , unless you take specific steps to combat them. when the core app team discussed this issue, there was a lot of disagreement as to whether long-lived feature branches were a good idea or a bad one. so this is my public argument against them and why i think they’re almost never a good idea.

merging your code back in frequently == communication with your team
people will often say that as long as you are rebasing your branches frequently from master, it’s not a big deal to have long-lived feature branches.

i disagree.

long-lived feature branches hide your work from the rest of your team. this is fine if no one else touches any aspect of the work you’re doing. (and having too much overlap may be a sign that it’s time to break up your app.) but the minute two people are working on the same section of the code, you’re each blind to how your work affects the other person.

think of your work as delta off of master. the size of the delta increases as your branch incorporates more and more work. as the size of your team grows, the amount of work hidden from each other increases. and the chances that your assumptions about the state of the code hold true decreases the more you use long-lived feature branches.

if you merge your code back to master frequently, you demonstrate the direction you’re taking the code. for example, you may add a conditional that shows the new code you’re working on and have it disabled by default. if some else is working on the code, they know what you’re doing with it.

you can argue this makes your code more complex. however the fact is, that even if you haven’t merged you code in, the conditional is in fact there. it’s just invisible to the rest of your team.

merging your code frequently avoids integration pain
when you merge your code more frequently to master, the pain of integration happens at the beginning instead of the end of your work. you can find issues faster and fix them at the earliest possible moment. this gives you time to react and communicate with colleagues, saving you a lot of pain and time.

merging your code back in forces incremental development
if you’re continually merging your code back to master, you are forced to do your work in increments that preserve existing functionality. you take smaller steps, which generally breaks less and leads to more stable development. and when you do break something, you can find it sooner and fix it faster, instead of waiting days or weeks. you’ll also have more context about how to fix things and acquire information about your assumptions of the code that much earlier. in general, being forced to take smaller steps is a good thing. a large part of our craft is about breaking work down into manageable chunks.

there are times you have to break code to make it work in a new way. generally, this is less common if you challenge whether breaking it is really necessary. but often it is absolutely necessary and in those cases, you merge things back in at the first time it is practical to do so.

feature branches are inventory
you can think of feature branches as inventory. in the same way that inventory is necessary to build something, feature branches (and sometimes even long term feature branches) can sometimes be necessary. the longer they run, however, the more cost your company incurs without any benefit being brought into the actual product.

to reduce the cost, merge frequently with feature flags. this has a nice side benefit. you can turn on the feature to a percentage of your customers and deliver that value earlier, or collect earlier feedback.

merging your code frequently results in fewer merge conflicts
when we started on our ruby 1.9 upgrade, our first task was to get the tests running on ruby 1.9. we began by making a branch for the upgrade and then we would rebase frequently.

we quickly ran into a problem. there was so much work on master that it was a huge pain to rebase our work every day. i believe i’m quite decent with git, but i quickly despaired of being able to keep the branch rebased. i was spending as much time with merge conflicts as i was spending on fixing tests. i merged the code to master and the process was much, much smoother after that.

some will argue that it isn’t always possible to do work outside a long-lived feature branch. for example, what about an upgrade from rails 2 to rails 3, a major architectural change? although it often seems impossible to make large changes incrementally, i think it’s worthwhile to push really hard to see if it is in fact possible. two of my colleagues are now embarking on a rails 3 upgrade and they’re frequently merging their work back to master. i’m sure they’ll have more to share about their experience in a future blog post, but it’s hugely simplifying their work. if they were doing it on a feature branch, they’d be spending much of their time fixing merge conflicts with all the work the rest of the engineers in the organization are making against master. and the rest of us would have no idea we were breaking all their work on the rails 3 upgrade. how much easier will it be when everyone else can see their work?

long-lived branches lead to larger chunks of work. is that a good thing?
there is one argument for long-lived branches that i initially sympathized with. long-lived branches allow you to commit one chunk of work that represents all the work, in one increment. and it’s immediately understandable what the work is because it’s in one commit.

admittedly, that’s very appealing. it can seem messy to have to deal with a pile of incremental commits. merging once a day per developer is a lot more commits to wade through. it also means you only need to review and integrate one chunk of work, which sounds great in practice. but it does mean that the size of that chunk is much larger.

increasing the size of changes increases risk. it’s essentially the same as deploying your code less frequently. the amount of change is larger and the risk is greater. instead of pipelining your process, you either hold it up and commit a huge block of code or you do it in isolated segments, where people are blind to your changes. the best practice, i believe, is to find ways to pipeline your process so every aspect of your engineering process is pipelined.

conclusion
so, that’s my argument against long-lived feature branches. what do you think? did i miss an important point? let me know if the comments below. i’d love to hear your thoughts and experiences.

jade works on the core team at new relic. he’s best known for his tweet: “if someone told you you had to kill someone to get a job at new relic, you should seriously consider it. it is that good here @newrelic”. if you’d like to join our team, no murders are necessary — contact us through newrelic.com/jobs