The Cost of Unmerge

one of the reasons you’re going to choose a trunk based development (tbd) model, is because you’re doing concurrent development of consecutive releases. maybe that’s not even your choice as a development team, but the people funding the development work already have a series of big releases planned, and even eighteen months out, they have a clear idea of the order of releases and are planning the marketing campaigns around them, and it’s pretty clear that you can’t delay the start of the each project because of the amount of work needed.

what if the unthinkable happens? specifically the order of releases is changed or a releases is cancelled, after usability testing shows it as popular be popular with end-users as dirt frosting on a donut.

i think we all know the solution i’ll propose, but lets drill into the problem is we were using non-trunk branching models.

cascading branches.

remember: your development teams are all working concurrently.

in this model the release 1.0 project team has it easiest. they just work towards their completion. periodically though the release 1.1 project team take merges (everything, not just cherry-picks). mostly likely a dev-pair will work for between half a day to a couple of days to run through all the merge conflict resolutions. this is especially true if the changes were happening to the same parts of a web page, or the same classes. it’s also true if the team has been authorized to do refactorings as they go. if the ‘hit’ is taken weekly it’ll be longer, whereas daily merges from ‘upstream’ might be easier. then again if the 1.0 team is 30 people, and the 1.1 is 20 people, then there is a likelihood that somebody in the 1.1 team is permanently arbitrating over merge activities. at least, that’s true for high-throughput teams.

release 1.1 being cancelled, or being re-sequenced after what would have been release 1.2, is pretty much a nuclear option. specifically is it pertains to part finished projects and merges you wish you’d not made. if either of those happens you’re going to have some down-time for development while many dozens of people are involved in work to neutralize (temporarily or permanently) work that’s been merged to 1.2 (and/or 1.3), then start merging that ‘neutralization’ again. this is the “unmerge” that is costly of the article title. the internal customer paying for the dev work, wonders why from their point of view, dev work essentially stops for a week or a month, yet salaries are still being paid.

it need not be a whole release that is scrapped or resequenced, it could be just some aspects of it.

classic clearcase branching

clearcase has a classic branching style, which counts a main-line branch as a pristine representation of previous released work, but actual development work-streams happen on a branch that is named for a particular release. as well as developer commits, actual releases happen from that release branch. because both are happening on the same branch, a code freeze incrementally kicks in to protect the release from mistakes. after the release, everything is merged back to the mainline. of course defects happen, so the release branch stays alive longer than is intended, and more merges happen after successive releases from there.

meanwhile, perhaps slightly in advance of the incremental code-freeze, a new release branch is cut, and preparatory work happens there towards that later release. after the release from the former branch, and the merge down to mainline, there will be a consequential merge out from mainline to the new release branch, bringing them up to date. these merges are “all” rather than cherry picks, and are not cheap.

the problem is when you’re canceling or delaying some components of the release. you’re going to have to devote some resources to commenting out code, or wrapping it around conditionals tied to a feature-toggle, when you’d not previously intended that. the latter is better of course. following that there’s a larger testing schedule and to ensure you got them all. again, that essentially costly unmerge.

the correct solution

the right thing to do is trunk based development together with branch by abstraction, and feature toggles.

when you have multiple teams working in one codeline, and they’re toggling everything because their code is going out long before marketing are ready to trumpet the arrival of the associated features, then there’s no cost of unmerge. the only cost is setting up another jenkins (substitute your choice of ci server) pipeline for a permutation of toggles that you’ve not previously tested. maybe there’s some small work after that to iron out the kinks.

this actually happened to a client of mine. we’d advocated for a shift to trunk based development for many months, and won it because of the huge pain or merging changes from a upstream branch (high throughput team), and subversion was actually breaking in a particular scenario , and we were struggling to find support for it. start a merge, abandon after two days, start over and abandon again after two days gets you to revaluate your branching strategy quite seriously.

source-control tool trends.

these numbers, from indeed.com, are gameable by those with vested interest, but let’s assume nobody would go to that huge effort:

subversion, git and tfs overtook clearcase in ‘09 to ’12. clearcase slowly trends downwards. thank goodness for that i say. i’ll be pleased when ibm/rational eventually pull the plug on it. rational now have “rational team concert” (rtc), which i have not shown. it is at the same level of interest as starteam (the black line at the bottom). starteam should also go the way of the dodo in my opinion. mercurial is as good as git really, but has been wrong-footed by the rise of git and github. even microsoft realized that git is unstoppable when it endorsed it in their tfs ecosystem . perforce chugs along as the darling of google, apple, and multi-platform games companies. tfs and subversion spent a lot of time matching the feature-set and mode of operation of perforce of course. subversion has peaked, obviously, but still remains the gorilla in the room. i think mercurial and subversion should merge, as selenium and webdriver did , and struts2 and webwork2 and rails3 & merb did.