Preflight Builds: There's A Time And A Place

Over the past several years, build management and Continuous Integration tool vendors have suggested that their tools should manage not just authoritative builds from source control, but also replace the test builds done on the developer’s desktop. While useful technology in specialized situations, these preflight (precommit) builds have the potential to undermine best practices -- doing more harm than good.

Unfortunately, the hype around preflight-build technology pushes more teams to use it than should. As Urbancode prepared to include preflight builds in the next release of AnthillPro, we took a good look at where they are actually useful and where they are detrimental.

What Are Preflight Builds?

The standard practice for teams employing Continuous Integration is that a developer should run the build and fast unit tests on his or her own machine prior to committing changes to source control. This practice cuts down on broken builds and prevents simple mistakes from interfering with the work of other developers. In recent years, a trend amongst Continuous Integration servers has been to offer tools that allow developers to run builds of their modified code on the authoritative build environment prior to committing to source control. This functionality is often called a “pre-commit”, “preflight”, “pre-tested” or “pre-integration” build. By running the build and tests in the official build environment, rather than the developer’s workstation, the developer has greater certainty in the quality of the change.

Preflight Hype and Reality

Tools supporting preflight builds come in two groups: The first runs as a developer attempts to commit to source control. It uses a preflight build to validate that commit and blocks any commit that fails the build or tests. The preflight build is used as a hard quality gate. Vendors supplying this functionality proudly proclaim, “unbreakable builds”, “Continuous Integration 2.0”, or that the source code “will never be broken again”. 

Unfortunately, reality doesn’t always follow hype. Consider this: Two developers commit changes to the same code base at the same time. The commits will need to be serialized (which does not scale). Otherwise, there is the possibility of integration errors (breaking the “unbreakable” build), and if the same files were modified, merge conflicts may occur that the build system is left to resolve.

Further, there may be times when breaking the build or tests is intentional. A developer may intentionally break the tests when a bug is discovered for which a failing test can be added more quickly than a fix. In this case, committing the broken tests to source control documents the defect in a useful way.

Preflight builds used as a hard quality gate to prevent broken builds remain vulnerable to breakage when they should not be -- merging changes into the source repository -- and are unbreakable when they should be. What are the gains? If project teams are healthy, and have true Continuous Integration implemented, a developer who breaks the build will be notified several minutes after committing. The developer can then choose to either rollback her/his changes or correct the mistake several minutes after that. In the rare case of a problem commit, the error should be rectified in less than 30 minutes. Preflight builds as a quality gate are a bad solution to a problem that teams should not be facing.

The second group of tools providing preflight builds -- AnthillPro included -- offers a more straightforward preflight option. Developers may run a production-style build on their uncommitted code.  If they like the results, they can commit after that, or keep on working if the results of the build are not acceptable. This is essentially identical to the preflight builds already discussed; however, there is no automatic commit, nor do these preflight builds act as a quality gate that forbids changes that break the build or tests. While a relatively simple offering, even vendors in this group are prone to hyperbole.

The message from vendors in both groups is essentially that preflight builds are fantastic, and everyone should be employing them. We politely disagree. They are a worthwhile option to have; however, only a small fraction of teams should actually use them.  For most teams, preflight builds slow the Continuous Integration cycle by discouraging commits, utilizing shared hardware and adding unnecessary roadblocks, or provide just enough "help" to allow bad practices such as fragile builds and poor development environments to remain hidden. 

Appropriate Uses for Preflight Builds

To understand when preflight builds are useful, and when they become redundant, it is important to examine why tool vendors now argue that traditional developer builds should be replaced by preflight builds.

The first argument relies on the incompetence or laziness of developers. These "bad" developers, it is argued, fail to adequately test their changes before committing, and are so lazy or incompetent that when the Continuous Integration build breaks, they do not fix it quickly. In this case, the problem is not tooling.

More likely, preflight builds are said to be necessary when the development team is transitioning from waterfall to Agile development practices. In this scenario, teams are often large, the build is long (more than 30 minutes), and integration errors become common. The result is that the build is always broken. And because Agile practices are still immature, build breakages are not discovered quickly, and it becomes very difficult to figure out where the fault is and/or who is responsible for the break. Often, this results in infrequent integrations because developers delay committing changes in order not to break the build. In this scenario, preflight builds can help developers test integrations without committing, and thus relax that cycle of fear during the time of transition.

Finally, preflight builds are said to be useful when addressing the differences between the individual developer's environment and that of the formal build environment. For many organizations, the formal build requires multiple platforms, or specialized hardware/software that cannot be included in the developer's environment. Under these conditions, a build on the developer's machine provides little information about the likelihood of success once the changes are committed. Indeed, developers writing software for custom hardware that cannot be located at each developer’s desk (like an aircraft or point of sale device) are the best candidates for the long-term use of preflight builds.

However, most development teams should be able to establish development environments that build practically the same way as the authoritative environment, and have no real need for preflight builds. In fact, relying on preflight builds may hamper the best practices required to institute such environments. Likewise, it’s often possible to break applications into components that build quickly and expose fewer developers to breakages -- without relying on preflight builds to mask procedural flaws. Once an organization has solved the problems with environmental differences, build failures will become less frequent, have a smaller impact on the teams, and be fixed more readily because developers will be following best practices when setting up their development environment. Preflight builds, however, can provide critical support to the team during the transition by reassuring developers that their changes are good on their own prior to integration.

The Risks of Preflight Builds

The inherent danger of preflight builds is that development teams using preflight builds to get past their worst pain points will never take the next steps to address the root causes of their problems (such as slow builds and mismatched environments). This is like treating a badly broken leg with just crutches, rather than resetting the bone and letting the leg fully heal. While the pain point (broken builds) is addressed, the actual problem may never be addressed when teams rely on preflight builds.

In the real world, teams that rely too heavily on preflight builds run the risk of never fully adopting Agile practices, and may never see the full benefits of Continuous Integration. A developer who has been reassured his changes work well in isolation may be slower to commit, or put in the effort to reconcile the integration problems or align his local development environment’s settings to that of the authoritative build environment. Therein lies the trepidation about preflight builds. For teams with unique hardware, or those transitioning to Continuous Integration and other Agile development practices, it has been shown that preflight builds are a useful tool. However, it is exactly those teams, by relying on preflight builds, that are in danger of using preflight builds to ignore the real and serious structural problems.

Moving Forward

In the end, vendors need to be more forthright with customers. While features like preflight builds are very powerful, vendors should acknowledge the dangers and pitfalls of preflight builds. Preflight builds, like most new technologies are neither a panacea nor a silver bullet. It is the responsibility of every tool vendor to give proper instruction on the use of preflight builds, and inform their customers when the use of preflight builds is inappropriate.

On the other side, organizations need to carefully examine whether they should be using preflight builds or not, and require the vendor to provide an alternative solution to using preflight builds. Organizations should be able to choose from different options, and not simply rely on the word of a vendor trying to sell their latest and greatest feature. A simple question to ask is: "Do preflight builds provide any value to our teams? If so, is the perceived value a sign that the teams need to improve their processes, or is it a sign of technical hurdles that only preflight builds can overcome?"