Monorepo vs. Bill of Materials: Shaping Projects

During most of my development career, the usual code organization pattern was monolithic repository or Monorepo. There is a single large version control repository, to which all developers on the project push their commits, sometimes breaking it. Every once in a while, this goliath is released as a set of artifacts. Before this may happen, all development activity ceases as we need to stabilize the code base and make sure every component is ready to be released.

At Chronicle Software, our development efforts are structured around the Maven Bill of Materials (BOM) paradigm. At first, I met this technology with skepticism. Thirty-five git repositories instead of just one? Having to update dependency versions in a separate BOM repository just to test cross-module changes? There was suspicion that it would only create busywork. A bit later, I have changed my mind significantly.

Ease of Use

First of all, it provides value for our users: they don’t need to track specific versions of Chronicle products. Instead, they pick a version of BOM dependency and import a few, or as many artifacts as they wish without specifying their versions, and always get compatible ones.

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>net.openhft</groupId>
                <artifactId>chronicle-bom</artifactId>
                <!--
                  Normally, this is the only place where the Chronicle version is specified.
                  It may always be easily bumped to the latest.
                -->
                <version>2.22ea50</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <!--
          For the following artifacts, the version is not specified so it is taken from BOM.
          These versions are mentioned in the same BOM file, so they are compatible,
          and will not conflict.
        -->
        <dependency>
            <groupId>net.openhft</groupId>
            <artifactId>chronicle-wire</artifactId>
        </dependency>        

        <dependency>
            <groupId>net.openhft</groupId>
            <artifactId>chronicle-queue</artifactId>
        </dependency>        

        <dependency>
            <groupId>net.openhft</groupId>
            <artifactId>chronicle-bytes</artifactId>
        </dependency>
    </dependencies>

Release Early, Release Often

Using Bill of Materials also provides value for us: We can ship new releases faster than having a Monorepo would ever allow, since only affected products need to be built, tested, and released. For most new releases, the majority of our codebase is not touched, with versions retained from the previous Bill of Materials release. Of course, we still run any production changes against the whole array of tests contained in all projects before merging to the main branch – which happens in the background so it does not hinder productivity. Before performing the release, we just need to bump all affected artifacts’ versions in the Bill of Materials POM file:

    <dependencyManagement>
        <dependencies>
            <!-- core projects -->
            <dependency>
                <groupId>net.openhft</groupId>
                <artifactId>chronicle-core</artifactId>
                <version>2.22ea15-SNAPSHOT</version> <!-- Was: 2.22ea14 -->
            </dependency>

            <dependency>
                <groupId>net.openhft</groupId>
                <artifactId>chronicle-threads</artifactId>
                <version>2.22ea8</version>
            </dependency>

            <dependency>
                <groupId>net.openhft</groupId>
                <artifactId>chronicle-bytes</artifactId>
                <version>2.22ea12-SNAPSHOT</version> <!-- Was: 2.22ea11 -->
            </dependency>
        </dependencies>
    </dependencyManagement>

Change, One Commit a Time

BOM allows us to gradually improve a core library without the risk of breaking all downstream dependencies at once. In the case of monorepo, API or behavior changes usually need to be done in a single large “code bomb” commit when a change affects a lot of code in depending projects. The changes may be pushed under SNAPSHOT version of the core library as they evolve, while the BOM will still refer to a previous, stable version of that library. We are free to decide when to release the library version, while still having the development process in the main branch, as opposed to keeping it in a feature branch that is bound to diverge from the main branch and needs repetitive manual merges.

Discussion: Benefits of Structured Dependencies

The legacy modular approach is to spin off common code, such as network layer, serialization, logging – by creating separate libraries with their own release life cycle. Still, developers often ended up copying the code from these libraries to improve it without waiting for the next release, then pushing it upstream later. Moreover, their users could end up with different, conflicting versions of a library.

A Monorepo was in many ways introduced to avoid code duplication between teams who often work on multiple related projects – now they could directly share code residing in a single repository, so the developers would always use the latest version. Not so simple for the users, since the code could only be released when the whole development repository has been stabilized.

With Bill of Materials we can further improve on that. There are separate libraries for low-level abstractions such as data access “Bytes” or marshalling “Wire”. We can immediately release a new version of a library together with a higher level product, such as distributed “Queue Enterprise”, all in one incremented version of “Chronicle BOM”. All the rest of our projects are unaffected, but still available to the user moving to that “Chronicle BOM” version in their latest stable form.

Conclusion

Chronicle Software’s ecosystem has a growing number of solutions with a focus on low-latency trading, so we had to make sure our development efforts have a short response loop as well. Using the Bill of Materials-based release system allows us to rapidly deliver new features by shipping them just in time.

Resources

• Chronicle Bill of Materials on GitHub (open-source)
• Chronicle Bill of Materials Maven Repository