Parallel Builds and Modularity for Faster Releases

Leading software development companies are now deploying software multiple times per day. In this sort of environment, even small periods of waiting can add up to significant disruptions. One of the bottlenecks we have heard from our customers lately is with their builds. If you are deploying your software multiple times per day, you are a building your software multiple times per day, the faster you build, the faster you can deploy your software. We recently reduced one of our clients build times by almost 70% using parallel builds and modularity, which allowed them to release their software faster.

You may also enjoy:  Patterns of Modular Architecture (RefCard)

How Do Parallel Builds Work?

Many popular tools support parallel builds including JFrog’s Conan, Maven, Gradle, and MSBuild. This allows you to utilize the full power of your hardware. Parallel builds analyses your project's dependencies and builds the modules in parallel based on the inter-module dependencies. According to Maven, your build performance can improve 20-50% by implementing parallel builds, but performance depends greatly on your software architecture (or how modular your software is). Below is an example representation of a software application’s dependencies. Each letter in the graph represents a module in the code.

This graph is calculated based on the inter-module dependencies in the source code. Assuming all the modules have equal running time, the build will run module A first, then run modules [D, F, C, E, B] in parallel, then run modules [H, I, G] in parallel, etc., until all the modules are built. In the example above, we are able to build a number of the modules in parallel, which is a significant performance boost to the build. But the performance depends greatly on your software architecture. Parallel building rewards modular architectures. If your software architecture has eroded (excessive inter-module dependencies) cleaning up your architecture (making it more modular) will boost your performance.

Using Modularity to Speed Up Parallel Builds

One of the best ways to make your architecture more modular is to use a dependency management tool (like Lattix). A dependency management tool will help you understand how and why various parts of your code are interconnected. They do this by providing you a visual representation of the dependencies in your code. As an example, we will use the Lattix Dependency Structure Matrix and a Graphviz graph view of some sample code to show how breaking the software into modules boosts build performance. Here is the same example from above in both graph and matrix format:

As you can see both the graph and matrix format give you the same information. You can easily see in both formats the hierarchy (levels) and which modules can be built in parallel. For small projects, we use both visualizations interchangeably. The problem is when the source code becomes large. Graph views do not scale well to large codebases which gives rise to the so-called “Deathstar Diagrams” as seen below:

Meanwhile, the Dependency Structure Matrix scales easily to large codebases. Here are the Dependency Structure Matrix and graph views for the Eclipse source code (over 8 million LOC and over 35,000 classes).

As you can see the Dependency Structure Matrix scales well, still giving you relevant information about your architecture (modules, coupling, and hierarchy) even for large codebases.

In our example, now that we have visualized the dependencies in the application, we realize that the architecture could be more modular. Based on the functionality of the individual components, we can group the components into three separate modules. This happens frequently as codebases mature what starts out as three separate modules each performing an individual function becomes one large module containing all the functionality. If we remove the dependencies between modules E->I and modules B->G, we now have a more modular architecture with three modules instead of one.

We have eliminated two entire levels of hierarchy (from 5 levels down to 3). More importantly, because we now have three independent modules, each one can be built, developed and test independently (and in parallel). You could even split up your development team into three separate teams and give each one a module to work on without fear that they would interfere with each other. This will boost performance not just in the build but in development and testing. You will now be able to release your application much faster. The quality of the entire application will also go up because the impact of an individual defect will be minimized because it will only affect one module.

Conclusion

Faster release cycles are better for a variety of reasons (faster time to market, reduced risk, etc). This is why most companies have or are in the process of taking a deeper dive into their release and development processes. Once you have put everything in a pipeline and automate everything that you can what is the next step? The next frontier is the changing the source code itself to make it more modular.

Further Reading

Parallelizing Tasks with Dependencies — Design Your Code to Optimize Performance

Java 9 Modular Development (Part 1)