Module Re-Use Under Version Control Using Mercurial Guest Repositories
The advantages of modularity are clear to every developer: separation of concerns, divide and conquer, sharing of the development tasks, and increased maintainability.
Another advantage of modularity is code re-use: the functionality provided by a module can be used by multiple applications. For this reason, an API should always strive to be modular, allowing its users take on board only the modules they need.
All these advantages from the development side have a price to pay when it comes to version control: the API modules have their own development path and their users must keep track of which particular version they are compatible with, each release build has to record the versions of the various modules used by the application.
Not as "famous" as the previous ones, is the NetBeans Platform Module System API, which provides an excellent framework for developing modular applications. The Module System API allows to clearly define a dependency from one module onto a second module in terms of its specification and implementation version.
Something the mentioned tools do not provide is version control at source code level. To achieve it, they have to be complemented by a "modules-aware" version control tool.
In this article, I will explain how I achieved this for the projects I am responsible for.
Where I work, our corporate tool for version control is Mercurial and, given this constraint, I had to devise a way that would allow efficient collaboration among projects sharing a code base while keeping, at the same time, strict version control of the source code and of the releases.
While Mercurial is very efficient as a distributed version control tool, by itself it is not module-aware and does not provide the functionality to handle them. Luckily, Mercurial has a number of plug-ins that expand its basic functionality.
The first idea I had was to use the subrepos extension, but reading the page that describes the extension, even Selenic defines it as a "feature of last resort" not leaving big hope about my chances to have it working correctly.
Luckily, looking further, a pretty new extension came to my help: guestrepos.
In order to demonstrate the solution I adopted, I have created a simple scenario showing two separate applications ("modules suites" in NetBeans teminology) that share a common module suite.
For the purposes of this demo, the Storage and Display classes in the Enhanced_RCP_Core module suite represent two services used by Application1:
In order to be able to access the Enhanced_RCP_Core classes, Application1 has to include the Enhanced_RCP_Core project as shown in the following screenshots:
Once the Application1 suite has got visibility of the Enhanced_RCP_Core suite, it is now possible for its modules to add dependencies to public modules contributed by the added project (only the packages explicitly made public are visible from outside the enclosing module).
Once the dependencies are set, the appropriate imports are available in the module classes:
And the application compiles correctly.
Following the same procedure, the other application Application2 uses the classes of the Enhanced_RCP_Core.
Let's now imagine that the Enhanced_RCP_Core is modified to provide a new storage system:
In order to make public that the new service is not available in version 1.0 of the module, the specification version number is increased to 2.0.
Application1 developers immediately make use of the new feature:
In order to declare the dependency on the modified version of the Enhanced_RCP_Core also the module dependency is set to require version 2.0 of the library.
From the version control point of view, we now have Application1 that depends on version 2.0 of the Enhanced_RCP_Core, while Application2 still depends on version 1.0.
NetBeans module system, is well aware of the dependencies, on the other end, the mercurial repositories of the three projects are completely independent and do not know anything about which change-set provides which version of the API.
Only using mercurial is not possible to reproduce a particular build after the applications have been released.
One possible solution to this problem would be to use mercurial tags, the following figure depicts a possible scenario:
|Release||Application Change-set||Platform Change-set|
|Release||Application Change-set||Platform Change-set|
Following the tags approach, platform change-sets are tagged with the identifiers of the releases of the applications that make use of it.
Using this system, the platform repository is polluted with information related to its users, such a procedure is cumbersome to maintain and difficult to automate in the build process.
Mercurial guest repositories allow to implement a more elegant solution.
From the guestrepos page:
This is an extension for enterprises needing to handle modules and components
Many products, multiple projects within those products, multiple developers within those projects, multiple branches per developer
Source code shared in a multiplicity of configurations
Third party code (possibly patched on some branches)
A given module may have multiple versions going into the same product
Hg subrepos do not handle the sharing of components well, due to the recursive merge from the top (super) repo and requirement to lock at a specific version.
Guestrepo's goal is to overcome these limitations.
The guestrepo extension does not change any existing Mercurial behavior. It only adds new commands.
The second part of the demo shows hot to make use guest repositories in our Application1 and Application2.
The steps are:
- downloading and enabling the extension,
- creating two files: .hggrmapping and .hgguestrepo.
To enable the extension, its path must be added to the hgrc file which is contained in the .hg folder of the project.
The .hggrmapping and .hgguestrepofiles contain the external and internal references of the guest repositories.
Once the extension has been enabled and the files created, the two files have to be committed to the repository.
The guest repositories are now setup and they can be pulled with the hg grpull command.
The grpull command creates a folder with the given name (Platform in this case) and pull the remote guest repository into it.
Opening the suite in NetBeans shows the original name and the tooltip shows the location in the file system.
The application must be now modified to point to its own private version of the Enhanced_RCP_Core suite. The procedure is the same explained above:
After these steps the guestrepos set-up is complete and the developers can continue to work on the project.
The following steps describe a typical workflow that exploit the guestrepos functionality.
Let's imagine that the developers of Application1 implement an improvement of the Enhanced_RCP_Core, they change the advertised specification version of the platform and make the change.
Once the changes in Application1 and in the Enhanced_RCP_Core have been completed, the code can be then committed to the two repositories (main and guest):
After the commit, the command hg grout shows that there are local changes to be pushed to the main location of the guest repository.
The changes can be pushed with the command hg grpush.
One aspect to remind is that the guestrepos commands do not affect the main repository and that if the changes in Application1 would need to be pushed to another repository a normal hg push would have to be made as well.
From Application2 (that has been configured to use guestrepos just like Application1), the command hg grin shows that the platform has some new change sets.
The hg grpull command pulls the changes in the local copy of the repository.
Until now, we have seen how the exchange of the change sets is performed, but what about the original problem of keeping the releases under version control?
The command hg grfreeze changes the content of the .hgguestrepo file to reflect the current change sets of the guest repositories.
Following the procedure suggested by the creators of the extension, the release is performed branching to the Release branch, and the performing a freeze and commit in the new branch.
The history in the Application1 repository shows how the .hgguestrepo file is changed in the Release branch:
In my experience, though, this is not practical because it introduced a new head in the repository, working with multiple heads is difficult, sooner or later one of the developers commits changes to the wrong branch.
The way I prefer to work using inactive bookmarks (as the output of the branch command suggests....)
The procedure I use is:
- perform a hg grfreeze to write the changeset references in the .hgguestrepo file,
- commit the change
- create a bookmark (inactive)
The release change-set is bookmarked for easy access in case the build has to be reproduced:
Either branching or using the bookmarks, only the application repositories are aware of the various releases.
Each release change-set contains the .hgguestrepo file with the hash reference of the platform change-set used for the release, using the bookmark it is straightforward to update the repository and its guests to a previous state .
To conclude, the NetBeans Module System API and Mercurial guestrepos extension provide, together, everything that is needed to keep a modular application under configuration control at source code level.
NetBeans allows to define for each module what are the modules, and in which version, it depends on.
Mercurial, together with the guestrepos extension, allows to keep all the source code under version control allowing to easily to update the state of the application (main and guest repositories) to a specific version.