Maintaining Multi-Branch Pipelines in Large Scale Enterprises
Creatining and maintaining multi-branch pipelines can be difficult for large enterprises without the appropriate considerations.
Join the DZone community and get the full member experience.Join For Free
Jenkins is a great tool for CI, CD, and DevOps. With its unique features, you can meet almost all the requirements of your business. One of these unique features is multi-branch pipelines, which enables the provisioning of pipelines dynamically. However, multi-branch pipeline alone will not fully cover all your needs as your company grows. Especially when it comes to large scale enterprises, you need to consider other things like centralized management, governance, stability, restrictions, and security for your pipelines. For having large-scaled CI/CD environment with Jenkins pipelines, you need to add more features that you have not thought before.
Dynamic Provisioning of Pipelines
When a developer creates a new branch and pushes it to a repository, Jenkins dynamically creates a pipeline for that new branch. Depending on the repository, even pull requests can be dynamically created as pipelines. This dynamic feature is very useful within the teams who are using Feature Branching or something similar. As the subject of this article is not multi-branch pipelines, you can find the details and some examples in end-to-end multi-branch pipeline project creation.
In multi-branch pipelines, pipeline scripts are stored within the project/code repository. This is very useful when it comes to the "pipeline-as-code" concept. Also, it is useful when you have small teams of developers or projects which do not have lots of branches. This way, developers can change the pipelines for their needs, push changes to branches and see the changes take effect immediately. This successful scenario fails when it comes to large scale enterprises with hundreds or thousands of developers with lots of projects.
While your teams or projects increase, it is time to create shared methods or commands which will be common and used among all the projects. This "centralized library" becomes very crucial in the long run because, as the scale grows, new requirements or changes in pipelines will arise. In this situation, changing every pipeline or script manually will be a nightmare for administrators. So, having that centralized library will be much more practical where you make changes in one place and every pipeline gets updated. This is where the Jenkins Shared Library concept comes in. For details you can visit the site.
You can still use a centralized library even if you have one pipeline.
Governance and Stability
Putting pipeline script within the code is great when you have a team of developers who have some knowledge about CI/CD and you are sure that they will not make big changes or script errors that will affect the stability of your environment. But can you be sure? Someone can accidentally delete the pipeline file or can make some typing errors. These kinds of little errors will affect the stability of CI/CD even if they do not break it all. Solving these errors if easy when you discover them early. If not, these little changes or mistakes will affect your CD more than you think. It will be spread to all branches or tags within different projects and this is the difficult part to solve.
You either need to push the correct pipeline script to all branches and/or repositories or ask every developer to pull the latest script. These kinds of problems can be solved by using the centralized library in a more advanced way even though it is not that easy to create such a level of an advanced library. Besides this, you will always have the risk of someone deleting the Jenkins file or making some typo, even in one line of script. Your environment will always be at risk.
Remote File Plugin
To eliminate the risks of unwanted changes and decrease the complexity of libraries that are used, we need to somehow separate pipeline scripts from the project/code repositories while still continuing to use multi-branch pipeline features. For that, we have the Remote File Plugin.
This plugin enables multi-branch pipelines to run/load pipeline scripts from other repositories instead of putting them withing the project/code repository. By this feature, you can have a separate repository where you put your all pipeline scripts and can give access only to yourself. This way you will have the centralized pipeline scripts repository same as the centralized library. Moreover, you can store your pipeline scripts within the centralized library itself.
The benefit of this feature is that no one will be able to make changes in pipeline scripts except the ones who have access. Any change you make in the centralized pipeline scripts will affect all multi-branch pipelines which are using that script file. This way you do not need to wait for all thedevelopers to get the updated version or to push scripts to all branches over all repositories.
Another benefit is that if you put your centralized pipeline scripts into repositories like BitBucket or GitHub, you will also have the code review function. This way you can share the repository with others while still restricting or reviewing the changes that others make.
Creating a CI/CD pipeline is not easy topic in large scale enterprises. You need to consider concepts like governance, restrictions, stability, and security. In this context, with the other features of Jenkins, Remote File Plugin provides a unique feature for centralizing, maintaining, and sharing pipeline scripts with others.
For details of the plugin, you can visit the wiki page of the plugin.
Opinions expressed by DZone contributors are their own.