Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Git Cache for Continuous Integration Jobs

DZone's Guide to

Git Cache for Continuous Integration Jobs

A quick code snippet on solving a large Git repository clone issue.

· DevOps Zone
Free Resource

“Automated Testing: The Glue That Holds DevOps Together” to learn about the key role automated testing plays in a DevOps workflow, brought to you in partnership with Sauce Labs.

In this article, I will explain how I solved a large Git repository clone issue we had during our continuous integration job.

The fact is that we have a Cordova project using multiple plugins. Most of these plugins are stored in git repositories and some of them or their dependencies are really large. This can slow our job execution or even make it fail (due to timeout) as it depends on our internet connection..

I decided to find a way to "cache" repository so the build jobs wouldn't use internet connection each time they need to clone the plugin repository without changing plugins files.

The solution I found was to use our  locally hosted GitLab as a mirror for the large repository and force Git to use it using the Git configuration "insteadOf".

I created a new repository in our gitlab to host the large plugin repository and I cloned it's content from remote location once. I then added the following configuration for git inside the CI job runner machine:

git config --global url."<url of local gitlab mirror>".insteadOf "<url of primary repository>"

Now I just need to regularly update the local mirror with a simple cron job for example. The Gitlab Enterprise Edition can also automatically update it's repository from remote origin.

The result is just impressive: one job that could take more than an hour before a large repository clone will now execute in less than 10 minutes.

CI Job successes

Learn about the importance of automated testing as part of a healthy DevOps practice, brought to you in partnership with Sauce Labs.

Topics:
git ,performance ,cache ,continuous integratinon

Published at DZone with permission of Mika Andrianarijaona. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}