Automating Bulk Transfer of Mercurial Repos To Git
With Mercurial sunsetting support, it was time to move our repos. But how do you do this en masse?
Join the DZone community and get the full member experience.Join For Free
When we received the first communication about Bitbucket sunsetting Mercurial support, there was quite a lot of excitement around the office. All of our source code was stored as Mercurial repositories on Bitbucket and — being a product company — a lot of the company’s value is in the source code.
A closer look at the problem presented us with the following challenge:
- 350 Repos across 5 teams to convert.
- Retaining the commit history for all projects.
- All branches need to be converted.
A solution was needed, but many searches and discussions later, nothing suitable was found. So, we decided to wait. There was still plenty of time and we trusted that a solution would surely present itself sooner or later.
However, when the reminder from Bitbucket arrived, no solution had as yet presented itself and time was starting to run out. A few searches later we found the fast-export converter by frej, a Python project on Github that we could use to convert our repos one-by-one. While each individual migration seems simple enough, our challenge was the fact that we had to do this in bulk.
The challenge to do this in bulk was a great use case for Linx. Linx excels at the rapid development and deployment of back-end automations. With its visual models, it accelerates the development process for developers to quickly build unattended automation without having to worry about setting up or scaling their backend infrastructure and the related services.
We decided to create an automated process that would perform the entire conversion against each of our repos without any intervention.
Within 3 days we developed and tested our Linx solution and it works great. The final solution automatically performed the following steps:
- Calling the Bitbucket API to fetch a list of hg repos and writing the info to a CSV file.
- Cloning each hg repo in the file.
- Converting each hg repo to git.
- Creating the git repo on Bitbucket.
- Pushing the git repo to Bitbucket.
The only limitations we are aware of are that closed branches and issues are not migrated. It seems that the closed branches concept does not exist in Git and we don’t use the Bitbucket Issues feature. However, if you need Issues, building the migration of these into the Linx solution should be quick and easy. Bitbucket has an Issue Export API call that you can implement in Linx to accomplish this.
Once ready, each of our teams would need to run this solution themselves as the conversion needed to fit into each team’s own release cycle. So, the process each team followed was:
- Everyone checked in all changes at the end of conversion day.
- We ran the Linx process overnight or over a weekend.
- In the morning, the teams pulled the new repos and just continued where they had left off.
Some conversions took a long time to run, but most teams have completed their conversions now and development has continued smoothly and without interruptions to date.
Some of our results were pretty impressive;
- The longest conversion process took 12 hours to run.
- The oldest project had commit history going back as far as 2006.
- The largest project has over 500MB of source code.
- Over 300 companies and over 80 personal repos converted.
Try It Out
Published at DZone with permission of Anthony Morris. See the original article here.
Opinions expressed by DZone contributors are their own.