Accelerate SaaS CI/CD With a Repository Manager
Accelerate SaaS CI/CD With a Repository Manager
If you're moving to the cloud and want to optimize your CI/CD workflow, see why a repository manager with a SaaS option is a good choice.
Join the DZone community and get the full member experience.Join For Free
Learn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market.
The trend to “cloudify” everything has spawned a variety of SaaS tools in every domain of the software development life-cycle. In fact, today, there are a variety of ways you can build a complete CI/CD pipeline in the cloud. Nevertheless, the basic elements that will be in any pipeline are:
- Version control — for developers to commit and manage the codebase
- CI/CD server — to run automated build and testing processes
- Build tools and dependency managers — to build the application packages
- Remote public repositories — to provide build dependencies
- Deployment tools — to push applications and services to production systems
- Cloud runtimes — to run your apps and provide your services to your customers
With this setup, you may feel confident that you’re making the best use of your resources. You’re already running everything on the cloud, so your on-prem hardware footprint is minimal, there are no servers to manage and maintain, you have the flexibility to scale up or down as needed, you have enterprise-grade infrastructure to provide availability, DR, unlimited storage, etc. You’re all set. Or are you?
Ask yourself these questions:
- How do you control who has access to your builds?
- What happens if someone decides to remove a dependency you need in your build. These things do happen.
- What happens if one of those remote repositories you depend on goes down? Yes, these things happen too.
- How can you be sure all those dependencies you are using in your builds are safe? Security vulnerabilities lurk everywhere.
What you’re missing in your cloud is a Binary Repository Manager.
A repository manager can be the answer to all the issues raised in these questions. Here is how one would fit into a SaaS CI/CD pipeline.
The diagram shows only some of the many SaaS repository managers available today to help you manage your binaries.
What Can a SaaS Repository Manager Give You?
There are many ways a SaaS repository manager can make your CI/CD pipeline both safer and faster, but not all repository managers are equal. Each has its advantages, and you need to match your choice of repository manager with your specific requirements. Here are some of the things you might be looking for.
Universal Support for All Major Package Formats
Any modern organization will use multiple technologies for its development efforts. Supporting all the different package formats, each with their respective package or dependency manager along with their unique repository layout requirements can be a nightmare to maintain. A universal repository manager that supports all major package formats can take the headache out of hosting all the different packages you work with. In most cases, all you will need to do is point your package manager at the repository manager to resolve dependencies and upload builds, and the repository manager should do the rest. Having all your organization’s packages in one place makes maintenance and support of your software workflow much easier. While several SaaS repository managers tout “universal” support for different package formats, some are more universal than others, while others are dedicated to a single technology. You should make sure that the repository manager you choose to work with fully supports all the development technologies you are using.
A repository manager should provide strict access control over the repositories it manages so that only authorized personnel can access artifacts that are hosted. For example, you may want to prevent developers from uploading artifacts directly to a production repository to make sure that anything that gets deployed to production has passed through all the necessary quality gates. So most repository managers will provide some kind of user-based permissions system which lets you specify who can access what with the best systems providing fine-grained access control. More advanced products will offer integration with corporate authentication mechanisms like LDAP or SAML, and allow different options for sign-on such as OAuth and SSO.
Efficient and Reliable Remote Access
Since much of your code is likely to be made up of dependencies downloaded from remote public repositories, you need those repositories to always be available for the many builds that your CI/CD server will be running each day. Network latency can slow down those builds. An outage in the network (beyond your trusted cloud of course), or a service interruption in one of those repositories will grind your builds to a halt.
A repository manager will overcome outages or network latency by acting as a proxy to the remote repository and caching artifacts that have already been downloaded. With artifacts cached locally in your SaaS environment, the repository manager has removed any dependence on the external network or the remote public repository. Not only will this prevent builds from breaking, but it will also help speed up builds since a cached artifact only needs to be downloaded once, and it is then available to all the different build processes that need it.
As companies grow and teams become distributed and spread across different global sites and time zones, a convenient way to share artifacts becomes increasingly important. A binary repository manager addresses this need in a number of ways.
First, a fundamental feature of repository managers is Local Repositories. These are locally-managed repositories where you can deploy artifacts, development builds, external releases and even 3rd party commercial components. Having all your internal resources available from a common URL eases the way towards collaboration between teams. By using a wise naming-convention for your repositories, you can make their purpose, ownership, and maturity of the artifacts they contain become implicitly obvious to their users.
Most repository managers will also have the concept of virtual repositories which aggregate any number of local and remote repositories (and in some cases, other virtual repositories) making the whole aggregation accessible through a single, commonly known URL. Basically, a virtual repository is a one-stop-shop for both uploading builds and internal artifacts, and downloading internal and external dependencies.
So, local and virtual repositories take care of the internal distribution of your artifacts. But how do you share and collaborate with teams that are using a different instance of the repository manager - possibly at geographically distant sites? That’s where repository replication comes into play. A good repository manager will offer different capabilities to replicate artifacts from one repository manager instance to another. Depending on different parameters, you may prefer push replication in which a source repository triggers pushing changes to a target repository, or pull replication in which a target repository triggers pulling changes from a source. The preferred mode of replication will depend on different parameters, but given the right capabilities, your repository manager should be able to support any replication topology to serve multiple sites of your organization.
Malware is everywhere. Year after year, the Black Duck (by Synopsys) annual report on open source software cites statistics that show the security risks in using open source software. Yes, those are the dependencies you download from all those remote public repositories on the cloud. A repository manager can provide scanning capabilities to detect security vulnerabilities and other issues in the dependencies you download. Some have scanning built-in while others are tightly integrated with complementary products that can provide a whole range of capabilities from blocking infected artifacts from being used to providing dependency graphs showing exactly how an infected artifact is used in different parts of an organization’s software. Whatever security scanning the repository manager you choose provides, you should ensure that the scanning capabilities also support all the technologies you work with and make your code safer on all fronts.
Choose the Cloud That’s Right for You
There are several SaaS repository managers available on the market today, and each one is run on a cloud computing provider - the most common ones being Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure (no abbreviation here :-) ). It’s important to check on which cloud computing provider your repository manager will be run. Having your complete CI/CD stack co-located with a single provider offers different benefits. On a technical level, integration of different services within a single cloud provider will be smoother than integrating services on different clouds (which in some cases, may not be feasible at all). The setup for each different service on a provider will be more consistent, and the interfaces between them better suited and coherent. On an administrative level, co-locating your services means dealing with fewer vendors (each of which needs legal approval, managing payments etc.), maintaining fewer user accounts, and basically, dealing with less overhead. Some SaaS repository managers have a multi-cloud offering letting you choose where your instance should be hosted, so if you’re already using different SaaS services for CI/CD, you can co-locate your repository manager with the same one.
Maximizing Strategic Choices
After weighing the pros and cons of SaaS vs. on-prem, you took the strategic decision to take your infrastructure to the cloud. But even in the cloud, you want to make the most out of your decision and optimize your CI/CD workflow. A binary repository manager should be in the center of your SaaS ecosystem. It speeds up your builds while adding security at different levels. There are several repository managers available on the market, but not all have a SaaS offering. You should find the ones that do and try them out to see which best fits your needs for continuous integration in the cloud.
Opinions expressed by DZone contributors are their own.