The 8 npm Registry Essentials You Can’t Live Without
Build processes are becoming increasingly more complex. But, with a Binary Repository Manager these eight essentials that make development with npm more reliable, stable, and efficient, are much easier to manage.
Join the DZone community and get the full member experience.Join For Free
The Short and Sweet of It
The software industry has evolved from developing software to assembling it from open source, commercial, and in-house components. This trend includes Node.js developers who download more than four billion packages per month from the public npm registry at npmjs.org. This move from “development” to “assembly” has well-known benefits in terms of cost and code quality, but it also presents several challenges. These challenges are met by a Binary Repository Manager—a central gateway to host and manage all binary resources in your organization which provides the eight npm registry essentials to make your node.js development more efficient.
The “Essentials” Response
Remote resources and network are unreliable. Build processes are long and network intensive.
“Remote repositories” store downloaded artifacts in a local cache so they are always available rendering you (and your build tools) independent of the remote resource or the network.
Controlling where developers download components from and who can access them
Security and access control
Sharing components on-site and across borders
A single source for internal packages. Replication synchronizes their content to distant sites.
Reproducing production issues
Fully reproducible builds
Using exhaustive build information, builds can be precisely reproduced and compared with “diff” tools.
Finding components by different criteria
Multiple search methods, from name search to simple but advanced “SQL-like” queries.
Your npm registry is a critical resource. It must never go down.
High availability configuration
Provides up to “five-nines” availability.
User plugins let you implement custom behavior for a long list of system events.
Managing components of other package formats
A universal repository that works with all package formats significantly reduces the complexity of managing all the different binary assets in your organization.
The public npm registry meets some of the needs of enterprises developing with Node.js, but it takes an advanced repository manager to provide the eight essentials that make development with npm reliable, stable, and efficient.
Software development is changing rapidly. Several years ago developers were writing most of their code in-house, while today most are using freely-available open-source, commercial, and in-house binary components. In fact, today, most enterprises use more open source software and assembled components than proprietary code. This trend includes Node.js developers who download more than four billion packages per month from the public npm registry at npmjs.org.
There are many advantages to creating code using assembled components, but you also need to address many challenges and pitfalls. Build processes are becoming increasingly more complex: while build cycles keep getting shorter, npm packages and their related metadata must be available offline, security and access control are always of paramount importance, packages must be shared across teams and across sites, binary versions must be tracked, specific packages must be found, and an organization's npm registry must be maintained at a high level of stability and reliability.
The answer to these challenges lie in a Binary Repository Manager; a central gateway that hosts and manages binary resources automatically as a collaboration hub. A repository manager speeds up Node.js development processes making them more efficient by managing internal and external npm registries for both developers and automated build processes.
An npm Registry Must Always Be Accessible
The recent #unpublishgate incident shows how critical it is for developers to have access to 3rd party npm packages. If a component is abruptly removed or an npm registry goes offline, things grind to a halt, and if we take a look at the official npmjs.org history page, we find that outages do happen in varying degrees of severity from time to time.
A repository manager insulates you from outages in remote public resources such as npmjs.org. It acts as an intermediary and downloads each component only once, storing it in an up-to-date, local cache of all software artifacts and their dependencies. This means that even if a remote npm registry goes down, your developers, and more importantly, your build servers, continue to operate, oblivious of the outage. As an added benefit, the local cache prevents multiple downloads of the same artifact, since the repository manager serves the single version currently resident in the local cache. This automatically reduces network traffic and provides a fast and reliable source of components, independent of the status of the internet connection. For developers, this means uninterrupted service, but for build tools and continuous integration (CI) servers, it also means considerably shorter build cycles.
In this example, repository manager JFrog Artifactory has cached a package originating from npmjs.org. The package is then available locally to developers and the build server, rendering them independent of the stability of the external network and npmjs.org as a remote resource.
Security Is Paramount
Security policies in a development organization are complex but are essential to enable controlled access to internal and external resources. A repository manager offers multiple levels of security and access control. Virtual repositories aggregate several local and remote resources to provide developers with a single URL from which they access all the components they need. In reality, the components may be located anywhere, in different remote and local repositories, but this is transparent to the developer. Through permissions at the level of the user and "Include" and "Exclude" patterns at the level of the repository, virtual repositories offer access control down to the level of an individual artifact. At a higher level, through integration with common server-level access control systems such as LDAP, Crowd, SAML and others, the repository manager makes it easy for administrators to comply with corporate-wide access policies.
Sharing Components Within the Organization and Across Borders
A repository manager offers different ways to share components across an organization whether teams are co-located, or sitting on opposite sides of the planet. A local repository is where a team will store the components it develops. One way to share npm packages in a local repository is to let another team proxy it. So for the sharing team, the resource is its local npm registry, for the receiving team, the resource is a remote repository which implements all the caching behavior to maintain accessibility as discussed before. Another way to share packages is through replication. A repository manager offers different ways to replicate repositories. Whether the sharing team push replicates a local repository to its distant counterpart, or the receiving team pull replicates the data, the end result is sharing of the organization's components by synchronizing repositories through replication across the globe.
Reproducing Builds to Manage Post-Production Issues
Production issues can be difficult to solve with the huge number of parameters that go into a build. Between system settings, environment variables, properties, dependency versions, licensing and much more, it can get extremely difficult to identify the source of an issue once the component is in production.
A repository manager stores detailed build information metadata. This exhaustive bill-of-materials, similar to that used by hardware engineers, lets you quickly reproduce a build precisely as it was originally created, and using "diff" tools, identify exactly what changed in the build to help find the source of the post-production issue.
As the number of components used by a software development organization continues to grow, and artifact storage starts reaching terabytes and even petabytes in size, finding a very specific component can become the proverbial needle-in-a-haystack. A repository manager can make that needle shine through the hay with a variety of different search options. Most repository managers will let you search by name, version, and timestamp; those are pretty standard. More advanced tools offer additional options such as searching through properties annotating the different components, or even searching on a component's checksum (this can be particularly useful when a component's name has been altered for some reason). The most advanced repository managers even offer a proprietary SQL-like query language which essentially lets you search for a component based on any number of complex search criteria to zero in on exactly the component you are looking for.
As a repository manager takes a central role in a software development organization, its own stability and availability become a critical factor in the organization's daily workflow. To keep developers (and more importantly, build servers) operational at all times, a repository manager can offer high availability. This is achieved by synchronizing multiple servers and providing access to them as a single unit through a load balancer so that no server is a single point of failure. This kind of configuration can offer unparalleled uptime guarantee with up to five-nines availability. Another option is to access your repository manager as a cloud service. This offers all the well-known advantages of cloud services such as minimal setup, hardware footprint, and maintenance within the organization, flexible and cost-effective pricing schemes, constant version updates and more.
The npm client offers different commands that let you upload (publish) packages to or download (install) packages from the public npm registry and work with them in different ways. While there are many commands, and how they work can even be modified by environment variables, there is still some limitation to what you can practically do with a fixed set of commands. If your organization's policies require some functionality that isn't available, you need to beg your local script-master to somehow, conjure up compliance. This is where an advanced repository manager can step in. Since a repository manager is central in your development environment, it is aware of everything that happens with an npm package and can provide corresponding hooks for any number of events. This lets you customize what happens when a package is uploaded, downloaded, moved, copied or modified in any way allowing you to support virtually any workflow required by your corporate policies. For example, you could invoke a vulnerability scan for any component downloaded from the public npm registry, send an email alert to the right administrator if a vulnerability is found and even prevent access to any suspicious component. This is the kind of functionality that only an extensible repository manager could offer.
Development with npm continues to be on the rise. While the public npm registry provided by npmjs.org may provide some of these essentials, and their enterprise offering may offer more, it takes an advanced repository manager to provide all of these eight essentials—for npm as well as for the other packaging formats your organization is likely to be using. The world's leading companies in every business sector have all realized that neither the file system nor any database can provide the universal component management they need and have understood that only an advanced repository manager paves the way to the success of their business.
Opinions expressed by DZone contributors are their own.