Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

The 8 npm Registry Essentials You Can’t Live Without

DZone's Guide to

The 8 npm Registry Essentials You Can’t Live Without

Build processes are becoming increasingly more complex. But, with a Binary Repository Manager these eight essentials that make development with npm more reliable, stable, and efficient, are much easier to manage.

· DevOps Zone
Free Resource

The DevOps Zone is brought to you in partnership with Sonatype Nexus. The Nexus Suite helps scale your DevOps delivery with continuous component intelligence integrated into development tools, including Eclipse, IntelliJ, Jenkins, Bamboo, SonarQube and more. Schedule a demo today

The Short and Sweet of It

The software industry has evolved from developing software to assembling it from open source, commercial, and in-house components. This trend includes Node.js developers who download more than four billion packages per month from the public npm registry at npmjs.org. This move from “development” to “assembly” has well-known benefits in terms of cost and code quality, but it also presents several challenges. These challenges are met by a Binary Repository Manager—a central gateway to host and manage all binary resources in your organization which provides the eight npm registry essentials to make your node.js development more efficient.


The Challenge

The “Essentials” Response

Remote resources and network are unreliable. Build processes are long and network intensive.

Accessibility

“Remote repositories” store downloaded artifacts in a local cache so they are always available rendering you (and your build tools) independent of the remote resource or the network.

Controlling where developers download components from and who can access them

Security and access control

  • Virtual repositories control which remote resources are accessible
  • “includes” and “excludes” directives provide fine-grained access control
  • Integration with server access protocols like LDAP, SAML, and Crowd.

Sharing components on-site and across borders

Local repositories

A single source for internal packages. Replication synchronizes their content to distant sites.

Reproducing production issues

Fully reproducible builds

Using exhaustive build information, builds can be precisely reproduced and compared with “diff” tools.

Finding components by different criteria

Searchability

Multiple search methods, from name search to simple but advanced “SQL-like” queries.

Your npm registry is a critical resource. It must never go down.

High availability configuration

Provides up to “five-nines” availability.

Custom behavior

Extensibility

User plugins let you implement custom behavior for a long list of system events.

Managing components of other package formats

Universality

A universal repository that works with all package formats significantly reduces the complexity of managing all the different binary assets in your organization.

The public npm registry meets some of the needs of enterprises developing with Node.js, but it takes an advanced repository manager to provide the eight essentials that make development with npm reliable, stable, and efficient.

Introduction

Software development is changing rapidly. Several years ago developers were writing most of their code in-house, while today most are using freely-available open-source, commercial, and in-house binary components. In fact, today, most enterprises use more open source software and assembled components than proprietary code. This trend includes Node.js developers who download more than four billion packages per month from the public npm registry at npmjs.org.

There are many advantages to creating code using assembled components, but you also need to address many challenges and pitfalls. Build processes are becoming increasingly more complex: while build cycles keep getting shorter, npm packages and their related metadata must be available offline, security and access control are always of paramount importance, packages must be shared across teams and across sites, binary versions must be tracked, specific packages must be found, and an organization's npm registry must be maintained at a high level of stability and reliability.

The answer to these challenges lie in a Binary Repository Manager; a central gateway that hosts and manages binary resources automatically as a collaboration hub. A repository manager speeds up Node.js development processes making them more efficient by managing internal and external npm registries for both developers and automated build processes.

An npm Registry Must Always Be Accessible

The recent #unpublishgate incident shows how critical it is for developers to have access to 3rd party npm packages. If a component is abruptly removed or an npm registry goes offline, things grind to a halt, and if we take a look at the official npmjs.org history page, we find that outages do happen in varying degrees of severity from time to time.

A repository manager insulates you from outages in remote public resources such as npmjs.org. It acts as an intermediary and downloads each component only once, storing it in an up-to-date, local cache of all software artifacts and their dependencies. This means that even if a remote npm registry goes down, your developers, and more importantly, your build servers, continue to operate, oblivious of the outage. As an added benefit, the local cache prevents multiple downloads of the same artifact, since the repository manager serves the single version currently resident in the local cache. This automatically reduces network traffic and provides a fast and reliable source of components, independent of the status of the internet connection. For developers, this means uninterrupted service, but for build tools and continuous integration (CI) servers, it also means considerably shorter build cycles.

An npm registry must always be accessible

In this example, repository manager JFrog Artifactory has cached a package originating from npmjs.org. The package is then available locally to developers and the build server, rendering them independent of the stability of the external network and npmjs.org as a remote resource.

Security Is Paramount

Security policies in a development organization are complex but are essential to enable controlled access to internal and external resources. A repository manager offers multiple levels of security and access control. Virtual repositories aggregate several local and remote resources to provide developers with a single URL from which they access all the components they need. In reality, the components may be located anywhere, in different remote and local repositories, but this is transparent to the developer. Through permissions at the level of the user and "Include" and "Exclude" patterns at the level of the repository, virtual repositories offer access control down to the level of an individual artifact. At a higher level, through integration with common server-level access control systems such as LDAP, Crowd, SAML and others, the repository manager makes it easy for administrators to comply with corporate-wide access policies.

Sharing Components Within the Organization and Across Borders

A repository manager offers different ways to share components across an organization whether teams are co-located, or sitting on opposite sides of the planet. A local repository is where a team will store the components it develops. One way to share npm packages in a local repository is to let another team proxy it. So for the sharing team, the resource is its local npm registry, for the receiving team, the resource is a remote repository which implements all the caching behavior to maintain accessibility as discussed before. Another way to share packages is through replication. A repository manager offers different ways to replicate repositories. Whether the sharing team push replicates a local repository to its distant counterpart, or the receiving team pull replicates the data, the end result is sharing of the organization's components by synchronizing repositories through replication across the globe.

Reproducing Builds to Manage Post-Production Issues

Production issues can be difficult to solve with the huge number of parameters that go into a build. Between system settings, environment variables, properties, dependency versions, licensing and much more, it can get extremely difficult to identify the source of an issue once the component is in production.

A repository manager stores detailed build information metadata. This exhaustive bill-of-materials, similar to that used by hardware engineers, lets you quickly reproduce a build precisely as it was originally created, and using "diff" tools, identify exactly what changed in the build to help find the source of the post-production issue.

Searchability

As the number of components used by a software development organization continues to grow, and artifact storage starts reaching terabytes and even petabytes in size, finding a very specific component can become the proverbial needle-in-a-haystack. A repository manager can make that needle shine through the hay with a variety of different search options. Most repository managers will let you search by name, version, and timestamp; those are pretty standard. More advanced tools offer additional options such as searching through properties annotating the different components, or even searching on a component's checksum (this can be particularly useful when a component's name has been altered for some reason). The most advanced repository managers even offer a proprietary SQL-like query language which essentially lets you search for a component based on any number of complex search criteria to zero in on exactly the component you are looking for.

High Availability

As a repository manager takes a central role in a software development organization, its own stability and availability become a critical factor in the organization's daily workflow. To keep developers (and more importantly, build servers) operational at all times, a repository manager can offer high availability. This is achieved by synchronizing multiple servers and providing access to them as a single unit through a load balancer so that no server is a single point of failure. This kind of configuration can offer unparalleled uptime guarantee with up to five-nines availability. Another option is to access your repository manager as a cloud service. This offers all the well-known advantages of cloud services such as minimal setup, hardware footprint, and maintenance within the organization, flexible and cost-effective pricing schemes, constant version updates and more.

Extensibility

The npm client offers different commands that let you upload (publish) packages to or download (install) packages from the public npm registry and work with them in different ways. While there are many commands, and how they work can even be modified by environment variables, there is still some limitation to what you can practically do with a fixed set of commands. If your organization's policies require some functionality that isn't available, you need to beg your local script-master to somehow, conjure up compliance. This is where an advanced repository manager can step in. Since a repository manager is central in your development environment, it is aware of everything that happens with an npm package and can provide corresponding hooks for any number of events. This lets you customize what happens when a package is uploaded, downloaded, moved, copied or modified in any way allowing you to support virtually any workflow required by your corporate policies. For example, you could invoke a vulnerability scan for any component downloaded from the public npm registry, send an email alert to the right administrator if a vulnerability is found and even prevent access to any suspicious component. This is the kind of functionality that only an extensible repository manager could offer.

Universality

While you may be developing in JavaScript, it's quite likely that your organization uses several different technologies to develop your products. Whether you're using Docker for virtualization, Bower for front-end development, and NuGet to cater to your .NET business, you need the same artifact repository management services for all of those technologies. While each component technology offers its own private and public registries with varying degrees of functionality, having a single universal repository that can cater to all the major packaging formats can significantly reduce the complexity of managing your organization's components while acting as a central hub for development and artifact management.

Development with npm continues to be on the rise. While the public npm registry provided by npmjs.org may provide some of these essentials, and their enterprise offering may offer more, it takes an advanced repository manager to provide all of these eight essentials—for npm as well as for the other packaging formats your organization is likely to be using. The world's leading companies in every business sector have all realized that neither the file system nor any database can provide the universal component management they need and have understood that only an advanced repository manager paves the way to the success of their business.

The DevOps Zone is brought to you in partnership with Sonatype Nexus. Use the Nexus Suite to automate your software supply chain and ensure you're using the highest quality open source components at every step of the development lifecycle. Get Nexus today

Topics:
npm ,node.js ,javascript

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}