Registry roadmap

At npm Inc., the term “npm” is pretty overloaded. In addition to being the name of the company, at the very least, it means three bits of software worked on by different teams:

npm the command-line tool. We call it just “CLI” (for “Command Line Interface”)
npm the website, aka “dubdub” (from “www”)
npm the registry, aka “the registry”

In the last few months we’ve written about the roadmap for the website as well as for the CLI (twice). That leaves only the roadmap for the registry itself, the least-visible of the three.

The state of the registry

The current status of the registry is easily summed up as “exponential growth”. You’ve probably seen a picture a lot like this graph before: it always stays the same shape, but the axes keep getting bigger.

We’ve hit a number of milestones recently:

500MM package downloads every 30 days
100,000 packages in the registry
3.7MM website visitors (since January)
peak traffic of over 2100 requests per second
99.99% uptime over the last six months, as measured by Pingdom

(the average outage was just 2 minutes long!)

The ever-increasing registry traffic has two distinct user groups: weekday and weekend users, who we can roughly categorize as “professionals” and “hobbyists”. Here’s the hobbyist traffic:

And here’s the professional traffic:

As you can see, both are growing, but the “professional” group is 2-3x as big. This isn’t news to some people, but to others it is: Node.js is a professional tool used by real companies to get stuff done.

Current registry architecture

We described the current architecture in some detail back when it was new 8 months ago, so we don’t need to go into a bunch of detail. The key goal of the changes back then was to help us scale, and they worked: registry traffic has tripled since then, and uptime has hugely improved. But the registry still has some major issues, and they all revolve around one thing: the registry is a single giant CouchDB application.

CouchDB is a pretty good database – we love how easily and reliably it replicates, in particular. But it is not meant to be an application platform, certainly not for anything like the scale and complexity of npm. It runs JavaScript – not Node – so the universe of Node modules is, weirdly, not available to the very application that is hosting them all. For this and other reasons, deploying new releases is hard, testing is hard, and writing new queries (which are map-reduce jobs) is very, very hard.

The future: Registry 2

To address all of these, we are shifting to a new, more modular architecture. The changes are simple:

All the logical components have been split into microservices which can be independently deployed and scaled
All application logic has been pulled from couchDB into fully-fledged Node applications
User data has been pulled out of CouchDB into a relational database, because it’s relational data

The new components that result from this split are also simple:

Front door: the start and end point of all requests, it handles routing, as well as input and output validation and sanitization.
Auth service: rapidly authenticates and/or authorizes all requests to frontdoor
Validate and store service: processes incoming package uploads, stores binaries and metadata separately

We call this collection of changes “Registry 2”. It will resolve some lingering inefficiency in the existing architecture and give us even more room to scale, but more importantly it will allow us to add a bunch of new features. We’re very, very confident that it will work, because it’s already in production: npm Enterprise already uses this architecture, and the Registry 2 project is largely a matter of taking this architecture and scaling it up.

Scoped packages

The biggest feature that comes with Registry 2 is scoped packages. Already supported in npm 2.x and npm Enterprise, they work very simply: instead of publishing packages to the global scope (e.g. “express”) you will be able to publish packages underneath a name that belongs to you (e.g. “@myuser/express”). To get more detail about how scoped packages work, you should read the docs about scoped packages on our shiny new documentation site.

Scoped packages are almost identical to global packages: they install the same way, they are required the same way, you publish them the same way. The only important difference is that a global package will not be able to depend on a scoped package. This lack of difference would make scoped packages a pretty boring feature if it wasn’t for the first real new feature of Registry 2: private packages.

Phase 1: private packages

In Registry 2, you will be able to publish scoped packages that are also private: only you, or people you designate as collaborators, will be able to see them and install them. Nobody else will even know a package with that name even exists in your scope.

The purpose of private packages is to remove a pain-point most Node developers have encountered: once you get into the “Node way” of building lots of small, single-purpose modules, you want to start doing it all the time, to share code between different parts of your application, or between different parts of your company. But if you write code that is closed-source, or is just too specific to your own application to make a good public module, your options right now are full of friction and split-brain thinking. Why can’t you just npm publish and npm install all the private packages in your app, the same way you can with public packages? People kept asking us to make it possible to do this, so we did.

Private packages will be part of the first release of Registry 2, expected early in 2015. If you would like to try this feature out earlier, you can sign up right here to request access to the beta, which will be available later this year.

Sign up for private modules beta

Your contact details will only ever be used to notify you about the availability of private modules.

Phase 2: orgs and teams

Of course, we know that most people using Node are using it at their day jobs, which means you’ll want to be able to share your packages easily with specific groups of people, and designate different levels of access in a granular way. We’ve already started building how this will work, and it should be released soon after private packages. We think these features together are going to make a huge difference to how easy it is to use Node as part of a team at smaller companies.

And after that…

We have a lot of fun ideas in mind, but in the interests of this not being too much like vaporware, we’re going to stick to announcing only the things we have firm plans for.

Update 2014-10-29: added mention that global packages can’t depend on scoped packages.