npm@5, specifications, and our RFC process

npm has one of the most active issue trackers on GitHub, so we receive constant exposure to your pain points and concerns. This means we have some pretty solid guesses of what we should build and how to prioritize it. However, given the huge number of users and projects who depend on npm, it’s essential that we understand everyone’s needs and build the right thing.

Since the release of npm 4, the team responsible for npm’s CLI has been hard at work on npm’s next major version. npm 4 was intended to clear out some breaking changes that had built up over time, but npm 5 will be a more ambitious release — so we’re trying a new process for soliciting feedback and hearing your concerns.

The process is broken down into a couple pieces. Let’s talk about them, and how you can get involved.

It 5!

Before the release of Node.js 8, we intend to ship npm@5:

a faster client, as measured by benchmarking and profiling tools available to everyone;
a much improved version of npm-shrinkwrap.json, npm’s lockfile. In particular, shrinkwrap was originally intended to be a finalizing tool used prior to deployment, but we intend the new shrinkwrap to be used for the entire lifecycle of your JavaScript applications;
new tools intended for use with upcoming registry features.

Specifications, or, how did we get here?

How did npm get to where it is? Well, not really on purpose.

npm was a product of rapid, iterative design, organically developed in response to immediate needs. A fair amount of what people have come to rely upon in the CLI were things that were either experiments on Isaac’s part, or contributions from community members trying to solve specific problems.

A great example of this is npm shrinkwrap. While Isaac was at Joyent, his colleague (and DTrace wizard) Dave Pacheco submitted npm shrinkwrap as a patch to solve some specific deployment issues that Joyent faced with npm.

This rapid iteration is part of what makes npm one of open source’s great success stories, but it also makes maintaining it a challenge. Absent design documents or specifications, developers assume that if they can get something to work with npm, that must be how it was designed to work. That’s a logical assumption, but as time passes and people build ever more sophisticated workflows on top of this assumption, it gets harder for us as maintainers to make even small simplifications to how npm works without breaking somebody’s workflow.

One conclusion I’ve drawn from this: we’re past the point where the CLI can afford to exist without some sort of specifications.

Writing down what we expect the CLI to do helps both maintainers and developers when there’s confusion, and makes it easier to integrate npm into your larger solutions for building and maintaining JavaScript applications.

Another conclusion I’ve come to: the specs we produce will be much better if you’re involved.

Every member of the CLI team has learned far more than we ever wanted about the intricacies of package management, but we’re not omniscient. We see only a tiny fraction of your use cases — and in most cases, we learn more about what we need to build by letting you talk directly to one other.

To support this, we’ve come up with a process to encourage you to help us figure out how to build the right functionality to meet your needs.

The RFC process

The flow is straightforward:

One or more of the CLI team engineers drafts a rough specification, following npm’s own in-house process.
The draft is posted for internal review in our Slack team in the #rfcs channel. This allows the other teams at npm to make sure we’re building the right thing, and that the spec makes sense. (This is not just engineering — feedback from product design, support, documentation, and marketing is important at this point too.)
The spec is posted to GitHub as a pull request to a new subdirectory, doc/specs/. Over time this will grow to include specifications of how we intend all of the CLI’s core components to behave.
This pull request will be up for community review for at least two full work weeks, and we’ll publicize new drafts via npm’s weekly community newsletter.
After the spec has had adequate review, it’ll be merged, and we will write the necessary code to implement the specified features.

This process shouldn’t look particularly new or exciting; we’ve borrowed pieces of it from other projects. We’ve also tried to balance our need to get a good level of feedback — from both the project’s collaborators and the broader community — with still keeping things moving along. To be ready for Node.js 8’s release, we need to finish the breaking changes for npm@5 by March of 2017. This is a tight deadline given our ambitions.

What’s next

Good news: we’re almost ready to put up the first set of RFCs!

Kicking things off, Rebecca and Kat have been working on a set of changes to how symbolic links and local dependencies work with shrinkwrap. At least one of those doesn’t work at all in shrinkwrap today, so their proposals are interesting and at least a little radical. We’ll get them up shortly after everyone returns from their winter holidays. After that, we’ll keep a steady stream coming until the release of npm@5.

We’re still striving to find newer and better ways to make clear what we’re doing, and to get your feedback. We’re looking forward to your input on these new proposals — I’m confident it will help us produce a much better new npm.

Thanks as always for your time and attention. From our family to yours: best wishes for the holidays and happy new year!