npm CLI roadmap: a periodic update

Last weekend, during NodeConf, many people asked me when various features were going to land in npm. I told them all that I was in the process of putting together a roadmap for the npm client, and so I have. There are big changes on the way for the CLI, some of which are new features, some of which are long-overdue efforts to improve the reliability and user experience of the client. Here’s a quick rundown of what we’ll be working to add to the CLI.

converting the CLI to an API

Sometimes it feels like there is an infinite number of features people would like to add to npm. Hooks for custom scripts, overridable behavior for front-end installers, process orchestration for OS services: there’s nearly no limit to what people would like to jam into npm, which is already an extremely featureful tool.

Instead of continuing to pile new features on to npm, I’d like to instead break the client down into a set of modules, with each module exporting an API for one of the tasks the client performs. It’s a lot simpler for everybody if we provide you with a set of components you can use in your own tools, especially if we do so in a way that allows these pieces to be used completely independently from npm. Another big benefit of this is that it will make npm much easier to test, which has long been a challenge of working with the codebase.

The first piece of redesign work we’re tackling is the npm cache. Some of you have seen the first piece of this redesign, which was a refactor that broke down lib/cache.js into a set of smaller pieces. The next goal is to turn it into a module with no dependencies on the rest of npm or npmconf and to test the heck out of it. Once it’s feature-compatible with the current cache functions, the plan is to incorporate multi-registry support into the cache, as well as making it a lot tougher for the cache to be corrupted.

After that, the next plan is to completely decouple npm-registry-client from the cache, so that it becomes what it should have been all along – an API library that knows how to talk to the npm registry, and which returns only package data. Having both the cache and the registry-client decoupled from the CLI proper will make it much easier to provide a true offline operation mode for npm, something which many people have been asking for.

There are many other pieces to pull out of npm (or in the case of npmconf, to pull back into npm), but these are the most important for now. It’ll be pretty easy to see what the next priority is once we get this work done.

dependency tree realization

Another thing that Isaac and I have been working towards for a while is a saner install process. Right now, it’s a sort of complicated and dynamic process figuring out what packages (and what dependencies of packages) need to be installed. We want to replace this complicated process with something much simpler and more deterministic:

check to see what’s currently installed
realize the tree that represents the ideal goal state of the local node_modules folder
generate a list of operations that must be done to transform the first tree into the second
actually perform the installation

This sounds straightforward, but it represents a major change to how npm works, and brings npm much closer to the approach followed by Dominic Tarr’s npmd (as do the changes to the caching strategy detailed above). It’s also a step closer to transactional npm installs (a goal near and dear to my heart), so we can get rid of the ridiculous tendency npm has to delete itself when an upgrade fails. That last piece is still a ways down the road, but I’m laying the groundwork now.

improved support for Windows paths

With the increasing popularity of front-end tooling within the Node ecosystem, there are cracks appearing in npm’s Windows support. The various plugin-based build tools all have an alarming tendency to create extremely long paths that cause tools like cmd.exe and git (pro tip #1: git config core.longpaths true) to fail. We want to address this, but research is necessary, because most of the possible approaches impose awkward tradeoffs.

One thing we know we’re going to do is make npm deduplicate more aggressively on install, in an effort to keep node_modules relatively flat (pro tip #2: npm dedupe is a good way to keep your node_modules directory relatively flat right now). Beyond that, we have to sort through alternatives, and we’ll have more to say on this change as the work gets closer to being finished. It seems likely that this will also entail some large changes to how npm works.

better error messages & reporting

When Isaac first built npm, its target audience was typical Node developers, which is to say mildly crazed server-side JavaScript engineers. Its error reporting is still optimized for that userbase – quick to dump detailed stacktraces with cryptic Unix error messages. As npm moves to being a part of the toolchain of front-end developers (who are more interested in shipping features than figuring out what’s going on with their OS), it’s time to work on making npm easier to understand when things go wrong.

At the same time, npm is about to reach an unfortunate milestone: sometime this week, npm will cross the 1,000-open-issue line. D: We’re lucky to have so many active community members, but we’re going to have to find a better way to separate support issues from genuine bugs in the client. We also want to improve how diagnostic information is captured and reported to us. We want to be able to clearly separate configuration and installation issues from actual bugs in the client. Isaac has been talking about adding an npm report command, and it’s about time we build it.

front-end tooling support

Speaking of front-end tooling, the front end is one of the fastest-growing use cases for npm, and many front-end frameworks are using tools written for Node. There’s a fair amount of work to be done here, and once again it’s a case of looking at the possible features and finding the simplest subset of functionality that will support a wide variety of use cases. We’ve talked to members of both the Ember and Angular teams (to say nothing of Browserify), and we’re optimistic we can come up with solutions to their problems with npm (and without overly weighing down the client with additional features).

nerf peerDependencies

In the not-too-distant past, we had a “discussion” about the problems caused by peerDependencies. They’re kind of a mess right now, since installing a plugin can not only implicitly install parent dependencies, but get an application into a completely invalid state. This will be fixed by making peerDependencies installed with missing parents raise a warning or error. We also intend to iron out the kinks with npm shrinkwrap and npm dedupe.

better CLI search

While the current npm search is all right (well, OK, it’s not total clownshoes), it could be a lot better. We’re putting a lot of work into improving the search experience on npmjs.org, and we’ll be bringing those improvements to the CLI as well (although the current CLI search will probably live on, because it can actually be useful offline).

other stuff

Perceptive readers will note that there are no dates tied to any of the above, and that there are also some pretty big projects in there. This is my current best thinking about what we can reasonably hope to get accomplished this year (and, with luck, I won’t be doing this work alone). Of course, startup life being what it is, it’s likely that something new and exciting will come along that requires some of these projects to be rescheduled or modified. However, this is the plan as of today, and if you see anything you think is missing, please let us know!