npm CLI Roadmap - Summer 2019

Motion on the npm CLI project has been accelerating, and we’re now moving forward with a clear direction and vision. This document outlines what’s in store for the remainder of the npm v6 line, and what to expect in v7 and v8.

Remaining npm v6 Releases

npm v6 is officially in “bugfix and minor enhancement” mode as work on npm v7 is getting into full swing. That doesn’t mean that improvements won’t be made! But the architectural changes for v7 will require quite a bit of attention, and will be the priority moving forward.

Expect weekly-to-biweekly releases to update dependencies and land pull requests, but major new features will likely be held off until v7. These releases will slow down precipitously once v7 is closer to release, but we do expect to continue to fix major bugs as long as people continue using it.

Bugfixes will focus on the most pervasive and persistent problems that users experience.

• Files with incorrect ownership (root files in user-owned folders, and user-owned files in root-owned folders, especially).
• Long-standing issues with the uid-number and cmd-shim modules.
• Any low-hanging fruit to make WSL work better than it currently does.
• Ensure that the urls stored in package-lock.json are always https when they’re supposed to be. (This is just a cosmetic issue, since the CLI always requests via HTTPS to the public registry, and the registry redirects to HTTPS, but it’s one that spooks people a lot.)
• Inconsistencies in npm ci (other than those that require significant rework, since v7 will replace this codebase anyway).

New features likely to land on the 6.x branch:

• Add support for the peerDependenciesMeta field. (#224)
• Add support for bundleDependencies: true (npm/read-package-json#81)
• Some other accepted rfcs, provided that they don’t involve breaking changes or conflict with other plans here.

Generally, not much in the way of new features or major changes. Bugfixes will continue, but anything that requires significant behavior or structural change will probably be put off until the next major.

npm v7

The headline theme for v7 is a refactor to the installer that adds consistency and speed, and makes lot of desirable features better and easier to implement.

Installer Refactor

Most of the installer logic is currently being factored out into @npmcli/arborist. This will enable teasing apart a lot of logic that has been developed organically over the years. The installer in the npm CLI itself will just handle passing arguments to @npmcli/arborist and listening to events to update the progress bar.

This will bring a lot of consistency to the various sorts of installation tasks (prune, update, etc.) as well as making package-lock.json more deterministically generated. It incorporates and replaces read-package-tree, read-logical-tree, libcipm, and almost all of the code in the CLI’s lib/install/ folder.

npm link

The Arborist module is being designed with first-class support for symbolic links. This will fix a class of edge cases where a symlinked folder in node_modules results in strange behavior and incorrect “missing dependency” warnings.

Links will be linked directly to their targets, instead of symlinking globally and then linking the global link into the local node_modules.

npm ls <path>

Currently, npm ls can take a package name, and show all the packages that depend on that package name.

In v7, npm ls /logical/path or npm ls node_modules/file/node_modules/path will be able to show you all of the packages whose dependency is met by that specific instance of a package, more clearly answering the “why is this here?” question, without resorting to digging through npm’s metadata manually.

Workspaces

First-class support for symbolic links means that workspaces will be trivial to implement. npm v7 will have at least the Workspace feature support of Yarn, and will set the stage for more advanced workspace features in v8.

At minimum, you’ll be able to keep multiple related packages all together in a single repository, and test changes in an integrated environment, without continually re-linking.

Once Workspaces land, it’ll be possible to add more advanced workspace management features. For example, building, versioning, managing permissions, and publishing all the packages within a workspace together with a single command. That likely won’t be in 7.0, and may be pushed out until v8, depending on feedback and user demand.

Proper peerDependencies Support

Part of the installer rewrite involves taking a fresh look at how peerDependencies are handled (and, quite often, not handled) to maximize the cases where the CLI does the right thing. Since npm v3, peerDependencies were not installed by default, putting the onus on users to resolve them. We’ll be bringing back this functionality in version 7.

This is one of the trickiest (not merely typing-intensive) parts of the refactor, because peerDependencies can actually conflict with one another in ways that create dependency hell. See npm/rfcs#43 for a discussion of how this will be implemented.

Resolution Overrides

Let’s say your project depends on foo, which depends on bar@1.0.0. There are sometimes cases where you need to give it a different version of bar than the one that foo requests in its package.json. For example, this can be useful to float security or bug fix patches, or to more aggressively deduplicate for the benefit of browser or serverless environments where memory is at a premium.

Resolution overrides give you a new tool to get out of these tricky scenarios without having to resort to manually mucking about in node_modules.

Play Nicer with Yarn

Right now, teams that switch between npm and Yarn (or use both!) have a lot of strife. npm reads and creates package-lock.json. Yarn reads and creates yarn.lock. These files easily get out of sync, and users get scolded by the tools that are supposed to be helping them. That’s not The npm Way.

npm v7 will treat yarn.lock as a valid source of information. If it sees a yarn.lock file, it’ll keep it up to date when modifying dependencies. If it sees a yarn.lock and no package-lock.json, it’ll use that as the authoritative source of dependency resolution. Also, Arborist will do its best to read the contents of a node_modules folder, even if it wasn’t the one to put it there.

This means that npm v7 will not add extra metadata to installed package.json files. Since we won’t depend on it, there’s no need for it.

More Intuitive Handling of files Array in package.json

Traditionally, npm has treated the files list in package.json as a sort of “un-ignore” list. That is, it gets logically treated as if it was a .npmignore file with * at the top, and ! prepended to all of the items.

This results in some odd edge cases, like including .DS_Store files if they are in an included folder, and has made it harder to strip out unwanted files like core dumps without having adverse side effects.

In version 7, npm-packlist will be changed to treat "files" specially, so that they can more comfortably coexist with ignore rules (including default-ignored files like .DS_Store and core dumps).

Drop Support for Node.js v6

Node.js v6 reached its end of life on 2019-04-30, a few months ago now.

Dropping support for Node 6 means that the CLI can make use of more advanced features, up to date modules, and keep up with the usage patterns of our community.

Other v7 Things

There are some other nice to have things that might make it into this release.

• Do away with the unsafe-perm and uid-switching issue entirely, as it causes more and worse problems than it solves. If someone’s running as root, just go ahead and run scripts as root
• Revisit the npm org and npm team commands to improve their usability and reliability.

Timing

Version 7 doesn’t have a release date yet, and most of the finer points here are still subject to change.

The installer refactor is underway, and will be the big breaking change that requires a SemVer major bump. If it gets to a stable releasable state before Workspaces and other features land, we will likely decide to release multiple iterations rather than try to squeeze everything into the 7.0.0 release. Initial release will almost certainly be this year, possibly as early as this fall.

Beyond Arborist, to Tink: npm v8

Version 8 will primarily be about integrating Tink. See Kat’s talk about Tink for more information about the whys and whats of that change. The Arborist refactor will provide a much more well-understood and stable footing for Tink to be merged in.

The tl;dr on Tink is:

• Dependencies don’t actually get unpacked in node_modules unless they have an install script, or an unwind is specifically requested. Instead, they are just added to the cache, and a package-lock.json or npm-shrinkwrap.json is generated.
• A fallback-fs overrides the Node.js fs module whenever a script is run via npm. This allows node programs to be served the files out of the npm cache for the specific dependency it would resolve to, instead of out of node_modules, without modifying anything else further up the stack.

Of course, it won’t work for cases where you want to run node server.js, but a new npm sh command will support running arbitrary commands as if they were an npm script.

Tink is still very experimental, and it’s 2 major releases out, so expect a lot of this to change in the finer details.