npm Blog (Archive)

The npm blog has been discontinued.

Updates from the npm team are now published on the GitHub Blog and the GitHub Changelog.

Rethinking JavaScript Test Coverage

This post was written by Benjamin Coe, Product Manager at npm, Inc. and lead maintainer of yargs and Istanbul for the Node.js Collection. It covers work that has gone into introducing native code coverage support to Node.js.

TLDR: You can now expose coverage output through Node.js by setting the environment variable NODE_V8_COVERAGE to the directory you would like coverage data output in. The tool c8 can be used to output pretty reports based on this coverage information.

A Bit of History: How Test Coverage Has Historically Worked

In JavaScript code, coverage has traditionally been facilitated by a clever hack: tools like Istanbul and Blanket parse JavaScript code inserting counters that (ideally) don’t change the original behavior of the application, for instance:

function foo (a) {
  if (a) {
  // do something with 'a'.
  } else {
  // do something else.
  }
}

gets rewritten as:

function foo(a) {
  cov_2mofekog2n.f[0]++;
  cov_2mofekog2n.s[0]++;
  if (a) {
    // do something with 'a'.
    cov_2mofekog2n.b[0][0]++;
  } else {
    // do something else.
    cov_2mofekog2n.b[0][1]++;
  }
}

cov_2mofekog2n.f[0]++ indicates that the function foo was executed, cov_2mofekog2n.s[0]++ indicates that a statement within this function was called, and cov_2mofekog2n.b[0][0]++ and cov_2mofekog2n.b[0][1]++indicate that branches were executed. Based on these counts, reports can be generated.

This transpilation approach works, but has shortcomings:

I found myself wishing there was a better way to collect code coverage…

Code Coverage in V8

I was chatting with Bradley Farias in August of 2017 about ESM module support in Node.js as ESM modules presented a problem for Istanbul. The problem? Bradley’s rewrite of Node.js’ loader to support ESM modules no longer supported hooking require statements, this made it difficult for Istanbul to detect that an ESM module had been loaded and instrument it. I made a strong case for adding this functionality back, Bradley had another suggestion:

What if we leverage V8’s new built in coverage functionality?

Using coverage built directly into the V8 engine could address many of the shortcomings facing the transpilation-based approach to code coverage. The benefits being:

I proceeded to investigate using Node.js’ inspector module for collecting coverage directly from V8; there were some hiccups:

These challenges aside, using V8’s coverage via the inspector felt promising. I was left wanting to help see things over the finish line.

Moving Towards a Proof of Concept

I reached out to Jakob Gruber on the V8 team regarding the bugs I was seeing integrating V8 coverage with Node.js. The folks at Google were also excited to see coverage support in Node.js and pitched in to start addressing bugs almost immediately.

After a discussion with several V8 maintainers, we determined that there was in fact a mechanism for enabling block level coverage:

  1. A program needed to be started with the --inspect-brk flag, such that the inspector terminated execution immediately.
  2. Coverage needed to be enabled.
  3. Runtime.runIfWaitingForDebugger needed to be run to kick off program execution.
  4. The event Runtime.executionContextDestroyed needed to be listened for, at which point coverage could be output.

I tested the approach outlined above and it worked!

I next asked Jakob if I could pitch in and start implementing some of the missing coverage functionality in V8. With the patient help of several folks on the V8 team, I was able to implement support for || and && expressions. This was my first time writing C++ in years and was a ton of fun for me.

At this point we had detailed V8 coverage information being output, but no easy way to output human readable reports. Two npm modules were written to facilitate this:

Leveraging these new libraries, we were finally able to see coverage reports!

This was a exciting milestone, but I still wasn’t satisfied. Here’s why:

Node-Core Implementation

I had an epiphany, what if Node.js could be placed into a mode that always dumped coverage?

In conversation with Anna Henningsen, it turned out that the Node.js inspector implementation lent itself to my idea:

  1. The inspector is actually always running in most environments, the websocket interface is just not enabled.
  2. There is an internal inspector protocol available that can interact with the inspector without creating a socket connection.

Excited, I sat down and implemented V8 test coverage as a feature of Node.js itself. Here’s what it looks like:

Next Steps

You can start using the Node.js built in coverage reporting today:

  1. Make sure you’ve upgraded to Node.js 10.10.0.
  2. Install the c8 tool, which can be used to convert V8 coverage output into human readable reports.
  3. Use c8 to execute your application, e.g., c8 node foo.js.

This feature is brand new, and I’d love for JavaScript communities to help see things over the finish line.

You can help by: