Avoiding the Tragedy of the Commons: Acceptable Use of the Public Registry

The npm public registry, like the JavaScript ecosystem at large, is experiencing exponential growth. The longevity of the registry and its continued availability as a public resource depends on awareness of what constitutes acceptable use by the entire JavaScript community.

Figure 1: Weekly Rolling Registry Requests 

As stewards of the public registry, we at npm, Inc. have a mandate to maintain a high quality of service for the entire community. Part of delivering on that mandate is to propose and enforce  standards of what constitutes acceptable use. If we don’t enforce such standards now, then the cost of the infrastructure required by the registry will rise, performance will suffer, and the vast majority of users will be impacted by the over-usage of a few.

While nearly all of the registry’s 11 million users fall well within the limits of acceptable use, there is a small number of outliers––all large commercial enterprises who use the registry for free––that consistently make tens to hundreds of millions of registry requests per month, falling well outside any reasonable scope of acceptable use. We have been in conversations with those organizations for some months and nearly all have responded responsibly to address their out-sized usage. In most cases, those organizations were unaware of their usage and were able to mitigate their impact by making a few simple changes once they were made aware of the situation.

Figure 2: Percentage of Monthly Registry Requests by Order of Magnitude

In order to protect the registry on behalf of our community, we are taking steps to enforce acceptable use, including blocking or rate-limiting requests that exceed acceptable limits. 

What’s Changing?

We are clarifying and enforcing the definition of acceptable use regarding excessive use of the public registry:

• Up to five million requests to the registry per month are considered acceptable at this time.

• A request rate above that threshold is considered excessive, unacceptable use.

• We will reach out to organizations that consistently exceed the threshold to help them bring their usage to acceptable levels.

• In the small number of cases where we are unable to reach an agreement to resolve the issue, we will be forced to implement rate limiting to protect the quality of service of the registry for the entire community.

When Will this Happen?

We have been working with organizations for some months to bring out-sized usage down. In nearly every case, we have been able to resolve the issues quickly. In the very small number of cases where we have been unable to reach an acceptable resolution, we will start rate limiting in September, 2019.

Does this Change Affect Me?

Probably not.

This change only affects a small number of commercial organizations that consistently make millions of requests per month. For the overwhelming majority of JavaScript users––99.99%––your usage falls well below that threshold, so this change will not affect you.

If you are one of 0.01% of organizations that consistently make tens to hundreds of millions of requests per month, we will contact you and work with you to bring your usage down to acceptable levels. In many cases that can be as simple as fixing mis-configured automation tooling or caching requests. 

How Will I Know if I’m Making Excessive Requests?

Since the number of organizations affected is so small, we’ve just been reaching out directly.  If we haven’t talked to your organization about this, you’re probably fine.

What Action Do I Need to Take?

Probably nothing.

If your use of the registry falls below the acceptable use threshold, you don’t need to do anything.

If your organization exceeds the acceptable use threshold, we will reach out to discuss ways to bring your use down. If your organization is unable to reduce usage, a commercial offering is available to raise the limit of acceptable use accordingly. This solution typically involves additional technical work on our end, which comes at a cost.

Why Are You Doing This?

The mission of npm, Inc. is to protect and support the public npm registry because it is a crucial resource for the JavaScript software development community. Our  mission means that we must avoid the “tragedy of the commons” where excessive use of a free resource by some parties makes the registry less reliable or performant for everyone else.

To assist the biggest users, we have offered them our professional services free of charge to help them identify the causes of their excessive use and implement solutions to reduce it. By pinpointing automated sources of requests and implementing effective caching, we’ve worked with several  organizations to reduce their request volumes by orders of magnitude, often from hundreds of millions of requests per month to well below the acceptable use threshold.

We would like to thank all of the organizations who have worked with us so far to help protect this common good. You can help, too, by examining your own use of the registry and buffering it from automated systems wherever they make requests above typical human-scale use. We hope this precedent to become the standard to protect the registry for the entire community.