Incident Report: 403 / 429 Errors for Some Users

At 11:06 UTC, our CDN partner deployed changes intended to detect spurious traffic by observing the “Referer” HTTP request header. This change caused some requests from the npm CLI to be flagged as suspect by the CDN.

To our monitoring systems, this deployment simply made it look like our CDN partner was helping us block harmful or abusive traffic. To our support staff, and to the initial technical investigators, this issue presented as a growing chorus of user feedback.

When we put the two together, we immediately contacted our content delivery partner. They traced the issue to a problem in a recent update to their service, where requests were being blocked if they contained HTTP Referer headers that were not fully qualified URLs. This led them to reject properly formatted npm traffic, notably “install” requests. By 13:00 UTC they had deployed a fix, resolving the issue.

Our technical staff, and many astute npm users, simultaneously observed that the npm CLI client does include Referer: install in requests to our registry. That fact resulted in engagement from experienced members of our community concerning HTTP standards. We appreciate the input from the community.

While information continued to pour in from developers around the world, simultaneously, the collaboration and response we got from our partners was extremely helpful and appreciated. In the end, npm and our partners found it easy to agree that we could use this incident as a learning experience towards better practices and automation on both sides.

We have deployed improvements that should lower mean time to recovery in the future. We were already in the process of deploying increased self-healing and monitoring systems, and we’ve further focused some of that monitoring on our edge to alert us as soon as possible to aberrations in traffic patterns.