npm Blog (Archive)

The npm blog has been discontinued.

Updates from the npm team are now published on the GitHub Blog and the GitHub Changelog.

Incident Report: npm Registry Service Degradation

From November 21-25, the npm registry experienced periodic service degradation. Alerted to increasing error rates from our monitoring systems and reports from the npm community, our incident response team began investigations on Thursday and has since identified the root cause and implemented mitigations that have stabilized registry service.

Root Cause 

Starting Thursday, an npm user launched a bot to aggressively crawl the registry––against our terms of service––querying a mix of existing and non-existing packages. One of the most important security aspects of the registry is to ensure that we do not acknowledge the existence or non-existence of private packages to unauthorized users. In order to do that while also making use of caching by our content delivery network, we have engineered systems that are capable of returning the required authorization information quickly and accurately.

The behavior by this user put undue strain on the systems that are used to authenticate whether a user may identify if a package exists, retrieve, and cache it for future retrievals.

Timeline

Nov 21, 18:20 UTC –– aggressive bot queries begin. The npm incident response team begins investigation and blocks bot activity while exploring long-term options. System load dissipates.

Nov 22, 19:50 UTC –– npm registry fully stable.

Nov 25, 01:20 UTC –– load spike on npm services returns.

Nov. 25, 12:18 UTC –– Further CDN updates deployed.

Nov. 25, 12:38 UTC –– npm registry fully stable.

Retrospective

As we have discussed here before, the npm public registry, like the JavaScript ecosystem as a whole, is experiencing exponential growth. Open systems like the registry require a basic level of mutual trust between the users and the operators of that system. Our terms of service are crafted to bolster that trust and to ensure quality of service to the entire JavaScript community. 

The npm operations team works continually to maintain quality of service and to enforce those terms of service.