The npm blog has been discontinued.
2014-01-28 Outage Postmortem
While making a change to simplify the Varnish VCL config on Fastly, we added a bug that caused all requests to go to Manta, including those that should have gone to CouchDB.
Since Manta doesn’t know how to handle requests like
/pkgname, these all returned
403 Forbidden responses.
Because Fastly is configured to not cache error codes, this proliferation of 403 responses led to a thundering herd which took a bit of time to get under control.
With the help of the Fastly support team, we have identified the root cause and it is now well understood. The issue is that we were setting the
req.backend in a
vcl_fetch function, and then calling
restart to re-parse the rules. However, calling
restart will reset the
req.backend to the first backed in the list, which in this case happened to be Manta, rather than the load balanced CouchDB servers.
Future changes to our Fastly VCL config will take this into account, and always explicitly set the
req.backend in the
vcl_recv function, even after a
Also, we are setting up a separate Varnish instance to stage changes like this, so that we can avoid being surprised by errors in our VCL configuration.