2014-01-28 Outage Postmortem

While making a change to simplify the Varnish VCL config on Fastly, we added a bug that caused all requests to go to Manta, including those that should have gone to CouchDB.

Since Manta doesn’t know how to handle requests like /pkgname, these all returned 403 Forbidden responses.

Because Fastly is configured to not cache error codes, this proliferation of 403 responses led to a thundering herd which took a bit of time to get under control.

With the help of the Fastly support team, we have identified the root cause and it is now well understood. The issue is that we were setting the req.backend in a vcl_fetch function, and then calling restart to re-parse the rules. However, calling restart will reset the req.backend to the first backed in the list, which in this case happened to be Manta, rather than the load balanced CouchDB servers.

Future changes to our Fastly VCL config will take this into account, and always explicitly set the req.backend in the vcl_recv function, even after a restart.

Also, we are setting up a separate Varnish instance to stage changes like this, so that we can avoid being surprised by errors in our VCL configuration.