Elevated Error Rates, 2014-02-27

Due to the same Manta HAProxy FD-leak bug that we were affected by 3 days ago, there were 5 minutes where we saw a spike in error rates from tarball fetches.

This time, our alerts caught it (just slightly before Joyent’s own alerts, in fact!) and we were able to get the situation addressed much faster.

We are still in the process of distributing the hosting of package tarballs such that this sort of hiccup is no longer possible.

Update: Apparently Manta is still having some problems. Joyent’s engineers are working on this presently.

They are upgrading HAProxy which should fix the problem permanently. There may be a few more 5-minute outage blips coming shortly. We’ll post an update once it’s completely resolved.

Update: We’ve worked with the Manta team to figure out a way to avoid this sort of issue in the future. They’ve been extremely helpful as always.

Jacques pushed a config change to Fastly that will always route tarball requests to a functioning Manta backend, even if one of their (several) backends is unreachable.