Broken Cooling Fan Takes Down Amazon's Cloud

By Matthew Dublin

Amazon Web Services experienced yet another outage last week, this time due to the failure of a cooling fan. Yup, you read that right — a cooling fan brought down the great Amazon cloud. The cloud outage affected a number of websites, including Heroku, Pinterest, Quora, and HootSuite.

The source of the outage occurred at a northern Virginia data center when the facility lost utility power. The center then switched to a backup generator power, but then nine minutes later, a defective cooling fan caused one of the backup generators to overheat and fail.

The incident began on June 14th at 8:44 pm PST and lasted until 10:19 pm PST.

According to the AWS Service Health Dashboard blog:

At this point, the EC2 instances and EBS volumes supported by this generator failed over to their secondary back-up power (which is provided by a completely separate power distribution circuit complete with additional generator capacity).

So it seems that cloud computing continues to receive doses of reality that counter its posturing as the ultimate replacement for the onsite data center as well as the promise of infinite resiliency and reliability.

Last week's outage was the third major outage in the last 14 months for the "US-East-1 availability zone" — Amazon's oldest availability zone based in a data center in Ashburn, Virginia.

Last April, the US-East-1 zone had a major outage as well as another less serious incident in March.

And back in 2010, the US East region experienced a series of four outages in a single week.

Matthew Dublin is a senior writer at Genome Technology.