Shuttered DoE Site Back In BusinessOpenEI.org Spent Nearly Two Days In Dark
Since early Thursday morning, visitors to the Energy Department website were greeted with the following message:
"OpenEI.org is a free, open-source, knowledge-sharing platform providing access to data, models, tools, and information that accelerate the transition to clean energy systems. Unfortunately, OpenEI's datacenter, Amazon EC2, is temporarily down. We are working aggressively to restore service as soon as possible, please check back soon."
Sunday night, Amazon on its web services health dashboard said service was basically restored to all affected sites using its northern Virginia cloud computing data center:
"The vast majority of affected volumes have now been recovered. We're in the process of contacting a limited number of customers who have EBS volumes that have not yet recovered and will continue to work hard on restoring these remaining volumes.
EBS, or Elastic Block Storage, provides block level storage volumes for use with Amazon EC2 service. On Friday evening, Amazon posted:
"The work we're doing to enable customers to be able to launch EBS backed instances and create, delete, attach and detach EBS volumes in the affected availability zone (northern Virginia) is taking considerably more time than we anticipated. The team is in the midst of troubleshooting a bottleneck in this process."
Earlier Friday, Amazon said it was making progress at restoring servers that hosts websites for businesses and government agencies. An outage early Thursday at a cloud computing server facility in northern Virginia caused a number of websites to go dark.
At 5:15 p.m. EDT Friday, Amazon said most of the servers were operational by 1:30 p.m., but a small number of volumes would require a more time-consuming process to recover. On its health dashboard, Amazon posted:
"We anticipate that those will take longer to recover." We're now starting to work on those. We're also now working to enable customers to be able to launch EBS backed instances and create, delete, attach and detach EBS volumes in the affected availability zone (northern Virginia). Our current estimate is that this will take three to four hours until full access is restored."
Friday morning, Amazon said it's started to see meaningful progress in restoring volumes, many of which had been restored earlier in the day, and expects this progress to continue over the next few hours. At 11:49 a.m. EDT, Amazon said on its dashboard:
"We ... have heard many additional customers confirm that they're recovering. Our current estimate is that the majority of volumes will be recovered over the next five to six hours."
On Thursday, Amazon said it struggled to restore service. At 9:18 p.m. EDT Thursday, Amazon posted the following update:
"We have stabilized the remaining availability zone, but recovery is taking longer than we originally expected. We have been working hard to add the capacity that will enable us to safely re-mirror the stuck volumes. We expect to incrementally recover stuck volumes over the coming hours, but believe it will likely be several more hours until a significant number of volumes fully recover and customers are able to create new EBS-backed instances in the affected Availability Zone. We will be providing more information here as soon as we have it."
According to published reports, the outage of Amazon's cloud-based Web service early Thursday morning took down a number of websites, including Reddit, HootSuite, Foursquare and Quora.