Resolved -
AWS has restored network connectivity so all the services are back online.
Update from AWS:
"Starting at 1:18 PM PDT we experienced connectivity issues to some EC2 instances, increased API errors rates, and degraded performance for some EBS volumes within a single Availability Zone in the EU-CENTRAL-1 Region. At 4:26 PM PDT, network connectivity was restored and the majority of affected instances and EBS volumes began to recover. At 4:33 PM PDT, increased API error rates and latencies had also returned to normal levels. The issue has been resolved and the service is operating normally. The root cause of this issue was a failure of a control system which disabled multiple air handlers in the affected Availability Zone. These air handlers move cool air to the servers and equipment, and when they were disabled, ambient temperatures began to rise. Servers and networking equipment in the affected Availability Zone began to power-off when unsafe temperatures were reached."
Jun 11, 02:17 UTC
Update -
Most of the services are back online - AWS is facing a loss of one of the Availability Zones in Frankfurt. Most of the workloads have been rescheduled. Certain workloads (e.g. Solr search servers) are bound to one zone and will remain unavailable.
Jun 10, 21:07 UTC
Identified -
Appears to be caused by an AWS issue:
"1:24 PM PDT We are investigating connectivity issues for some EC2 instances in a single Availability Zone (euc1-az1) in the EU-CENTRAL-1 Region"
Jun 10, 20:50 UTC
Investigating -
We are currently investigating an issue causing website outages in DE3.
Jun 10, 20:37 UTC