Issues on dev1-compact

Incident Report for amazee.io

Postmortem

The service provider of dev1-compact (Digital Ocean) had an issue with their block storage in the region where the server resides (FRA1). After being made aware of the issue, they were able to resolve it on their side, and when restarted, dev1-compact was brought fully back online.

If you were connecting via SSH to dev1-compact, you will see an error similar to this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

This is expected, please delete the old line from your known_hosts file and re-attempt your connection.

Posted Nov 17, 2016 - 22:18 UTC

Resolved

dev1-compact is back fully operational, we also noticed an issue with deploys to the server after it was fixed, and have resolved that as well.

Posted Nov 17, 2016 - 18:44 UTC

Update

The issue was caused by a problem with the providers Block Storage. They have fixed the issue and dev1-compact appears to be stable, we will monitor for a while before closing this incident.

Posted Nov 17, 2016 - 18:24 UTC

Monitoring

dev1-compact is now up again, we are awaiting a report of the outage cause from the service provider

Posted Nov 17, 2016 - 17:46 UTC

Update

We have just heard back from the service provider, their engineer team is now investigating the issue.

Posted Nov 17, 2016 - 17:37 UTC

Identified

We have identified an issue with one of the volumes on the server, we are working with the service provider to resolve this currently.

Posted Nov 17, 2016 - 17:22 UTC

Investigating

The dev1-compact is currently offline, we are investigating the issues at this time.

Posted Nov 17, 2016 - 17:19 UTC