us1 Lagoon cluster instability

Incident Report for amazee.io

Resolved

This incident has been resolved.

Posted Nov 01, 2019 - 22:45 CET

Monitoring

After rebalancing the cluster and restarting affected nodes, our monitoring is green across the cluster. We are monitoring for continued stability.

Posted Nov 01, 2019 - 20:17 CET

Update

We are attempting to rebalance the cluster onto nodes which are not affected by this issue.

Posted Nov 01, 2019 - 19:53 CET

Identified

The issues appears to stem from some nodes being unable to correctly access their EBS volumes. We are working on a resolution.

Posted Nov 01, 2019 - 19:39 CET

Investigating

We are seeing instability in the us1 cluster at this time. Some sites are indicating down.

Posted Nov 01, 2019 - 19:38 CET

This incident affected: USA (us1.lagoon).