us1 Lagoon cluster instability

Incident Report for amazee.io

Resolved

This incident has been resolved.
Posted Nov 01, 2019 - 21:45 UTC

Monitoring

After rebalancing the cluster and restarting affected nodes, our monitoring is green across the cluster. We are monitoring for continued stability.
Posted Nov 01, 2019 - 19:17 UTC

Update

We are attempting to rebalance the cluster onto nodes which are not affected by this issue.
Posted Nov 01, 2019 - 18:53 UTC

Identified

The issues appears to stem from some nodes being unable to correctly access their EBS volumes. We are working on a resolution.
Posted Nov 01, 2019 - 18:39 UTC

Investigating

We are seeing instability in the us1 cluster at this time. Some sites are indicating down.
Posted Nov 01, 2019 - 18:38 UTC