On Monday 21st October, 2019 we experienced heavily degraded server performance across some of our enterprise customers.
The outage lasted approximately 2 hours, with servers responding erratically and returning intermittent 502 errors over the duration.
The issue was caused by one of our cloud database servers, relied on by our high-volume customers, becoming overloaded and unable to respond to higher-than-usual traffic. Our monitoring systems failed to report any anomalies, which meant it took us longer to diagnose and identify the cause of the degraded performance.
We’ve substantially upgraded the resources available to this database server which has restored operations.
We will immediately investigate our monitoring system and put in place measures to prevent this type of failure from happening again in the future, as well as implementing a higher level of isolation, to prevent issues of this nature from manifesting as broadly.
To those affected, we apologise for this significant window of downtime/degraded performance, and we’ll be contacting you directly with followup measures and apologies.
Thank you for your patience,