On the 13th of May at 13:41 UTC and the 14th of May at 13.40 UTC the team received sporadic reports from Datto RMM partners on the Pinotage (EU1) platform that alerts were not loading successfully in the Web Interface.
The root cause of this issue was identified to be with the backend service handling Alert related requests from the front end. The Alert service has been observed to have run out of resources, reported by our internal monitoring system.
The short term mitigation was a combination of doing a failover on the Alerts database, and restarting one of the downstream services. The long term resolution that was implemented was to permanently increase the available resources for the Alert service.