On December 6th, 2022 at 16:15 UTC, Datto Workplace Partners on the US data center experienced a service interruption which caused logins and navigation to fail in Workplace Manager and Workplace Online, as well as the failure of a portion of app connections. 

The root cause for this service interruption was identified to be engineering change deployed on the day addressing a defect. This change caused login related queries to queue up and overload the database servicing these requests. 

The issue did not surface during QA or in production on smaller data centers because a key trigger was the volume of requests in a short amount of time. While the components are equipped to deal with this volume, the engineering change deployed with the release on the day along with the general usage on the US data center caused a degradation. 

Our Engineering team deployed a hotfix to correct the problem on December 6th, 2022 at 18:49 UTC and proceeded to monitor the health of the service. The incident was resolved at 20:20 UTC after no further issues surfaced during monitoring.