On March 26th, 2024, Datto SaaS Protection and Datto Backupify customers started to experience a service degradation. Starting April 9th, 2024, some end domains began to experience a complete service interruption.
The root cause for this service degradation was identified to be a Microsoft 365 Exchange release, which was confirmed by Microsoft. Microsoft had patched their Exchange servers and part of the deployment process involved reindexing all Exchange data in a way which affects APIs which Independent Software Vendors (ISVs) like Datto, utilize for backup and caused all items to be resynchronized. In this case the Exchange API which Datto uses to backup data was affected, resulting in a 14x increase in daily data backup volume compared to the average daily load. This increase also had an impact on Microsoft API throttling due to the large increase in requests to the Microsoft API to resynchronize each Exchange folder.
Datto was in direct communication with Microsoft from the outset of this problem, and on April 20th, 2024, Microsoft stopped the re-indexing process for all Datto customers. Microsoft confirmed that they have found an alternative approach to complete their software deployment and will not resume re-indexing Exchange folders moving forward.
Datto expedited deployment of a significant amount of additional infrastructure to process the additional data and deployed a number of software changes to process this large data volume.
As of May 16th, 2024, the issue is resolved.