RMM - Concord and Merlot - Delayed Audit
Incident Report for Datto
Postmortem

On May 26th, 2022 at 19:00 UTC, Datto RMM Partners on the Concord and Merlot platforms experienced an issue which caused slow processing of device audits.

Engineering has determined that the resources were exhausted after the release on servers handling audit requests.

Our Engineering team added additional instances with increased resource capacity to resolve the issue. The problem was confirmed to be fixed on May 27th, 2022 at 02:13 UTC.

Additional changes are planned to improve audit processing, and prevent the issue from re-occurring.

Posted Jun 22, 2022 - 10:54 UTC

Resolved
This incident has been resolved.
Posted May 27, 2022 - 19:15 UTC
Monitoring
The team have made some adjustments and provided the below update;
- Concord’s audit processing is keeping pace with audit submission rate. Audits are working as expected in a timely manner.
- Daily audits are expected to run correctly
- Root cause analysis will be released when available
Posted May 27, 2022 - 02:13 UTC
Update
We are continuing to investigate this issue. Merlot Audits should now be processing at a normal rate. Policy pushes on Concord may also be delayed.
Posted May 26, 2022 - 22:24 UTC
Investigating
Our teams are currently investigating Device Audits not completing for Datto RMM on Concord and Merlot.

Thank you for your patience!
Posted May 26, 2022 - 19:00 UTC
This incident affected: Datto RMM (Concord (US East)).