Datto EDR - EDR may unexpectedly isolate services/policies
Incident Report for Datto
Postmortem

On the morning of July 6th, Datto EDR Partners experienced a service interruption which caused automated threat response policies to isolate devices when there was an update to Datto AV.

The root cause for this service interruption was when the Datto EDR Content Engineering team pushed a new behavioral detection rule to all EDR tenants. This rule included a recommended response action to isolate the host when an alert was raised.

Prior to the production release testing of the rule was limited and we did not discover that the Datto AV agent can trigger the alert when performing updates. Once this rule was pushed to product tenants with automated threat response policies assigned to any locations with Datto AV there was a potential the alert would trigger and automatically isolate the endpoints.

Our Engineering team deployed a fix to correct the problem on the morning of July 8th that removed this new rule's auto response actions and stopped it from generating any more alerts.

To ensure issues like this do not happen in the future: The Datto EDR team is expanding the testing procedures, adding stakeholders from multiple departments into the change management process, and formalizing a partner notification process to provide updates prior to any production changes to existing or new behavioral rules.

Posted Jul 10, 2024 - 19:19 UTC

Resolved
This incident has been resolved.
Posted Jul 09, 2024 - 17:03 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 08, 2024 - 13:59 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 08, 2024 - 13:59 UTC
Identified
We are aware of a problem where Datto EDR may unexpectedly isolate certain services/policies.


The Kaseya R&D Team is actively working on a solution for this problem.


To prevent the behavior from occurring you can disable the following rule within Datto EDR
-Endpoint Protection Disable Attempted


If you require assistance with this workaround please engage our Support team by creating a ticket at https://helpdesk.kaseya.com/


Subscribe to the Datto Status Page for up-to-date information at https://status.datto.com/
Posted Jul 08, 2024 - 13:43 UTC
This incident affected: Datto EDR (Detection Engine).