PSA - CLEP email to ticket creation delays (AE, AE2, AW, and AW2)
Incident Report for Datto
Postmortem

Start and End Date/Time (UTC):

15:49 February 19th, 2020 - 16:30 February 19th, 2020

Zone(s) Affected:

America West and America East 2

Root Cause Analysis:

On Wednesday, February 19th, 2020 at approximately 15:49 UTC, Clients in the AE2 and AW zones started to experience email delays. These delays were due to the need to take some mail cluster members down in order to increase CPU resources. Unfortunately, this caused a higher than normal load on the mail cluster resources which resulted in the delays experienced by the customers. After the increase in CPU resources, email processing was caught up by 16:30 UTC. While pre-deployment estimates indicated that we would not be resource-constrained in this instance, in the future, we will over-provision mail hosts in the mail cluster in order to avoid low-resource situations in that infrastructure tier.

Posted Mar 10, 2020 - 17:30 UTC

Resolved
This incident has been resolved.
Posted Feb 19, 2020 - 19:07 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 19, 2020 - 16:57 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 19, 2020 - 16:32 UTC
Investigating
Our teams are currently investigating email to ticket creation delays for PSA on AE, AE2, AW, and AW2. An update will be posted here as we progress with this investigation.
Posted Feb 19, 2020 - 16:20 UTC
This incident affected: Autotask PSA (America East, America East 2, America West, America West 2).