PSA - OTP Email delay - Under Investigation
Incident Report for Datto
Postmortem

Start and End Date/Time (UTC):

05:13 February 5th, 2020 - 11:30 February 5th, 2020

Zone(s) Affected:

Limited Release, America East, America East 2, America West, America West 2, Spanish (Español), Australia / New Zealand, UK (United Kingdom), UK2 (United Kingdom 2) and UK Limited Release (United Kingdom).

Root Cause Analysis:

On the evening of February 11th, during a scheduled maintenance, we added new infrastructure and security measures to our mail clustering environment in our LON and PHL Data Centers. While our testing was successful after the change, we started to receive reports several hours later of email delays and processing issues. At approximately 6:30 AM Eastern, we made the decision to revert all changes made the evening before to give us more time for research and to temporarily resolve the email processing issues. After we reverted the change, there was a backlog of emails to be processed that took approximately an hour to catch up.

After further investigation, we discovered that our emails were being greylisted due to the new IP addresses used to send the emails from the Autotask PSA. Although the new IPs were within our published range they had no established IP reputation and were therefore greylisted by some recipients’ filtering providers. We researched the greylisting issue and made the decision to stay with the established external mail IPs. The following night we re-implemented the new infrastructure and security measures but maintained the established external mail IPs.

Posted Mar 10, 2020 - 17:27 UTC

Resolved
This Incident has been resolved.
Posted Feb 05, 2020 - 15:04 UTC
Monitoring
All mail should now be caught up.

Thank you for your patience and our apologies for this impact this has had on your business.
Posted Feb 05, 2020 - 14:28 UTC
Update
Dear partners, We have reverted the DKIM changes and we are monitoring the email traffic. Please note that we are processing a substantial backlog and emails may take some time before being received.
Posted Feb 05, 2020 - 11:53 UTC
Update
We are aware that some partners are still experiencing issues with the delivery of their emails. We are still investigating this issue as a matter of critical priority.
Posted Feb 05, 2020 - 11:29 UTC
Update
Dear valued customers,

We have confirmed that the email service is fully operational. Partners impacted by the email delays are likely to be related to the DKIM changes that were deployed yesterday as per the schedule below:

When: Tuesday 2/4/2020 9:00 PM Local time for each Datacenter
Who:
Munich Datacenter (DE1, EU1) at 3:00 PM Eastern
London Datacenter (UK, UK2, UKLR) at 4:00 PM Eastern
Philadelphia Datacenter (LR, PR, AE, AW, AU, ES, AT, AE2, AW2) at 9:00 PM Eastern

If you have followed the Whitelisting Autotask IP addresses page in Autotask Help, you will NOT be impacted and no further action is needed. https://www.autotask.net/help/Content/AdminSetup/TechnicalStuff/Whitelist.htm?Highlight=whitelist

If you have NOT followed the “Whitelisting Autotask IP addresses” page in Autotask Help, you may have issues with your emails going out from Autotask. (Example: If you have Whitelisted the individual IPs and not the range)
Posted Feb 05, 2020 - 10:32 UTC
Update
Our Teams have confirmed that emails are currently being delayed. Although emails are being delayed pleased rest assured that they are being processed.
Posted Feb 05, 2020 - 09:35 UTC
Investigating
Our teams are currently investigating OTP Email delay for PSA on multiple zones. Updates will be posted here with the status of this investigation.

Thank you for your patience!
Posted Feb 05, 2020 - 08:21 UTC
This incident affected: Autotask PSA (America East, America East 2, America West, America West 2, UK (United Kingdom), UK 2 (United Kingdom), EU1 (English Europe and Asia)).