PSA - LR, AE, AE2, AW, AW2 and AU - Partners may experience duplicate ticket & ticket note creation in PSA.
Incident Report for Datto
Postmortem

On 30-September-2021, Thursday at 6am UTC, Autotask PSA Partners hosted in our Philadelphia datacenter experienced a service interruption which caused incoming emails to not create tickets or add notes to existing tickets, and in some cases to create duplicate tickets or duplicate notes.

Our Infrastructure team worked to correct the problem and it has been resolved by 9:30am UTC on the same day.

The service interruption was caused by the procedure deleting emails marked as already processed failing to delete the emails. This resulted in the emails being queued up to be processed again, causing an increasingly longer queue and multiplicate processing of a single email.

Autotask uses a clustered environment to handle incoming email processing.  Incoming emails are parsed and the associated information is used to create or update associated tickets.  Once an email is processed, it is then deleted and the next email is processed.

Analysis revealed that the mailbox database had entered a bad state which led to the issue.  We created a new mailbox and switched Autotask to utilize it.  Incoming emails began immediately processing and deleting properly.  We then migrated all unprocessed emails to this new mailbox so that the backlog could be cleared.

We have added logic to verify that emails are properly deleted upon processing.  If an email is not deleted, the service is paused and we are immediately alerted. 

In the coming weeks, we will migrate to a new email processing technology to further harden the incoming email processing flows.

Posted Oct 07, 2021 - 11:47 UTC

Resolved
This incident has been resolved.
Posted Sep 30, 2021 - 15:48 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Emails are currently being processed, it will take approximately an hour for the email service to be up to date.
Posted Sep 30, 2021 - 09:43 UTC
Update
We are continuing to investigate this issue.
Posted Sep 30, 2021 - 07:19 UTC
Update
We are continuing to investigate this issue.
Posted Sep 30, 2021 - 06:47 UTC
Update
Our engineers are continuing to investigate the cause of the duplicate ticket and ticket note creation in PSA.

Partners may also experience delays in creation of tickets due to this issue.

We apologise for the continued inconvenience and will update this status within 30 minutes.
Posted Sep 30, 2021 - 06:42 UTC
Investigating
Our engineers are currently investigating reports of duplicate ticket creation in PSA.

We apologise for any inconvenience caused and will update this status within the next 30 minutes.
Posted Sep 30, 2021 - 06:01 UTC
This incident affected: Autotask PSA (Limited Release, America East, America East 2, America West, America West 2, Australia / New Zealand).