PSA - DE1 Zone (WW18) - Latency/Long loading time - Under Investigation
Incident Report for Datto

Beginning  February 16, 2020 at 08:21UTC, and over the next two days, partners in the DE1 zone began to experience increased latency followed by an error page. The first service interruption lasted 27 minutes. The second interruption, on February 17 lasted three minutes and the last, on February 18 lasted 24 minutes, for a total of 55 minutes over three consecutive days.

The underlying cause of the interruptions was an inefficient method in the Autotask application code, used for querying Installed Products. The method was returning ALL Products whenever a user queried for or navigated to a control that queried Products. Historically, this has not been an issue, however, shortly prior to these incidents a partner mistakenly imported over 993,000 products manually into their database.  When a user in that partner’s database queried Products, the inefficient method queried ALL 993,000+ Products, causing race conditions on the web server making the call. As the web server became overloaded, the query was passed to the next web server, which was also overwhelmed by the query. The query would continue passing to the next web server until all of the web servers for the zone were overloaded, bringing the zone down.

Datto’s Development Engineers have inspected and replaced nine inefficient calls that formerly would try to return all rows for items such as Products. These calls now only return a portion of the data and will not cause any load increase on web servers. 

We will continue to serve our partners in order to allow them to use Autotask PSA to run their business reliably without undue limitations. We will undertake edge-case testing of large-scale data sets to determine if there are other opportunities to increase efficiency or eliminate causes of service interruption.

Posted Feb 24, 2021 - 15:33 UTC

This incident has been resolved.
Posted Feb 18, 2021 - 11:33 UTC
A fix has been implemented and we are monitoring the results.
Posted Feb 18, 2021 - 10:36 UTC
The issue has been identified and a fix is being implemented.
Posted Feb 18, 2021 - 10:24 UTC
Our teams are currently investigating long loading times for PSA on the DE1 (WW18) Zone. Updates will be posted here with the progress of this investigation.

Thank you for your patience!
Posted Feb 18, 2021 - 10:15 UTC
This incident affected: Autotask PSA (German (Deutsch)).