The Autotask AW zone (about 10% of Autotask partners) experienced some periods of high latency and a brief service interruption lasting less than 5 minutes during the morning of Wednesday June 23.

Our research revealed that the cause was due to a defect in the 2021.1 release which had been deployed the night before. This release included a new user experience for the Company/Account pages.

One of the features of this new UIX was the ability to view the number of active contacts for a company. An inefficient method was used to count the number of active contacts within a company.

A partner in the AW zone had an extremely high number of active contacts associated with a single account (>150,000). When resources at that partner began using the new Company user experience, this inefficient method overwhelmed the web tier and caused the process to run out of memory.

Our Development and Infrastructure teams began gathering data and mitigating the impact immediately. The edge-case was identified and a hotpatch to address the inefficient active contact count was deployed early the following morning (Thursday June 24)