Platform(s) Affected:

US Tunnel Grid (West & East)

User Impact:

Users will have experienced slower tunneled connections as they were routed through other parts of the grid in Europe and South America. Users who had only whitelisted specific sections of the grid may have been able to establish connections. Additionally, users with incorrectly configured DNS servers may have been unable to establish a connection during the window.

Root Cause Analysis:

The number of concurrent connections that a tunnel can support is, in part, restricted by the open file limit on Ubuntu. A misconfiguration in the previous deployment of new tunnel servers meant this open file limit was not increased, and resulted in the tunnel servers being unable to accept new connections.

Modifications to the configuration were rolled out, correctly increasing the open file limit.