User Impact: Devices reconnecting and general connection instability, causing dropped sessions and false offline alerts.
Root Cause Analysis: A buildup of job related "action flags" caused a slow down in processing other messages as the existing flags were not being cleared out as expected. The slow down resulted in the failure of some devices to receive a response prior to their timeout. As such, the Ping response message had failed to reach the platform and triggered a reconnect and offline alerts.
We have cleared out the backlog of flags and an investigation is underway to determine the best method of avoiding a this behavior in the future.