The backlog of events has been processed. Event processing latency is back to normal.
Posted Sep 13, 2023 - 21:22 PDT
Monitoring
A fix has been deployed and we seem to be making progress in processing the backlog of events. Event processing latency seems to be going down overall, though some customers might still be seeing high latency. We will continue to monitor and update on progress.
Posted Sep 13, 2023 - 16:40 PDT
Identified
The revert of yesterday's change did not fix the issue. We identified a dependency on a downstream service as another cause of high latency. We are removing this dependency and will update here when we have deployed the fix.
Also, customers should note that aux input-triggered lockdowns and unlocks will also be slowed down by this issue, as will panic button-triggered lockdowns in Alarms.
Posted Sep 13, 2023 - 13:36 PDT
Monitoring
There are significant delays in Access Control event processing. Recent events may not appear in Command, either in the door event view, the user event view, the Reports view, or Alert Inbox. These delays will also delay notifications for alerts based on any events whose processing is delayed.
The root cause was an attempted fix for intermittent slowness in event processing for some customers, which unfortunately made processing slower for all customers.
The change has already been reverted and we are monitoring. We will update here when event processing latency is back to normal.
Posted Sep 13, 2023 - 09:55 PDT
This incident affected: Events and Notifications, Access Control, and Alarms.