This incident was caused by an excessive volume of user updates generated by one of our automated testing systems running against a Verkada-managed test account in our production cluster, which we use to test for regressions in production. This behavior of the testing system is new and the system was not provisioned to handle the load. We increased the provisioning of our user update processing system to handle this load, which cleared the user update backlog and allowed user update latency to return to normal.
In the near future we will be making the following changes to make our user processing system more robust and help avoid these issues in the future:
Exclude user updates generated by our testing systems from realtime user update processing.
Consolidate user updates for faster processing.
Posted Jun 05, 2024 - 02:35 PDT
Resolved
This incident has been resolved.
Posted Jun 05, 2024 - 02:28 PDT
Update
Our current estimate for when the backlog will finish processing and user processing latency will return to normal is 2 hours.
Posted Jun 04, 2024 - 17:39 PDT
Monitoring
The system that processes Access user profile changes for display in Command is experiencing a processing backlog. We have increased system capacity to process the backlog, which is having the desired effect of reducing the backlog. During this time, changes made to users will be slow to appear in Command. Changes will still take effect on devices as usual.