Access Control User updates slow to appear in Command

Incident Report for Verkada Command

Postmortem

This incident was caused by an accidental underprovisioning of one of the services used to process user updates and propagate them to the low-latency storage used to display user profile and user list in Command. During this outage, all changes made to users (for example access suspensions, card assignments, etc.) actually were persisted and propagated to devices, so devices were functioning properly. However, changes were showing up in Command only after a long delay.

The underprovisioning was caused by an incorrect deployment configuration for the affected service. We have addressed this issue by fixing bug in the deployment configuration. We will prevent future instances of this issue by adding monitoring for the deployment configuration. That work is in progress.

Posted Aug 12, 2024 - 13:12 PDT

Resolved

This incident has been resolved.
Posted Aug 12, 2024 - 13:08 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 12, 2024 - 11:32 PDT

Update

The root cause seems to have been an underprovisioning of one of the services that processes user updates. We have increased the provisioning of this service and the user profile update latency seems to have improved to expected values. We are monitoring to see if the improvement is consistent and durable.
Posted Aug 12, 2024 - 11:32 PDT

Investigating

We are currently investigating this issue.
Posted Aug 12, 2024 - 11:19 PDT
This incident affected: Access Control.