Search Service Issue

Incident Report for Verkada Command

Postmortem

Post-Incident Report: Camera Search Features Disruption

Between the evening of September 30 and the afternoon of October 2, we experienced a service disruption that impacted several camera search features within our Command platform in the United States region. The issue, which began on the evening of September 30, impacted LPR, AI-powered, Attribute, and Face Search. Importantly, all core camera services, including live video streaming, system alerts, and archives, were unaffected by this incident, and no customer data was lost as a result.

The initial cause is believed to be a routine update that unintentionally reduced system capacity, leading to a service overload. Our team worked in close collaboration with our cloud provider to restore service, and restored impacted features on a rolling basis.

Now that the impacted analytics features have been restored, we are continuing to conduct a thorough joint investigation with our cloud provider in an effort to further support the resilience of our platform. We sincerely apologize for this disruption and thank you for your continued partnership.

Posted Oct 03, 2025 - 19:20 PDT

Resolved

All camera analytics search features, including AI-powered search, attribute search, face search, and license plate recognition (LPR) search, have been fully restored in the United States region. Indexing is fully caught up, and the system is performing normally. No customer data was lost during this incident, and we appreciate your patience throughout the disruption.

Posted Oct 02, 2025 - 23:45 PDT

Update

We are continuing to monitor for any further issues. We will post a final update once indexing for AI-powered search and attribute search are fully caught up in the United States region. Thank you for your patience.

Posted Oct 02, 2025 - 15:36 PDT

Monitoring

All search functionality has been restored and is working. We are currently catching up on indexing for AI-powered search and attribute search, which may cause some results to be not fully up to date. Search results will continue to improve as indexing completes. Our team is monitoring the system closely to ensure normal performance in the United States region. Other search features and regions remain unaffected.

Posted Oct 02, 2025 - 14:36 PDT

Update

Face search and attribute search have been partially restored. Searches are working as expected, but indexing has not yet resumed, which may cause some results to appear out of date. Our engineering team continues to restore normal indexing performance in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues.

Posted Oct 02, 2025 - 13:25 PDT

Update

We have restored License Plate Recognition (LPR) search functionality. Our engineering team continues to work on fully restoring AI-powered search and attribute search features in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to resolve this issue.

Posted Oct 02, 2025 - 10:44 PDT

Update

We are close to restoring License Plate Recognition (LPR) search functionality and expect it to be available soon. Our engineering team continues to work on fully restoring AI-powered search and attribute search features in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to resolve this issue.

Posted Oct 02, 2025 - 09:51 PDT

Update

Face search has been restored and is now functioning normally. Our engineering team is actively working to restore AI-powered search, attribute search, and license plate recognition (LPR) search features. Other search features remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to fully resolve this issue.

Posted Oct 01, 2025 - 14:30 PDT

Update

LPR unlocks are not affected. We are continuing to work on a fix for this issue.

Posted Oct 01, 2025 - 06:50 PDT

Identified

We are currently experiencing a partial outage affecting our camera analytics in the United States region, including AI-powered search, attribute search, face search, and license plate recognition (LPR) features. Other search features are not impacted at this time. Our engineering team is actively working on a fix and will provide further updates as progress continues.

Posted Sep 30, 2025 - 21:23 PDT

This incident affected: Camera Analytics.