Verkada Command Status

Sep 21, 2023

No incidents reported today.

Sep 20, 2023

Resolved - This incident has been resolved.
Sep 20, 14:06 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 20, 13:25 PDT

Identified - The issue has been identified and a fix is being implemented.
Sep 20, 13:19 PDT

Investigating - Some users may be experiencing issues loading Verkada Command. We are actively investigating this issue and will provide an update shortly.
Sep 20, 13:14 PDT

Sep 19, 2023

No incidents reported.

Sep 18, 2023

Degraded Performance

Resolved - This incident has been resolved.
Sep 18, 23:27 PDT

Update - We are continuing to work on a fix for this issue.
Sep 18, 20:52 PDT

Update - We are continuing to work on a fix for this issue.
Sep 18, 16:15 PDT

Identified - We are currently experiencing connectivity issues due to an ongoing Amazon Web Services outage in our primary region. Our engineering team is actively monitoring the situation. Some services may be intermittently unavailable. We apologize for any inconvenience and are working to resolve the issue as quickly as possible. Updates from AWS can be found here: https://health.aws.amazon.com/health/status
Sep 18, 16:06 PDT

Update - We are continuing to monitor for any further issues.
Sep 18, 15:35 PDT

Update - We are continuing to monitor for any further issues.
Sep 18, 14:46 PDT

Update - We are continuing to monitor for any further issues.
Sep 18, 14:41 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 18, 14:35 PDT

Identified - The issue has been identified and a fix is being implemented.
Sep 18, 12:41 PDT

Investigating - Some users may be experiencing degraded performance loading certain parts of Verkada Command. We are actively investigating this issue and will provide an update shortly.

Requests to create camera archives are are not being processed.
Sep 18, 11:29 PDT

Notification Delays

Resolved - This incident has been resolved.
Sep 18, 11:29 PDT

Update - We are continuing to work on a fix for this issue.
Sep 18, 10:55 PDT

Identified - The issue has been identified and a fix is being implemented.
Sep 18, 10:13 PDT

Update - We are continuing to investigate this issue.
Sep 18, 09:32 PDT

Investigating - We are currently experiencing issues with high latency on notifications. We are actively investigating this issue and will provide an update shortly.
Sep 18, 09:28 PDT

Sep 17, 2023

No incidents reported.

Sep 16, 2023

No incidents reported.

Sep 15, 2023

No incidents reported.

Sep 14, 2023

Notification Delays

Resolved - This incident has been resolved.
Sep 14, 13:29 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 14, 13:08 PDT

Identified - We have identified an issue with our person of interest notification pipeline. Some notifications may be delayed. We are actively investigating this issue and will provide an update shortly.
Sep 14, 11:58 PDT

Sep 13, 2023

Access Control Events Slow to Populate

Resolved - The backlog of events has been processed. Event processing latency is back to normal.
Sep 13, 21:22 PDT

Monitoring - A fix has been deployed and we seem to be making progress in processing the backlog of events. Event processing latency seems to be going down overall, though some customers might still be seeing high latency. We will continue to monitor and update on progress.
Sep 13, 16:40 PDT

Identified - The revert of yesterday's change did not fix the issue. We identified a dependency on a downstream service as another cause of high latency. We are removing this dependency and will update here when we have deployed the fix.

Also, customers should note that aux input-triggered lockdowns and unlocks will also be slowed down by this issue, as will panic button-triggered lockdowns in Alarms.
Sep 13, 13:36 PDT

Monitoring - There are significant delays in Access Control event processing. Recent events may not appear in Command, either in the door event view, the user event view, the Reports view, or Alert Inbox. These delays will also delay notifications for alerts based on any events whose processing is delayed.

The root cause was an attempted fix for intermittent slowness in event processing for some customers, which unfortunately made processing slower for all customers.

The change has already been reverted and we are monitoring. We will update here when event processing latency is back to normal.
Sep 13, 09:55 PDT

Notification Delays

Resolved - This incident has been resolved.
Sep 13, 10:03 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 13, 09:57 PDT

Investigating - We are currently experiencing issues with high latency on notifications. We are actively investigating this issue and will have an update shortly.
Sep 13, 09:01 PDT

Sep 12, 2023

Access Control is experiencing long event processing times

Resolved - This incident has been resolved.

The root cause was a change to our event processing code meant to better spread processing load between nodes, which increased pressure on caches and, under high load, resulted in processing nodes timing out. We have fixed the issue.
Sep 12, 04:28 PDT

Identified - The issue has been identified and a fix is being implemented.
Sep 11, 08:34 PDT

Sep 11, 2023

Sep 10, 2023

No incidents reported.

Sep 9, 2023

No incidents reported.

Sep 8, 2023

Cellular Connectivity Issues

Resolved - All LTE devices are functioning normally at this stage.
Sep 8, 13:53 PDT

Update - Most cellular backup modules should have normal connectivity at this stage. However, a small number of devices appear to still be experiencing connectivity issues. We are working with our upstream partner to resolve these cases.
Sep 6, 19:38 PDT

Update - Our provider has indicated that they have implemented a fix and impacted devices should be gradually coming back online. Further updates are available here: https://status.hologram.io.
Aug 30, 09:29 PDT

Update - Our provider has indicated that the previously identified fix is not viable. A new fix is being implemented, however the timeline for full resolution is not certain at this stage. Further updates are available here: https://status.hologram.io.
Aug 29, 20:55 PDT

Identified - Our LTE networking provider has identified the root cause of the issue and is actively resolving. They expect full resolution between 0700 and 0800 UTC. Further updates are available here: https://status.hologram.io.
Aug 29, 16:28 PDT

Investigating - Some cellular backup modules are unable to connect to the network due to an issue with our SIM provider. Our provider has found the root cause and is progressively fixing affected devices. They expect the issue to be resolved fully later this evening.
Aug 29, 12:46 PDT

Delayed Notifications

Resolved - This incident has been resolved.
Sep 8, 09:29 PDT

Monitoring - A fix has been implemented and we are monitoring the results.
Sep 8, 08:47 PDT

Investigating - We are currently experiencing an issue with delayed notifications. We are actively investigation this issue and will provide an update shortly.
Sep 8, 08:24 PDT

Sep 7, 2023

No incidents reported.