This incident was due to misconfiguration of one of the services responsible for delivering alerts for Environmental Sensor’s events.

During the outage, alerts generated by sensor devices did not get delivered to downstream systems responsible for displaying events on Command UI and sending email, SMS and push notifications out to users.

We have fixed the incorrect configuration and all systems are functioning as expected.

We will prevent such issues in the future by adding tests and monitoring relevant metrics to detect problems before impacting customers. Specifically we will monitor alert propagation from service to service and raise alarms when the sending of alerts falls outside of normal traffic patterns.