Summary 

On October 31, 2022, from approximately 22:15 UTC until 22:40 UTC, a few customers in the US service region received 500 errors for events sent to the Events API. Events receiving errors were retried successfully within minutes. Webhooks, notifications, inbound email events, and the REST API were not impacted at all by this incident.

What Happened 

The errors were precipitated by the phased rollout of a configuration change, which began at 21:02 UTC and continued until 22:40 UTC when the rollout was paused. During this rollout hosts were marked as "healthy" but were repeatedly restarting, causing allocations to fail. Those allocations were transferred to other functioning hosts, so indicators of degradation in our system did not appear until 22:25 UTC when a monitor for excessive failed allocations was triggered. As the configuration change continued to roll out, additional hosts began to repeatedly restart. Customer impact began at 22:15 UTC when enough hosts were restarting that there were insufficient healthy hosts to process all Events API requests. Alerts notified responders, and a major incident was triggered at 22:25 UTC after the first few 500 errors. The deployment was paused and failed events were immediately re-queued for processing. At 22:40 UTC, we observed recovery and an end to delayed customer events. The configuration change was rolled forward to a known good configuration and additional hosts were provisioned. Clean-up actions and decommissioning affected hosts continued until 02:42 UTC November 1st.

What We Are Doing About This

We have identified the change that caused hosts to restart and have removed it from the host configuration. We also plan to improve our monitoring for failed allocations to catch this issue before it becomes customer-impacting in future rollouts. We apologize for the inconvenience that this has caused. For any questions, comments, or concerns, please contact us at support@pagerduty.com.