Notification Outage

Incident Report for PagerDuty

Postmortem

Summary

On April 12th, 2023 between 22:12 UTC to 23:20 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing a delay in the delivery of SMS, phone, and push notifications. The email notification channel was unaffected during this time.

What Happened

On April 12th, 2023 at 22:12 UTC our teams noticed an increase in notification processing times caused by an infrastructure (server) mis-configuration that prevented network communication between our notification delivery services. Service was restored by 23:20 UTC which involved rolling back the configuration to a previous known state. Users affected would have received notifications delayed up to 20 minutes.

What Are We Doing About This

We are currently improving our infrastructure scripts to catch this error condition and will add additional testing for future roll-outs of this nature. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com

Posted Apr 18, 2023 - 22:00 UTC

Resolved

We have resolved an incident where all PagerDuty customers in the US service region experienced issues with notification delivery. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.

Posted Apr 12, 2023 - 23:26 UTC

Monitoring

We have seen an almost full recovery to notification delivery for US service region customers. We are currently working to address delayed notifications for a small subset of customers and will provide another update within 20 mins.

Posted Apr 12, 2023 - 23:17 UTC

Identified

We have identified a fix and are rolling it out to US service region
customers. We will continue to monitor for improvements and provide another update within 20 mins.

Posted Apr 12, 2023 - 22:58 UTC

Update

We are continuing to investigate an incident where all of PagerDuty customers are experiencing issues with notification delivery. We will provide further updates within 20 minutes.

Posted Apr 12, 2023 - 22:38 UTC

Investigating

We are investigating a potential issue within PagerDuty where Push SMS and Phone notification are not working with in the US service Region.

Posted Apr 12, 2023 - 22:25 UTC

This incident affected: Notification Delivery (SMS (US), Voice (US), Push (US)).