US SMS Notification Delivery
Incident Report for PagerDuty
Postmortem

Summary

On May 18, 2023, from 17:21 UTC to 18:17 UTC, PagerDuty experienced a service degradation that resulted in a small subset of United States based users on the AT&T carrier network, being unable to receive SMS notifications. SMS users in the United States on other carriers (not AT&T-based) were unaffected. Other notification channels such as Push, Voice and Email were also unaffected during this time.  

What Happened

On May 18, 2023 at 17:21 UTC, we noticed an increase in SMS notification delivery failures to users in the United States on the AT&T carrier network due to a third-party outage. 

Users affected would not have received SMS notifications during the outage period. Service was restored at 18:17 UTC by the third-party carrier and we saw a decrease in SMS notification delivery failures. 

What Are We Doing About This

Outages from third-party carriers are outside of our control and we highly recommend that users configure multiple channels (SMS, Push, Voice, Email) for notifications, so they can still receive notifications in the event that one of their carriers has an outage. We are updating our public documentation to highlight this best practice.

We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com.

Posted May 24, 2023 - 21:11 UTC

Resolved
We have resolved an incident where some PagerDuty users using AT&T in the US experienced issues receiving SMS notifications. The incident is now resolved, and there is no ongoing impact on customers. Please reach out to support@pagerduty.com if you have any concerns.
Posted May 18, 2023 - 18:37 UTC
Identified
We are investigating an incident where some U.S PagerDuty users are not receiving SMS notifications. We will provide further updates within 20 minutes.
Posted May 18, 2023 - 18:18 UTC
Investigating
We are investigating a potential issue within PagerDuty. If we confirm an impact, we will update within 15 minutes. If there is no impact this notification will be removed.
Posted May 18, 2023 - 18:13 UTC
This incident affected: Notification Delivery (SMS (US)).