Summary

On February 13th, 2023, from 14:03 UTC to 14:13 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing delayed email notifications in the US service region. This affected email notifications only. Other notification channels such as Push, SMS, Voice and Slack were unaffected during this time.  

What Happened

On February 13th, 2023 at 13:55 UTC, we noticed an increase in notification processing times due to a new configuration being rolled out to the notification system. This resulted in a small backup of notifications on the PagerDuty Notification Service at 14:03. Users affected would have received notifications delayed by 3-10 minutes. Service was restored at 14:15 UTC by rolling back the configuration to a previous known state.

What Are We Doing About This

We are currently improving both the code and our service architecture requirements to address this issue. We are also adding new procedures to prevent such an issue in the future. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com