Investigating a Potential Issue
Incident Report for PagerDuty
Postmortem

Summary

On February 13th, 2023, from 14:03 UTC to 14:13 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing delayed email notifications in the US service region. This affected email notifications only. Other notification channels such as Push, SMS, Voice and Slack were unaffected during this time.  

What Happened

On February 13th, 2023 at 13:55 UTC, we noticed an increase in notification processing times due to a new configuration being rolled out to the notification system. This resulted in a small backup of notifications on the PagerDuty Notification Service at 14:03. Users affected would have received notifications delayed by 3-10 minutes. Service was restored at 14:15 UTC by rolling back the configuration to a previous known state.

What Are We Doing About This

We are currently improving both the code and our service architecture requirements to address this issue. We are also adding new procedures to prevent such an issue in the future. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com

Posted Feb 16, 2023 - 22:26 UTC

Resolved
We have identified and resolved an incident where there were delays in sending notifications in the US service region. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.
Posted Feb 13, 2023 - 14:33 UTC
Investigating
We are investigating a potential issue within PagerDuty. If we confirm an impact, we will update within 15 minutes. If there is no impact this notification will be removed.
Posted Feb 13, 2023 - 14:12 UTC
This incident affected: Notification Delivery (Notification Delivery (US)).