On April 12th, 2023 between 22:12 UTC to 23:20 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing a delay in the delivery of SMS, phone, and push notifications. The email notification channel was unaffected during this time.
On April 12th, 2023 at 22:12 UTC our teams noticed an increase in notification processing times caused by an infrastructure (server) mis-configuration that prevented network communication between our notification delivery services. Service was restored by 23:20 UTC which involved rolling back the configuration to a previous known state. Users affected would have received notifications delayed up to 20 minutes.
We are currently improving our infrastructure scripts to catch this error condition and will add additional testing for future roll-outs of this nature. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com