Summary

On June 9th, 2022 from 12:15 UTC through 18:25 UTC, a temporary failure of a downstream communications provider caused delays in delivery of specific PagerDuty emails. In the US and EU service regions, administrative emails for password resets, user invitations, and responder readiness reports were affected. EU service region status update emails were also affected.

Email notifications for all other types (incident notifications, responder requests, on-call handoff), and all other channels, including Push, SMS and Voice were unaffected by this incident.

What Happened

On June 9th, 2022 at 13:24 UTC, PagerDuty internal monitoring systems alerted that one of our downstream communication providers’ service was failing to accept and send outbound administrative emails and status update emails to a handful of recipients. The downstream communications provider was able to re-establish our connections with them at 13:48 UTC, and our systems were able to automatically retry requests that previously failed. It took a while longer for the communications provider to recover from the backlog of email requests that had built up during the earlier downtime. Newer email notifications were prioritized for delivery and were sent in real-time as of 15:30 UTC. Full recovery of all systems (including the backlog of requests) completed at 18:25 UTC.

What We Are Doing About This

Following this incident, our teams conducted a thorough investigation to discover ways of mitigating this issue both at the technical and organizational level. We had good monitoring in place to catch this issue, but we have identified gaps and potential areas of improvement and are actively addressing those concerns. We will be revisiting our retry and availability strategy for all outbound notifications in order to ensure that you get your message when you need it most.

We sincerely apologize for the delayed notifications you or your teams experienced. We understand how vital our platform is for our customers. As always, we stand by our commitment to providing the most reliable and resilient platform in the industry. If you have any questions, please reach out to support@pagerduty.com.