iOS notification delays
Incident Report for PagerDuty
Postmortem

Summary

On Aug 9th, 2021, from 20:51 UTC to 22:36 UTC, PagerDuty’s notification service experienced an issue resulting in iOS users on a few accounts experiencing delayed iOS push notifications. Android push notifications, SMS, email, and voice channels were unaffected.

What Happened

On Aug 9th, 2021, at 20:51 UTC, we noticed some outdated connections between our system and the Apple Push Notification Service which resulted in a small backup of iOS push notifications on the PagerDuty Notification Service. Service was restored at 22:36 UTC by re-establishing these connections.

What Are We Doing About This

We are working on a few improvements that further enhance our resiliency to stale connections with 3rd party services, so we can retry and re-establish connections automatically. We are also adding additional metrics that will help us prevent the same in the future and respond faster in the event of such an issue. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com.

Posted Aug 16, 2021 - 21:44 UTC

Resolved
We have implemented a fix and are now recovered. iOS Push notifications are now being sent successfully.
Posted Aug 09, 2021 - 22:40 UTC
Update
We are still investigating delays for Push notifications for iOS devices in the US service region. The EU service region is unaffected.
Posted Aug 09, 2021 - 22:23 UTC
Investigating
We are experiencing delays with Push notifications for iOS devices and are currently investigating the cause. Other notification channels are unaffected.
Posted Aug 09, 2021 - 22:14 UTC
This incident affected: Notification Delivery (Notification Delivery (US), Notification Delivery (EU)).