Voice (Phone) notifications delivered without audio for certain users
Incident Report for PagerDuty
Postmortem

Summary

From May 24th, 21:35 UTC, to May 26th, 21:24 UTC, PagerDuty experienced a service degradation that resulted in a subset of users experiencing voice (phone) notifications being delivered without an audio prompt for calls in the US service region. This incident only affected voice notifications, other notification channels such as Push, SMS, and Email were unaffected during this time.

What Happened

On May 24th 21:35 UTC we rolled out a new network configuration and we noticed an increase in connection errors for voice calls, resulting from that change, on May 26th.  This resulted in the voice notifications being delivered without an audio prompt (example: “You have an incident…. Press ‘“..’” to respond…”). Due to the nature of that network issue, the errors were not immediately caught by our metrics. Certain users affected by this issue would have received voice notifications without an audio prompt. We restored normal service on May 26th 21:24 UTC by rolling back the configuration to a previously known state.

What Are We Doing About This

We have moved the implementation out of the network servers and into our application servers for increased visibility and immediate detection of issues. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com

Posted Jun 01, 2023 - 23:14 UTC

Resolved
We have resolved an incident where some PagerDuty customers in the US service region were experiencing intermittent issues with voice calls. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.
Posted May 26, 2023 - 21:36 UTC
Monitoring
We are investigating an incident where some PagerDuty customers in the US service region are experiencing intermittent issues with voice calls. Impacted customers may see voice calls that immediately hang up on when responded to. All other forms of notification are working normally and the underlying incidents are not impacted by the intermittent voice call failures. We will provide further updates within 20 minutes.
Posted May 26, 2023 - 21:26 UTC
Update
We are continuing to investigate this issue.
Posted May 26, 2023 - 21:10 UTC
Update
We are continuing to investigate this issue.
Posted May 26, 2023 - 21:09 UTC
Investigating
We are investigating a potential issue within PagerDuty where audio calls associated with an incident are not relaying information. If we confirm an impact, we will update within 15 minutes. If there is no impact this notification will be removed. No other elements are impacted.
Posted May 26, 2023 - 21:04 UTC
This incident affected: Notification Delivery (Voice (US), Voice (EU)).