Summary

From May 24th, 21:35 UTC, to May 26th, 21:24 UTC, PagerDuty experienced a service degradation that resulted in a subset of users experiencing voice (phone) notifications being delivered without an audio prompt for calls in the US service region. This incident only affected voice notifications, other notification channels such as Push, SMS, and Email were unaffected during this time.

What Happened

On May 24th 21:35 UTC we rolled out a new network configuration and we noticed an increase in connection errors for voice calls, resulting from that change, on May 26th.  This resulted in the voice notifications being delivered without an audio prompt (example: “You have an incident…. Press ‘“..’” to respond…”). Due to the nature of that network issue, the errors were not immediately caught by our metrics. Certain users affected by this issue would have received voice notifications without an audio prompt. We restored normal service on May 26th 21:24 UTC by rolling back the configuration to a previously known state.

What Are We Doing About This

We have moved the implementation out of the network servers and into our application servers for increased visibility and immediate detection of issues. We apologize for the inconvenience that this has caused. For any questions, comments or concerns, please reach out to support@pagerduty.com