Delays in legacy reporting

Incident Report for PagerDuty

Postmortem

Summary

On Tuesday, May 16th between 08:06 UTC and ending on Wednesday, May 17th 04:30 UTC, our legacy reports had stale data in the US service region. At its peak, data was up to 7 hours and 40 minutes stale. There was no interruption to accessing the legacy reports UI or any incident response functionality. EU production was not affected by this incident.

What Happened

On Tuesday, May 16th at 08:06 UTC, a large query caused a table lock which prevented further transactions from occurring. Responders identified the root problem and killed the problem query, which allowed transactions to complete again. On Tuesday, May 16th at 12:41 UTC, responders replayed the failed and incomplete transactions. At 04:30 UTC on Wednesday, May 17th we successfully caught up and were back to near real-time data.

What We Are Doing About This

We are building a new Insights reporting suite.

We have also taken steps to be able to identify and remediate issues with stale data in our legacy reports in the event that these issues recur. We sincerely apologize for any inconvenience that this has caused. For any questions, comments, or concerns, please contact us at support@pagerduty.com.

Posted May 24, 2023 - 21:07 UTC

Resolved

We have resolved an incident where all PagerDuty customers in the US service region experienced delays in our legacy reports. The incident is now resolved, customers should expect some delays before a full recovery. Please reach out to support@pagerduty.com if you have any concerns.

Posted May 16, 2023 - 12:41 UTC

Update

We are continuing to investigate an incident where all PagerDuty customers in the US service region are experiencing issues with delays in our legacy reports. We will provide further updates within 20 minutes.

Posted May 16, 2023 - 12:23 UTC

Identified

We are investigating an incident where all PagerDuty customers in the US service region are experiencing issues with delays in our legacy reports. We will provide further updates within 20 minutes.

Posted May 16, 2023 - 12:04 UTC

Investigating

We are investigating a potential issue within PagerDuty. If we confirm an impact, we will update within 15 minutes. If there is no impact this notification will be removed.

Posted May 16, 2023 - 11:49 UTC