On Tuesday, May 16th between 08:06 UTC and ending on Wednesday, May 17th 04:30 UTC, our legacy reports had stale data in the US service region. At its peak, data was up to 7 hours and 40 minutes stale. There was no interruption to accessing the legacy reports UI or any incident response functionality. EU production was not affected by this incident.
On Tuesday, May 16th at 08:06 UTC, a large query caused a table lock which prevented further transactions from occurring. Responders identified the root problem and killed the problem query, which allowed transactions to complete again. On Tuesday, May 16th at 12:41 UTC, responders replayed the failed and incomplete transactions. At 04:30 UTC on Wednesday, May 17th we successfully caught up and were back to near real-time data.
We are building a new Insights reporting suite.
We have also taken steps to be able to identify and remediate issues with stale data in our legacy reports in the event that these issues recur. We sincerely apologize for any inconvenience that this has caused. For any questions, comments, or concerns, please contact us at support@pagerduty.com.