Incident Timeline and Alert Log issues
Incident Report for PagerDuty
Resolved
This incident has been resolved.
Posted Oct 14, 2021 - 02:51 UTC
Update
We have confirmed that the affected systems are now functioning normally, and are continuing to monitor to verify full recovery
Posted Oct 14, 2021 - 01:43 UTC
Update
We are observing full recovery of incident log entries functionality. We will continue to monitor the state of our systems and will share additional updates within an hour.
Posted Oct 14, 2021 - 01:19 UTC
Update
We are observing progress in processing of the delayed incident log entries, but will continue to actively monitor the outcome of the fix. We will be following up with hourly updates until we confirm the full recovery.
Posted Oct 14, 2021 - 00:16 UTC
Update
We have completed our fix for the log entry issues. We are now focused on processing delayed log entries. Customers may still experience delayed entries in their timeline. We will continue to post hourly updates until we have fully recovered.
Posted Oct 13, 2021 - 23:27 UTC
Update
We continue to monitor our fix underway. We anticipate the resolution for incident timeline and alert logs will complete within the next hour. However, it may still take several hours after that before we process all delayed log entries. There will be no data loss. We will continue to provide updates on progress hourly until this has been fully remediated.
Posted Oct 13, 2021 - 22:39 UTC
Update
We are continuing to monitor for any further issues.
Posted Oct 13, 2021 - 22:38 UTC
Monitoring
We are monitoring a fix that is underway. Log entries for new incidents are still impacted. We will continue to provide updates on progress hourly until this has been fully remediated.
Posted Oct 13, 2021 - 21:30 UTC
Update
A fix is currently being applied. We are still estimating that this may take several hours. We will continue to provide updates on progress hourly until this has been fully remediated.
Posted Oct 13, 2021 - 20:22 UTC
Identified
PagerDuty conducted a migration of our Incidents field to Bigint experienced an issue related to the incidents timeline. The team is working to restore this service. The planned migration/fix will take up to two hours. During that time any new incidents that are created may not show up in the incidents timeline and alert log. No data will be lost, but may be delayed in appearing in the page or in search results. We will provide an update in 30 minutes on the progress of the fix. Notifications and incident creation are not affected.
Posted Oct 13, 2021 - 19:53 UTC
This incident affected: REST API (REST API (US)), Mobile Application (Mobile Application (US)), and Web Application (Web Application (US)).