Notification Delays
Incident Report for PagerDuty

Please refer to this postmortem to view postmortem information for this incident.

Posted 15 days ago. Oct 04, 2017 - 18:12 UTC

Resolved
We are fully recovered with no delays in the delivery of notifications or webhooks. At this time, we have taken all reasonable measures to ensure the stability of our system and are closing this incident. Once we have completed a thorough analysis and investigation, we will publish a full postmortem.
Posted 28 days ago. Sep 21, 2017 - 03:28 UTC
Update
At this time, we have fully recovered and are not seeing delays in the delivery of notifications.

As per our previous update, we have successfully made the change/upgrade to our distributed data infrastructure.

We are now monitoring the system to ensure we have long-term stability. As always, we will send another update on any material changes.
Posted 28 days ago. Sep 21, 2017 - 03:02 UTC
Update
We are now starting to make a change/upgrade to our distributed data infrastructure in order to improve both short-term and long-term stability. We believe this change will cause temporary delays in notifications. We have all hands on deck working and monitoring the situation, in order to minimize the window of delays in notifications.
Posted 28 days ago. Sep 21, 2017 - 01:17 UTC
Update
At this time, we have temporarily recovered and are no longer seeing delays in delivery of notifications. That being said, we are working on making a change to our distributed data infrastructure to permanently remediate the issue. We believe that this change will cause additional temporary delays in the delivery of notifications. We will provide an additional update just before undertaking this change.
Posted 28 days ago. Sep 20, 2017 - 23:37 UTC
Update
We continue to work to remediate the delay in notifications and webhooks.
Posted 29 days ago. Sep 20, 2017 - 20:25 UTC
Update
We are still working at remediation of this issue but continue to see a delay in delivery of notifications and webhooks averaging about 30 minutes.
Posted 29 days ago. Sep 20, 2017 - 18:49 UTC
Update
We are still experiencing delays in notification processing and delivery at an average of 30 minutes. At this time, we are exploring multiple paths to remedy and bring our systems back to an acceptable and expected state. The core of our issue resides in our distributed data infrastructure with respect to maintaining acceptable throughput and performance of data across regions. At this time we do not have an ETA for full recovery but rest assured we have around the clock coverage to mitigate this issue.
Posted 29 days ago. Sep 20, 2017 - 17:39 UTC
Update
Notifications and webhooks are still delayed but flowing. We are continuing to investigate and monitor the situation.
Posted 29 days ago. Sep 20, 2017 - 16:52 UTC
Update
We are continuing to investigate and monitor the situation. Notifications and webhooks are still delayed but flowing.
Posted 29 days ago. Sep 20, 2017 - 16:19 UTC
Update
Notifications and webhooks are delayed and we are actively working to mitigate the issue.
Posted 29 days ago. Sep 20, 2017 - 15:43 UTC
Update
Notifications are delayed and we are actively working to mitigate the issue. We are continuing to monitor and investigate the situation.
Posted 29 days ago. Sep 20, 2017 - 15:08 UTC
Update
We are continuing to monitor and investigate the situation. Notifications are going through but delivery times are delayed and still fluctuating.
Posted 29 days ago. Sep 20, 2017 - 14:35 UTC
Update
No significant changes at this time. Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Posted 29 days ago. Sep 20, 2017 - 13:57 UTC
Update
Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Posted 29 days ago. Sep 20, 2017 - 13:17 UTC
Update
Notification delivery times are fluctuating. We’re still monitoring and investigating the situation.
Posted 29 days ago. Sep 20, 2017 - 12:36 UTC
Monitoring
Notifications are going out in a timely manner. We are still monitoring the situation. All other systems are functional.
Posted 29 days ago. Sep 20, 2017 - 12:01 UTC
Investigating
We are currently experiencing delays in notifications. Our engineering team is aware and working to restore notification delivery to full performance.
Posted 29 days ago. Sep 20, 2017 - 11:30 UTC