Notification Delays
Incident Report for PagerDuty

Please refer to this postmortem to view postmortem information for this incident.

Posted 2 months ago. Oct 04, 2017 - 18:12 UTC

Resolved
We are fully recovered with no delays in the delivery of notifications or webhooks. At this time, we have taken all reasonable measures to ensure the stability of our system and are closing this incident. Once we have completed a thorough analysis and investigation, we will publish a full postmortem.
Posted 3 months ago. Sep 21, 2017 - 03:28 UTC
Update
At this time, we have fully recovered and are not seeing delays in the delivery of notifications.

As per our previous update, we have successfully made the change/upgrade to our distributed data infrastructure.

We are now monitoring the system to ensure we have long-term stability. As always, we will send another update on any material changes.
Posted 3 months ago. Sep 21, 2017 - 03:02 UTC
Update
We are now starting to make a change/upgrade to our distributed data infrastructure in order to improve both short-term and long-term stability. We believe this change will cause temporary delays in notifications. We have all hands on deck working and monitoring the situation, in order to minimize the window of delays in notifications.
Posted 3 months ago. Sep 21, 2017 - 01:17 UTC
Update
At this time, we have temporarily recovered and are no longer seeing delays in delivery of notifications. That being said, we are working on making a change to our distributed data infrastructure to permanently remediate the issue. We believe that this change will cause additional temporary delays in the delivery of notifications. We will provide an additional update just before undertaking this change.
Posted 3 months ago. Sep 20, 2017 - 23:37 UTC
Update
We continue to work to remediate the delay in notifications and webhooks.
Posted 3 months ago. Sep 20, 2017 - 20:25 UTC
Update
We are still working at remediation of this issue but continue to see a delay in delivery of notifications and webhooks averaging about 30 minutes.
Posted 3 months ago. Sep 20, 2017 - 18:49 UTC
Update
We are still experiencing delays in notification processing and delivery at an average of 30 minutes. At this time, we are exploring multiple paths to remedy and bring our systems back to an acceptable and expected state. The core of our issue resides in our distributed data infrastructure with respect to maintaining acceptable throughput and performance of data across regions. At this time we do not have an ETA for full recovery but rest assured we have around the clock coverage to mitigate this issue.
Posted 3 months ago. Sep 20, 2017 - 17:39 UTC
Update
Notifications and webhooks are still delayed but flowing. We are continuing to investigate and monitor the situation.
Posted 3 months ago. Sep 20, 2017 - 16:52 UTC
Update
We are continuing to investigate and monitor the situation. Notifications and webhooks are still delayed but flowing.
Posted 3 months ago. Sep 20, 2017 - 16:19 UTC
Update
Notifications and webhooks are delayed and we are actively working to mitigate the issue.
Posted 3 months ago. Sep 20, 2017 - 15:43 UTC
Update
Notifications are delayed and we are actively working to mitigate the issue. We are continuing to monitor and investigate the situation.
Posted 3 months ago. Sep 20, 2017 - 15:08 UTC
Update
We are continuing to monitor and investigate the situation. Notifications are going through but delivery times are delayed and still fluctuating.
Posted 3 months ago. Sep 20, 2017 - 14:35 UTC
Update
No significant changes at this time. Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Posted 3 months ago. Sep 20, 2017 - 13:57 UTC
Update
Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Posted 3 months ago. Sep 20, 2017 - 13:17 UTC
Update
Notification delivery times are fluctuating. We’re still monitoring and investigating the situation.
Posted 3 months ago. Sep 20, 2017 - 12:36 UTC
Monitoring
Notifications are going out in a timely manner. We are still monitoring the situation. All other systems are functional.
Posted 3 months ago. Sep 20, 2017 - 12:01 UTC
Investigating
We are currently experiencing delays in notifications. Our engineering team is aware and working to restore notification delivery to full performance.
Posted 3 months ago. Sep 20, 2017 - 11:30 UTC