Resolved -
We are fully recovered with no delays in the delivery of notifications or webhooks. At this time, we have taken all reasonable measures to ensure the stability of our system and are closing this incident. Once we have completed a thorough analysis and investigation, we will publish a full postmortem.
Sep 21, 03:28 UTC
Update -
At this time, we have fully recovered and are not seeing delays in the delivery of notifications.
As per our previous update, we have successfully made the change/upgrade to our distributed data infrastructure.
We are now monitoring the system to ensure we have long-term stability. As always, we will send another update on any material changes.
Sep 21, 03:02 UTC
Update -
We are now starting to make a change/upgrade to our distributed data infrastructure in order to improve both short-term and long-term stability. We believe this change will cause temporary delays in notifications. We have all hands on deck working and monitoring the situation, in order to minimize the window of delays in notifications.
Sep 21, 01:17 UTC
Update -
At this time, we have temporarily recovered and are no longer seeing delays in delivery of notifications. That being said, we are working on making a change to our distributed data infrastructure to permanently remediate the issue. We believe that this change will cause additional temporary delays in the delivery of notifications. We will provide an additional update just before undertaking this change.
Sep 20, 23:37 UTC
Update -
We continue to work to remediate the delay in notifications and webhooks.
Sep 20, 20:25 UTC
Update -
We are still working at remediation of this issue but continue to see a delay in delivery of notifications and webhooks averaging about 30 minutes.
Sep 20, 18:49 UTC
Update -
We are still experiencing delays in notification processing and delivery at an average of 30 minutes. At this time, we are exploring multiple paths to remedy and bring our systems back to an acceptable and expected state. The core of our issue resides in our distributed data infrastructure with respect to maintaining acceptable throughput and performance of data across regions. At this time we do not have an ETA for full recovery but rest assured we have around the clock coverage to mitigate this issue.
Sep 20, 17:39 UTC
Update -
Notifications and webhooks are still delayed but flowing. We are continuing to investigate and monitor the situation.
Sep 20, 16:52 UTC
Update -
We are continuing to investigate and monitor the situation. Notifications and webhooks are still delayed but flowing.
Sep 20, 16:19 UTC
Update -
Notifications and webhooks are delayed and we are actively working to mitigate the issue.
Sep 20, 15:43 UTC
Update -
Notifications are delayed and we are actively working to mitigate the issue. We are continuing to monitor and investigate the situation.
Sep 20, 15:08 UTC
Update -
We are continuing to monitor and investigate the situation. Notifications are going through but delivery times are delayed and still fluctuating.
Sep 20, 14:35 UTC
Update -
No significant changes at this time. Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Sep 20, 13:57 UTC
Update -
Notifications are going through, although delivery times continue to fluctuate. We will continue to monitor and investigate the situation.
Sep 20, 13:17 UTC
Update -
Notification delivery times are fluctuating. We’re still monitoring and investigating the situation.
Sep 20, 12:36 UTC
Monitoring -
Notifications are going out in a timely manner. We are still monitoring the situation. All other systems are functional.
Sep 20, 12:01 UTC
Investigating -
We are currently experiencing delays in notifications. Our engineering team is aware and working to restore notification delivery to full performance.
Sep 20, 11:30 UTC