Microsoft Teams Integration Issue

Incident Report for PagerDuty

Postmortem

Summary

During a period beginning at 21:44 UTC on Monday, January 9th, and ending at 13:16 UTC on Tuesday, January 10th, PagerDuty experienced an incident that prevented customers in the EU Service Region from authorizing new Microsoft Teams and Slack integrations. Integrations that were authorized prior to this incident were not affected, as well as new authorizations from other integrations such as Jira, Live Call Routing, Zendesk, and ServiceNow.

What Happened

After receiving customer reports of not being able to authorise the Microsoft Teams and Slack integration, we initiated our major incident response at 12:01 UTC Tuesday, January 10th. During our investigation, our engineers found that a code change to build redirect URLs intended for our non-production environments inadvertently affected new authorizations for Microsoft Teams and Slack integrations in our EU production environment. As soon as this was identified, around 12:50 UTC Tuesday, January 10th, the code change was reverted and completed at 13:16 UTC Tuesday, January 10th. Both the Microsoft Teams and Slack authentications have been operating normally since the fix, and the incident was closed at 13:22 UTC Tuesday, January 10th.

What Are We Doing About This

Our engineers are adding extra measures in testing and monitoring our login and integration workflows. We apologise for any inconvenience this caused. ⁠For any questions, comments, or concerns, please contact us at ⁠support@pagerduty.com⁠.

Posted Jan 18, 2023 - 19:40 UTC

Resolved

We have identified and resolved an incident with our Microsoft Teams and Slack Integration in the EU service region. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.

Posted Jan 10, 2023 - 13:21 UTC

Update

During our investigation, we have also established that our Slack integration along with our Microsoft Teams Integration within PagerDuty in the EU region are having authentication issues. We will update with further impact and severity within 15 minutes.

Posted Jan 10, 2023 - 13:00 UTC

Update

We are still continuing to investigate the issue with our Microsoft Teams Integration within PagerDuty in the EU region. We will update with further impact and severity within 15 minutes.

Posted Jan 10, 2023 - 12:46 UTC

Update

We are continuing to investigate the issue with our Microsoft Teams Integration within PagerDuty in the EU region. On confirmation, we will update with further impact and severity within 15 minutes.

Posted Jan 10, 2023 - 12:31 UTC

Investigating

We are investigating potential issues with our Microsoft Teams Integration within PagerDuty in the EU region. On confirmation, we will update with further impact and severity within 15 minutes.

Posted Jan 10, 2023 - 12:13 UTC

This incident affected: Integrations (Microsoft Teams (EU), Slack (EU)).