Issue with Google Auth SSO
Incident Report for PagerDuty
Postmortem

Summary

On June 8th, between 6:00 PM UTC and 9:00 PM UTC, PagerDuty experienced an incident with Google Single Sign-On (SSO) in both the US and EU regions. For much of this window, customers who use Google SSO to sign in to PagerDuty were unable to do so. The incident was discovered to be a direct result of a code upgrade we had just performed. A rollback of the change was immediately kicked off, and by the end of the rollout, we had completely recovered from the incident.

What Happened

As part of our ongoing efforts to make sure our code base is up to date, we have been performing upgrades to parts of our system that are using older technology. On the day of the incident, we shipped an upgrade that, unbeknownst to us, affected how Google SSO worked. We started receiving customer reports about being unable to sign in to PagerDuty using Google SSO and established that our recent change had caused the issue. Once the code in question was reverted, we were able to verify that signing in to PagerDuty using Google SSO no longer had any issues.

What We Are Doing About This

Following the incident, our teams conducted a thorough investigation into the factors leading up to the incident and have identified several action items for us to undertake to ensure incidents like these don't happen in the future. The action items include the following:

  • Enhancing our test suite and manual testing procedures for the services in question to ensure we detect any similar issue before deploying code
  • Identifying the underlying cause of the issue so that we can continue to maintain an up-to-date codebase

We apologize for our failure and the impact on you and your teams. As always, we stand by our commitment to providing the industry's most reliable and resilient platform. If you have any questions, please reach out to support@pagerduty.com

Posted Jun 14, 2022 - 22:23 UTC

Resolved
We have resolved an incident where PagerDuty customers using Google Auth SSO to connect to PagerDuty in both the US and EU service regions were unable to login. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.
Posted Jun 08, 2022 - 20:56 UTC
Identified
We’ve identified the issue that is preventing customers from using Google Auth SSO to login to PagerDuty. We are working to remediate the issue and will continue to update.
Posted Jun 08, 2022 - 20:51 UTC
Update
We are continuing to investigate an issue where PagerDuty customers are experiencing issues with using Google Auth SSO to login. We will provide further updates within 20 minutes.
Posted Jun 08, 2022 - 20:44 UTC
Investigating
We are investigating issues with using Google Auth SSO to connect to PagerDuty. We will update with further impact and severity within 15 minutes.
Posted Jun 08, 2022 - 20:26 UTC
This incident affected: Mobile Application (Mobile Application (US), Mobile Application (EU)) and Web Application (Web Application (US), Web Application (EU)).