OAuth authorization failure for third-party apps

Summary

On Tuesday, January 20th, between 15:10 UTC and 23:48 UTC, PagerDuty experienced an issue in which a select number of third-party OAuth apps (apps created through Developer Mode) failed on token authorization flows. PagerDuty was notified of the issue through customer reports at 21:25 UTC. The issue was limited to authorizing new tokens and did not affect the use of existing previously authorized applications. PagerDuty-created integrations such as the PagerDuty Mobile Apps, Slack, and Zendesk apps were not affected. 

Over the course of this period, apps may have experienced a bad credentials error when trying to retrieve an OAuth token after authentication. Affected apps may have noticed issues at any point during the period depending on when their configuration was updated. After identifying the issue, the team reverted the change causing unexpected configuration updates, and restored old OAuth configurations for affected apps. This remediated the bad credentials issue.

What Happened

The incident occurred following an update to the service used for managing OAuth clients. Any OAuth configurations for third-party apps that were accessed following this update had their configurations unexpectedly changed, resulting in previous OAuth client secrets being invalidated. Any subsequent attempts by end-users to authorize affected OAuth clients would fail with a bad credentials error, affecting roughly 4% of registered third-party OAuth applications.

While we do monitor many forms of authorization failure, this particular case wasn’t picked up and our team instead escalated the issue in response to reports from third-party developers.

The issue was remediated after rolling back the affected service to a previous known good state and restoring affected configurations.

What We’re Doing

OAuth functionality is critical to third-party developers building on PagerDuty as a platform. To improve our reliability going forward we are in the process of:

• Fixing our monitors to better detect anomalies during all steps of OAuth client management, authorization and token exchange.
• Improving our testing guidelines with additional emphasis on OAuth client management to prevent unexpected changes in the future.

For any questions, comments, or concerns, please reach out to support@pagerduty.com.