Access Denied Errors for Advanced Permissions
Incident Report for PagerDuty
Postmortem

Summary

On January 17 2018, from 19:55 to 21:50 UTC, users on accounts with advanced permissions only had access to a subset of their teams. As a result, users were denied access to objects that did not belong to any team, or belonged to teams that they did not have explicit permission to access.

What Happened?

As part of our work on extending the advanced permissions feature, we were refactoring the database model. The data model change exposed a previously undetected bug in our production code that led to restricted visibility for users of that feature. Once discovered, we rolled back the data additions to fix the issue.

What Are We Doing About This?

We fixed the bug in the advanced permissions feature before re-migrating the data to the new model.

Also, we will review our migration testing procedures to identify the short-comings that led to us this bug before it impacted customers and make necessary adjustments.

We would like to express our regret for the service interruption. For any questions, comments, or concerns, please reach out to support@pagerduty.com

Posted 9 months ago. Feb 06, 2018 - 18:23 UTC

Resolved
We've resolved the issue causing "Access Denied" errors for customers using advanced permissions.
Posted 10 months ago. Jan 17, 2018 - 21:57 UTC
Monitoring
Our engineers are currently working to resolve the issue causing "Access Denied" errors for customers using advanced permissions.
Posted 10 months ago. Jan 17, 2018 - 21:46 UTC
Identified
We're currently investigating an issue causing "Access Denied" errors for customers using advanced permissions. Our team has identified the issue and is working to resolve it.
Posted 10 months ago. Jan 17, 2018 - 21:22 UTC