On January 16, 2019, at 22:31 UTC we received reports of 404 responses from users trying to access configuration pages. We found that the issue was isolated to Global Admins on accounts with advanced permissions. All other user roles and accounts without this feature were not affected.
Following a deployment, a logic error was introduced into a permissions check method. The execution of the method in question was in a code path that required a Global Admin user to request certain configuration pages, while also using advanced permissions. The logic error introduced on this code path resulted in 400s being returned. The logic error, coupled with how we were handling these types of error messages, ultimately caused a delay in identifying a root cause.
Once the pattern of a user role and feature was found and confirmed, we were able to identify the source of the issue. The deployment was rolled back to a healthy state.
First, we will implement a set of tests to catch these types of logic errors going forward. Secondly, we will implement better logging to identify these types of issues faster. Lastly, we will work to improve the messaging on our 4XX error pages.
We would like to apologize for this service interruption. For any questions, comments, or concerns, please reach out to firstname.lastname@example.org.