CSS Not Loading in Web App
Incident Report for PagerDuty
Postmortem

Summary

On February 22, 2019, 14:31 UTC, a subset of pages in the Web UI were unable to render correctly due to missing CSS assets. An incident initiated on February 24, 2019, 2:02 UTC, with the issue being resolved in 3 hours at 5:04 UTC. The listing pages for Schedules, Service, Users and Teams were primarily affected as well as the login page of our Web UI.

What Happened?

A deploy of the web application included changes to the front end assets, which triggered a rebuild of the assets. However, this asset rebuild produced incorrect results on some hosts, causing requests to these hosts for the assets to fail. Any pages that were affected by this would load, but would not render correctly. Because we did not have monitoring in place to alert us of this type of problem, we were not aware of it until receiving customer reports, by which point it had been ongoing for approximately 2.5 days. In order to restore full functionality to the web application as quickly as possible, a temporary fix was implemented to copy the correct asset files to all affected hosts. Subsequently, triggering a rebuilding of all assets on all hosts fully resolved the issue.

What Are We Doing About This?

We have implemented monitoring and alerting to immediately notify us if an issue of this type is to occur, and have documented the remediation steps which could then be put into action immediately.

Our investigation into the reason why the asset rebuild process produced different results on different hosts is ongoing, and once completed the appropriate steps will be taken to ensure that it does not recur.

Posted 7 months ago. Mar 05, 2019 - 17:10 UTC

Resolved
We have fully recovered. Our Engineers have successfully deployed the fix, and have completed mitigation actions necessary to ensure the incident does not recur. CSS functionality is now fully restored in the Web Application for all users.
Posted 7 months ago. Feb 25, 2019 - 05:15 UTC
Identified
We've identified the root cause to the issue, and our Engineers are currently implementing a fix to restore CSS loading in the Web application.
Posted 7 months ago. Feb 25, 2019 - 02:58 UTC
Investigating
Some users located in the APJ region are experiencing CSS issues being loaded in our website application. The mobile app and notification functionality is not impacted. Our Engineers are currently investigating the issue.
Posted 7 months ago. Feb 25, 2019 - 02:27 UTC