On February 22, 2019, 14:31 UTC, a subset of pages in the Web UI were unable to render correctly due to missing CSS assets. An incident initiated on February 24, 2019, 2:02 UTC, with the issue being resolved in 3 hours at 5:04 UTC. The listing pages for Schedules, Service, Users and Teams were primarily affected as well as the login page of our Web UI.
A deploy of the web application included changes to the front end assets, which triggered a rebuild of the assets. However, this asset rebuild produced incorrect results on some hosts, causing requests to these hosts for the assets to fail. Any pages that were affected by this would load, but would not render correctly. Because we did not have monitoring in place to alert us of this type of problem, we were not aware of it until receiving customer reports, by which point it had been ongoing for approximately 2.5 days. In order to restore full functionality to the web application as quickly as possible, a temporary fix was implemented to copy the correct asset files to all affected hosts. Subsequently, triggering a rebuilding of all assets on all hosts fully resolved the issue.
We have implemented monitoring and alerting to immediately notify us if an issue of this type is to occur, and have documented the remediation steps which could then be put into action immediately.
Our investigation into the reason why the asset rebuild process produced different results on different hosts is ongoing, and once completed the appropriate steps will be taken to ensure that it does not recur.