Issue with UI displaying a "Trial has expired" banner
Incident Report for PagerDuty
Postmortem

Summary

On November 30, 2022 between 16:53 and 17:43 UTC, PagerDuty experienced an incident that resulted in all customers being presented with a banner message that read: “Oh no! Your trial has expired. Check out our plans to keep using PagerDuty,” even when the customer’s account was not an expired trial.

This did not result in customers losing any functionality within the app, but the resulting messaging was confusing and could lead customers to think that their accounts were deactivated or had otherwise lost functionality. Event ingestion, Web and REST APIs were not impacted by this incident.

What Happened

The Nurture Bar, which is a banner-based frontend component that we use to display onboarding messages to trial, expired, and freemium customers, was updated with a series of interface improvements. Among these improvements was a design change to the messaging we present to customers whose free PagerDuty trials had expired. When we made this specific change to the expired messaging, we inadvertently removed some display logic around whom this message should be displayed to.

As a result, the “Your trial has expired” message was shown to any customer that was not on a free trial or a freemium account. This includes expired customers as well as all active subscribers to our product.

Our engineers were able to remediate this issue by performing an emergency rollback of the Nurture Bar component. This allowed us to quickly remove the incorrect messaging and prepare a more permanent solution to ensure that customers are not incorrectly shown this kind of messaging in the future. The permanent fix was then deployed on the same day as the incident itself.

What We Are Doing About This

Following this incident, our teams conducted a thorough post-mortem investigation which identified a series of events that led to a failure of this nature. Our engineering teams have worked diligently to address these findings and ensure we've adequately guarded against this manner of failure from now on. The corrective actions included the following:

  • Code changes that explicitly prevent the display of expired trial messaging to customers that are not expired trials, or any other inappropriate messaging to actively subscribed customers.
  • Additions to our comprehensive automated and manual testing workflows that confirm that customers are not shown incorrect messaging from the Nurture Bar.

We apologize for the impact this incident had on you and your teams. As always, we stand by our commitment to providing the most reliable and resilient platform in the industry. If you have any questions, please reach out to support@pagerduty.com.

Posted Dec 09, 2022 - 18:44 UTC

Resolved
We have resolved an incident where all PagerDuty customers in both the US and EU service regions experienced issues with the UI displaying a "Trial has expired" banner. The incident is now resolved, and there is no ongoing impact to customers. Please reach out to support@pagerduty.com if you have any concerns.
Posted Nov 30, 2022 - 17:48 UTC
Monitoring
We are monitoring improvement in an incident with the UI displaying a "Trial has expired" banner. We have deployed a fix, and we expect systems to continue to improve.
Posted Nov 30, 2022 - 17:46 UTC
Investigating
We are investigating potential issues with the PagerDuty UI incorrectly displaying a "Trial has expired" banner. On confirmation, we will update with further impact and severity within 15 minutes.
Posted Nov 30, 2022 - 17:40 UTC
This incident affected: Web Application (Web Application (US), Web Application (EU)).