Summary

On November 30, 2022 between 16:53 and 17:43 UTC, PagerDuty experienced an incident that resulted in all customers being presented with a banner message that read: “Oh no! Your trial has expired. Check out our plans to keep using PagerDuty,” even when the customer’s account was not an expired trial.

This did not result in customers losing any functionality within the app, but the resulting messaging was confusing and could lead customers to think that their accounts were deactivated or had otherwise lost functionality. Event ingestion, Web and REST APIs were not impacted by this incident.

What Happened

The Nurture Bar, which is a banner-based frontend component that we use to display onboarding messages to trial, expired, and freemium customers, was updated with a series of interface improvements. Among these improvements was a design change to the messaging we present to customers whose free PagerDuty trials had expired. When we made this specific change to the expired messaging, we inadvertently removed some display logic around whom this message should be displayed to.

As a result, the “Your trial has expired” message was shown to any customer that was not on a free trial or a freemium account. This includes expired customers as well as all active subscribers to our product.

Our engineers were able to remediate this issue by performing an emergency rollback of the Nurture Bar component. This allowed us to quickly remove the incorrect messaging and prepare a more permanent solution to ensure that customers are not incorrectly shown this kind of messaging in the future. The permanent fix was then deployed on the same day as the incident itself.

What We Are Doing About This

Following this incident, our teams conducted a thorough post-mortem investigation which identified a series of events that led to a failure of this nature. Our engineering teams have worked diligently to address these findings and ensure we've adequately guarded against this manner of failure from now on. The corrective actions included the following:

• Code changes that explicitly prevent the display of expired trial messaging to customers that are not expired trials, or any other inappropriate messaging to actively subscribed customers.
• Additions to our comprehensive automated and manual testing workflows that confirm that customers are not shown incorrect messaging from the Nurture Bar.

We apologize for the impact this incident had on you and your teams. As always, we stand by our commitment to providing the most reliable and resilient platform in the industry. If you have any questions, please reach out to support@pagerduty.com.