Kenyan SMS Response Temporarily Affected
Incident Report for PagerDuty
Postmortem

Summary

On June 30th, 2022 starting at 3:00AM UTC and ending July 2nd, 2022 at 1:20AM UTC, what was believed to be a version of telco fraudulent activity occurred in the Kenya region. Upon discovery, we mitigated the event by temporarily disabling inbound SMS messages to the region. Recipients in the Kenya region would have been unable to temporarily respond to SMS notifications. Once this action was taken, the event subsided.

Other notifications in the Kenya region, such as stakeholder notifications, on-call handoffs, and responder requests, as well as notifications and other communications in all other regions, were not impacted.

What Happened

On June 30th, 2022 at 3:00AM UTC, a telco carrier in the Kenya region began to engage in what is recognized as SMS telco fraud. Upon discovery, and after confirming fraudulent activity, we disabled incoming replies in the Kenya region to temporarily mitigate the effects on PagerDuty infrastructure.

Once this mitigation was put into place, recipients in Kenya would have not been able to respond to SMS notifications but were able to receive SMS notifications with no degradation of service. Recipients could respond to notifications through the PagerDuty app or website.

Kenya SMS replies were blocked until our engineering teams were able to put permanent mitigations in place. Once permanent mitigations were in place, the temporary block was lifted and full SMS service in Kenya was restored without further incident.

What Are We Doing About This

Following this event, our teams conducted a thorough investigation and put in place additional protections and monitoring:

  • Worked with our providers to share the data we gathered during this fraud incident
  • Enhanced internal logic to detect and handle the traffic associated with this unique type of carrier fraud
  • Added additional monitoring of traffic to detect this pattern in the future

We sincerely apologize for the impact that you or your teams have experienced due to this event. We understand how vital our platform is for our customers and thank you for your patience. If you have any questions, please reach out to support@pagerduty.com.

Posted Jul 29, 2022 - 21:21 UTC

Resolved
This incident has been resolved.
Posted Jul 18, 2022 - 22:51 UTC
Monitoring
We are experiencing an incident where PagerDuty customers with recipients in Kenya (+254 Country Code) are unable to respond to SMS messages. Impacted customers will still receive all incident notifications by SMS. Customers impacted are encouraged to acknowledge or resolve notifications by other channels such as the PagerDuty mobile application, website, or third party integrations. Phone and email notifications and the ability to respond via those channels are not impacted. Additional communications have been sent to account owners that send SMS notices to these country codes that includes further details and workaround instructions.
Posted Jul 02, 2022 - 00:06 UTC
This incident affected: Notification Delivery (Notification Delivery (US)).