Between 19:05 UTC and 20:13 UTC on April 23rd, PagerDuty experienced an incident that caused event ingestion to operate in a degraded state.
During this period some invalid event submissions were rejected with an HTTP 500 response when they should have been rejected with an HTTP 400 response.
A change to the Events API service was deployed that caused invalid V1 generic events to be rejected at an earlier stage than they should have been. As a result, some event rejection responses indicated an HTTP 500 response code rather than an HTTP 400 response code.
To restore the expected behavior, the change to the Events API service was rolled back.
In addition to rolling back the offending change, we are working on addressing the following factors that contributed to this issue:
We will do everything we can to learn from this event and make the improvements necessary to uphold the high standard of availability we have to serve the needs of our customers.
Finally, we’d like to apologize for any impact this had on our customers. If you have any further questions, please contact firstname.lastname@example.org.