17:39 UTC | 10:39 PT
We believe this incident is now resolved. Please let us know if you see any further issues.
17:06 UTC | 10:06 PT
Our mitigation is in place and affected pod 5 customers should now be seeing recovery.
16:50 UTC | 09:50 PT
We believe we've identified the cause of these performance issues, and are working towards resolving it.
16:31 UTC | 09:31 PT
We've received reports of all components being slow in Pod 5. We're still investigating.
16:19 UTC | 09:19 PT
We’re receiving reports of performance issues on Pod 5. Investigation is underway.
During this incident customers experienced slow and degraded performance with their Zendesk. A customer used our SDK to implement a search which resulted in a very expensive MySQL query. This action, when paired with high traffic from that customer, resulted in the app servers hanging as they waited for query responses. To resolve the issue, the DBA team issued a query killer and we temporarily blocked the customer while working with them to resolve the issue. To prevent this from happening in future, we are looking to limit the number of SDK Anonymous identities for a given user.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.