10:54 UTC | 03:54 PT
We are happy to announce that the performance issues affecting POD 13 are now resolved.
10:17 UTC | 03:17 PT
We're seeing improvements in pod13.
10:09 UTC | 03:09 PT
We're working on remediating an issue affecting pod13 accounts. More to follow shortly.
CPU utilization on the db slaves on a pod 13 cluster spiked due to an expensive organizations query rendering both slaves unavailable to serve apps until those queries were killed to restore normalcy. The issue occurred again 24 hours later. In order to prevent this from happening again in the future, we will add conditional rate limiting, improve internal db monitoring and alerting for the given pod, and improve organization query performance.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.