17:27 UTC | 10:27 PT
The slowness issues in Pod 14 are now resolved.
17:11 UTC | 10:11 PT
We are seeing improved performance on Pod 14. We’re continuing to monitor and will update shortly.
16:58 UTC | 09:58 PT
We're continuing to investigate the root cause of the performance issues impacting pod 14. More updates to follow.
16:43 UTC | 09:43 PT
We are investigating issues with Pod 14 slowness and email delays
Ticket creation on POD 14 was delayed due to contention over obtaining event IDs from a database. ID generation was taking over 1 second, causing processes like email ticket creation to slow when trying to commit Ticket events (Comments, etc). ID generation was delayed due to binlogs being enabled and consuming the available capacity. The issue was exacerbated with top of the hour automation runs, and the overall volume on POD 14. In order to prevent this from happening again in the future, we have disabled binlogs on POD 14 as well as other PODs.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.