08:52 UTC | 00:52 PT
We have resolved the performance issue affecting pod 6 customers using our Support product.
08:43 UTC | 00:43 PT
We are experiencing performance issues for some customers on pod 6 affecting our Support product.
During a scheduled database configuration change, 2 slaves became unavailable due to 1 being offline as part of the change and 1 becoming overloaded with excessive queries. This left only master available and resulted in delayed response times and slowness in the Zendesk agent. After this occurred, the 1st slave completed the configuration change and the 2nd slave was restarted bringing both slaves back into rotation, performance then recovered and the issue resolved. In order to prevent this from happening again in the future, we will be updating our database change configuration process to increase redundancy for slaves and improve DBA team response times through process improvements when similar issues occur.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.