19:00 UTC | 12:00 PT
Performance has stabilized on PODs 5 and 6. Thanks for your patience.
18:35 UTC | 11:35 PT
We continue to see improvements in performance and are monitoring progress until full resolution.
18:15 UTC | 11:15 PT
We have rolled back the deploy and are seeing positive improvements in performance. We are continuing to monitor progress of full resolution
18:01 UTC | 11:01 PT
Current slowness on Pods 5 and 6 appear related to recent deploy, we are working on rollback.
17:47 UTC | 10:47 PT
We are experiencing slowness and other performance issues on pod 5 and 6. We are investigating.
During this incident customers received green screens when trying to connect to their Zendesk accounts. A weekly code deploy contained a bug which resulted in a high number of poorly performing database queries. This caused a backup of transactions across the platform, ultimately resulting in errors for customers. This issue was resolved by reverting the deploy.
As part of our remediation efforts, we are planning a review of our build and release management process to identify improvements we can make on the whole.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.