16:44 UTC | 09:44 PT
We're happy to report that the performance degradation some customers experienced in Pods 13, 14, 19 and 20 is resolved. Please let us know if you're still experiencing issues.
16:20 UTC | 09:20 PT
We are beginning to see performance improvement on Pods 13, 14, 19 and 20. Please reach out to us if you are seeing any continuing issues.
16:00 UTC | 09:00 PT
We are continuing to monitor performance in Support and Guide on Pods 13, 14, 19 and 20. Please reach out to us if you are seeing any issues.
15:37 UTC | 08:37 PT
We are continuing to investigate degraded performance in Support and Guide products on Pods 13, 14, 19 and 20. We apologize for the inconvenience this may have caused.
15:10 UTC | 08:10 PT
We’re investigating reports that customers are experiencing slowness and 502 errors when loading pages in Zendesk.
During this incident, customers experienced slow or failing requests when trying to access Support, Guide, Talk, and Explore across pods 13, 14, 17, 19, and 20.
This incident appeared because of a code change that was intended to aid in Side Conversation connections by opening a connection to the API endpoint. That code change was run on all Support instances to account for Side Conversations’ configuration, and this resulted in an increased load that eventually put pressure on the proxy that helps with those kinds of requests. The proxy eventually began rejecting new connections due to the heightened load, which is why customers also saw effects on these other product areas.
Once the heightened load resulted in throttling for Side Conversation containers, our teams responded by deploying a blocker to that endpoint at approximately 9:00 AM PT. Services were restored at 9:07 AM. The code change was reverted, and the All Clear was called for this incident shortly after.
To remediate this, our teams are improving our monitoring metrics and our alerts for Side Conversations-focused changes, and we are performing tests on our autoscaling with regard to these particular types of connections.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.