10:52 UTC | 02:52 PT
We are happy to report that the performance issues on Pod 3 have been resolved.
09:50 UTC | 01:50 PT
We are working with our network partners to resolve performance issues affecting Pod 3.
09:16 UTC | 01:16 PT
We are continuing to monitor the performance issues on Pod 3. More information to follow
08:56 UTC | 00:56 PT
We are seeing improvements with the performance issues on Pod 3. We still are keeping the situation monitored. Updates will follow.
08:40 UTC | 00:40 PT
We are currently investigating reports of performance issues affecting some customers in Pod 3.
During this incident, some customers experienced slow and intermittent connectivity issues to their Zendesk Support account. A bug was found in Pod 3's routing policies between a security service and the various ISP providers we use. This security service traffic was being routed through a highly trafficked ISP, which was almost at its peak, instead of a secondary ISP, which is less utilized. This caused the initial circuit to become overloaded, causing network performance issues. During the investigation we also confirmed that restrictions were made on the firewall in China, further impacting performance. For the performance degradation we saw in Pod 6, it was most likely related to the Performance Issues Pod 6 incident on the previous day. Routing changes with our security service were made in both pods and connections were reset, which provided relief and performance improvements in both pods, resolving the incident.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.