08:38 UTC | 00:38 PT
We’re happy to report that the access issues to the affected pods have now been resolved.
08:17 UTC | 00:17 PT
We're seeing improvements on affected pods and are continuing to work to stabilise services. Our teams are working on mitigating the issues.
07:49 UTC | 23:49 PT
We are investigating some packet loss issues affecting talk customers on Pod 2/4/8. More to come
During this incident customers experienced general slowness and sporadic timeouts when trying to connect to their Zendesk.
The timeouts were caused by a series of events that began with infrastructure maintenance carried out by one of our vendors. That maintenance caused current sessions to disconnect and then a major spike of firewall traffic as disconnected clients tried to reconnect. The spike of reconnection requests surpassed the maximum allowed and were rejected. To allow time for the firewall to recover, we executed a failover to additional resources. This action provided the necessary relief while the firewall stabilized and the packet loss subsided. The firewall infrastructure in Sacramento is scheduled for a significant capacity upgrade in January that will minimize the risk of this type of event in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.