As of October 29th, 15:00 GMT / 8AM PST we are working to resolve the following service incident:
We are receiving reports of performance issues. We are currently investigating this.
15:30 GMT / 8:30 AM PST:
We are seeing some improvement to the current issue impacting performance. More information to follow.
15:48 GMT / 8:48 AM PST:
We are happy to report that the issue impacting performance is now fully resolved. Post-mortem to follow below.
This incident was a result of a DDoS (Distributed Denial of Service) attach against one of the Zendesk IPs, causing accessibility issues to one of our data centres. Actual DDoS attack flood time was between 14:37 - 14:40 UTC. Once the attack subsided, due to the time of day and volume of customers online, there was a reconnection volume which seemed to overwhelm the edge firewalls and caused routing protocol issues, which caused a second outage.
Due to the short duration of the attack, our DDoS detection tools did not trigger actionable alerts. These thresholds have been adjusted, accordingly.
DDoS attack subsided on its own, network convergence and routing protocols instability settled once flood of legitimate reconnection activity normalized.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.