16:57 UTC | 08:57 PT
Service issues impacting accounts on Pod 14 have been resolved.
16:42 UTC | 08:42 PT
Performance on Pod 14 appears to have stabilized but we are continuing to monitor to confirm the issue is resolved.
16:11 UTC | 08:11 PT
We're continuing to work towards a fix for service issues impacting Pod 14. Next update in 30 minutes.
15:50 UTC | 07:50 PT
We most likely found the cause of the service outage and we are working to fix it as soon as possible. We will follow up shortly
15:30 UTC | 07:30 PT
We keep investigating performance issues affecting Pod 14, we appreciate your patience. We will update you as we have more information
15:12 UTC | 07:12 PT
We are investigating performance issues affecting Pod 14. More information to follow
During this incident customers experienced performance degradation and service unavailability. This incident was caused by a DDoS attack on a Zendesk IP address. Since the attack happened to a customer on one of our AWS pods, our primary DDoS mitigation procedures were unavailable. This is a known deficiency of these environments that is undergoing work. Once the attack subsided, customers' service and availability returned to normal. To reduce impact of these types of attacks on AWS pods, we will implement better monitoring and alerts.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.