17:31 UTC | 10:31 PT
The issue affecting access to PODS 3, 7, and 10 has now been resolved.
17:18 UTC | 10:18 PT
We are seeing account access being restored to the subset of customers that were impacted on PODS 3 and 7.
17:09 UTC | 10:09 PT
We are investigating reports of account access issues for some customers from specific geographical locations. More information to come.
During this incident, some customers in the West Coast USA and APAC regions were unable to access their Zendesk support accounts and Guide on PODs 3, 7, 10, and 11. Our operations team detected packet loss from within our network, originating from our DDoS mitigation service. Upon disabling the service, routing traffic over our internet circuits mitigated the issue and customers were then able to access their accounts. The vendor completed their investigation of the incident and confirmed that there was a routing issue within their network between the times of 16:42 - 17:42 UTC. The issue was caused by a change implemented by one of the vendor’s internal teams. The change underwent the vendor’s change control process and was planned as being non-impacting. A human error during the change implementation resulted in the change not being implemented as planned. This resulted in a portion of backbone traffic being misrouted within the vendor’s network. We will be working with our vendor to improve their change control process to prevent this from happening again in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.