10:55 UTC | 02:55 PT
We are happy to report that the service disruption to your Zendesk account is now resolved.
10:29 UTC | 02:29 PT
We begin to see improvement on all pods. Our operations team continue to mitigate impact. We apologize for the inconvenience.
10:13 UTC | 02:13 PT
Our operations team are working hard to mitigate impact to your Zendesk account. We apologize for the inconvenience. Next update in 30 mins.
09:45 UTC | 01:45 PT
We apologize for the disruption to your Zendesk account. Our operations team are working to mitigate the issue. Next update in 30 mins.
09:26 UTC | 01:26 PT
We are investigating access issues / green screens affecting some customers. We will provide further information shortly.
During this incident customers initially experienced performance issues and green screens when connecting to their Zendesk support account. Before discovering a wider service incident, we received a few reports of customers experiencing missing translations. These reports were addressed as they came in but did not point to a larger issue. We continued receiving missing translation reports and normal remediation actions did not resolve the problems. This led to a more active investigation into what was happening. A hotfix was made that worked temporarily, but more action was taken to provide a better fix. By this point, the load on our translation service skyrocketed. This caused the translation service to have incredibly high response times and return error messages. These errors resulted in customers seeing green screens when trying to connect to their Zendesk support account. After additional research, some duplicate translations were removed and the translations service was restarted. Error rates immediately started to go down and customers started reporting that the Zendesk service was working as expected again. To prevent this from happening in the future we will be working on validation for the translation service and notifications when validation fails.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.