08:27 UTC | 01:27 PT
We are happy to advise that the issue affecting Search for customers on POD17 is now resolved.
08:04 UTC | 01:04 PT
We continue to investigate and take mitigating steps on the Search issues impacting Search for Support and Help Center accounts on Pod 17.
07:50 UTC | 00:50 PT
Our Operations team are investigating an issue affecting Search for Support and Help Center accounts on POD17. More details to follow.
Around 07:26 UTC, our Operations team investigated exceptions in our Search service that were caused by a large amount of ticket indexing occurring on a data node cluster. The cause of this issue was too many concurrent account moves to POD 17 that impacted the cluster’s capacity, which resulted in Search timing out for users on this POD, impacting queries happening on Support and Guide as well as some general API queries. To fix this issue, capacity was increased on this cluster, and customers saw recovery within 30 minutes. To prevent this issue from occurring in the future, we are improving circuit breakers to automatically slow down indexing when search latencies are increasing.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.