01:02 UTC | 18:02 PT
Search services in POD 14 have returned to normal. If you experience any issues, please let us know. We apologize for any inconvenience this may have caused.
23:52 UTC | 16:52 PT
We continue to investigate issues with Search Services on POD 14. More information will be shared as we get it.
23:14 UTC | 16:14 PT
Our team is investigating issues while using search services for some customers on Pod 14. More information to follow.
On June 7 4:59 UTC, June 8 02:50 UTC, June 8 22:43 UTC, and June 11 14:30 UTC, customers on Pod 14 experienced multiple incidents resulting in search becoming unavailable and causing 502 errors resulting in service unavailability.
Our developers took several actions in order to mitigate the search performance issues including:
• Isolating a slow search index shard to dedicated data node
• Migrate Guide search traffic to a separate endpoint so that Support traffic isn't impacted
• Disable cjk_bigram token filter for Korean languages
• Added circuit breakers to Guide to further reduce the impact of search overloading
• Split the article index into multiple indices to compartmentalize abusive accounts
Our developers have identified the root cause to be an elasticsearch CJK bigram filter used to optimize Korean language search queries. Our developers have since disabled the CKJ bigram filter on Pod 14.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.