UPDATE - June 14, 2018 10:00AM PT.
The Contextual Help for the Web Widget feature is now re-enabled for Pod 14 Accounts.
20:08 UTC | 13:08 PT
The search issues in POD 14 are resolved with the exception of contextual help in the Web Widget. We will provide an update once contextual help is restored.
19:21 UTC | 12:21 PT
We are seeing improvements for Guide Search in Pod 14, but we’re continuing to investigate issues with embeddable search.
18:18 UTC | 11:18 PT
Our investigation of issues affecting Pod 14 Guide Search continues. We are narrowing down potential causes, and we will provide further updates as we have more information.
17:11 UTC | 10:11 PT
We are seeing improvements in Search on Pod 14 as our investigation into the root cause continues. We’ll continue to post updates every hour, or when we have additional information. Thanks for your patience!
16:09 UTC | 09:09 PT
Our team is still working to identify the cause of Search issues affecting Pod 14.
15:35 UTC | 08:35 PT
Our team continues to investigate the cause of the issues affecting Search on Pod 14.
15:00 UTC | 08:00 PT
We continue to investigate and take mitigating steps on the Search issues impacting Guide on Pod 14.
14:45 UTC | 07:45 PT
Our Operations team are investigating an issue affecting Guide accounts on POD14. Thank you for your patience.
On June 7 4:59 UTC, June 8 02:50 UTC, June 8 22:43 UTC, and June 11 14:30 UTC, customers on Pod 14 experienced multiple incidents resulting in search becoming unavailable and causing 502 errors resulting in service unavailability.
Our developers took several actions in order to mitigate the search performance issues including:
- Isolating a slow search index shard to dedicated data node
- Migrate Guide search traffic to a separate endpoint so that Support traffic isn't impacted
- Disable cjk_bigram token filter for Korean languages
- Added circuit breakers to Guide to further reduce the impact of search overloading
- Split the article index into multiple indices to compartmentalize abusive accounts
Our developers have identified the root cause to be an elasticsearch CJK bigram filter used to optimize Korean language search queries. Our developers have since disabled the CKJ bigram filter on Pod 14.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.