15:16 UTC | 08:16 PT
Our search service is fully stable in pod3. Thanks for your patience.
14:57 UTC | 07:57 PT
Search performance has improved across Pod 3 accounts. We continue to work on stabilizing this service.
14:12 UTC | 07:12 PT
We continue to investigate the search issues impacting Pod 3 customers.
13:53 UTC | 06:53 PT
We are currently experiencing search issues on Pod 3 accounts. More info to follow
At 12:35pm UTC on Tuesday, October 18th, a server in the Elasticsearch cluster in Pod 3 crashed due to a kernel panic. This caused a short but significant latency spike. When the server came back online, the resulting cluster rebalancing led to high loads on other servers in the cluster which caused high latencies and errors for Pod 3 searches. To prevent a reoccurring incident in the future, we are building a new search cluster for Pod 3.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.