Service Incident - July 2nd, 2018 - Search issues on Pod 13

SUMMARY

15:25 UTC | 08:25 PT
We have fixed the search issues affecting some of our customers on POD13. Please contact us if you continue to see issues.

15:11 UTC | 08:11 PT
We are seeing improvement with Search on Pod 13. Our teams continue to investigate. Thank you for your patience.

15:11 UTC | 08:11 PT
We are seeing improvement with Search on Pod 13. Our teams continue to investigate. Thank you for your patience.

14:48 UTC | 07:48 PT
Some customers on Pod 13 are experiencing a degraded level of service. Please bear with us as we investigate.

POST-MORTEM

On July 2nd, 2018 at 14:20 UTC / 07:20 PT, customers on Pod 13 experienced issues with search in Support and Guide. The incident was caused by a configuration error within our search backend that resulted in high indexing activity. Due to unadjusted throttles, this overwhelmed a search cluster resulting in search timeouts across Support and Guide. Inbuilt recovery mechanisms caught up and ensured latencies went back to normal levels resolving the search issues in Support and Guide. Once our Operations team investigated the issue, we scheduled ongoing work to adjust the search cluster throttles and implement automatic failover to a backup cluster.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.