As of 04:07 UTC / 20:07 PST we're investigating the following service incident:
We received several reports from some of our customers that the search functionality in customer's Help Center is not working. We are investigating now.
04:27 UTC / 20:27 PST:
Shortly after the issue was reported our team detected the root cause and immediately resolved the issue.
06:28 UTC / 22:28 PST:
The search issue that was resolved earlier has returned and our team has jumped backed on the issue. We are looking to find what caused this reoccurrence and working on a permanent fix for the issue. More updates coming.
This incident affected all search services for customers whose accounts are located in our US West Coast datacenter. This led to Help Centre search not working for around 30 minutes, as well as inability to search within Zendesk. In addition, the outage affected our Voice service as Voice Dashboard functionality relies on search services.
This was caused by a failure of our search process on two database hosts in the affected datacenter. A significant amount of unassigned data began to appear in the search cluster; a problem related to stability in the search cluster. To remediate, we restarted the search service on the affected hosts and started to reroute unassigned data, using scripts and later back indexers; after which reports of issues subsided.
We're looking at improving our monitoring of issues with the search service: as a result of this incident, we will be improving our monitoring of unassigned data in on the search clusters.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.