02:00 UTC | 19:00 PT
The root cause of this incident has been identified and remediated, Search services on POD 13 are now stable. If you're unable to Search for tickets older than 7 days, please contact us and we’ll prioritize remediation for you.
23:30 UTC | 16:30 PT
We continue to investigate issues searching within Support on Pod 13. We will provide more updates within the hour or as soon as more information is available
22:08 UTC | 15:08 PT
We’re still investigating issues searching within Support on Pod 13. We will provide more updates within the hour or as soon as more information is available.
21:06 UTC | 14:06 PT
We continue to investigate search issues affecting Support on Pod 13.
20:32 UTC | 13:32 PT
We're investigating reports of issues searching in Support on Pod 13.
On Friday June 29th we kicked off a ticket reindexing process on Pod 13 in order to improve search performance and capacity. Due to an escaped defect, the reindexing configuration was incorrectly set and tickets were reindexed improperly, resulting in customers losing the ability to search through tickets created prior to the reindex process starting for each customer’s data. Once the misconfiguration was identified, our search operations team worked to backfill the incorrectly reindexed tickets, beginning with the newest tickets first. The backfill was successfully completed July 8, 2018. In order to prevent this from happening again in the future, we will be implementing additional guardrails to our reindexing operations process and additional indexing logging.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.