17:08 UTC | 09:08 PT
Search performance has returned to normal on P6. The team continues to monitor the situation.
16:23 UTC | 08:23 PT
We've added additional capacity to search on P6 to handle the performance issue. Search is returning to normal. We are currently monitoring
16:10 UTC | 08:10 PT
We are currently experiencing performance and search issues on POD 6. The issue is under investigation, we will provide an update shortly.
This is a reoccurrence of the same incident that occurred on the 22nd of February, however occurred on a different filesystem.
A surge in memory usage on data filesystems in Pod 6 led to server instability and a high timeout rate. We responded by restarting the data filesystems and re-routing traffic to a different server while the original server recovered, then once that was stable re-routed the traffic back. To prevent this from happening in the future we're improving our monitoring tools to better visualize traffic, as well as investigating if it's possible to make it so that each filesystem isn't a single point of failure for the server.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.