23:42 UTC | 15:42 PT
All known issues impacting POD 6 customers have been resolved.
23:09 UTC | 15:09 PT
Search on POD 6 continues to be stable. We're monitoring the situation closely.
22:33 UTC | 14:33 PT
We have received reports the system is stabilizing. Search functionality has returned on POD 6. We're confirming the disruption has ended.
22:20 UTC | 14:20 PT
We are receiving reports of issues with search and slowness for customers on POD 6. Investigation is underway.
A surge in memory usage on data filesystems in Pod 6 led to server instability and a high timeout rate. We responded by restarting the data filesystems and re-routing traffic to a different server while the original server recovered, then once that was stable re-routed the traffic back. To prevent this from happening in the future we're improving our monitoring tools to better visualize traffic, as well as investigating if it's possible to make it so that each filesystem isn't a single point of failure for the server.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.