19:48 UTC | 11:48 PT
The issues with slow views and delayed automations on Pod 6 are now resolved
19:13 UTC | 11:13 PT
We've isolated the Pod 6 issue and performance has improved. We are continuing to monitor the situation and will update shortly.
18:53 UTC | 10:53 PT
Some Pod 6 customers are experiencing slow load times for views and delayed automations. We are still investigating.
18:38 UTC | 10:38 PT
We are currently experiencing performance problems in pod 6. The issue is under investigation, we will provide an update shortly.
During this incident, some customers experienced slow performance connecting and using their Zendesk.
The syslog configuration which leverages standard output to coordinate logging to various servers had inadequate buffers to handle the extremely high volume of logging at the time. When the buffers filled out, the services writing to them, in this case Occam, experienced degraded performance. To resolve the issue, we switched the configuration to not use standard output to write to the buffer logs and allowed Occam to write the logs directly to disk. To prevent this from happening again in the future, we plan to implement improvements to logging architecture and performance.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.