16:48 UTC | 08:48 PT
The performance issues on Pod 3 are resolved. We're still investigating the root cause
16:28 UTC | 08:28 PT
Based on our monitoring the performance issue in Pod 3 should be recovering. We're still narrowing down exact root cause and are striving to get that ASAP. Please let us know if you're seeing improvements.
15:15 UTC | 07:15 PT
We're continuing to mitigate the degradation that some customers on Pod 3 are experiencing. We are seeing some improvements.
14:09 UTC | 06:09 PT
We apologize for the disruption that you are experiencing on Pod 3 today. Our teams are working hard to mitigate the issue.
13:05 UTC | 05:05 PT
We are still investigating performance issues with POD3. Next update in 60 mins. Apologies.
12:03 UTC | 04:03 PT
Our team are still investigating performance issues with POD3. Next update in 60 mins. Apologies
11:06 UTC | 03:06 PT
Our team are actively working on stabilising the performance issues you may be experiencing in POD3. Next update in 60 minutes.
10:28 UTC | 02:28 PT
We are still working towards a resolution for the performance issues in POD3, apologies for the inconvenience
10:09 UTC | 02:09 PT
Our team are continuing to investigate performance issues with POD3, we will update you again shortly.
09:51 UTC | 01:51 PT
We are currently investigating reports of performance issues affecting POD 3.
During this incident, some customers on Pod 3 reported performance degradation in Zendesk Support, along with difficulty taking and wrapping up calls in Zendesk Talk. This issue was caused by a misconfiguration in our health check monitoring, surfacing a latent bug that overloaded some application services. This overload, coupled with insufficient capacity to handle said overload, resulted in service degradation. To address the issue, we're exploring ways to re-architect our health checks so that they have a smaller network footprint while remaining effective. We're also extensively investigating capacity issues and adding more resources to alleviate any stress.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.