16:19 UTC | 08:19 PT
Issues affecting performance on Pod 3 have been resolved.
15:54 UTC | 07:54 PT
We are continuing to monitor the performance issues on Pod 3 and the service looks now stable again. More info shortly.
15:05 UTC | 07:05 PT
Slowness in Pod 3 continues to improve - thanks for your patience while we continue to progress on our remediation work.
14:16 UTC | 06:16 PT
We are adding resources to one of our core applications to handle a larger load of work to fix performance issues on Pod 3. More info soon.
13:38 UTC | 05:38 PT
We are seeing improvements on performance issues affecting Pod 3. We will update you shortly.
13:00 UTC | 05:00 PT
We narrowed down the root cause and we are working on several mitigation strategies for the performance issues in Pod 3. More info to follow
12:01 UTC | 04:01 PT
We are continuing to work towards a resolution. Updates will follow.
11:30 UTC | 03:30 PT
We have now identified some potential causes for the performance issues on Pod 3 and we are working on a resolution. More info to follow
10:56 UTC | 02:56 PT
We are continuing to investigate the performance issues affecting Pod 3. More updates shortly
10:23 UTC | 02:23 PT
We are progressing with our investigation on performance issues on Pod 3. More info to follow
10:02 UTC | 02:02 PT
We are investigating reports of slowness for our Pod 3 customers. Updates to follow
During this incident, customers experienced a degradation in performance for their Views in Support, as well as dropped calls in Talk. As part of preparation work for some infrastructure improvements, we saw an increase in the number of API requests to some nodes on Pod 3, causing degraded performance for our customers on that pod. To resolve the incident, we increased the number of nodes available to better handle the influx of API requests. This action, combined with a code tweak, allowed services to recover. To prevent this from reoccurring, we're improving our the API rate limiting for this particular infrastructure work. We're also ensuring better cross-training between our infrastructure and operations engineers so we can respond more quickly to this type of issue.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.