19:18 UTC | 11:18 PT
The performance issues impacting talk on pod 6 have been resolved.
19:03 UTC | 11:03 PT
We're investigating performance issues impacting talk on Pod 6. More info to come.
During this incident, some customers on Pod 6 experienced Talk errors and dropped calls. The issue was caused by a software change that added a service to a datacenter. The software change included an escaped defect that caused network connections to overload without alerting our monitoring systems - this in turn caused the newly added service to fail, leading to dropped calls and Talk errors. Once our team pinpointed the error, the software change was rolled back and the previous working version deployed in its place, which resolved the issue. To help prevent this type of issue from recurring, our teams are investigating improved monitoring parameters as well as modifications to our internal change management processes.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.