As of October 5th, 2015 12:26 GMT+1 / 04:26 PST, we are working to resolve the following service incident:
We are investigating some issues affecting some customer using Zendesk Voice. More information to come.
13:00 GMT+1 / 05:00 PST:
Our team is still investigating issues affecting some customers using voice. Apologies for the inconvenience.
13:45 GMT+1 / 05:45 PST:
Our team has deployed a fix to solve the current issue impacting some of our Voice customers.
13:56 GMT+1 / 05:56 PST:
Issues relating to voice have now been resolved. A post-mortem will be posted shortly.
This incident began when some code changes introduced to help us measure our error rate malfunctioned after a server restart.
As part of improvement initiatives in the few weeks preceding this incident, we have some servers dedicated to the Voice "high" queue. Thanks to these changes, critical call-related jobs (dequeueing, ticket creation etc.) were not affected by the problem today. Instead, jobs on the "low" queue (e.g. voicemail creation, billing jobs, stats jobs etc.) were delayed for a period. Since this error only occurred when the servers were restarted, it was unlikely QA could have found this prior to deploy. It was, in fact. subsequent deploys which caused this problem which added to the confusion.
To resolve the issue, we found the error in the code and fixed it. We have planned some further improvement initiatives to help troubleshoot similar issues more effectively, such as improving our logging of Voice errors.
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.