23:44 UTC | 16:44 PT
All issues with the voice incident have been resolved. We will publish a post-mortem here: http://zdsk.co/1TGI8Tv
22:52 UTC | 15:52 PT
We're continuing to investigate voice connectivity issues internally and will investigate with our vendor as needed.
22:28 UTC | 15:28 PT
We're continuing to investigate the voice connectivity issue with our voice provider.
22:09 UTC | 15:09 PT
We are receiving reports of connectivity issues with Zendesk Voice. Investigation is underway.
Customers reported connectivity issues with our Voice service, including complete inability to connect to the Voice service. Root cause investigation determined that the issue was caused by the recent introduction of a new stats endpoint for our load balancer software. With the standard setup, there is no conflict between the front-end and the stats port.
In the case of the Voice server, the front-end port for the voice application had been changed from the standard port to a different port. This caused both the Voice app and the stats reporting to use the same port on the Voice servers. This overlap caused ambiguity for requests sent to the Voice hosts, which resulted in the Voice app ceasing to function.
Additionally, the Voice requests were failing with an unexpected error code which was not monitored by our alerting system. Remediation efforts include an audit of the stats endpoint to ensure it no longer conflicts with any application endpoints, as well as improving internal status alerts to ensure that all error codes are caught by our monitoring.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.