22:07 UTC | 14:07 PT
We're happy to report issues impacting Voice for some customers are resolved. Post mortem to follow:
21:36 UTC | 13:36 PT
We’ve identified a fix for people impacted by Voice issues and are working towards resolution.
21:02 UTC | 13:02 PT
We are investigating reports of Voice issues for some customers. More information to come.
A database migration was run, which required a more up-to-date version of our Business Hours service to be installed across all Zendesk features. Voice was using an outdated version, which caused the application to break and cause almost every request to fail, leading to many customers being unable to complete calls.
The problem should have been detected on staging after the migration was run there as soon as we run any set of automated QA tests, or even just used the environment. However, it seems the migration was run in staging and deployed to production with no gap between. Our expectation is that all migrations are deployed to staging and allowed stew there for a few days at least. Had that occurred, there is a good chance this outage could have been avoided.
To resolve the incident, we bumped to the required version of the service in the Voice app. This was then deployed and Voice performance recovered.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.