17:58 UTC | 09:58 PT
The intermittent voice issues have been resolved. Postmortem to follow here.
Our team is continuing their investigation into the intermittent voice issues on our east coast data center. More information to come.
17:03 UTC | 09:03 PT
Some customers on our east coast data center are experiencing intermittent issues with Voice. We’re investigating.
This incident was caused by a fan failure in one of the proxies. The affected host was removed from rotation for maintenance, while the proxy was nearing capacity thresholds. Once this server was removed from rotation, our SSL TPS licensing limits were reached for the US East Coast data center, inducing failures in connections to the Voice service and resulting in errors for some Voice customers, whose accounts are located in the affected data center.
We'e working with our hosting provider on ensuring there are checks in place prior to removal of any host for maintenance, to ensure there aren't capacity concerns and also a health check of redundant servers to ensure there is enough resiliency to perform maintenance activities. From our side, we've already added more proxies as part of overall data center expansion initiatives.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.