As of July 25th, 2:08pm PDT, we are working to resolve the following service incident:
2:08pm PDT: Network interruption at our US east coast data center.
2:59pm PDT: We are still looking at the issue.
3:42pm PDT: Some customers had Voice issues. Our team is investigating and monitoring.
4:24pm PDT: All known issues related to this incident have been resolved. Postmortem will be posted here once available.
During a scheduled and planned maintenance window to replace a failed network core chassis, we encountered a network disruption due to an unexpected routing loop. Faulty network elements are meant to be taken offline during planned maintenance, while this time, the chassis was left powered on. We turned the faulty chassis off as soon as we received first reports of connectivity issues. Such incidents are rare but we will be working to refine our internal procedures to ensure this kind of error doesn't recur going forward.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.