18:21 UTC | 10:21 PT
We’re happy to report the access issues surrounding the US east coast data center is now stable.
18:12 UTC | 10:12 PT
Access issues impacting our East Coast US data center have stabilized and we are continuing to investigate.
17:51 UTC | 09:51 PT
We are investigating access issues for some customers in east coast data center.
This incident began as a mySQL database in our East Coast data center became unresponsive due to the number of connections reaching the maximum allowed number. This resulted in no write activity for a significant number of customers on the affected cluster for about 20 minutes.
Our database admin team worked with the support team in our 3rd party DB hosting to determine the root cause. Exact root may not be pin-pointed but seems very likely due to either non-uniform memory access being enabled on the server and causing a spike in connections; a bug in the version of mySQL currently installed on the server; or a combination of both.
The database was restarted and activity returned to normal. We're working with our vendor to install the latest version of mySQL on all servers. We've also updated the configuration of non-uniform memory access enabled servers to remain stable.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.