11:00am UTC+1/3:00am PST:
We're experiencing a few issues which may affect some customers and are investigating. Apologies for the inconvenience.
11:29am UTC+1/3:29am PST:
We are working at fixing the issue impacting some of our customers. Apologies for the disruption.
12:00pm UTC+1/4:00am PST:
We're still working at fixing the issue impacting our European datacenter. We apologies for the inconvenience.
12:45pm UTC+1/4:45am PST:
Our team is working hard at deploying a patch to resolve the current performance issues. Please bear with us.
13:50pm UTC+1/5:50am PST:
We are still experiencing issues on our European data center. Our team is working on it.
14:45pm UTC+1/6:45am PST:
We’re continuing to work toward mitigating the performance issues affecting customers in our European datacenter. Thanks for your patience.
15:45pm UTC+1/7:45am PST:
We actively working on resolving the issue impacting some of our customers. We will post more info here shortly
16:14pm UTC+1/8:14am PST:
We are working hard on the issues impacting our European Data Center. Next update in 30 minutes.
16:43pm UTC+1/8:43am PST:
The service incident should now be fully resolved. Thank you for your patience. Post-mortem to follow below when ready.
The cause of this issue was found to be a lack of optimization in our app framework in the way we process queries from our apps. This caused an immense load the databases in the datacenter to where the services could not keep up and essentially became unresponsive.
Once the issue was found, aggressive corrective actions were taken to to resolve this issue and a multi-step approach was needed. First Operations needed to limit the number of queries originating from the offending Apps during this time period. They then reigned in the existing queries that were not performing efficiently and degrading the system and then lastly, they added additional resources to help process the backlogged services in an effort to return to a normal operating state.
A number of efforts are underway to improve the optimization of the App framework and how queries are processed from the framework. We are adding additional resources around caching, query optimization, and framework enhancements around app functionality. This is a priority but is also a work in progress.
How to stay up to date and receive help related to this event:
If you are not subscribed to our Twitter feed, I encourage you to do so in order to get the most current information about any service issues. We record all site outages on our system status page where you can also see recent activity.
If you have any questions or continue to see behavior you suspect to be related to this incident, please send an email to: firstname.lastname@example.org