As of 10:15 UTC+1 on June 15th, 2015, we are experiencing a disruption in our EU datacenter.
10:20 UTC+1 - We're experiencing a few performance issues which may affect some customers. Apologies for the inconvenience.
10:40 UTC+1 - Our Operations team are actively troubleshooting issues affecting our EU datacentre. More to follow.
10:57 UTC+1 - We continue to troubleshoot performance issues in the EU datacentre.
11:20 UTC+1 - We're still actively working on fully resolving issues in EU data centre and seeing some signs of improvement.
11:45 UTC+1 - Service should be restoring for most customers. We continue to monitor until full resolution
12:25 UTC+1 - We're continuing to monitor performance issues in our EU data centre. Things are looking up and we'll communicate when fully resolved.
13:00 UTC+1 - All performance issues in the EU data centre have now been mitigated.
The cause of this issue was found to be similar to the issues experienced on June 11th which can be found here. Essentially the issue is due to a lack of optimization in our app framework in the way we process queries from our apps. This caused an immense load the databases in the datacenter to where the services could not keep up and essentially became unresponsive.
Once the issue was identified, our Operations team implemented a number of actions to reduce the load on our services in this datacenter in order to get our customers back up and operating at normal expected service levels.
We are continuing to work on a number of efforts to improve the optimization of the App framework and how queries are processed from the framework. We are adding additional resources around caching, query optimization, and framework enhancements around app functionality. This is a priority but is also a work in progress. We are also working with some of our App partners to help with optimization of their apps as well.
How to stay up to date and receive help related to this event:
If you are not subscribed to our Twitter feed, I encourage you to do so in order to get the most current information about any service issues. We record all site outages on our system status page where you can also see recent activity.
If you have any question or continue to see behavior you suspect to be related to this incident, please send an email to: firstname.lastname@example.org