As of 11:05 GMT / 03:05 PST we are working to resolve the following service incident:
We are investigating performance issues with Zopim. We will provide more information shortly.
11:24 GMT / 03:24 PST:
We are happy to report the performance issues affecting Zopim are now resolved. A Postmortem will be available here shortly.
This incident began due to a database pile-up, which caused by an emergency backup job and resulted in performance and login issues across the Zopim application.
An emergency database backup routine was executed to generate formal point-in-time backup to repair MySQL read services. High parallel thread concurrency from a backup tool caused memory and encrypted disk starvation which resulted in a pile-up of SQL queries.
To resolve the problems, we killed the offending backup routine as soon as we identified the root cause, and restarted MySQL to ensure full resolution.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.