As of 6:30 AM GMT / 22:30 PM PST we are working to resolve the following service incident:
We are investigating accessibility issues in our US-West Data Center. More information to follow.
As of 7:18 AM GMT / 23:18 PM PST
We're seeing improvements in accessing Zendesk from our end.
As of 7:28 AM GMT / 23:28 PM PST
Our team has confirmed that the outage experienced by our US-West Data Center has now been resolved.
A combination of delete operations from regular account deletes and inbound emails purge operation (older than 1 month) seemed to have taken system resources on one of our databases in the US West Coast data center by increasing the response times of the normal queries and rendered the server inaccessible. This affected the ability to access Zendesk for a subset of customers hosted in said data center.
As the master database devolved into an unresponsive state, our database admin team tried to prepare to failover the database host to an alternative DB. However, the failover was not successful. A query kill was then initiated on the master server, which stabilized the cluster and brought it back into its original state. We're working on certain precautionary measures to prevent this from happening again; we've modified the job that carries our the deletion of inbound email records such that those queries do not consume an inordinate amount of database resources, as well as changed our failover mechanism to ensure service continuity should failures occur.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.