20:04 UTC | 13:04 PT
We can now report that the login issue for our standalone Chat customers is resolved
19:53 UTC | 12:53 PT
Customers should no longer be impacted by ongoing chat issues. Still working on root cause. Next update in 60 minutes.
19:21 UTC | 12:21 PT
We are starting to see improvements for our standalone chat customers. We are continuing to monitor the situation and will update shortly.
18:58 UTC | 11:58 PT
The scope of the chat login issue appears to only be impacting a small subset of customer accounts. We’re working towards a fix.
18:25 UTC | 11:25 PT
We're continuing to investigate the Chat issue but have not yet isolated a root cause. Next update in 30 minutes
18:04 UTC | 11:04 PT
Impact is currently isolated to standalone chat accounts for the dashboard login issues. We are continuing to investigate.
17:45 UTC | 10:45 PT
We're investigating reports of some customers not able to access the Chat Dashboard
During this incident, a subset of Chat customers could not access their dashboard. A misconfiguration in a Chat JS API caused an infinite loop for every Chat registration, which quickly exhausted the resources on one of our Live Chat servers. We moved accounts from the impacted server to another server until we determined that one customer account was causing the issue. Once we identified the offending account, we moved it to an isolated Live Chat server so other accounts would no longer be affected. We've worked with that account to remedy the issue.
To remedy this going forward, there will be long-term planning discussions around our Live Chat server capacity and unintentional platform abuse situations.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.