We are working to resolve the following service incident:
22:00 UTC | 14:00 PST
Issues impacting Help Center for our West Coast data center customers have been resolved. Post-mortem to follow here once available.
21:19 UTC | 13:19 PST
We are continuing to investigate Help Center performance issues for some of our West Coast data center customers.
20:38 UTC | 12:38 PST
Voice and agent collision services improving for our West Coast data center customers. Help Center is still being investigated.
20:04 UTC | 12:04 PST
We are investigating issues with errors on Help Center as well as calls with Voice calls connecting for some customers. More info to come
This incident affected services such as Voice, agent collision and caused some general slow performance for customers hosted in our US West Coast data center. A process in one of our servers hit a memory limit, causing a memory leak and a subsequent crash of this process.
The reconnection logic for the clients connected to the affected server process created an increased load on the cluster, generating a high rate of commands to the server. The extra load caused a datacenter-wide slowdown of server response.
Additionally, the increase in client reconnects created increased memory pressure for the remaining server processes, causing additional servers to crash, creating a reconnection problem and putting the entire cluster into an unstable state. A simultaneous restart of all affected services helped clear the issue.
We're currently working on setting up better monitoring for memory usage by such processes so that we can ensure better response and recovery in such cases.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.