18:32 UTC | 10:32 PT
We're happy to report that Chat Real Time Monitors are displaying correct values. If you're still experiencing any issues, please reach out to us.
18:18 UTC | 10:18 PT
Our team is still actively investigating issues with the RTM Chat Dashboard. We’ve seen some improvements in stability, but will confirm once we know it is completely resolved.
17:57 UTC | 09:57 PT
Our Operations team is currently investigating some reports of incorrect values displaying in the Chat Real Time Monitor Dashboard
On February 12, 2019 at 16:49 UTC / 08:49 PST, we started receiving reports of Chat Monitor displaying incorrect values.
This incident was caused by an issue with a third-party real-time data pipeline not consuming all Chat events, causing partial aggregation of events leading to incorrect numbers being reported in the Chat Monitor. During the investigation, our team identified some data pipeline consumers were disconnected and restarted those services resulting in full recovery.
To prevent this issue from occurring in the future, we have attached a critical priority to the following remediation tasks:
- Additional metrics, monitoring and alerting on our data pipeline consumers.
- Additional logging and expanded log retention.
In addition to the above tasks, we have scheduled work to increase Chat data pipeline resilience.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.