01:28 UTC | 17:28 PT
We will complete a review of this incident as soon as possible.
01:04 UTC | 17:04 PT
We have identified and resolved an issue affecting delayed inbound email creation on Pod 20 via the Gmail Connector.
On December 6, during a routine change adding additional worker nodes in Pod 20, a deployment of our mail processing queue was only partially completed. This resulted in a backlog of undelivered emails from the Gmail connector because our Mail Fetcher was downloading emails from Gmail via Gmail Connector but the mail parser queue that is required to parse these emails was missing. Because MPQ wasn't deployed, the MPQ directory/queue was not being monitored and therefore build up of mail didn't trigger alerts. As a result when MPQ was deployed these emails were delivered with up to 5 days delay. In order to prevent this from happening again in the future, we will be improving our deploy runbook, alerting, and monitoring.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.