16:58 UTC | 09:58 PT
We're happy to report that queues have returned to normal levels and are processing emails without delay. If you're still experiencing delays in processing emails, please reach out to us.
16:01 UTC | 09:01 PT
Emails are continuing to process and queues are going down. We will monitor the situation until emails queues are back to normal. The next update will be in an hour or when queues are normal.
15:21 UTC | 08:21 PT
Emails are processing for most customers. We are continuing to monitor the situation while queues catch up. No emails were lost during this incident, only delayed.
14:13 UTC | 07:13 PT
We’ve identified the issue which is impacting email delivery for customers on Pods 14, 17, 18 & 19. We continue to implement a fix.
13:42 UTC | 06:42 PT
Our teams continue to investigate the email delays impacting multiple pods. We apologise for the inconvenience.
13:22 UTC | 06:22 PT
Customers on Pods 14, 17, 18 & 19 may experience Inbound email delays. Our teams are working to remediate.
During this incident, we received an influx of SPAM emails on pods 14, 17, 18, and 19. Due to a change to our mail parsing code, mail started to queue and back-up on the email processing hosts. We blocked, filtered, and began to drop these emails. After sorting and re-queuing emails into our email processor, the backlog of emails and mail processing returned to normal processing times. An hour later, we received another batch of malicious emails that that resulted in a small 15-20 minute delay on those emails. In order to prevent this from happening again in the future, we have patched the mail parsing code defect and will be improving our SPAM identification and filters.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.