10:46 UTC | 03:46 PT
We’re happy to confirm the backlog of emails have been cleared, and will be delivered shortly.
03:05 UTC | 20:05 PT
We’re still working through the email backlog for POD14 customers. Next update once backlog is cleared.
02:10 UTC | 19:10 PT
We’re still working through a way of processing the backlog of emails that were not delivered to customers that were affected in Pod14.
00:49 UTC | 17:49 PT
We are processing new email notifications, and working on a way to deliver emails from our backlog. More info to follow.
00:11 UTC | 17:11 PT
We are currently investigating email deliverability issues on Pod 14. More information to come.
The MX record for outbound email on POD 14 was deleted as part of a scheduled DNS configuration change. After the DNS record expired, the outbound hosts started receiving DNS errors when trying to send emails. These DNS errors did not trigger any alerts, since DNS errors are fairly common when sending emails.
We then started receiving reports from customers that outbound emails were not being received. We identified the missing MX record as the root cause and re-created the records to resolve the issue.
At this point, outbound email service was restored, but emails that were sent while the MX record was missing were still undelivered. Over the course of several hours, we then re-sent the emails manually to ensure they were being sent with proper headers and with DMARC signatures intact. Emails deliveries fully completed around 05:31 CT / 10:31 UTC.
In order to prevent this from happening again in the future, we will be improving how we manage and monitor DNS configuration changes, particularly in regards to email deliverability
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.