22:39 UTC | 14:39 PT
Email delivery issues on Pod 3 are resolved. Please let us know if you are still have issues
22:22 UTC | 14:22 PT
Email delivery on Pod 3 for new incoming emails is back to normal. Emails sent during the incident may be still be delayed.
21:15 UTC | 13:15 PT
We have had reports of improved email delivery times for Pod 3, but there may still be delays until we determine the root cause of the issue
20:17 UTC | 12:17 PT
The issue appears to be with our internal email processing as emails are getting to us in a timely manner. Update in an hour.
19:41 UTC | 11:41 PT
We are still investigating the intermittent delays of up to an hour for inbound email on Pod 3. Next update in 30 minutes.
19:11 UTC | 11:11 PT
We are continuing to investigate the delays with some inbound email on Pod 3. Delays have ranged from 20 minutes to just over an hour.
18:49 UTC | 10:49 PT
We are seeing some delays with inbound emails on Pod 3
During this incident, customers experienced delays of up to 1 hour for inbound Emails. When work postfix servers receive emails, they relay it to log servers for archiving purposes before writing the email to a directory that email processing code pulls from. One of our logs servers became unresponsive and didn’t write emails to disk for processing to start. Eventually, the delivery to log servers timed out and moved to a deferred queue, which cleared the path for new emails coming in to go through quickly. To resolve the issue, the offending logs server was removed from the list of destinations to relay emails, allowing emails to start flowing to disk again. To avoid this from happening in the future, our long term plan is to replace the logs server on Pod 3.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.