19:29 UTC | 11:29 PT
The outbound email delay in Pod 3 has now been fully resolved.
18:54 UTC | 10:54 PT
We are seeing improvements in Pod 3 outbound email delivery. We are continuing to work on adding additional capacity to handle this issue.
18:18 UTC | 10:18 PT
We are investigating potential customer abuse for the Pod 3 outbound email issue. Will update shortly.
17:46 UTC | 09:46 PT
We are still investigating reports of delays on outbound email in Pod 3. Will update in 30 minutes.
17:20 UTC | 09:20 PT
We are currently investigating reports of delays on outbound email in Pod 3. We will update shortly.
During this incident our Pod 3 customers' outbound emails were delayed. An influx of trial accounts created for the purposes of sending spam emails through our services caused several outbound email servers' IP addresses on Pod 3 to be blacklisted. This caused massive backups in the outbound email queue. To resolve the incident, the spammers' trial accounts were suspended, their spamming emails scrubbed from the queues, and the IP addresses of the outbound mail servers were rotated to non-blacklisted addresses. This returned email service to normal for valid customers while giving us some time to implement additional changes and fixes to further prevent spammers from abusing our system. Short-term and long-term solutions have been discussed and are in progress to minimize this type of malicious activity from happening in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.