As of Sept 14th at 14:03 PM GMT+1, we are working to resolve the following service incident:
Ticket creation delays for our European customers. More info to follow.
14:21 GMT+1 /6.21 PST
Our team is working at mitigating the delays impacting the ticket creation on our European data center.
14.42 GMT+1 / 6.42 PST
We have identified the cause of delays impacting ticket creation and are working on a resolution.
15.03 GMT+1 / 7.03 PST
We are continuing to work on a resolution to the ticket creation delays our European data centre customers are currently experiencing.
15.38 GMT+1 / 7.38 PST
Our team are currently implementing a fix to help resolve ticket creation delays impacting some customers. More information to follow.
16.23 GMT+1 / 8.23 PST
We have seen a decrease in ticket creation delays since implementing a fix. We will update once we have confirmed all issues are resolved.
16.33 GMT+1 / 8.33 PST
The issues impacting ticket creation have now been resolved. Post-mortem to follow on this article.
This incident was caused by a sudden surge of incoming bounce notifications, causing incoming email delay of up to 15 minutes for some customers. Our mail processors worked properly and marked sender as blacklisted. However, the step where we check for blacklisting is quite far down in the chain, which means it takes time for our servers to recognise the blacklist. The additional time spent caused a backup in emails processing and therefore delays. To fully resolve the incident, we've had to block a specific IP number for STMP traffic in the firewall. To prevent similar recurrences, we're improving the course of action to address email flood vulnerability faster.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.