At approximately 18:45 GMT we began to receive reports from customers of inbound email processing delays.
18:15 GMT email delays in email to ticket conversion began
18:45 GMT initial reports of email to ticket conversion
19:20 GMT our Operations team had diagnosed the issue, and had taken steps to restore normal behavior
21:38 GMT all all enqueued email was delivered properly
An incorrect configuration resulted in emails not being delivered to the correct service. This was spotted in QA but pushed to production before the error was rectified.
The email ticket creator service could be updated to increment a counter each time an email is processed. If such metrics are added, our Operations team could then create a metric alert to notify us if the current processing rate drops below its expected value. A regression test has been added to internal instructions to ensure that services are being configured correctly going forward.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.