As of 12:20am UTC / 16:20 PST we're investigating the following service incident:
We're currently investigating reports of an issue impacting ticket creation for inbound emails. More information to follow.
12:30am UTC /16:30 PST:
We have identified an issue with delays in ticket creation from inbound email & some Help Center errors. We are currently implementing a fix.
12:50am UTC /16:50 PST:
We are continuing to work on delays with ticket creation via inbound email and Help Centre errors, Next update in 30 minutes.
01:30am UTC /17:30 PST:
We are happy to report that the issue with the inbound email ticket creation delay and Help Center errors is now resolved. A Post Mortem will be available shortly.
This incident was caused by an emergency change that was carried out to update some of our network certificates, which resulted in failures of some jobs (mainly inbound email processing), as well as runtime failures for Help Center theme edits.
After finding the root cause, we immediately reverted the emergency change. Once validated that this has resolved the errors, we put a recovery mechanism in place to allow the jobs to process as they remained queued at this point. During recover, some users began reporting 500 errors when editing Help Center themes. As the change revert propagated on all mirrors, errors subsided and a full resolution was confirmed as inbound email queues returned to normal.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.