As of 07:10 UTC / 23:30 PST we're investigating the following service incident:
We're currently investigating reports of issues with updating tickets for some customers. More information to follow.
07:32 UTC / 23:32 PST:
We are still investigating the bulk update issue reported earlier.
08:24 UTC / 00:24 PST:
We have identified the likely cause of the bulk update issue, and are working on solutions to fix it.
09:10 UTC / 01:10 PST:
We are continuing to work on resolving issues with bulk updates that some of our customers are currently experiencing. More info to follow
10:16 UTC / 02:16 PST:
Our Operations team are continuing to work on the root cause of bulk update issues affecting some accounts.
11:24 UTC / 03:24 PST:
We are continuing to work on the bulk update issues some customers are still experiencing. More updates to follow.
11:43 UTC / 03:43 PST:
We are happy to report that the issue with ticket bulk updates is now resolved. A Post Mortem will be available shortly.
This incident was caused by a backlog of bulk update jobs in the queue. An unexpected surge in ticket update activity caused delays to all other bulk update jobs. Our investigation found that the ticket update job was making outbound synchronous http requests on a per ticket basis. Since some bulk updates affect many tickets, a large sequence of jobs is put in queue for one single customer. One customer's instance was responding so slow that our queue was clogging up. During the incident, we had to remove several jobs from the queue to ensure it begins processing updates again. To prevent this, we are looking for a more intelligent approach to outbound requests.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.