01:02 UTC | 17:02 PT
The bulk update delay issue has been resolved.
00:16 UTC | 16:16 PT
Mitigation efforts are underway to alleviate bulk ticket update delays in our European, West, and East coast data centers.
23:48 UTC | 15:48 PT
We're receiving increased reports of bulk updating ticket delays in EMEA/West coast USA. Our operations team is currently investigating.
This incident was brought about when our bulk job processing queues increased dramatically, causing bulk ticket updates to be significantly delayed for data centers in Europe and US West Coast. The queue build up was caused by a ticket sharing loop on a number of customer accounts. The common denominator in this case was that all the accounts in question were using "Apply macro after update", a now-deprecated beta feature that was known to cause issues in the past. To remediate, we had to temporarily disable ticket sharing on the affected accounts.
While we do have a loop detection mechanism for sharing agreements, it is only designed to detect loops between two accounts; where more than two instances are involved, the loop will not be detected. We're working on changing this to catch larger loops. We're also working on implementing dynamic rerouting of queues to prevent queue buildup from affecting performance for our customers.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.