As of December 26th at 20:09 GMT / 12:09 PST, we are working to resolve the following service incident:
Some customers are unable to bulk update tickets.
20:27 GMT / 12:27 PST
We’re investigating reports of some customers unable to bulk update tickets. More information soon.
20:47 GMT / 12:47 PST
Our engineers have identified issues affecting bulk updates and are working towards resolution.
21:08 GMT / 01:08 PST
We are still working on root cause identification and complete resolution.
21:28 GMT / 01:28 PST
All bulk update service has been restored. Post-mortem to follow.
This issue was caused by connectivity issues experienced by an external provider on our infrastructure. This affected multiple customers, blocking bulk updates to all accounts located in our US-East data centre. The root cause analysis from our provider points to a line card issue in their core network.
Since the issue happened at their core network degrading performance, and not a disruption at the customer premise layer, our routers couldn't automatically detect the failure and converge to different upstreams. This has been made manually by our Operations team, who migrated the traffic away from the affected network. Once the service migration was completed, the impacted traffic stabilized and stuck jobs began to process.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.