19:55 GMT / 12:55 PDT:
We are currently investigating formatting issues with the content of some email notifications.
20:40 GMT / 1:40 PDT:
We have identified the likely cause of this issue, and are working on solutions to fix it. More to come…
00:00 GMT / 16:00 PDT:
We’ve confirmed Liquid Markup formatting issues are resolved. We continue to investigate a fix for unexpected line breaks in email reported by a few customers.
01:15 GMT / 17:15 PDT:
Investigation and work to resolve the linebreak formatting issue continues, we have not yet found the resolution. This issue is not wide-spread, but is impacting a small subset of customers in one of our data North American data centers. We will continue to focus on this issue until it is resolved.
Next update will be in approx. 2 hours in this article. Twitter followers have also been referred to this article for additional updates.
02:52 GMT/ 19:51 PDT:
The issue with linebreak formatting was resolved at approximately 01:18 GMT/19:17 PDT. A post-mortem will be provided here.
During this incident, some end-users were seeing the following error in the bodies of email notifications:
"Liquid error: no implicit conversion of nil into String"
This affected some of our customers hosted in the West Coast data center. The issue was caused by a recent deploy, which aimed to improve the way we handle HTML in ticket comment bodies. However, some of the deploy code had a bug. Agents were not affected, but end-users were.
This bug was difficult to surface in our QA process before the deploy because it only affected accounts without a help center or web portal; a combination we do not often encounter or test. To recover, we rolled back the deploy that had the bug, then shipped the deploy with the fix. We're currently reviewing our testing scenarios to ensure these bugs are easier to surface.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.