An environment variable used to configure Help Center with a list of Kafka brokers in development was set in staging with an incorrect value but didn’t show up in staging tests. Inadvertently the environment variable change was deployed to production, causing the Kafka client to crash and bringing down Help Center with it, due to a new dependency in Help Center code. Once we realized this, we rolled back the code, resolving the issue.
Future efforts to mitigate this type of issue include better automated testing and investigating a quicker rollback process to decrease customer down time.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.