Service Incident - July 14th, 2017 - Pod 6 Performance Issues

SUMMARY

04:26 UTC | 21:26 PT
We are happy to report that the Performance Issue on Pod 6 has been resolved.

03:52 UTC | 20:52 PT
We are still investigating the issue affecting Performance for some accounts on Pod 6. Thank you for your patience.

03:28 UTC | 20:28 PT
We are currently experiencing Performance Issues for some accounts on Pod 6. More updates to follow.

POST-MORTEM

A large user import caused significant database performance issues due to each user being checked against an customer blacklist with a large number of entries, which lead to new user creations being significantly more expensive. The combination of high volume and expensive checking lead to high load on the database servers. Our developers worked to mitigate the impact of the import by rate limiting the instance and removing the blacklist. These two actions resolved the excessive load on the database and stabilized performance. To prevent this from happening again in the future, we will be introducing performance improvements to how we query customer blacklists during user imports, optimize partner user imports, and query db slaves instead of the master db.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.