18:27 UTC | 11:27 PT
We’ve identified the cause of this incident. Our teams are working on remediating the incident. If you are impacted by this, please contact support by emailing firstname.lastname@example.org.
17:50 UTC | 10:50 PT
We are currently investigating an incident with light agents downgrading to end users.
During the process of migrating from Light Agents to the Collaboration Tool add-on in our internal subscription management system, the software was intended to use a “silent” communications profile that would not result in any changes made to Light Agents on customer accounts. Though the profile was tested in a staging environment without any issues, a configuration error meant that that profile did not completely match the same profile in the production environment. As a result, updates were made to Light Agent roles on customer accounts. Our billing development team was able to revert those changes and have made adjustments to all affected customer accounts to restore configurations that were in place prior to this incident. To prevent this issue from reoccurring in the future, our team is conducting a thorough review of all profiles between our staging and production environments to ensure they match, and implementing logging tools that help us more easily note dependencies affected by role changes.
Please note that light agents that were restored who had the viewing permissions set to All OR all requests by users in agent's org, now have the permission set to "only within agent group(s)”. The admins will need to set these permissions manually.
Professional Plans: Admin> Settings> Agents
Enterprise Plans: Admin> People> Roles> Light Agents
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.