DUO38 and DUO48 Deployments: SSO Authentication Issues
Incident Report for Duo
Postmortem

DUO38 and DUO48 Deployments: SSO Authentication Issues

Summary

On June 10, 2021 from 4:23 p.m. EDT to 5:13 p.m. EDT, Duo’s Cloud Service experienced issues affecting the DUO38 and DUO48 deployments which resulted in end-user authentication failures for some Duo Single Sign-On (SSO) customers. The Duo Engineering Team was able to fully resolve this issue at 5:13 p.m. EDT on June 10.

Details

At 4:23 p.m. EDT on June 10, the Duo Engineering Team was alerted of a connectivity issue for one of our redundant database servers. The initial connectivity issue arose from a hardware issue with our hosting provider. SSO authentications that attempted to use the unresponsive read replica were unable to complete Duo-protected SSO authentication during the impacted time period.

Proactive service monitoring immediately alerted the Duo Engineering Team to these issues, and once notified of the problem and status at the vendor’s facilities, we were able to begin working on a solution. At 5:03 p.m. EDT, we were able to begin a rollout of our solution, and at 5:13 p.m., we were able to confirm the restoration of our services. At this point, Duo SSO authentication issues were fully resolved.

The Duo Engineering Team has identified improvements that can be made that will allow us to continue to service SSO authentications if we again experience connectivity issues with our read replicas. Every authentication session currently uses one read replica. Engineering completed code changes that allow the SSO server to detect when that read replica has failed and switch mid-session to a different replica. This change is in testing now, with additional work planned.

Posted Jun 15, 2021 - 11:17 EDT

Resolved
The issue regarding authentication failures with SSO on our DUO38 and DUOO48 deployments is fully resolved and all services are now fully functional.

We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.

Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Jun 10, 2021 - 17:41 EDT
Monitoring
Our engineering team has made some changes that have resulted in the issue being fixed.

We will continue to monitor the issue and will post any updates when the incident is considered fully resolved.

Please check back here or subscribe here for further updates.
Posted Jun 10, 2021 - 17:19 EDT
Investigating
We are currently investigating an issue causing authentications to our SSO product on our DUO48 deployment to fail, and are working to correct the issue as soon as possible.

Please check back here or subscribe to updates for any changes.
Posted Jun 10, 2021 - 17:06 EDT
This incident affected: DUO48 (SSO) and DUO38 (SSO).