Summary
On June 10, 2021 from 4:23 p.m. EDT to 5:13 p.m. EDT, Duo’s Cloud Service experienced issues affecting the DUO38 and DUO48 deployments which resulted in end-user authentication failures for some Duo Single Sign-On (SSO) customers. The Duo Engineering Team was able to fully resolve this issue at 5:13 p.m. EDT on June 10.
Details
At 4:23 p.m. EDT on June 10, the Duo Engineering Team was alerted of a connectivity issue for one of our redundant database servers. The initial connectivity issue arose from a hardware issue with our hosting provider. SSO authentications that attempted to use the unresponsive read replica were unable to complete Duo-protected SSO authentication during the impacted time period.
Proactive service monitoring immediately alerted the Duo Engineering Team to these issues, and once notified of the problem and status at the vendor’s facilities, we were able to begin working on a solution. At 5:03 p.m. EDT, we were able to begin a rollout of our solution, and at 5:13 p.m., we were able to confirm the restoration of our services. At this point, Duo SSO authentication issues were fully resolved.
The Duo Engineering Team has identified improvements that can be made that will allow us to continue to service SSO authentications if we again experience connectivity issues with our read replicas. Every authentication session currently uses one read replica. Engineering completed code changes that allow the SSO server to detect when that read replica has failed and switch mid-session to a different replica. This change is in testing now, with additional work planned.