On February 25, 2021 from 2:35 a.m. EST to 3:21 a.m. EST, Duo's Cloud Service experienced issues affecting the DUO1 deployment which resulted in end-user authentication failures for some customers. Some customers were also unable to access the Duo Admin Panel via admin.duosecurity.com. The Duo Engineering Team was able to fully resolve this network issue at 3:21 a.m. EST.
At 2:35 a.m. EST on February 25, the Duo Engineering Team was conducting routine maintenance on our DUO1 deployment. A previously undiscovered configuration defect prevented our load balancer failover logic from working normally, resulting in a service load balancer becoming unreachable from the public internet. This resulted in some customers being unable to complete Duo-protected authentication or access the Duo Admin Panel.
Proactive service monitoring immediately alerted the Duo Engineering Team to these issues, and we were able to begin working on a solution at 2:36 a.m. EST. At 3:21 a.m. EST, we were able to restore our load balancing solution, and both authentication and administration issues were fully resolved.
The Duo Engineering Team is currently identifying other potential configuration defects so this failover logic error does not cause a service issue again. We are committed to identifying additional process improvements so we can deliver an even more reliable, resilient service moving forward.