On April 27, 2022, from 1:35 p.m. EST to 2:06 p.m. EST, Duo's cloud service experienced issues affecting some deployments, which caused failures with SMS passcode and phone call delivery, blocking authentication for customers that attempted to use those methods.
At 1:35 p.m. EST on April 27, the Duo Engineering Team was conducting routine maintenance on our cloud service responsible for telephony.
During the maintenance, a previously undiscovered configuration defect caused SMS and phone call authentication requests to fail. This resulted in blocking end-users from completing authentication to Duo-protected applications, and prevented administrators from accessing the Duo Admin Panel using those authentication methods.
Proactive service monitoring alerted the Duo Engineering Team to these issues, and we were able to begin working on a solution immediately.
At 2:06 p.m. EST, we were able to remove the configuration defect, restoring the functionality of SMS passcode and phone call authentications.
The Duo Engineering Team is currently improving fault detection systems to prevent invalid configuration from impacting services. We are committed to identifying additional process improvements so we can deliver an even more reliable, resilient service moving forward.
Affected deployments: DUO38, DUO48, DUO53, DUO57, DUO66, DUO67, DUO68, DUO69, and DUO70
| 2022-04-27 13:35 | Maintenance activity deployed application incompatible with configuration.
| 2022-04-27 13:40 | Duo Site Reliability Engineering (SRE) is informed by alerts from the monitoring of the systems and begins triage.
| 2022-04-27 13:45 | Duo Engineering identifies the cause of failure.
| 2022-04-27 13:58 | Duo Engineering tests configuration update required to address the issue.
| 2022-04-27 14:06 | Duo Engineering deploys configuration update, failures cease.