Duo Telephony-based Authentication Issue
Incident Report for Duo
Postmortem

Summary

On April 27, 2022, from 1:35 p.m. EST to 2:06 p.m. EST, Duo's cloud service experienced issues affecting some deployments, which caused failures with SMS passcode and phone call delivery, blocking authentication for customers that attempted to use those methods.

Details

At 1:35 p.m. EST on April 27, the Duo Engineering Team was conducting routine maintenance on our cloud service responsible for telephony.

During the maintenance, a previously undiscovered configuration defect caused SMS and phone call authentication requests to fail. This resulted in blocking end-users from completing authentication to Duo-protected applications, and prevented administrators from accessing the Duo Admin Panel using those authentication methods.

Proactive service monitoring alerted the Duo Engineering Team to these issues, and we were able to begin working on a solution immediately.

At 2:06 p.m. EST, we were able to remove the configuration defect, restoring the functionality of SMS passcode and phone call authentications.

The Duo Engineering Team is currently improving fault detection systems to prevent invalid configuration from impacting services. We are committed to identifying additional process improvements so we can deliver an even more reliable, resilient service moving forward.

Affected deployments: DUO38, DUO48, DUO53, DUO57, DUO66, DUO67, DUO68, DUO69, and DUO70

Timeline of events

| 2022-04-27 13:35 | Maintenance activity deployed application incompatible with configuration.

| 2022-04-27 13:40 | Duo Site Reliability Engineering (SRE) is informed by alerts from the monitoring of the systems and begins triage.

| 2022-04-27 13:45 | Duo Engineering identifies the cause of failure.

| 2022-04-27 13:58 | Duo Engineering tests configuration update required to address the issue.

| 2022-04-27 14:06 | Duo Engineering deploys configuration update, failures cease.

Posted Apr 29, 2022 - 14:43 EDT

Resolved
We have identified an issue disrupting end-user telephony-based authentications. After monitoring issues affecting telephony authentications, we have resolved the issue and deployed a solution. As of 7:00 pm UTC, telephony-based authentications are operating normally
Posted Apr 27, 2022 - 15:09 EDT
This incident affected: DUO67 (Phone Call Delivery, SMS Message Delivery), DUO68 (Phone Call Delivery, SMS Message Delivery), DUO57 (Phone Call Delivery, SMS Message Delivery), DUO38 (Phone Call Delivery, SMS Message Delivery), DUO53 (Phone Call Delivery, SMS Message Delivery), DUO48 (Phone Call Delivery, SMS Message Delivery), and DUO66 (Phone Call Delivery, SMS Message Delivery).