Postmortem
Summary
On February 13, 2025 at 18:00 UTC, some Duo customers experienced an unexpected service disruption, affecting services including:
Core Authentication Service
Passwordless Authentication
Risk-based Authentication
Telephony Authentication
Admin Panel logins
Device Management Portal
Device Health Policies via Duo Desktop
New user enrollments
Engineering teams manually restored services around 22:30 UTC. During the outage, no data was lost, and active sessions remained valid.
Details
Duo's cloud service is internally implemented by multiple microservice applications, each responsible for a product feature. A service manager takes in a configuration describing each service and automates deploying them to run on cloud infrastructure.
The issue began with the planned removal of a deprecated application. Due to a bug, after the removal the configuration for service manager’s own application had an error. The manager interpreted this error to mean it should remove itself and all services it manages, instead of removing only the deprecated service.
Duo SRE received alerts that critical services were no longer running. After diagnosing the problem, Duo SRE redeployed the service manager, followed by all services. This process took several hours because lower-level services (like network management) had to be redeployed before features like telephony authentication could run on top of them.
The bug affects an older version of Duo’s service manager which is responsible for telephony, risk-based authentication, self-service device management, and some passwordless authentication factors. These features were completely unavailable until each one was restored by region.
This bug did not occur when we removed the deprecated service from any of our staging environments which use the same service manager version.
Users experienced higher than normal error rates with push and other authentication services as their dependencies recovered. These services run on a newer version of the service manager.
What is Duo doing to prevent this in the future?
Duo is removing the ability for removal of one service to automatically remove other service dependencies. Each removal will have to be reviewed and approved individually, even in versions of the service manager that did not have the bug. This prevents the issue from happening again while making service removal more explicit and thus safer.
Duo had prior plans to migrate all services to our newer service manager. In response to this event, we plan to expedite the migration effort. This new service manager is more reliable and has better isolation to minimize the impact of any potential incidents. It also uses a smaller service mapping eliminating a single point of failure, resulting in a more resilient architecture paradigm than the legacy service manager. Duo will also review Authorization and Scope of Access Control to ensure permissions and control so that changes to one service cannot negatively impact other services.
Posted Feb 14, 2025 - 18:08 EST
Resolved
We can now confirm that all services have been completely restored. A full RCA will be provided as soon as possible. Thank you for your patience, while we worked to resolve the issues.
Posted Feb 13, 2025 - 20:43 EST
Update
Device Management Portal services have now been restored for all deployments.
We will provide further details as more progress on restoration is made.
Posted Feb 13, 2025 - 20:29 EST
Update
Core Authentication services have now been restored for the following deployments:
DUO1
DUO2
DUO4
DUO5
DUO6
DUO7
DUO9
DUO10
DUO13
DUO14
DUO15
DUO16
DUO17
DUO18
DUO19
DUO20
DUO21
DUO22
DUO23
DUO24
DUO28
DUO31
DUO33
DUO35
DUO36
DUO37
DUO39
DUO40
DUO41
DUO42
DUO45
DUO49
DUO50
DUO52
DUO55
DUO56
DUO58
DUO60
DUO62
DUO63
DUO64
DUO65
DUO72
DUO73
DUO74
DUO75
DUO76
DUO77
DUO78
DUO79
DUO80
We will provide further details as more progress on restoration is made.
Posted Feb 13, 2025 - 19:54 EST
Update
Device Management Portal services have now been restored for the following deployments:
DUO1
DUO2
DUO4
DUO5
DUO6
DUO7
DUO9
DUO10
DUO13
DUO14
DUO15
DUO16
DUO17
DUO18
DUO19
DUO20
DUO21
DUO22
DUO23
DUO24
DUO28
DUO31
DUO33
DUO35
DUO36
DUO37
DUO39
DUO40
DUO41
DUO42
DUO45
DUO49
DUO50
DUO52
DUO55
DUO56
DUO58
DUO60
DUO62
DUO63
DUO64
DUO65
DUO72
DUO73
DUO74
DUO75
DUO76
DUO77
DUO78
DUO79
DUO80
We will provide further details as more progress on restoration is made.
Posted Feb 13, 2025 - 19:26 EST
Update
We are seeing successful SMS and Phone Call Delivery for the DUO70 deployment.
We are still actively working toward restoring all other services and will provide updates as soon as possible.
Posted Feb 13, 2025 - 19:16 EST
Update
Telephony services and the Device Management Portal services have now been restored for the DUO38 and DUO48 deployments. We will provide further details as more progress on restoration is made.
Posted Feb 13, 2025 - 19:08 EST
Update
We are seeing successful recovery of Admin Panel login for all deployments.
We are still actively working toward restoring all other services and will provide updates as soon as possible.
Posted Feb 13, 2025 - 17:53 EST
Update
We are seeing successful SMS and Phone Call Delivery for the additional following deployments below. Other deployments not in the list may still be impacted:
DUO1
DUO2
DUO4
DUO5
DUO6
DUO7
DUO10
DUO13
DUO14
DUO15
DUO16
DUO18
DUO19
DUO20
DUO21
DUO23
DUO24
DUO28
DUO31
DUO33
DUO35
DUO36
DUO37
DUO41
DUO60
DUO66
DUO68
DUO79
DUO80
We are still actively working toward restoring all other services and will provide updates as soon as possible.
Posted Feb 13, 2025 - 17:37 EST
Update
We are seeing successful SMS and Phone Call Delivery for the following deployments below. Other deployments not in the list may still be impacted:
DUO9
DUO17
DUO22
DUO39
DUO40
DUO42
DUO45
DUO49
DUO50
DUO52
DUO55
DUO56
DUO58
DUO62
DUO63
DUO64
DUO65
DUO72
DUO73
DUO74
DUO75
DUO76
DUO77
DUO78
We are still actively working toward restoring all other services and will provide updates as soon as possible.
Posted Feb 13, 2025 - 17:10 EST
Update
We have identified the issue causing Core Authentication Service, Admin Panel login, Device Management Portal, Risk-based Authentication, and SMS/Phone Call delivery failures across all deployments and are working to deploy a fix.
If you have your Admin Panel URL saved, you can navigate to the URL (ex. admin-xxxxxxxx.duosecurity.com/) to get direct access.
When accessing the Admin Panel, please utilize non-telephony-based MFA methods to ensure proper access. Admin's currently logged in can activate Duo Push for other admins following this documentation: https://duo.com/docs/administration-admins#updating-an-administrator's-secondary-authentication-methods
Posted Feb 13, 2025 - 15:12 EST
Identified
We have identified the issue causing Admin Panel login, Device Management Portal, Risk-based Authentication, and SMS/Phone Call delivery failures across all deployments and are working to deploy a fix.
If you have your Admin Panel URL saved, you can navigate to the URL (ex. admin-xxxxxxxx.duosecurity.com/) to get direct access.
When accessing the Admin Panel, please utilize non-telephony-based MFA methods to ensure proper access. Admin's currently logged in can activate Duo Push for other admins following this documentation: https://duo.com/docs/administration-admins#updating-an-administrator's-secondary-authentication-methods
Posted Feb 13, 2025 - 14:01 EST
Investigating
We are currently investigating an issue causing Admin Panel login and SMS/Phone Call delivery failures across all deployments.
If you can access your Admin Panel, please utilize our other MFA methods such as Push, Passcodes, Hardware Tokens, and Security Keys in the meantime. https://duo.com/docs/policy#authentication-methods
Please check back here or subscribe to updates for any changes.
Posted Feb 13, 2025 - 13:45 EST
This incident affected: DUO1 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO2 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO3 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO4 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO5 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO6 (Admin Panel, Core Authentication Service, Phone Call Delivery, SMS Message Delivery), DUO7 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO8 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO47 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO10 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO11 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO12 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO13 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO14 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO15 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO16 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO17 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO18 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO19 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO20 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO21 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO22 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO23 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO24 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO25 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO26 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO27 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO28 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO29 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO30 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO31 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO32 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO33 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO34 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO36 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO37 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO38 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO39 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO40 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO41 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO42 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO43 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO44 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO45 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO46 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO48 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO9 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO49 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO50 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO51 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO52 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO53 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO54 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO55 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO56 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO57 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO58 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO59 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO60 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO61 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO62 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO63 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO64 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO65 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO66 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO67 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO68 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO69 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO70 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO71 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO72 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO73 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO74 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO75 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO76 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO77 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO78 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO79 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO80 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), DUO81 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery), and DUO35 (Core Authentication Service, Admin Panel, Phone Call Delivery, SMS Message Delivery).