Postmortem
Summary
On February 24, 2025, 9:47 am EST, Duo experienced a service disruption affecting the Admin Panel. The incident impacted customers using the discovery service, which is primarily used by managed service provider (MSP) administrators for managing subaccounts in a multi-tenant environment.
The incident was mitigated at 11:15 am. After monitoring the success of the fix, the incident was resolved at 12:00 pm.
Details
The disruption resulted from a performance issue with database queries in the discovery service, leading to problems logging into the Admin Panel login and searching or navigating subaccount-related features. The slow running query caused excessive database load during subaccount searches. The query's performance degraded with growing customer subaccount search, leading to slow response times.
Customers without multi-tenancy and/or not using discovery service were not affected, and the outage did not impact any authentication services. The issue was mitigated by scaling up the database instance to provide more headroom and fixing the slow database query.
The issue was identified prior to this outage and the development team had fixes that were planned for the next Duo release version. However, a spike in customer traffic caused this incident before that version was rolled out.
What is Duo doing to prevent this in the future?
In the short term, Duo has further optimized the database query to enhance performance and will add additional read replicas to alleviate load on the primary database. In the long term, Duo is implementing a range of performance improvements, including caching on the admin panel, to further reduce database load during subaccount operations.
Posted Feb 28, 2025 - 13:38 EST
Resolved
We have confirmed that the implemented fix has resolved the issue. Admin Panel logins are now fully operational. An RCA will be posted once it has been completed.
Posted Feb 24, 2025 - 11:57 EST
Monitoring
A fix has been implemented and we're starting to see recovery. We will continue to monitor the results.
Posted Feb 24, 2025 - 11:13 EST
Identified
The issue causing problems with Admin Panel logins has been identified and a fix is being worked on.
Posted Feb 24, 2025 - 10:38 EST
Investigating
We are currently investigating an issue causing login issues with the Duo Admin Panel on all of our deployments and are working to correct the issue as soon as possible.
Please check back here or subscribe to updates for any changes.
Posted Feb 24, 2025 - 10:17 EST
This incident affected: DUO6 (Admin Panel), DUO1 (Admin Panel), DUO2 (Admin Panel), DUO3 (Admin Panel), DUO4 (Admin Panel), DUO5 (Admin Panel), DUO7 (Admin Panel), DUO8 (Admin Panel), DUO47 (Admin Panel), DUO10 (Admin Panel), DUO11 (Admin Panel), DUO12 (Admin Panel), DUO13 (Admin Panel), DUO14 (Admin Panel), DUO15 (Admin Panel), DUO16 (Admin Panel), DUO17 (Admin Panel), DUO18 (Admin Panel), DUO19 (Admin Panel), DUO20 (Admin Panel), DUO21 (Admin Panel), DUO22 (Admin Panel), DUO23 (Admin Panel), DUO24 (Admin Panel), DUO25 (Admin Panel), DUO26 (Admin Panel), DUO27 (Admin Panel), DUO28 (Admin Panel), DUO29 (Admin Panel), DUO30 (Admin Panel), DUO31 (Admin Panel), DUO32 (Admin Panel), DUO33 (Admin Panel), DUO34 (Admin Panel), DUO36 (Admin Panel), DUO37 (Admin Panel), DUO38 (Admin Panel), DUO39 (Admin Panel), DUO40 (Admin Panel), DUO41 (Admin Panel), DUO42 (Admin Panel), DUO43 (Admin Panel), DUO44 (Admin Panel), DUO45 (Admin Panel), DUO46 (Admin Panel), DUO48 (Admin Panel), DUO9 (Admin Panel), DUO49 (Admin Panel), DUO50 (Admin Panel), DUO51 (Admin Panel), DUO52 (Admin Panel), DUO53 (Admin Panel), DUO54 (Admin Panel), DUO55 (Admin Panel), DUO56 (Admin Panel), DUO57 (Admin Panel), DUO58 (Admin Panel), DUO59 (Admin Panel), DUO60 (Admin Panel), DUO61 (Admin Panel), DUO62 (Admin Panel), DUO63 (Admin Panel), DUO64 (Admin Panel), DUO65 (Admin Panel), DUO66 (Admin Panel), DUO67 (Admin Panel), DUO68 (Admin Panel), DUO69 (Admin Panel), DUO70 (Admin Panel), DUO71 (Admin Panel), DUO72 (Admin Panel), DUO73 (Admin Panel), DUO74 (Admin Panel), DUO75 (Admin Panel), DUO76 (Admin Panel), DUO77 (Admin Panel), DUO78 (Admin Panel), DUO79 (Admin Panel), DUO80 (Admin Panel), DUO81 (Admin Panel), and DUO35 (Admin Panel).