[Federal Deployments] Authentication issues using Yubikey tokens
Postmortem
Summary
On Nov 4, 2024, between 6:37 p.m. and 7:20 p.m. ET, FedRAMP customers using YubiKey tokens for authentication experienced failed authentications. The root cause was identified as a configuration error introduced during the rollout of a new software version for the internal proxy service.
Deployments Impacted
Timeline of Events (ET)
2024-11-04 6:38:35 - Duo Site Reliability Engineering (SRE) is alerted by an automated monitoring system of a high number of Python tracebacks.
2024-11-04 7:05:00 - The issue is identified as related to the recent software upgrade to the internal proxy service.
2024-11-04 7:11:00 - The software rollback process is initiated.
2024-11-04 7:20:00 - Rollback process completes, and Python tracebacks decrease.
2024-11-04 7:22:00 - The Duo status page is updated to inform customers of the downtime.
Details
The root cause was identified as a configuration mismatch between two services, introduced during the new software deployment. The issue was identified and resolved within 40 minutes, impacting 494 users.
As a short-term solution, Duo rolled back to the previous stable version of code to mitigate customer impact. After authentications were confirmed to be successful again, Duo released a permanent fix to the affected deployments to ensure long-term stability.
Posted Nov 06, 2024 - 19:09 EST
Resolved
The issue causing authentication failures using Yubikey tokens is now resolved and full functionality is restored.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Posted Nov 04, 2024 - 20:04 EST
Monitoring
We have implemented a fix for the authentication issue using Yubikey tokens.
We will continue to monitor the issue and will post any updates when the incident is considered fully resolved.
Posted Nov 04, 2024 - 19:46 EST
Identified
We have identified the cause of the authentication issue using Yubikey tokens and are actively working to restore service.
Posted Nov 04, 2024 - 19:33 EST
Investigating
We are currently investigating an issue that is affecting authentication using Yubikey tokens on Federal Deployments. We are working on remediation and will provide more updates as soon as we have more information.
Posted Nov 04, 2024 - 19:24 EST
This incident affected: DUO1 (Core Authentication Service), DUO2 (Core Authentication Service), DUO3 (Core Authentication Service), DUO4 (Core Authentication Service), DUO5 (Core Authentication Service), DUO7 (Core Authentication Service), DUO8 (Core Authentication Service), DUO47 (Core Authentication Service), DUO10 (Core Authentication Service), DUO11 (Core Authentication Service), DUO12 (Core Authentication Service), DUO13 (Core Authentication Service), DUO14 (Core Authentication Service), DUO15 (Core Authentication Service), DUO16 (Core Authentication Service), DUO17 (Core Authentication Service), DUO18 (Core Authentication Service), DUO19 (Core Authentication Service), DUO20 (Core Authentication Service), DUO21 (Core Authentication Service), DUO22 (Core Authentication Service), DUO23 (Core Authentication Service), DUO24 (Core Authentication Service), DUO25 (Core Authentication Service), DUO26 (Core Authentication Service), DUO27 (Core Authentication Service), DUO28 (Core Authentication Service), DUO29 (Core Authentication Service), DUO30 (Core Authentication Service), DUO31 (Core Authentication Service), DUO32 (Core Authentication Service), DUO33 (Core Authentication Service), DUO34 (Core Authentication Service), DUO36 (Core Authentication Service), DUO37 (Core Authentication Service), DUO38 (Core Authentication Service), DUO39 (Core Authentication Service), DUO40 (Core Authentication Service), DUO41 (Core Authentication Service), DUO42 (Core Authentication Service), DUO43 (Core Authentication Service), DUO44 (Core Authentication Service), DUO45 (Core Authentication Service), DUO46 (Core Authentication Service), DUO48 (Core Authentication Service), DUO9 (Core Authentication Service), DUO49 (Core Authentication Service), DUO50 (Core Authentication Service), DUO51 (Core Authentication Service), DUO52 (Core Authentication Service), DUO53 (Core Authentication Service), DUO54 (Core Authentication Service), DUO55 (Core Authentication Service), DUO56 (Core Authentication Service), DUO57 (Core Authentication Service), DUO58 (Core Authentication Service), DUO59 (Core Authentication Service), DUO60 (Core Authentication Service), DUO61 (Core Authentication Service), DUO62 (Core Authentication Service), DUO63 (Core Authentication Service), DUO64 (Core Authentication Service), DUO65 (Core Authentication Service), DUO66 (Core Authentication Service), DUO67 (Core Authentication Service), DUO68 (Core Authentication Service), DUO69 (Core Authentication Service), DUO70 (Core Authentication Service), DUO71 (Core Authentication Service), DUO72 (Core Authentication Service), DUO73 (Core Authentication Service), DUO74 (Core Authentication Service), DUO75 (Core Authentication Service), DUO76 (Core Authentication Service), DUO77 (Core Authentication Service), DUO78 (Core Authentication Service), DUO79 (Core Authentication Service), DUO80 (Core Authentication Service), DUO81 (Core Authentication Service), DUO6 (Core Authentication Service), and DUO35 (Core Authentication Service).