Authentication failures when requiring Duo Device Health App presence
Postmortem
On May 3, 2023, at approximately 1:14 pm ET, Duo received reports from customers about end-users blocked from authentication. Customers were affected on multiple deployments and had Duo Device Health application (DHA) policies in common. By 1:30 pm, Duo’s Engineering team verified that error log events coinciding with blocked authentications were associated with a production update to backend services that began shortly before the customer reports, at 11:08 am. Duo’s Engineering team concluded that the backend services update was the root cause of the DHA authentication issue. Duo’s Engineering team began rolling back the update at 1:37 pm.
Deployments Impacted
- DUO1, DUO2, DUO3, DUO4, DUO6,, DUO7, DUO47, DUO10, DUO13, DUO14, DUO15, DUO16, DUO17, DUO18, DUO19, DUO20, DUO21, DUO22, DUO23, DUO24, DUO25, DUO27, DUO28, DUO29, DUO30, DUO31, DUO32, DUO33, DUO36, DUO37, DUO38, DUO39, DUO40, DUO41, DUO42, DUO44, DUO45, DUO48, DUO9, DUO49, DUO50, DUO52, DUO53, DUO54, DUO55, DUO56, DUO57, DUO58, DUO60, DUO62, DUO63, DUO64, DUO65, DUO70, DUO71, DUO72, DUO73, and DUO35 .
(All times Eastern)
Timeline
- 2023-05-03 11:08am - Update to backend services started
- 2023-05-03 1:14pm - Received the first report of authentication issues
- 2023-05-03 1:30pm - Verified the errors in our logging coincided with the start of the update
- 2023-05-03 1:35pm - Status page updated to ‘investigating’
- 2023-05-03 1:37pm - Decision made to roll back the update
- 2023-05-03 1:43pm - Status page updated to ‘identified’
- 2023-05-03 1:55pm - Changes made to the code started to roll out to internal test deployments
- 2023-05-03 2:47pm - Confirmed the rollback resolved the issue through acceptance testing with our test deployment
- 2023-05-03 5:45pm - Rollback of the update to customer deployments complete
- 2023-05-03 5:50pm - Confirmed absence of errors in our logging after the rollback
- 2023-05-03 6:15pm - Status page updated to ‘monitoring’
- 2023-05-03 9:32pm - Status page set to ‘resolved’ after our metrics confirmed the errors stopped for affected customers
Details
On Wednesday, May 3, 2023, at 11:08 am EST, Duo Security started rolling out an update to one of our backend services. As this update reached customer deployments, end-users logging into applications protected by policies requiring the Duo Device Health application were blocked from authenticating.
At 1:14 pm, Duo Support received the first reports from customers about end-users blocked from authentication. This impacts all Advantage and Premier edition customers in the specified deployments that require DHA in their policies.
The Duo Endpoint Health team verified that the issue was due to the update, and a rollback of the changes was initiated. This fix reached all impacted customer deployments by May 3, 2023, at 5:45 pm EST.
Root cause analysis identified a need for additional alerting as well as changes in processes to our acceptance testing and end-to-end testing prior to releasing changes to the affected services. Our phased-release strategy allows us to identify issues early on. In this case, we were able to get customer feedback and were able to mitigate the issue before it had a larger customer impact.
Posted May 05, 2023 - 15:37 EDT
Resolved
The issue that is causing authentication failures when DHA health checks are required by policy is fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted May 03, 2023 - 21:32 EDT
Monitoring
We have implemented a fix for the issue that is causing authentication failures when DHA health checks are required by policy and are monitoring the results.
Posted May 03, 2023 - 18:15 EDT
Identified
We have identified the issue that is causing authentication failures when DHA health checks are required by policy and are working on a fix.
Posted May 03, 2023 - 16:43 EDT
Investigating
We are currently investigating an issue that is causing authentication failures for authentication that require DHA health checks by policy
Posted May 03, 2023 - 16:34 EDT
This incident affected: DUO1 (Core Authentication Service), DUO2 (Core Authentication Service), DUO3 (Core Authentication Service), DUO4 (Core Authentication Service), DUO6 (Admin Panel), DUO7 (Core Authentication Service), DUO47 (Core Authentication Service), DUO10 (Core Authentication Service), DUO13 (Core Authentication Service), DUO14 (Core Authentication Service), DUO15 (Core Authentication Service), DUO16 (Core Authentication Service), DUO17 (Core Authentication Service), DUO18 (Core Authentication Service), DUO19 (Core Authentication Service), DUO20 (Core Authentication Service), DUO21 (Core Authentication Service), DUO22 (Core Authentication Service), DUO23 (Core Authentication Service), DUO24 (Core Authentication Service), DUO25 (Core Authentication Service), DUO27 (Core Authentication Service), DUO28 (Core Authentication Service), DUO29 (Core Authentication Service), DUO30 (Core Authentication Service), DUO31 (Core Authentication Service), DUO32 (Core Authentication Service), DUO33 (Core Authentication Service), DUO36 (Core Authentication Service), DUO37 (Core Authentication Service), DUO38 (Core Authentication Service), DUO39 (Core Authentication Service), DUO40 (Core Authentication Service), DUO41 (Core Authentication Service), DUO42 (Core Authentication Service), DUO44 (Core Authentication Service), DUO45 (Core Authentication Service), DUO48 (Core Authentication Service), DUO9 (Core Authentication Service), DUO49 (Core Authentication Service), DUO50 (Core Authentication Service), DUO52 (Core Authentication Service), DUO53 (Core Authentication Service), DUO54 (Core Authentication Service), DUO55 (Core Authentication Service), DUO56 (Core Authentication Service), DUO57 (Core Authentication Service), DUO58 (Core Authentication Service), DUO60 (Core Authentication Service), DUO62 (Core Authentication Service), DUO63 (Core Authentication Service), DUO64 (Core Authentication Service), DUO65 (Core Authentication Service), DUO70 (Core Authentication Service), DUO71 (Core Authentication Service), DUO72 (Core Authentication Service), DUO73 (Core Authentication Service), and DUO35 (Core Authentication Service).