Device Health Application detection issues for Chromium-based browsers
Incident Report for Duo


On September 19, 2022, Google begins rolling out Chrome Root Store and Certificate validation program to Chrome 105 on macOS and Windows. On September 23, 2022, at around 12:48 EST, the Duo Care Team was alerted that a new certificate store was released for Mac and Windows and the Duo Health Application (DHA) was not being recognized as a trusted certificate store. 

The issue was resolved on September 23, 2022, after rolling out updates to manage the issue.

Timeline of Events EST

2022-09-19 00:00 Google begins rolling out Chromium Secure Store and Certificate validation program to Chrome 105 on macOS and Windows.

2022-09-23 12:00 Internal logs indicate we are beginning to see a trend of increased DHA failures, indicating a potential issue.

2022-09-23 12:15 A number of support escalations surface with invalid certificate issues on Windows.

2022-09-23 12:48 Duo Engineering (DE) is made aware that Chromium is rolling out a new Certificate Store, which lines up with the problems we are seeing.

2022-09-23 14:14 After confirming the issue, Engineering identifies and tests a solution for the problem with local testing. This solution is rolled out internally to confirm that it is a fix.

2022-09-23 14:49 After confirming the efficacy of the solution internally, Engineering begins enabling it for select users.

2022-09-23 15:51 Engineering gets confirmation from a customer that the fix has resolved the issue.

2022-09-23 17:46 Engineering continues to monitor the solution’s effect on customers who have it enabled and begins planning a mass rollout.

2022-09-23 19:28 Status page updated to Identified.

2022-09-23 21:08 A fix is manually enabled for the customers most impacted according to our logs.

2022-09-24 10:21 Begin rollout of fix to all customers for MFA prompt.

2022-09-24 12:04 Passwordless Engineering is brought in after realizing the Passwordless product is affected separately.

2022-09-25 09:04 Passwordless fix is validated.

2022-09-25 09:25 Begin rolling out Passwordless fix to all customers.

2022-09-25 11:24 Engineering continues monitoring the fix in production as it rolls out.

2022-09-25 21:54 Fix release is completed for all customers.

2022-09-26 01:00 Status page updated to Monitoring.

2022-09-27 02:09 Status page updated to Resolved.


Chrome introduced a new program for the Chrome Root Store and Certificate validation. When this started slowly rolling out to users, we were alerted that Device Health Application (DHA)’s HTTPS localhost communication was failing. This is similar to the behavior we see today in Firefox, where Chrome started rejecting our localhost certificate and thus causing HTTPS localhost calls from the prompt to the app to fail.

We use feature flags to quickly manage application behavior. Engineering created a specific flag to manage DHA and Chromium HTTP connection, which allows us to use HTTP localhost as a fallback if HTTPS fails for whatever reason in Chromium-based browsers. We enabled this feature flag for all customers but realized that the passwordless flow wasn’t consuming it. We fixed the passwordless issue and rolled out the passwordless fix to customers.

This impacts all Access and Beyond customers that require DHA as part of their policy. It’s difficult to know the exact count, but we did receive direct communication from a few impacted customers. Any customer using Device Health as Trusted or enforcing any Health State via Device Health policy would be affected as their users would be blocked for being unable to complete a health check. Our release strategy allows us to identify issues early on. In this case, we were able to get customer feedback and were able to mitigate the issue before it had a larger customer impact.

This issue underscores a need for better monitoring of vendor announcements within Duo. Our software interacts with and relies upon software from Google, Apple, Microsoft, Mozilla, and others. We need to do a better job of monitoring announcements about changes and surfacing them to teams that may be impacted.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.

Posted Sep 27, 2022 - 02:20 EDT

We have finished monitoring the applied fix, and all the Device Health applications for Chromium-based browsers on the latest version of macOS and Windows are now operational. We will provide an RCA as soon as it is available.
Posted Sep 27, 2022 - 02:09 EDT
We have finished rolling out the fix to all customers. We are currently monitoring the performance of the solution.
Posted Sep 26, 2022 - 01:00 EDT
We have identified the communication issue between the Duo prompt and the Device Health Application for Chromium-based browsers on macOS and Windows, which will make it so that users running the latest version of the browsers will be prompted to install the Device Health Application even when it is already installed, and will not allow authentication, if your Policy settings require it.

The issue is caused by a new Chrome certificate store and acceptable Certificate Authorities.

Google is rolling out this specific change to their certificate store as a rolling release, so not everyone on the latest version of Chrome has this change yet.

Other browsers, such as Firefox and Safari, are currently unaffected. All other services are fully operational and customers who do not use the Device Health Application are unaffected.

We are currently working on rolling out a fix for all customers.
Posted Sep 23, 2022 - 19:28 EDT