On September 19, 2022, Google begins rolling out Chrome Root Store and Certificate validation program to Chrome 105 on macOS and Windows. On September 23, 2022, at around 12:48 EST, the Duo Care Team was alerted that a new certificate store was released for Mac and Windows and the Duo Health Application (DHA) was not being recognized as a trusted certificate store.
The issue was resolved on September 23, 2022, after rolling out updates to manage the issue.
2022-09-19 00:00 Google begins rolling out Chromium Secure Store and Certificate validation program to Chrome 105 on macOS and Windows.
2022-09-23 12:00 Internal logs indicate we are beginning to see a trend of increased DHA failures, indicating a potential issue.
2022-09-23 12:15 A number of support escalations surface with invalid certificate issues on Windows.
2022-09-23 12:48 Duo Engineering (DE) is made aware that Chromium is rolling out a new Certificate Store, which lines up with the problems we are seeing.
2022-09-23 14:14 After confirming the issue, Engineering identifies and tests a solution for the problem with local testing. This solution is rolled out internally to confirm that it is a fix.
2022-09-23 14:49 After confirming the efficacy of the solution internally, Engineering begins enabling it for select users.
2022-09-23 15:51 Engineering gets confirmation from a customer that the fix has resolved the issue.
2022-09-23 17:46 Engineering continues to monitor the solution’s effect on customers who have it enabled and begins planning a mass rollout.
2022-09-23 19:28 Status page updated to Identified.
2022-09-23 21:08 A fix is manually enabled for the customers most impacted according to our logs.
2022-09-24 10:21 Begin rollout of fix to all customers for MFA prompt.
2022-09-24 12:04 Passwordless Engineering is brought in after realizing the Passwordless product is affected separately.
2022-09-25 09:04 Passwordless fix is validated.
2022-09-25 09:25 Begin rolling out Passwordless fix to all customers.
2022-09-25 11:24 Engineering continues monitoring the fix in production as it rolls out.
2022-09-25 21:54 Fix release is completed for all customers.
2022-09-26 01:00 Status page updated to Monitoring.
2022-09-27 02:09 Status page updated to Resolved.
Chrome introduced a new program for the Chrome Root Store and Certificate validation. When this started slowly rolling out to users, we were alerted that Device Health Application (DHA)’s HTTPS localhost communication was failing. This is similar to the behavior we see today in Firefox, where Chrome started rejecting our localhost certificate and thus causing HTTPS localhost calls from the prompt to the app to fail.
We use feature flags to quickly manage application behavior. Engineering created a specific flag to manage DHA and Chromium HTTP connection, which allows us to use HTTP localhost as a fallback if HTTPS fails for whatever reason in Chromium-based browsers. We enabled this feature flag for all customers but realized that the passwordless flow wasn’t consuming it. We fixed the passwordless issue and rolled out the passwordless fix to customers.
This impacts all Access and Beyond customers that require DHA as part of their policy. It’s difficult to know the exact count, but we did receive direct communication from a few impacted customers. Any customer using Device Health as Trusted or enforcing any Health State via Device Health policy would be affected as their users would be blocked for being unable to complete a health check. Our release strategy allows us to identify issues early on. In this case, we were able to get customer feedback and were able to mitigate the issue before it had a larger customer impact.
This issue underscores a need for better monitoring of vendor announcements within Duo. Our software interacts with and relies upon software from Google, Apple, Microsoft, Mozilla, and others. We need to do a better job of monitoring announcements about changes and surfacing them to teams that may be impacted.
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.