DUO55: Issue with Native LDAP integrations communicating to Duo
Incident Report for Duo
Postmortem

Summary

On February 28, 2022, at around 15:53 EST, Duo's Engineering Team was alerted by monitoring that customers with Cisco ASA LDAP integrations on DUO55 were unable to connect to Duo. The root cause was identified as a change to the root CA for a certificate used to verify connectivity between Cisco ASA devices and Duo's service.

The issue was resolved on the same day by restoring the previous trusted CA.

Deployments Impacted

DUO55 (impacting only three customers)

Timeline of Events EST

  • 2022-02-28 15:53 - Duo Site Reliability Engineering (SRE) is informed by Duo Customer Support (CS) that three customers are reporting issues with Cisco ASA devices failing to connect to Duo. SRE begins triage.
  • 2022-02-28 16:29 - After collecting customer information and completing basic troubleshooting, Duo Support escalates to Engineering to begin incident mitigation processes.
  • 2022-02-28 16:52 - Status page updated to Investigating.
  • 2022-02-28 16:45 - Duo SRE determines the rollout of a new certificate was the cause of the issue.
  • 2022-02-28 17:01 - Duo SRE initiates rollback to the previous certificate.
  • 2022-02-28 17:22 - Status page updated to Monitoring.
  • 2022-02-28 17:50 - Duo SRE starts the deployment of the previous certificate.
  • 2022-02-28 18:10 - Customer and internal testing confirm the incident is remediated.
  • 2022-02-28 18:46 - Status page updated to Resolved.
  • 2022-02-28 19:02 - Root cause identified.
  • 2022-02-28 20:04 - Root cause fixed.

Details

Duo SRE had recently rolled out a renewed X509 certificate with a change in the certificate chain as part of the certificate renewal process. There was an update in the root CA that Digicert uses to sign the new certificate request. The previous certificate was based on "CN=DigiCert High Assurance EV Root CA" while the new certificate was based on "CN=DigiCert Global Root CA". If the new root CA was missing in the trusted certificates pool relied on by ASA devices, then communication would fail with the LDAP endpoint.

After discovering the change in root CA, Duo updated public documentation (KB article) to reflect the new structure of the certificate chain.

As a short-term solution, Duo rolled back to the previous certificate signed by DigiCert High Assurance EV Root CA, which resolved connectivity issues. Duo is working to reissue the certificate so that it continues to work with the original root CA, in this case, DigiCert High Assurance EV Root CA, rather than the Global Root CA.

This incident was limited to three customers.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.

Posted Mar 09, 2022 - 11:20 EST

Resolved
After monitoring issues affecting authentication with LDAPS integrations, our engineers have mitigated the issue by deploying a solution. If anyone is still experiencing issues, please contact us at support@duo.com
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Posted Feb 28, 2022 - 22:46 EST
Monitoring
The issue has been resolved and we are seeing successful communications from the affected LDAPS integrations. We will actively keep this incident in monitoring mode to observe for any recurrence before resolving.
Posted Feb 28, 2022 - 19:49 EST
Identified
We have identified the source of the issue and are working on a resolution.

Affected Cisco ASA LDAPS integrations can work around this issue by ensuring the following two DigiCert CA certs are trusted by the ASA:
https://cacerts.digicert.com/DigiCertHighAssuranceEVRootCA.crt
https://cacerts.digicert.com/DigiCertGlobalRootCA.crt

Log on to your Cisco ASA administrator web interface (ASDM).
Click the Configuration tab and then click Device Management in the left menu.
Navigate to Certificate Management → CA Certificates.
Click the Add button.
In the "Install Certificate" window, select the Install from a file option and then click the "Browse..." button.
Posted Feb 28, 2022 - 16:51 EST
Investigating
We are currently investigating an issue affecting a limited subset of users authenticating with Native LDAP integrations, such as our Cisco ASA LDAPS integration, failing to communicate to Duo's cloud.
Posted Feb 28, 2022 - 16:19 EST
This incident affected: DUO55 (Core Authentication Service).