DUO61: Limited User and Admin Panel Authentication Failures

Incident Report for Duo

Postmortem

Summary

On February 10, 2022, at around 16:05 EST, Duo’s Engineering Team was made aware of two customers who were unable to access the Duo Admin Panel and were failing to MultiFactor in the DUO61 deployment. No other customers or deployments were impacted.

The issue was resolved on the same day by updating Web Application Firewall (WAF) rules to allow for discrete Customer entry.

Deployments Impacted

DUO61

Timeline of Events EST

2022-02-12 16:05 Duo Site Reliability Engineering (SRE) is informed by Duo Customer Support (CS) that two customers are having issues accessing Admin Panel and are unsuccessful with MultiFactor Authentication. SRE begins triage.

2022-02-12 16:29 After collecting customer information and completing basic troubleshooting, Duo Support escalates to Engineering to begin incident mitigation processes

2022-02-12 16:52 Status page updated to Investigating.

2022-02-12 16:45 Duo SRE determines the cause to be the WAF Disallow List WAF has IP Address entries for the two customers who are affected. The incident is remediated by adding the Customer IPs to the Allow List.

2022-02-12 16:57 Duo CS confirms blocked IPs.

2022-02-12 17:19 Duo SRE updates the Allow list of the confirmed Customer IP Addresses.

2022-02-12 17:22 Status page updated to Monitoring.

2022-02-12 18:10 The customer confirms the incident is remediated.

2022-02-12 18:46 Status page updated to Resolved.

2022-02-12 19:02 Root cause identified.

2022-02-12 20:05 Root cause fixed.

Details

The root cause of these errors was determined to be information collected from the Chrome browser causing false positives, triggering the AWS WAF SQL injection rule. We modified our SQL injection rules to balance proper execution and identification without compromising the integrity of the rules. The root cause was identified and fixed. This incident was limited to two customers.

Upon completion of Updating the Allow List on the WAF, we were able to verify that the customer was successful in accessing the Admin Panel.

After remediation, Duo’s Engineering Team performed a deep-dive investigation on this issue and is working on implementing more discrete alerting and enhancing WAF Deny List Rules

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.

Posted Feb 11, 2022 - 17:35 EST

Resolved

The issue is now confirmed to be fully resolved. A root-cause analysis (RCA) will be provided as soon as it is available.
Posted Feb 10, 2022 - 18:46 EST

Monitoring

The issue has been resolved and all user authentications and admin panel logins are now working as expected. We are monitoring to ensure no recurrence of the issue.
Posted Feb 10, 2022 - 17:22 EST

Investigating

We are currently investigating an issue affecting a limited subset of users and admins from successfully authenticating.
Posted Feb 10, 2022 - 16:52 EST
This incident affected: DUO61 (Core Authentication Service, Admin Panel).