On February 10, 2022, at around 16:05 EST, Duo’s Engineering Team was made aware of two customers who were unable to access the Duo Admin Panel and were failing to MultiFactor in the DUO61 deployment. No other customers or deployments were impacted.
The issue was resolved on the same day by updating Web Application Firewall (WAF) rules to allow for discrete Customer entry.
DUO61
2022-02-12 16:05 Duo Site Reliability Engineering (SRE) is informed by Duo Customer Support (CS) that two customers are having issues accessing Admin Panel and are unsuccessful with MultiFactor Authentication. SRE begins triage.
2022-02-12 16:29 After collecting customer information and completing basic troubleshooting, Duo Support escalates to Engineering to begin incident mitigation processes
2022-02-12 16:52 Status page updated to Investigating.
2022-02-12 16:45 Duo SRE determines the cause to be the WAF Disallow List WAF has IP Address entries for the two customers who are affected. The incident is remediated by adding the Customer IPs to the Allow List.
2022-02-12 16:57 Duo CS confirms blocked IPs.
2022-02-12 17:19 Duo SRE updates the Allow list of the confirmed Customer IP Addresses.
2022-02-12 17:22 Status page updated to Monitoring.
2022-02-12 18:10 The customer confirms the incident is remediated.
2022-02-12 18:46 Status page updated to Resolved.
2022-02-12 19:02 Root cause identified.
2022-02-12 20:05 Root cause fixed.
The root cause of these errors was determined to be information collected from the Chrome browser causing false positives, triggering the AWS WAF SQL injection rule. We modified our SQL injection rules to balance proper execution and identification without compromising the integrity of the rules. The root cause was identified and fixed. This incident was limited to two customers.
Upon completion of Updating the Allow List on the WAF, we were able to verify that the customer was successful in accessing the Admin Panel.
After remediation, Duo’s Engineering Team performed a deep-dive investigation on this issue and is working on implementing more discrete alerting and enhancing WAF Deny List Rules
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.