On December 3, 2021 at around 10am ET, Duo’s Engineering Team was made aware of intermittent errors on Admin Panel’s Reporting pages that prevented authentication logs from being displayed properly on the UI. All other authentications and applications were unaffected and were operating as normal during this time.
The issue was resolved on the same day by a code change deployed for all affected customers.
The root cause of these errors was determined to be a bug which caused our reporting APIs to fail intermittently on requests. Duo’s Engineering Team conducted an investigation to identify the root cause of the issue and corrected the problem with a code change. The fix was deployed to all impacted deployments by around 6:55pm ET on December 3.
The bug affected the following deployments - DUO9, DUO17, DUO22, DUO39, DUO40, DUO42, DUO45, DUO49, DUO50, DUO52, DUO55, DUO56, DUO58, DUO62, DUO63, DUO64, and DUO65.
Upon completion of deploying the fix, we were able to verify that the Admin Panel reporting pages were working as expected. Duo’s Engineering Team monitored the event for a couple of days and set the status of the issue to be resolved by around 8:00pm ET on December 5 as no further issues were detected or reported.
After remediation, Duo’s Engineering Team performed a deep-dive investigation on this issue and is working on implementing additional logging, monitoring and testing in our services to more proactively avoid and detect such problems in the future.
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.