Multiple Deployments: Intermittent Admin Panel Reporting Issues
Incident Report for Duo
Postmortem

Intermittent Errors on Duo Admin Panel Reporting pages

Summary

On December 3, 2021 at around 10am ET, Duo’s Engineering Team was made aware of intermittent errors on Admin Panel’s Reporting pages that prevented authentication logs from being displayed properly on the UI. All other authentications and applications were unaffected and were operating as normal during this time.

The issue was resolved on the same day by a code change deployed for all affected customers.

Details

The root cause of these errors was determined to be a bug which caused our reporting APIs to fail intermittently on requests. Duo’s Engineering Team conducted an investigation to identify the root cause of the issue and corrected the problem with a code change. The fix was deployed to all impacted deployments by around 6:55pm ET on December 3.

The bug affected the following deployments - DUO9, DUO17, DUO22, DUO39, DUO40, DUO42, DUO45, DUO49, DUO50, DUO52, DUO55, DUO56, DUO58, DUO62, DUO63, DUO64, and DUO65.

Upon completion of deploying the fix, we were able to verify that the Admin Panel reporting pages were working as expected. Duo’s Engineering Team monitored the event for a couple of days and set the status of the issue to be resolved by around 8:00pm ET on December 5 as no further issues were detected or reported.

After remediation, Duo’s Engineering Team performed a deep-dive investigation on this issue and is working on implementing additional logging, monitoring and testing in our services to more proactively avoid and detect such problems in the future.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.

Posted Dec 10, 2021 - 15:03 EST

Resolved
The issues with intermittent Admin Panel reporting have been resolved. An RCA will be provided as soon as it is available.
Posted Dec 05, 2021 - 20:00 EST
Monitoring
We have implemented a fix that has resolved the issue, and are monitoring to ensure no further occurrences are observed.
Posted Dec 03, 2021 - 19:08 EST
Identified
The issue has been identified and a fix is being implemented.
Posted Dec 03, 2021 - 14:36 EST
Investigating
We are currently investigating an issue that is causing intermittent errors on the reporting pages of the Duo Admin Panel for the following deployments:

DUO9
DUO17
DUO22
DUO39
DUO40
DUO42
DUO45
DUO49
DUO50
DUO52
DUO55
DUO56
DUO58
DUO62
DUO63
DUO64
DUO65

We are working urgently on correcting this issue, and will provide an update as soon as it is available. All other authentications and applications are unaffected and are operating as normal.
Posted Dec 03, 2021 - 14:20 EST
This incident affected: DUO39 (Admin Panel), DUO22 (Admin Panel), DUO17 (Admin Panel), DUO56 (Admin Panel), DUO9 (Admin Panel), DUO58 (Admin Panel), DUO50 (Admin Panel), DUO64 (Admin Panel), DUO52 (Admin Panel), DUO55 (Admin Panel), DUO65 (Admin Panel), DUO40 (Admin Panel), DUO45 (Admin Panel), DUO63 (Admin Panel), DUO62 (Admin Panel), DUO49 (Admin Panel), and DUO42 (Admin Panel).