On-Premises Active Directory Sync erroring out for some customers

Incident Report - 2020-10-26

Summary

On October 26, 2020, at 1:57 pm EDT, Duo’s Engineering Team was made aware of an issue with on-premises Active Directory syncs that resulted in some customers’ syncs failing to run. Customers on the following deployments were potentially affected: DUO38, DUO64, DUO55, and DUO66.

These failures resulted in some new users being unable to authenticate with Duo. Additionally, users who were supposed to be removed from Duo via directory sync may not have been removed as intended. After this issue was resolved, subsequent directory syncs added and removed users as intended.

Details

The root cause of this error was determined to be a bug introduced in Duo’s late October release, which began deploying to customers at 11:00 am EDT on October 26, 2020. This bug was introduced during updates to Duo’s directory sync code to remove reliance on a deprecated library.

A fix for the issue was added to the release and redeployed to all affected customers. This was finished by 5:24 pm EDT on October 26. Duo’s Engineering Team has enhanced automated test coverage in this area to prevent this issue in the future.

Posted Oct 27, 2020 - 15:38 EDT

Resolved

Following a monitoring period, we have confirmed that the issues impacting AD Sync are now fully resolved.

We will be posting a Root-Cause Analysis (RCA) to this incident as soon as it is available.

Posted Oct 26, 2020 - 20:23 EDT

Monitoring

Our engineering team successfully deployed a change to resolve the issue causing AD Sync failures on the affected deployments. We are continuing to monitor this issue.

Posted Oct 26, 2020 - 17:33 EDT

Identified

We have identified the cause of issue causing AD Sync to fail on the affected deployments. We are actively working to restore functionality.

Please check back here or subscribe to updates for any changes.

Posted Oct 26, 2020 - 15:30 EDT

Investigating

We are currently investigating an issue with AD Sync failures on multiple deployments. We are working to correct the issue as soon as possible.

Affected customers may see the following error in their Directory Sync settings page in the Duo Admin Panel:
"Test query using the configured Base DN Searching the directory failed, possibly due to an incorrect base DN."

Please check back here or subscribe to updates for any changes.

Posted Oct 26, 2020 - 14:50 EDT

This incident affected: DUO38 (Admin Panel), DUO55 (Admin Panel), DUO64 (Admin Panel), and DUO66 (Admin Panel).