MultipleDeployments: AD Sync Issues
Incident Report for Duo
Postmortem

Summary

On March 3, 2022, at around 3:00 pm EST, Duo's Engineering Team was alerted by monitoring that azure directory syncs were failing for some customers. The root cause was identified as the Duo Directory Sync process encountering failures for certain API calls to the Microsoft Graph API used during Directory Sync. Authentication was not impacted during this time period.

The issue was resolved on the same day by deploying a code change that removed the problematic API calls to the Microsoft Graph API during sync.

Deployments Impacted

  • DUO9, DUO39, DUO40, DUO49, DUO50, DUO55, DUO56, DUO58, DUO62, DUO63, DUO57

Timeline of Events EST

  • 2022-03-03 15:07 - Duo Engineering observes Azure AD syncs failing on select Duo deployments and starts investigating
  • 2022-03-03 16:26 - Engineering submitted updates for a complete workaround as Engineering narrowed down the scope of failures
  • 2022-03-03 17:58 - Status page updated to Investigating.
  • 2022-03-03 18:18 - Scope of affected deployments was identified. Engineering began planning for immediate deployment. Status page updated to Identified.
  • 2022-03-03 19:00 - Plan for pushing the fix to affected deployments complete.
  • 2022-03-04 00:30 - Fix pushed to all affected deployments.
  • 2022-03-04 00:35 - Status page updated to Monitoring.
  • 2022-03-04 03:42 - Status page updated to Resolved.

Details

Duo Engineering observed Azure AD full Directory Syncs failing. Only Customers that use Microsoft Azure AD were impacted in our identified Deployment list. Duo Engineering eliminated unnecessary API calls to Microsoft Graph endpoint that were occasionally failing. The unnecessary API calls will no longer be part of the sync process, so future failures from these endpoints will not affect directory sync. We are improving our monitoring to detect this type of issue earlier.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.

Posted Mar 09, 2022 - 11:27 EST

Resolved
We have completed monitoring of the Azure sync and confirmed that services have returned to normal. We have not seen any evidence of ongoing issues.
Posted Mar 04, 2022 - 03:42 EST
Monitoring
Our engineering team successfully deployed a change to resolve the issue causing Azure AD Sync failures on the affected deployments.

We are continuing to monitor this issue.
Posted Mar 04, 2022 - 00:35 EST
Identified
We have identified the cause of issue causing Azure AD Sync to fail on the affected deployments. We are actively working to restore functionality.

Please check back here or subscribe to updates for any changes.
Posted Mar 03, 2022 - 18:18 EST
Investigating
We are currently investigating an issue with Azure AD Sync failures on multiple deployments. We are working to correct the issue as soon as possible.

Affected customers may see the following error in the Azure Directory Sync page in the Duo Admin Panel when running a Full Sync:
"Sync error. Synced 0 users and 0 groups. (1 error; see Administrator Actions report.)"

Syncing individuals users is not impacted.

Authentications are not impacted and logins to Duo protected integrations will continue as expected.

Please check back here or subscribe to updates for any changes.
Posted Mar 03, 2022 - 17:58 EST
This incident affected: DUO50 (Admin Panel), DUO55 (Admin Panel), DUO49 (Admin Panel), DUO62 (Admin Panel), DUO63 (Admin Panel), DUO40 (Admin Panel), DUO39 (Admin Panel), DUO56 (Admin Panel), and DUO57 (Admin Panel).