MultipleDeployments: AD Sync Issues

Summary

On March 3, 2022, at around 3:00 pm EST, Duo's Engineering Team was alerted by monitoring that azure directory syncs were failing for some customers. The root cause was identified as the Duo Directory Sync process encountering failures for certain API calls to the Microsoft Graph API used during Directory Sync. Authentication was not impacted during this time period.

The issue was resolved on the same day by deploying a code change that removed the problematic API calls to the Microsoft Graph API during sync.

Deployments Impacted

• DUO9, DUO39, DUO40, DUO49, DUO50, DUO55, DUO56, DUO58, DUO62, DUO63, DUO57

Timeline of Events EST

• 2022-03-03 15:07 - Duo Engineering observes Azure AD syncs failing on select Duo deployments and starts investigating
• 2022-03-03 16:26 - Engineering submitted updates for a complete workaround as Engineering narrowed down the scope of failures
• 2022-03-03 17:58 - Status page updated to Investigating.
• 2022-03-03 18:18 - Scope of affected deployments was identified. Engineering began planning for immediate deployment. Status page updated to Identified.
• 2022-03-03 19:00 - Plan for pushing the fix to affected deployments complete.
• 2022-03-04 00:30 - Fix pushed to all affected deployments.
• 2022-03-04 00:35 - Status page updated to Monitoring.
• 2022-03-04 03:42 - Status page updated to Resolved.

Details

Duo Engineering observed Azure AD full Directory Syncs failing. Only Customers that use Microsoft Azure AD were impacted in our identified Deployment list. Duo Engineering eliminated unnecessary API calls to Microsoft Graph endpoint that were occasionally failing. The unnecessary API calls will no longer be part of the sync process, so future failures from these endpoints will not affect directory sync. We are improving our monitoring to detect this type of issue earlier.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.