All Deployments: Investigating reports of Azure CA Authentication Failures
Postmortem
Azure Conditional Access Issues - All Deployments
Incident Report - 2018/11/29
Summary:
From 01:14 UTC to 02:54 UTC on November 30th, all Duo deployments experienced intermittent failures with the Azure Conditional Access (CA) integration. During this window, the majority of Azure CA authentication requests failed for customers using this integration, accounting for 3% of total authentication requests during that time. The root cause of this outage has been identified and resolved to prevent similar issues going forward.
Details:
The Duo SaaS platform integrates with Azure CA as a custom control. Authentication requests are routed to Duo from Microsoft Azure based on a CA rule, and Duo receives a request as a JSON Web Token (JWT). In order to reject fraudulent authentication requests from services sending otherwise legitimate JWTs, we implemented a whitelist of domains, accepting authentication requests only from trustworthy domains managed by Microsoft.
At 02:02 UTC on November 30th, customers reached out to Duo support to report that the Azure CA integration was failing to process user authentication requests. Upon further investigation, we determined that these authentication failures were due to erroneously rejected JWTs and that these failures had started occurring at 01:14 UTC. This timeline corresponded with a dramatic uptick in authentication requests from a new domain owned by Microsoft. At 02:38 UTC, our engineering team determined that the JWTs were rejected as invalid because the domain issuer of the tokens did not match the whitelisted domain on Duo’s side.
At 02:47 UTC, we implemented a change to accept the additional domain issuer from Microsoft as valid JWTs and restarted the Azure CA integration service across Duo deployments. By 02:54 UTC, we confirmed that authentications were being successfully processed and completed.
Additional monitoring improvements have been implemented in order to allow Duo’s systems to better detect this scenario in the future prior to customer impact. We have also opened discussions with our partner Microsoft regarding improved SLA for updates regarding Azure, so customers do not experience service disruptions.
Posted Dec 03, 2018 - 17:05 EST
Resolved
Our changes have been completed and the issue is resolved.
Posted Nov 29, 2018 - 22:09 EST
Identified
Our Engineering Team has identified the cause of the error with users being unable to authenticate to Azure Conditional Access on our deployments and are actively working to restore service. An update has been deployed and is being rolled out.
Please check back here or subscribe to updates for any changes.
Posted Nov 29, 2018 - 21:53 EST
Investigating
We are currently investigating an issue causing an error with users being unable to authenticate to Azure Conditional Access on our all of our deployments and are working to correct the issue as soon as possible.
Please check back here or subscribe to updates for any changes.
Posted Nov 29, 2018 - 21:14 EST
This incident affected: DUO1 (Core Authentication Service), DUO2 (Core Authentication Service), DUO3 (Core Authentication Service), DUO4 (Core Authentication Service), DUO5 (Core Authentication Service), DUO6 (Admin Panel, Core Authentication Service), DUO7 (Core Authentication Service), DUO8 (Core Authentication Service), DUO47 (Core Authentication Service), DUO10 (Core Authentication Service), DUO11 (Core Authentication Service), DUO12 (Core Authentication Service), DUO13 (Core Authentication Service), DUO14 (Core Authentication Service), DUO15 (Core Authentication Service), DUO16 (Core Authentication Service), DUO17 (Core Authentication Service), DUO18 (Core Authentication Service), DUO19 (Core Authentication Service), DUO20 (Core Authentication Service), DUO21 (Core Authentication Service), DUO22 (Core Authentication Service), DUO23 (Core Authentication Service), DUO24 (Core Authentication Service), DUO25 (Core Authentication Service), DUO26 (Core Authentication Service), DUO27 (Core Authentication Service), DUO28 (Core Authentication Service), DUO29 (Core Authentication Service), DUO30 (Core Authentication Service), DUO31 (Core Authentication Service), DUO32 (Core Authentication Service), DUO33 (Core Authentication Service), DUO34 (Core Authentication Service), DUO36 (Core Authentication Service), DUO37 (Core Authentication Service), DUO38 (Core Authentication Service), DUO39 (Core Authentication Service), DUO40 (Core Authentication Service), DUO41 (Core Authentication Service), DUO42 (Core Authentication Service), DUO43 (Core Authentication Service), DUO44 (Core Authentication Service), DUO45 (Core Authentication Service), DUO46 (Core Authentication Service), DUO48 (Core Authentication Service), DUO9 (Core Authentication Service), DUO49 (Core Authentication Service), DUO50 (Core Authentication Service), DUO51 (Core Authentication Service), DUO52 (Core Authentication Service), DUO53 (Core Authentication Service), DUO54 (Core Authentication Service), DUO55 (Core Authentication Service), DUO56 (Core Authentication Service), DUO57 (Core Authentication Service), DUO58 (Core Authentication Service), DUO59 (Core Authentication Service), DUO60 (Core Authentication Service), DUO61 (Core Authentication Service), and DUO35 (Core Authentication Service).