Phone Call Authentications Delayed
Incident Report for Duo Security
Postmortem

Summary:

From 14:47 UTC to 15:58 UTC on November 5, 2018, all DUO deployments experienced a delay in placing phone calls for customers using the phone callback authentication method. During this window, roughly 50% of phone callback requests were delayed long enough to cause corresponding authentications to timeout and subsequently fail. The root cause of this outage has been identified and resolved to prevent similar issues going forward.

Details:

The Duo SaaS platform uses many telephony providers for generating phone calls and SMS text messages. Our platform routes telephony interactions across these multiple providers to ensure phone calls and messages reach the intended recipient. If a request sent to one of these providers fails, it is automatically routed to another provider, protecting against provider specific service degradation.

Starting at 14:47 UTC on November 5, 2018, one of these providers began experiencing internal issues, which caused some requested phone calls to be significantly delayed. Duo’s monitoring systems notified our engineering team of these delays at 14:49 UTC. The engineering team investigated the issue and determined that phone callback requests were being processed successfully by Duo but were being queued in the provider’s system. The provider continued to accept requests from Duo without errors, preventing automatic failover from being triggered.

The Duo team initiated a manual failover to a secondary telephony provider for phone services on a test deployment. After successfully testing and verifying this process, the team triggered a manual failover across all deployments. This process completed at 15:58 UTC when all phone callback authentications began completing as expected. We have since confirmed that the provider in question has identified and resolved the root cause of these issues, and we have reverted our temporary failover configuration.

Additional monitoring improvements have been implemented in order to allow Duo’s monitoring systems to better detect this scenario in the future. We will also enhance our automated failover mechanisms to better handle situations where providers are incorrectly reporting successful operation.

Posted 7 days ago. Nov 07, 2018 - 15:00 EST

Resolved
The issue regarding phone call authentications is fully resolved and all services are now fully functional.

We will be posting a root-cause analysis (RCA) here once our engineering team has conducted a thorough investigation of the issue with our telephony providers.

Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted 9 days ago. Nov 05, 2018 - 12:00 EST
Update
Our migration to an alternative telephony provider has completed and phone call authentications are now completing successfully.

We will continue to monitor the issue and will post and updates when the incident is considered fully resolved.

Please check back here or subscribe here for further updates.
Posted 9 days ago. Nov 05, 2018 - 11:14 EST
Monitoring
Our migration to an alternative telephony provider has completed and phone call authentications are now completing successfully.
Posted 9 days ago. Nov 05, 2018 - 11:06 EST
Update
We are continuing to investigate this issue.
Posted 9 days ago. Nov 05, 2018 - 10:56 EST
Update
We are continuing to investigate this issue.
Posted 9 days ago. Nov 05, 2018 - 10:55 EST
Investigating
We are currently investigating an issue causing authentication errors with phone calls on all Duo deployments and are working to correct the issue as soon as possible.
Posted 9 days ago. Nov 05, 2018 - 10:36 EST
This incident affected: DUO14 (Phone Call Delivery), DUO59 (Phone Call Delivery), DUO44 (Phone Call Delivery), DUO31 (Phone Call Delivery), DUO38 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO43 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO33 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO45 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO36 (Phone Call Delivery), DUO50 (Phone Call Delivery), DUO46 (Phone Call Delivery), DUO29 (Phone Call Delivery), DUO42 (Phone Call Delivery), DUO39 (Phone Call Delivery), DUO41 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO40 (Phone Call Delivery), DUO32 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO9 (Phone Call Delivery), DUO51 (Phone Call Delivery), DUO53 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO61 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO57 (Phone Call Delivery), DUO37 (Phone Call Delivery), DUO30 (Phone Call Delivery), DUO55 (Phone Call Delivery), DUO34 (Phone Call Delivery), DUO58 (Phone Call Delivery), DUO52 (Phone Call Delivery), DUO56 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO48 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO54 (Phone Call Delivery), DUO49 (Phone Call Delivery), DUO60 (Phone Call Delivery), DUO1 (Phone Call Delivery), and DUO35 (Phone Call Delivery).