Widespread Telephony Authentication Issue
Incident Report for Duo
Postmortem

On October 7, 2015, between approximately 1:10PM UTC and 1:47PM UTC, Duo Security users were unable to authenticate via phone callback and SMS passcodes due to an outage at our primary telephony provider. Users were able to authenticate using other methods such as Duo Push, Duo Mobile generated passcodes, and hardware tokens.

At 1:47 UTC Duo re-routed calls through a different provider, restoring service. Some users later received phone calls that read them the following error message: "An application error has occurred. Goodbye."

Duo automatically detects and re-routes requests if a telephony provider fails to accept the call. During this event, calls were not automatically re-routed because the provider accepted the calls without indicating there was an error. We have deployed monitoring to detect this specific type of dropped call and are improving general dropped call detection to quickly identify similar issues moving forward.

On October 8, 2015, Duo's telephony provider found the root cause (a configuration issue on their side) and began working to correct it. We continue to work closely with them to ensure successful remediation.

We sincerely apologize and recognize the impact this service interruption may have had on your operations. High-availability access to our systems is a top priority at Duo Security and we will continue to make improvements to our product and support processes.

Posted Oct 08, 2015 - 17:07 EDT

Resolved
We are changing the status of our Telephony Authentication issue to Resolved.

We are developing a post-mortem on our status page once root cause analysis has been completed.
Posted Oct 07, 2015 - 16:21 EDT
Update
We are still Monitoring the service, and current attempts for Telephony Authentications are completing successfully.

Authentication attempts that occurred during the outage may cause the end-user to receive a phone call containing the message "An application error has occurred. Good Bye". We are currently working on a way to stop these expired authentication calls.
Posted Oct 07, 2015 - 11:17 EDT
Monitoring
We have restored Telephony Authentication service, and will continue to monitor the situation.
Posted Oct 07, 2015 - 10:50 EDT
Identified
We have identified an issue with one of our primary telephony providers, and have begun routing calls to a fallback provider.
Posted Oct 07, 2015 - 10:12 EDT
Investigating
We are currently experiencing some issues with Telephony Authentications. We are actively investigating the root cause of this issue.
We will update our status as soon as this has been resolved.
Posted Oct 07, 2015 - 09:54 EDT
This incident affected: DUO1 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO14 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO29 (Phone Call Delivery), and DUO30 (Phone Call Delivery).