All Deployments: Issue with Delivery of Phone Call Authentications
Incident Report for Duo
Postmortem

United States VOIP Outage - All Deployments

Incident Report - 2019/11/07

Summary

On Thursday, November 7, 2019 from 10:05 AM EST to 4:09 PM EST, authentication and enrollment phone calls to US-based telephone numbers intermittently failed. During this time US telephony carriers rejected some calls made from Duo’s toll-free domestic number. At 4:09 PM EST, Duo completed a change to a non-toll-free number and normal telephony functionality was restored.

Details

Duo’s platform uses a default telephone number for US-based phone calls. This telephone number follows a toll-free format. Because of the nature of Duo’s service, we place high volumes of short-duration calls. In concert with a toll-free number, these calls can mimic the style and format of unwanted robocalls.

On Thursday, November 7, 2019 at 10:05 AM EST, a number of US telephony carriers started rejecting calls made from Duo’s toll-free domestic number. Upon being notified of these issues, Duo’s engineering team immediately began triaging the issue and exploring alternatives. Upon consulting with our downstream telephony providers, we decided that the best course of action was to switch to using a non-toll-free number (+1-669-201-9315).

After testing out this solution on a small subset of affected customers, we applied the change broadly. This immediately restored telephony service to normal operations upon completion at 4:09 PM EST.

We understand that the sudden change to a new number may affect customers who have whitelisted or performed internal user education to expect the old number. We did not make the decision lightly and chose to prioritize immediate restoration of telephony quality.

Duo’s engineering team already had initiated a project prior to this incident to expand the resiliency of our telephony services. Today’s events have added further aspects for us to incorporate into this work. Additionally, we continue to work with our telephony providers to determine whether it will be possible to safely and reliably switch back to the old number. If so, we will provide advance communication to our customers about these changes.

Posted Nov 08, 2019 - 09:54 EST

Resolved
We can confirm that the phone authentication delivery issue has been remediated. The issue will continue to be monitored to ensure it is completely resolved.

Due to issues at the carrier level, it is not recommend to modify the outbound caller ID in the Admin Panel to reflect the old phone number. The new caller ID that your users will see is +1-669-201-9315.

An RCA will be posted detailing the issue once available.

Please check back here or subscribe for further updates.
Posted Nov 07, 2019 - 20:30 EST
Monitoring
The change made to update the caller ID from which phone call authentications are sent has been deployed, and we have seen the number of failed phone call authentications return to normal at this time.

An RCA will be posted detailing the issue once available.

Please check back here or subscribe for further updates.
Posted Nov 07, 2019 - 16:41 EST
Update
We have made a change to the default number from which phone call authentications are sent.

The new caller ID that users will see is +1-669-201-9315.

Please check back here or subscribe for any further updates.
Posted Nov 07, 2019 - 16:08 EST
Identified
We have made a change to the default number from which phone call authentications are sent.

The new caller ID that users will see is +1-669-201-9315.

Please check back here or subscribe for any further updates.
Posted Nov 07, 2019 - 15:33 EST
Update
We are continuing to investigate the issue affecting phone call authentication and are working with our telephony providers and carriers on a resolution.

Please check back here or subscribe for more updates.
Posted Nov 07, 2019 - 13:18 EST
Update
We are continuing to investigate the issue affecting delivery of phone call authentications and are in the process of making changes to our telephony routing to resolve the problem.

Please check back here or subscribe for any further updates.
Posted Nov 07, 2019 - 12:43 EST
Investigating
We are receiving reports of phone call authentication attempts not arriving to user's devices and are currently working to determine the cause of this issue.

Please check back here or subscribe to updates for any changes.
Posted Nov 07, 2019 - 11:53 EST
This incident affected: DUO9 (Phone Call Delivery), DUO49 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO54 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO48 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO52 (Phone Call Delivery), DUO63 (Phone Call Delivery), DUO56 (Phone Call Delivery), DUO58 (Phone Call Delivery), DUO34 (Phone Call Delivery), DUO30 (Phone Call Delivery), DUO37 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO31 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO61 (Phone Call Delivery), DUO32 (Phone Call Delivery), DUO40 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO41 (Phone Call Delivery), DUO39 (Phone Call Delivery), DUO42 (Phone Call Delivery), DUO29 (Phone Call Delivery), DUO46 (Phone Call Delivery), DUO50 (Phone Call Delivery), DUO36 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO44 (Phone Call Delivery), DUO45 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO59 (Phone Call Delivery), DUO65 (Phone Call Delivery), DUO64 (Phone Call Delivery), DUO62 (Phone Call Delivery), DUO43 (Phone Call Delivery), DUO33 (Phone Call Delivery), DUO14 (Phone Call Delivery), DUO57 (Phone Call Delivery), DUO51 (Phone Call Delivery), DUO1 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO55 (Phone Call Delivery), DUO38 (Phone Call Delivery), DUO60 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO53 (Phone Call Delivery), and DUO35 (Phone Call Delivery).