All Deployments: Issue with Delivery of Phone Call Authentications
Incident Report for Duo
Postmortem

United States Telephony Outage - All Deployments

Incident Report - 2019/11/08

Summary

On Thursday, November 7, 2019 from 10:30 AM EST to 1:46 PM EST, authentication and enrollment phone calls to US-based telephone numbers intermittently failed. During this time US telephony carriers rejected some calls made from Duo’s single outbound domestic number. At 1:46 PM EST, Duo completed a change to expand the pool out of outbound domestic numbers used for telephony.

Details

Duo’s platform uses a default telephone number for US-based phone calls. Because of the nature of Duo’s service, we place high volumes of short-duration calls. These calls can mimic the style and format of unwanted robocalls and can trigger automatic mitigation at various carriers.

On Friday, November 7, 2019 at 10:30 AM EST, two US telephony carriers started rejecting calls made from Duo’s new domestic number (+1-669-201-9315). Upon being notified of these issues, Duo’s engineering team immediately began provisioning additional outbound numbers and engaged our telephony partners to directly contact the affected carriers.

Duo’s Engineering team completed a change at 1:46 PM EST to utilize the additional domestic numbers for outbound telephony service. Telephony service returned to normal operations upon completion of this change.

We understand that the sudden change to a new number may affect customers who have whitelisted or performed internal user education to expect the old number. We did not make the decision lightly and chose to prioritize immediate restoration of telephony quality. The full list of possible numbers can be found at https://help.duo.com/s/article/5516?language=en_US. We realize that this is a very large pool of numbers and customers may have difficulty whitelisting all of the numbers provided. The link above also contains instructions to set an outbound number in the Duo Admin Panel if desired.

Duo’s engineering team already had initiated a project prior to this incident to expand the resiliency of our telephony services. These events have added further aspects for us to incorporate into this work. Additionally, we continue to work with our telephony providers and their partners to mitigate this from reoccurring in the future.

Posted Nov 09, 2019 - 12:35 EST

Resolved
After yesterday's change to the new caller ID number for phone call authentications, we have observed issues with calls to U.S. devices being blocked by several carriers.

We have reached out to our telephony providers to whitelist the new numbers. We have also added more numbers from which the phone call authentication can originate. The Knowledge Base article that lists the new numbers is here:

https://help.duo.com/s/article/5516?language=en_US

For more information on yesterday's incident, please see https://status.duo.com/incidents/pb4twk50clsl.

An RCA will be released detailing the issue once available.

Please check back here or subscribe to updates for any changes.
Posted Nov 08, 2019 - 17:12 EST
Update
After yesterday's change to the new caller ID number for phone call authentications, we have observed issues with calls to U.S. devices being blocked by several carriers.

We have reached out to our telephony providers to whitelist the new number. We are additionally adding more numbers from which the phone call authentication can originate. The Knowledge Base Article that lists the new numbers is here:

https://help.duo.com/s/article/5516?language=en_US

For more information on yesterday's incident, please see https://status.duo.com/incidents/pb4twk50clsl.

Please check back here or subscribe to updates for any changes.
Posted Nov 08, 2019 - 13:56 EST
Monitoring
After yesterday's change to the new caller ID number for phone call authentications, we have observed issues with calls to U.S. devices being blocked by several carriers.

We have reached out to our telephony providers to whitelist the new number. We are additionally adding more numbers from which the phone call authentication can originate. The Knowledge Base Article that lists the new numbers is here:

https://duo.my.salesforce.com/articles/FAQ/5516

For more information on yesterday's incident, please see https://status.duo.com/incidents/pb4twk50clsl.

Please check back here or subscribe to updates for any changes.
Posted Nov 08, 2019 - 13:51 EST
Identified
After yesterday's change to the new caller ID number for phone call authentications, we have observed issues with calls to U.S. devices being blocked by several carriers.

We have reached out to our telephony providers to whitelist the new number. We are additionally adding more numbers from which the phone call authentication can originate, and will update this post with those numbers as they're available.

For more information on yesterday's incident, please see https://status.duo.com/incidents/pb4twk50clsl.

Please check back here or subscribe to updates for any changes.
Posted Nov 08, 2019 - 13:13 EST
Investigating
After yesterday's change to the new caller ID number for phone call authentications, we have observed issues with calls to U.S. devices being blocked by several carriers.

We have reached out to our telephony providers to whitelist the new number.

For more information on yesterday's incident, please see https://status.duo.com/incidents/pb4twk50clsl.

Please check back here or subscribe to updates for any changes.
Posted Nov 08, 2019 - 12:17 EST
This incident affected: DUO45 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO63 (Phone Call Delivery), DUO31 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO52 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO48 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO54 (Phone Call Delivery), DUO49 (Phone Call Delivery), DUO32 (Phone Call Delivery), DUO9 (Phone Call Delivery), DUO53 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO60 (Phone Call Delivery), DUO1 (Phone Call Delivery), DUO14 (Phone Call Delivery), DUO40 (Phone Call Delivery), DUO30 (Phone Call Delivery), DUO62 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO64 (Phone Call Delivery), DUO65 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO44 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO61 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO43 (Phone Call Delivery), DUO41 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO39 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO51 (Phone Call Delivery), DUO57 (Phone Call Delivery), DUO38 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO42 (Phone Call Delivery), DUO33 (Phone Call Delivery), DUO29 (Phone Call Delivery), DUO46 (Phone Call Delivery), DUO50 (Phone Call Delivery), DUO37 (Phone Call Delivery), DUO36 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO34 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO58 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO55 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO56 (Phone Call Delivery), DUO59 (Phone Call Delivery), and DUO35 (Phone Call Delivery).