United States Telephony Outage - All Deployments

Incident Report - 2019/11/08

Summary

On Thursday, November 7, 2019 from 10:30 AM EST to 1:46 PM EST, authentication and enrollment phone calls to US-based telephone numbers intermittently failed. During this time US telephony carriers rejected some calls made from Duo’s single outbound domestic number. At 1:46 PM EST, Duo completed a change to expand the pool out of outbound domestic numbers used for telephony.

Details

Duo’s platform uses a default telephone number for US-based phone calls. Because of the nature of Duo’s service, we place high volumes of short-duration calls. These calls can mimic the style and format of unwanted robocalls and can trigger automatic mitigation at various carriers.

On Friday, November 7, 2019 at 10:30 AM EST, two US telephony carriers started rejecting calls made from Duo’s new domestic number (+1-669-201-9315). Upon being notified of these issues, Duo’s engineering team immediately began provisioning additional outbound numbers and engaged our telephony partners to directly contact the affected carriers.

Duo’s Engineering team completed a change at 1:46 PM EST to utilize the additional domestic numbers for outbound telephony service. Telephony service returned to normal operations upon completion of this change.

We understand that the sudden change to a new number may affect customers who have whitelisted or performed internal user education to expect the old number. We did not make the decision lightly and chose to prioritize immediate restoration of telephony quality. The full list of possible numbers can be found at https://help.duo.com/s/article/5516?language=en_US. We realize that this is a very large pool of numbers and customers may have difficulty whitelisting all of the numbers provided. The link above also contains instructions to set an outbound number in the Duo Admin Panel if desired.

Duo’s engineering team already had initiated a project prior to this incident to expand the resiliency of our telephony services. These events have added further aspects for us to incorporate into this work. Additionally, we continue to work with our telephony providers and their partners to mitigate this from reoccurring in the future.