United States VOIP Outage - All Deployments

Incident Report - 2019/11/07

Summary

On Thursday, November 7, 2019 from 10:05 AM EST to 4:09 PM EST, authentication and enrollment phone calls to US-based telephone numbers intermittently failed. During this time US telephony carriers rejected some calls made from Duo’s toll-free domestic number. At 4:09 PM EST, Duo completed a change to a non-toll-free number and normal telephony functionality was restored.

Details

Duo’s platform uses a default telephone number for US-based phone calls. This telephone number follows a toll-free format. Because of the nature of Duo’s service, we place high volumes of short-duration calls. In concert with a toll-free number, these calls can mimic the style and format of unwanted robocalls.

On Thursday, November 7, 2019 at 10:05 AM EST, a number of US telephony carriers started rejecting calls made from Duo’s toll-free domestic number. Upon being notified of these issues, Duo’s engineering team immediately began triaging the issue and exploring alternatives. Upon consulting with our downstream telephony providers, we decided that the best course of action was to switch to using a non-toll-free number (+1-669-201-9315).

After testing out this solution on a small subset of affected customers, we applied the change broadly. This immediately restored telephony service to normal operations upon completion at 4:09 PM EST.

We understand that the sudden change to a new number may affect customers who have whitelisted or performed internal user education to expect the old number. We did not make the decision lightly and chose to prioritize immediate restoration of telephony quality.

Duo’s engineering team already had initiated a project prior to this incident to expand the resiliency of our telephony services. Today’s events have added further aspects for us to incorporate into this work. Additionally, we continue to work with our telephony providers to determine whether it will be possible to safely and reliably switch back to the old number. If so, we will provide advance communication to our customers about these changes.