Summary

On December 10, 2021, from 15:20 EST to 16:30 EST, all Duo deployments began to see SMS and VOIP authentication failures in North America. This was caused by our primary telephony service provider experiencing high latency and returning errors. While Duo Engineering was evaluating routing calls to North American numbers to our backup telephony services provider, our primary provider fixed the issue. All SMS and VOIP issues were resolved by 16:30 EST.

Details

At 15:20 EST on December 10, 2021, Duo was alerted via internal monitoring tools to issues with telephony authentications on multiple Duo deployments. By 15:39 EST, our telephony provider confirmed VOIP and SMS delivery issues.

By 15:54, our telephony provider identified the issue as being related to latency with its REST API and Duo Engineering began evaluating routing telephony to our alternate provider.

However, by 16:09, the number of failures had fallen greatly and our primary telephony provider reported by 16:20 that they solved the issue.

The Duo Engineering Team will continue to work with existing and new telephony partners to ensure redundancy and uptime in the future. In addition, we have made the following changes:  

• Duo Engineering has decreased our timeout value for provider APIs such as Twilio. This will allow us to fail over automatically to a backup provider quicker in the event of failures such as this. 
• We are also now routing our telephony traffic through regionalized edge locations to improve resiliency in case of a regional failure.