All Deployments: Intermittent Phone and SMS Delivery Delays
Incident Report for Duo
Postmortem

Degraded Phone Callback and SMS Text Authentication - All Deployments

Summary

On February 26, 2021 from 8:02 a.m. EST to 8:30 a.m. EST, all Duo deployments experienced phone callback and SMS (passcode and activation link) delivery failures to all destination regions. From 8:30 a.m. EST to 11:20 a.m. EST, phone calls and SMS continued to fail intermittently to some destinations including Canada, India, and the United Kingdom (CA, IN, and GB regions). No other authentication methods or features were affected.

Duo switched to a secondary telephony provider to place phone calls at 8:30 a.m. EST, which resolved issues outside of the CA, IN, and GB regions. Our primary provider was returned to normal service at 11:20 a.m. EST, which restored call and SMS service for all regions.

Details

At 8:02 a.m. EST on February 26, the Duo Engineering Team was alerted to an increased failure rate for phone callback and SMS passcode delivery and began investigating. After determining that failures were limited to a specific telephony provider, the decision was made to fail over to alternate providers.

At 8:30 a.m. EST, the team routed impacted calls to alternate telephony providers and confirmed that this improved deliverability for most calls and SMS texts, but intermittent delivery issues to destinations in the CA, IN, and GB regions continued.

At 11:20 a.m. EST, after internal validation, the primary provider was returned to normal service, and phone callback and SMS passcode delivery were restored in all supported geographies.

The Duo Engineering Team will continue to work with existing and new telephony partners to ensure redundancy and uptime in the future.

Posted Mar 02, 2021 - 15:05 EST

Resolved
The issue regarding delivery of calls for Phone Callback authentication and SMS Message Delivery on our deployments is fully resolved and all services are now fully functional.

We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.

Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Feb 26, 2021 - 12:17 EST
Update
We are seeing successful delivery of calls for Phone Callback authentications and are still investigating issues with SMS delivery for some customers.
Posted Feb 26, 2021 - 11:27 EST
Update
We are continuing to investigate this issue.
Posted Feb 26, 2021 - 10:29 EST
Investigating
We are currently investigating an issue causing phone and SMS delivery issues on all of our deployments and are working to correct the issue as soon as possible.

Please check back here or subscribe to updates for any changes.
Posted Feb 26, 2021 - 10:24 EST
This incident affected: DUO1 (Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO40 (Phone Call Delivery, SMS Message Delivery), DUO33 (Phone Call Delivery, SMS Message Delivery), DUO14 (Phone Call Delivery, SMS Message Delivery), DUO17 (Phone Call Delivery, SMS Message Delivery), DUO6 (Phone Call Delivery, SMS Message Delivery), DUO37 (Phone Call Delivery, SMS Message Delivery), DUO26 (Phone Call Delivery, SMS Message Delivery), DUO4 (Phone Call Delivery, SMS Message Delivery), DUO36 (Phone Call Delivery, SMS Message Delivery), DUO9 (Phone Call Delivery, SMS Message Delivery), DUO12 (Phone Call Delivery, SMS Message Delivery), DUO10 (Phone Call Delivery, SMS Message Delivery), DUO65 (Phone Call Delivery, SMS Message Delivery), DUO62 (Phone Call Delivery, SMS Message Delivery), DUO64 (Phone Call Delivery, SMS Message Delivery), DUO27 (Phone Call Delivery, SMS Message Delivery), DUO46 (Phone Call Delivery, SMS Message Delivery), DUO49 (Phone Call Delivery, SMS Message Delivery), DUO43 (Phone Call Delivery, SMS Message Delivery), DUO44 (Phone Call Delivery, SMS Message Delivery), DUO41 (Phone Call Delivery, SMS Message Delivery), DUO23 (Phone Call Delivery, SMS Message Delivery), DUO18 (Phone Call Delivery, SMS Message Delivery), DUO28 (Phone Call Delivery, SMS Message Delivery), DUO60 (Phone Call Delivery, SMS Message Delivery), DUO59 (Phone Call Delivery, SMS Message Delivery), DUO31 (Phone Call Delivery, SMS Message Delivery), DUO19 (Phone Call Delivery, SMS Message Delivery), DUO54 (Phone Call Delivery, SMS Message Delivery), DUO15 (Phone Call Delivery, SMS Message Delivery), DUO52 (Phone Call Delivery, SMS Message Delivery), DUO7 (Phone Call Delivery, SMS Message Delivery), DUO3 (Phone Call Delivery, SMS Message Delivery), DUO58 (Phone Call Delivery, SMS Message Delivery), DUO57 (Phone Call Delivery, SMS Message Delivery), DUO47 (Phone Call Delivery, SMS Message Delivery), DUO8 (Phone Call Delivery, SMS Message Delivery), DUO38 (Phone Call Delivery, SMS Message Delivery), DUO25 (Phone Call Delivery, SMS Message Delivery), DUO22 (Phone Call Delivery, SMS Message Delivery), DUO63 (Phone Call Delivery, SMS Message Delivery), DUO29 (Phone Call Delivery, SMS Message Delivery), DUO16 (Phone Call Delivery, SMS Message Delivery), DUO24 (Phone Call Delivery, SMS Message Delivery), DUO13 (Phone Call Delivery, SMS Message Delivery), DUO21 (Phone Call Delivery, SMS Message Delivery), DUO48 (Phone Call Delivery, SMS Message Delivery), DUO66 (Phone Call Delivery, SMS Message Delivery), DUO50 (Phone Call Delivery, SMS Message Delivery), DUO51 (Phone Call Delivery, SMS Message Delivery), DUO56 (Phone Call Delivery, SMS Message Delivery), DUO53 (Phone Call Delivery, SMS Message Delivery), DUO34 (Phone Call Delivery, SMS Message Delivery), DUO30 (Phone Call Delivery, SMS Message Delivery), DUO55 (Phone Call Delivery, SMS Message Delivery), DUO20 (Phone Call Delivery, SMS Message Delivery), DUO2 (Phone Call Delivery, SMS Message Delivery), DUO39 (Phone Call Delivery, SMS Message Delivery), DUO45 (Phone Call Delivery, SMS Message Delivery), DUO42 (Phone Call Delivery, SMS Message Delivery), DUO11 (Phone Call Delivery, SMS Message Delivery), DUO32 (Phone Call Delivery, SMS Message Delivery), DUO5 (Phone Call Delivery, SMS Message Delivery), DUO61 (Phone Call Delivery, SMS Message Delivery), and DUO35 (Phone Call Delivery, SMS Message Delivery).