All Deployments: SMS 2FA delivery delay on Canadian Logins
Postmortem
Summary
On March 7, 2022 from 10:11 a.m. EST to 11:53 a.m. EST, all Duo deployments experienced SMS (passcode and activation link) intermittent delivery failures to the Canada region. No other authentication methods or features were affected.
Duo switched to a secondary telephony provider to send SMS at 11:10 a.m. EST, which did not solve the issue. Our primary provider was returned to normal service at 11:53 a.m. EST, which restored SMS service for the Canada region.
Details
At 10:11 a.m. EST on March 7, the Duo Engineering Team was alerted to an increased failure rate for SMS passcode in Canada delivery and began investigating. After determining that failures were limited to a specific telephony provider, the decision was made to failover to alternate providers.
At 11:10 a.m. EST, the team routed impacted calls to alternate telephony providers and confirmed that this did not improve deliverability for SMS texts, intermittent delivery issues to destinations in the CA region continued.
At 11:53 a.m. EST, after internal validation, the primary provider was returned to normal service, and SMS passcode delivery was restored in the Canada region.
The Duo Engineering Team is currently working on improving observability and implementing automation around this type of outage to increase the resiliency of our telephony services. What is more, we will continue to work with existing and new telephony partners to ensure redundancy and uptime in the future.
Posted Mar 10, 2022 - 18:15 EST
Resolved
The issue with SMS delivery delays to our Canadian-based deployments is now confirmed to be fully resolved.
A root-cause analysis (RCA) will be provided as soon as it is available.
Posted Mar 07, 2022 - 17:23 EST
Monitoring
We are observing recovery in SMS delivery delays to our Canadian-based deployments with a small percentage of failures.
Please subscribe to the status page for further updates.
Posted Mar 07, 2022 - 14:21 EST
Identified
We have identified the issue causing delayed SMS delivery of 2FA on some of our Canadian-based deployments. Delays were noted primarily between 3:40 pm to 5:40 pm UTC.
Please subscribe to the status page for further updates.
Posted Mar 07, 2022 - 14:15 EST
Investigating
We are currently investigating an issue causing delayed SMS delivery of 2FA on some of our Canadian based deployments and are working to correct the issue as soon as possible.
Please check back here or subscribe to updates for any changes.
Posted Mar 07, 2022 - 13:11 EST
This incident affected: DUO1 (Phone Call Delivery, SMS Message Delivery), DUO2 (Phone Call Delivery, SMS Message Delivery), DUO3 (Phone Call Delivery, SMS Message Delivery), DUO4 (Phone Call Delivery, SMS Message Delivery), DUO5 (Phone Call Delivery, SMS Message Delivery), DUO6 (Phone Call Delivery, SMS Message Delivery), DUO7 (Phone Call Delivery, SMS Message Delivery), DUO8 (Phone Call Delivery, SMS Message Delivery), DUO47 (Phone Call Delivery, SMS Message Delivery), DUO10 (Phone Call Delivery, SMS Message Delivery), DUO11 (Phone Call Delivery, SMS Message Delivery), DUO12 (Phone Call Delivery, SMS Message Delivery), DUO13 (Phone Call Delivery, SMS Message Delivery), DUO14 (Phone Call Delivery, SMS Message Delivery), DUO15 (Phone Call Delivery, SMS Message Delivery), DUO16 (Phone Call Delivery, SMS Message Delivery), DUO17 (Phone Call Delivery, SMS Message Delivery), DUO18 (Phone Call Delivery, SMS Message Delivery), DUO19 (Phone Call Delivery, SMS Message Delivery), DUO20 (Phone Call Delivery, SMS Message Delivery), DUO21 (Phone Call Delivery, SMS Message Delivery), DUO22 (Phone Call Delivery, SMS Message Delivery), DUO23 (Phone Call Delivery, SMS Message Delivery), DUO24 (Phone Call Delivery, SMS Message Delivery), DUO25 (Phone Call Delivery, SMS Message Delivery), DUO26 (Phone Call Delivery, SMS Message Delivery), DUO27 (Phone Call Delivery, SMS Message Delivery), DUO28 (Phone Call Delivery, SMS Message Delivery), DUO29 (Phone Call Delivery, SMS Message Delivery), DUO30 (Phone Call Delivery, SMS Message Delivery), DUO31 (Phone Call Delivery, SMS Message Delivery), DUO32 (Phone Call Delivery, SMS Message Delivery), DUO33 (Phone Call Delivery, SMS Message Delivery), DUO34 (Phone Call Delivery, SMS Message Delivery), DUO36 (Phone Call Delivery, SMS Message Delivery), DUO37 (Phone Call Delivery, SMS Message Delivery), DUO38 (Phone Call Delivery, SMS Message Delivery), DUO39 (Phone Call Delivery, SMS Message Delivery), DUO40 (Phone Call Delivery, SMS Message Delivery), DUO41 (Phone Call Delivery, SMS Message Delivery), DUO42 (Phone Call Delivery, SMS Message Delivery), DUO43 (Phone Call Delivery, SMS Message Delivery), DUO44 (Phone Call Delivery, SMS Message Delivery), DUO45 (Phone Call Delivery, SMS Message Delivery), DUO46 (Phone Call Delivery, SMS Message Delivery), DUO48 (Phone Call Delivery, SMS Message Delivery), DUO9 (Phone Call Delivery, SMS Message Delivery), DUO49 (Phone Call Delivery, SMS Message Delivery), DUO50 (Phone Call Delivery, SMS Message Delivery), DUO51 (Phone Call Delivery, SMS Message Delivery), DUO52 (Phone Call Delivery, SMS Message Delivery), DUO53 (Phone Call Delivery, SMS Message Delivery), DUO54 (Phone Call Delivery, SMS Message Delivery), DUO55 (Phone Call Delivery, SMS Message Delivery), DUO56 (Phone Call Delivery, SMS Message Delivery), DUO57 (Phone Call Delivery, SMS Message Delivery), DUO58 (Phone Call Delivery, SMS Message Delivery), DUO59 (Phone Call Delivery, SMS Message Delivery), DUO60 (Phone Call Delivery, SMS Message Delivery), DUO61 (Phone Call Delivery, SMS Message Delivery), DUO62 (Phone Call Delivery, SMS Message Delivery), DUO63 (Phone Call Delivery, SMS Message Delivery), DUO64 (Phone Call Delivery, SMS Message Delivery), DUO65 (Phone Call Delivery, SMS Message Delivery), DUO66 (Phone Call Delivery, SMS Message Delivery), and DUO35 (Phone Call Delivery, SMS Message Delivery).