All Deployments: Phone Call Delivery
Postmortem
Degraded Phone Callback Authentication - All Deployments
Summary
On March 1, 2021 from 3:03 a.m. EST to 3:46 a.m. EST, all Duo deployments experienced phone callback authentication failures to all destination regions. From 3:46 a.m. EST to 4:27 a.m. EST, phone authentications to only the United Kingdom (GB region) were degraded. No other authentication methods or features were affected.
Duo switched to a secondary telephony provider to place phone calls at 3:46 a.m. EST, which resolved issues for all non-GB calls. Our primary provider was returned to normal service at 4:27 a.m. EST, restoring call service for all regions.
Details
At 3:03 a.m. EST, the Duo Engineering Team was alerted to an increased failure rate for phone callback authentications and began investigating. After determining that failures were limited to a specific telephony provider, the decision was made to fail over to a secondary provider.
At 3:46 a.m. EST, the team routed impacted calls to a secondary telephony provider and confirmed that new phone callback authentications had completed successfully, but this secondary provider also had separate issues delivering calls to the GB region.
At 4:27 a.m. EST, after internal validation, the primary provider was returned to normal service, and phone callback authentications were restored in all supported geographies.
The Duo Engineering Team will continue to work with existing and new telephony partners to ensure redundancy and uptime in the future.
Posted Mar 02, 2021 - 15:07 EST
Resolved
The issue regarding delivery of Phone Callback authentication on our deployments is fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Mar 01, 2021 - 06:47 EST
Monitoring
Our Engineering team's remedial actions have resulted in restoring the phone call authentication services for all regions.
We are going to continue monitoring and update this incident if required.
Posted Mar 01, 2021 - 05:42 EST
Update
Our Engineering team's remedial actions have resulted in restoring the phone call authentication services for a majority of regions.
We are still observing, however, phone call authentication failures in certain countries and are currently working towards restoring the remaining affected regions.
Please continue to use alternative authentication methods like Push or SMS and check back here or subscribe to updates for any changes.
Posted Mar 01, 2021 - 04:51 EST
Identified
We are currently investigating an issue causing phone delivery issues on all of our deployments and are working to correct the issue as soon as possible.
Please, where possible, use alternative authentication methods, like push or SMS.
Please check back here or subscribe to updates for any changes.
Posted Mar 01, 2021 - 03:58 EST
This incident affected: DUO1 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO14 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO29 (Phone Call Delivery), DUO30 (Phone Call Delivery), DUO31 (Phone Call Delivery), DUO32 (Phone Call Delivery), DUO33 (Phone Call Delivery), DUO34 (Phone Call Delivery), DUO36 (Phone Call Delivery), DUO37 (Phone Call Delivery), DUO38 (Phone Call Delivery), DUO39 (Phone Call Delivery), DUO40 (Phone Call Delivery), DUO41 (Phone Call Delivery), DUO42 (Phone Call Delivery), DUO43 (Phone Call Delivery), DUO44 (Phone Call Delivery), DUO45 (Phone Call Delivery), DUO46 (Phone Call Delivery), DUO48 (Phone Call Delivery), DUO9 (Phone Call Delivery), DUO49 (Phone Call Delivery), DUO50 (Phone Call Delivery), DUO51 (Phone Call Delivery), DUO52 (Phone Call Delivery), DUO53 (Phone Call Delivery), DUO54 (Phone Call Delivery), DUO55 (Phone Call Delivery), DUO56 (Phone Call Delivery), DUO57 (Phone Call Delivery), DUO58 (Phone Call Delivery), DUO59 (Phone Call Delivery), DUO60 (Phone Call Delivery), DUO61 (Phone Call Delivery), DUO62 (Phone Call Delivery), DUO63 (Phone Call Delivery), DUO64 (Phone Call Delivery), DUO65 (Phone Call Delivery), DUO66 (Phone Call Delivery), and DUO35 (Phone Call Delivery).