All Deployments: Telephony Auth Delivery
Postmortem
Summary
On March 28, 2022, at around 11:06 EDT, Duo's Engineering Team was alerted by monitoring that Authentication Phone Call and SMS Passcode Delivery started failing. The root cause was identified as an outage affecting one of our telephony providers for multiple platforms.
The issue was resolved on the same day by switching to alternative telephony providers and later switching back to the primary telephony provider.
Timeline of Events EDT
2022-03-28 11:06 - Duo Site Reliability Engineering (SRE) is informed by alerts from the monitoring of the systems. SRE begins triage.
2022-03-28 11:35 - After collecting all the information and basic troubleshooting, Duo SRE determines our primary telephony provider’s outage caused the issue.
2022-03-28 11:38 - Duo SRE rolls out failover to secondary telephony provider for US, IN and GB.
2022-03-28 12:04 - Status page updated to Investigating.
2022-03-28 12:49 - Duo SRE routes are restored to primary telephony provider.
2022-03-28 12:54 - Status page updated to observing successful telephony authentications.
2022-03-28 13:15 - Status page updated to Resolved.
Details
Duo SRE alerted for Authentication Phone Call and SMS Passcode Delivery failures in GB, IN and US. During investigation, Duo SRE discovered the alert was caused due to the primary telephony provider’s multi-platform outage. Duo SRE switched to secondary telephony provider for the affected customers in GB, IN and US. Duo SRE switched back to the primary telephony provider in GB, IN and US after their services were fully restored.
The outage affected callees in GB, US and IN.
Duo Engineering is working with primary telephony providers to identify opportunities to remove blocking dependencies on components to improve resiliency.
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.
Posted Apr 05, 2022 - 17:17 EDT
Resolved
The issue with telephony auth delivery failures is now confirmed to be fully resolved.
Posted Mar 28, 2022 - 13:15 EDT
Update
We are observing successful telephony authentications on our deployments, we will continue to monitor until authentication return to their usual levels
Please subscribe to the status page for further updates.
Posted Mar 28, 2022 - 12:54 EDT
Monitoring
We are observing successful telephony authentications on our deployments, we will continue to monitor until authentication return to their usual levels
Please subscribe to the status page for further updates.
Posted Mar 28, 2022 - 12:52 EDT
Investigating
We are currently investigating an issue causing telephony authentication delivery failures from one of our telephony providers. This is affecting all of our deployments and we are working with the provider to correct the issue as soon as possible.
Please check back here or subscribe to updates for any changes.
Posted Mar 28, 2022 - 12:04 EDT
This incident affected: DUO1 (Phone Call Delivery, SMS Message Delivery), DUO2 (Phone Call Delivery, SMS Message Delivery), DUO3 (Phone Call Delivery, SMS Message Delivery), DUO4 (Phone Call Delivery, SMS Message Delivery), DUO5 (Phone Call Delivery, SMS Message Delivery), DUO6 (Phone Call Delivery, SMS Message Delivery), DUO7 (Phone Call Delivery, SMS Message Delivery), DUO8 (Phone Call Delivery, SMS Message Delivery), DUO47 (Phone Call Delivery, SMS Message Delivery), DUO10 (Phone Call Delivery, SMS Message Delivery), DUO11 (Phone Call Delivery, SMS Message Delivery), DUO12 (Phone Call Delivery, SMS Message Delivery), DUO13 (Phone Call Delivery, SMS Message Delivery), DUO14 (Phone Call Delivery, SMS Message Delivery), DUO15 (Phone Call Delivery, SMS Message Delivery), DUO16 (Phone Call Delivery, SMS Message Delivery), DUO17 (Phone Call Delivery, SMS Message Delivery), DUO18 (Phone Call Delivery, SMS Message Delivery), DUO19 (Phone Call Delivery, SMS Message Delivery), DUO20 (Phone Call Delivery, SMS Message Delivery), DUO21 (Phone Call Delivery, SMS Message Delivery), DUO22 (Phone Call Delivery, SMS Message Delivery), DUO23 (Phone Call Delivery, SMS Message Delivery), DUO24 (Phone Call Delivery, SMS Message Delivery), DUO25 (Phone Call Delivery, SMS Message Delivery), DUO26 (Phone Call Delivery, SMS Message Delivery), DUO27 (Phone Call Delivery, SMS Message Delivery), DUO28 (Phone Call Delivery, SMS Message Delivery), DUO29 (Phone Call Delivery, SMS Message Delivery), DUO30 (Phone Call Delivery, SMS Message Delivery), DUO31 (Phone Call Delivery, SMS Message Delivery), DUO32 (Phone Call Delivery, SMS Message Delivery), DUO33 (Phone Call Delivery, SMS Message Delivery), DUO34 (Phone Call Delivery, SMS Message Delivery), DUO36 (Phone Call Delivery, SMS Message Delivery), DUO37 (Phone Call Delivery, SMS Message Delivery), DUO38 (Phone Call Delivery, SMS Message Delivery), DUO39 (Phone Call Delivery, SMS Message Delivery), DUO40 (Phone Call Delivery, SMS Message Delivery), DUO41 (Phone Call Delivery, SMS Message Delivery), DUO42 (Phone Call Delivery, SMS Message Delivery), DUO43 (Phone Call Delivery, SMS Message Delivery), DUO44 (Phone Call Delivery, SMS Message Delivery), DUO45 (Phone Call Delivery, SMS Message Delivery), DUO46 (Phone Call Delivery, SMS Message Delivery), DUO48 (Phone Call Delivery, SMS Message Delivery), DUO9 (Phone Call Delivery, SMS Message Delivery), DUO49 (Phone Call Delivery, SMS Message Delivery), DUO50 (Phone Call Delivery, SMS Message Delivery), DUO51 (Phone Call Delivery, SMS Message Delivery), DUO52 (Phone Call Delivery, SMS Message Delivery), DUO53 (Phone Call Delivery, SMS Message Delivery), DUO54 (Phone Call Delivery, SMS Message Delivery), DUO55 (Phone Call Delivery, SMS Message Delivery), DUO56 (Phone Call Delivery, SMS Message Delivery), DUO57 (Phone Call Delivery, SMS Message Delivery), DUO58 (Phone Call Delivery, SMS Message Delivery), DUO59 (Phone Call Delivery, SMS Message Delivery), DUO60 (Phone Call Delivery, SMS Message Delivery), DUO61 (Phone Call Delivery, SMS Message Delivery), DUO62 (Phone Call Delivery, SMS Message Delivery), DUO63 (Phone Call Delivery, SMS Message Delivery), DUO64 (Phone Call Delivery, SMS Message Delivery), DUO65 (Phone Call Delivery, SMS Message Delivery), DUO66 (Phone Call Delivery, SMS Message Delivery), DUO67 (Phone Call Delivery, SMS Message Delivery), DUO68 (Phone Call Delivery, SMS Message Delivery), and DUO35 (Phone Call Delivery, SMS Message Delivery).