Postmortem
Summary
On April 1st, 2022, at around 11:19 EDT, Duo's Engineering Team was alerted by monitoring that Authentication Phone Call Delivery started failing for GB and IN regions. The root cause was identified as an outage affecting one of our telephony providers’ downstream carriers.
The issue was almost immediately resolved for IN by switching to alternative telephony providers, however a failure on our GB alternative provider led us to further investigations and escalation. The incident with our primary provider was resolved later at 16:32 EDT
Timeline of Events EDT
2022-04-01 11:19 Duo Site Reliability Engineering (SRE) is informed by alerts from the monitoring of the systems. SRE begins triage.
2022-04-01 11:59 After collecting all the information and basic troubleshooting, Duo SRE determines that the issue is due to an outage affecting our primary telephony provider.
2022-04-01 12:10 Duo SRE rolled out failover to secondary telephony provider for IN and GB
2022-04-01 12:19 Duo SRE kept the failover for IN as it improves our deliverability but found GB did not help
2022-04-01 12:53 Duo SRE has escalated to providers and work with them actively to fix the issue
2022-04-01 12:55 Status page updated to Investigating.
2022-04-01 13:25 Duo SRE routes are working again for GB region with our primary provider
2022-04-01 14:52 Status page updated to observing successful telephony authentications.
2022-04-01 16:32 Status page updated to Resolved.
Details
Duo SRE was alerted for Authentication Phone Call Delivery failures in GB and IN. During investigation, Duo SRE discovered the alert was caused due to an outage affecting one of our telephony providers’ downstream carriers.
Duo SRE switched to secondary telephony provider for the affected customers in GB, IN. Duo SRE saw restoration of service almost immediately for IN but not for GB and hence escalated to both providers. Later, the issue was fixed and telephony routes were restored back to our primary provider.
Duo Engineering is working with telephony providers to ensure our routes are resilient to our customers whether through primary or secondary. The Duo SRE team in particular will be soon releasing a new system that will load balance traffic through our different providers to ensure resiliency for SMS and VOIP factors.
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.
Posted Apr 05, 2022 - 17:25 EDT
Resolved
This issue is now fully resolved.
Posted Apr 01, 2022 - 16:32 EDT
Monitoring
The issues with Phone Call Delivery to customers in the Great Britain region are resolved, and we are continuing to monitor to ensure a full recovery.
Posted Apr 01, 2022 - 14:52 EDT
Identified
We have identified the issue and are working to correct it as soon as possible.
Posted Apr 01, 2022 - 13:36 EDT
Investigating
We are currently investigating an issue causing some Phone Call Delivery failures for customers who have service in the Great Britain region on all of our deployments and are working to correct the issue as soon as possible.
All other authentication methods are unaffected.
Please check back here or subscribe to updates for any changes.
Posted Apr 01, 2022 - 12:55 EDT
This incident affected: DUO1 (Phone Call Delivery), DUO2 (Phone Call Delivery), DUO3 (Phone Call Delivery), DUO4 (Phone Call Delivery), DUO5 (Phone Call Delivery), DUO6 (Phone Call Delivery), DUO7 (Phone Call Delivery), DUO8 (Phone Call Delivery), DUO47 (Phone Call Delivery), DUO10 (Phone Call Delivery), DUO11 (Phone Call Delivery), DUO12 (Phone Call Delivery), DUO13 (Phone Call Delivery), DUO14 (Phone Call Delivery), DUO15 (Phone Call Delivery), DUO16 (Phone Call Delivery), DUO17 (Phone Call Delivery), DUO18 (Phone Call Delivery), DUO19 (Phone Call Delivery), DUO20 (Phone Call Delivery), DUO21 (Phone Call Delivery), DUO22 (Phone Call Delivery), DUO23 (Phone Call Delivery), DUO24 (Phone Call Delivery), DUO25 (Phone Call Delivery), DUO26 (Phone Call Delivery), DUO27 (Phone Call Delivery), DUO28 (Phone Call Delivery), DUO29 (Phone Call Delivery), DUO30 (Phone Call Delivery), DUO31 (Phone Call Delivery), DUO32 (Phone Call Delivery), DUO33 (Phone Call Delivery), DUO34 (Phone Call Delivery), DUO36 (Phone Call Delivery), DUO37 (Phone Call Delivery), DUO38 (Phone Call Delivery), DUO39 (Phone Call Delivery), DUO40 (Phone Call Delivery), DUO41 (Phone Call Delivery), DUO42 (Phone Call Delivery), DUO43 (Phone Call Delivery), DUO44 (Phone Call Delivery), DUO45 (Phone Call Delivery), DUO46 (Phone Call Delivery), DUO48 (Phone Call Delivery), DUO9 (Phone Call Delivery), DUO49 (Phone Call Delivery), DUO50 (Phone Call Delivery), DUO51 (Phone Call Delivery), DUO52 (Phone Call Delivery), DUO53 (Phone Call Delivery), DUO54 (Phone Call Delivery), DUO55 (Phone Call Delivery), DUO56 (Phone Call Delivery), DUO57 (Phone Call Delivery), DUO58 (Phone Call Delivery), DUO59 (Phone Call Delivery), DUO60 (Phone Call Delivery), DUO61 (Phone Call Delivery), DUO62 (Phone Call Delivery), DUO63 (Phone Call Delivery), DUO64 (Phone Call Delivery), DUO65 (Phone Call Delivery), DUO66 (Phone Call Delivery), DUO67 (Phone Call Delivery), DUO68 (Phone Call Delivery), and DUO35 (Phone Call Delivery).