Summary

On April 1st, 2022, at around 11:19 EDT, Duo's Engineering Team was alerted by monitoring that Authentication Phone Call Delivery started failing for GB and IN regions. The root cause was identified as an outage affecting one of our telephony providers’ downstream carriers.

The issue was almost immediately resolved for IN by switching to alternative telephony providers, however a failure on our GB alternative provider led us to further investigations and escalation. The incident with our primary provider was resolved later at 16:32 EDT

Timeline of Events EDT

2022-04-01 11:19 Duo Site Reliability Engineering (SRE) is informed by alerts from the monitoring of the systems. SRE begins triage.

2022-04-01 11:59 After collecting all the information and basic troubleshooting, Duo SRE determines that the issue is due to an outage affecting our primary telephony provider.

2022-04-01 12:10 Duo SRE rolled out failover to secondary telephony provider for IN and GB

2022-04-01 12:19 Duo SRE kept the failover for IN as it improves our deliverability but found GB did not help

2022-04-01 12:53 Duo SRE has escalated to providers and work with them actively to fix the issue

2022-04-01 12:55 Status page updated to Investigating.

2022-04-01 13:25 Duo SRE routes are working again for GB region with our primary provider

2022-04-01 14:52 Status page updated to observing successful telephony authentications.

2022-04-01 16:32 Status page updated to Resolved.

Details

Duo SRE was alerted for Authentication Phone Call Delivery failures in GB and IN. During investigation, Duo SRE discovered the alert was caused due to an outage affecting one of our telephony providers’ downstream carriers.

Duo SRE switched to secondary telephony provider for the affected customers in GB, IN. Duo SRE saw restoration of service almost immediately for IN but not for GB and hence escalated to both providers. Later, the issue was fixed and telephony routes were restored back to our primary provider.

Duo Engineering is working with telephony providers to ensure our routes are resilient to our customers whether through primary or secondary. The Duo SRE team in particular will be soon releasing a new system that will load balance traffic through our different providers to ensure resiliency for SMS and VOIP factors.

Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.