Summary:

From 14:47 UTC to 15:58 UTC on November 5, 2018, all DUO deployments experienced a delay in placing phone calls for customers using the phone callback authentication method. During this window, roughly 50% of phone callback requests were delayed long enough to cause corresponding authentications to timeout and subsequently fail. The root cause of this outage has been identified and resolved to prevent similar issues going forward.

Details:

The Duo SaaS platform uses many telephony providers for generating phone calls and SMS text messages. Our platform routes telephony interactions across these multiple providers to ensure phone calls and messages reach the intended recipient. If a request sent to one of these providers fails, it is automatically routed to another provider, protecting against provider specific service degradation.

Starting at 14:47 UTC on November 5, 2018, one of these providers began experiencing internal issues, which caused some requested phone calls to be significantly delayed. Duo’s monitoring systems notified our engineering team of these delays at 14:49 UTC. The engineering team investigated the issue and determined that phone callback requests were being processed successfully by Duo but were being queued in the provider’s system. The provider continued to accept requests from Duo without errors, preventing automatic failover from being triggered.

The Duo team initiated a manual failover to a secondary telephony provider for phone services on a test deployment. After successfully testing and verifying this process, the team triggered a manual failover across all deployments. This process completed at 15:58 UTC when all phone callback authentications began completing as expected. We have since confirmed that the provider in question has identified and resolved the root cause of these issues, and we have reverted our temporary failover configuration.

Additional monitoring improvements have been implemented in order to allow Duo’s monitoring systems to better detect this scenario in the future. We will also enhance our automated failover mechanisms to better handle situations where providers are incorrectly reporting successful operation.