All Deployments: Intermittent Push Notifications Errors
Incident Report for Duo
Postmortem

Summary

From 6:34 a.m. EDT to 9:01 a.m. EDT on June 24, 2019, Duo customers may have experienced intermittent issues accessing the Duo Admin Panel or completing authentication to Duo-protected applications across all deployments. This issue was caused by a disruption in internet routing impacting traffic destined for multiple Internet destinations. Therefore, impact to customers varied depending on which route their traffic took to Duo’s infrastructure.

Details

Duo’s Engineering team was alerted to customers experiencing intermittent issues completing authentication attempts or accessing the Duo Admin Panel at 8:09 a.m. EDT. At this time Duo’s Engineering team began investigating the cause of the issue.

Duo utilizes many premier cloud partners as part of our SaaS platform, including Amazon AWS. Per an update at 8:28 a.m. EDT to Amazon’s public status page (https://status.aws.amazon.com), AWS began to experience a network issue with an external provider outside its network at 6:34 a.m. EDT. This issue prevented connectivity between multiple AWS regions.

The investigation by Duo’s Engineering team concluded that the issue reported by AWS was the cause of issues being experienced by customers. In an update at 9:25 a.m. EDT, AWS confirmed that the issue with the external provider had been resolved at 9:01 a.m. EDT.

Duo’s Engineering team will be implementing further monitoring to detect this issue more quickly in the future so we can provide proactive updates to our customers.

Posted Jun 24, 2019 - 17:29 EDT

Resolved
We have received confirmation from AWS that the issue with Amazon's external provider that affected connectivity between some customer networks and multiple AWS regions has been resolved: https://status.aws.amazon.com/

We have confirmed that Duo Push authentication and Duo Admin Panel access are now functioning correctly.

We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.

Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Jun 24, 2019 - 09:48 EDT
Investigating
We are currently investigating reports that some customers are unable to access Duo or other AWS-hosted services, affecting Duo Push authentication and Duo Admin Panel access. AWS has confirmed an issue with an external provider outside of their network. See https://status.aws.amazon.com/

For two-factor authentication purposes, we recommend using cellular data instead of WiFi to potentially work around the issue.

We are closely tracking the issue to assess and minimize impact to Duo customers.

Please check back here or subscribe to updates for any changes.
Posted Jun 24, 2019 - 08:56 EDT
This incident affected: DUO53 (Push Delivery), DUO23 (Push Delivery), DUO50 (Push Delivery), DUO49 (Push Delivery), DUO34 (Push Delivery), DUO51 (Push Delivery), DUO45 (Push Delivery), DUO43 (Push Delivery), DUO38 (Push Delivery), DUO11 (Push Delivery), DUO42 (Push Delivery), DUO40 (Push Delivery), DUO62 (Push Delivery), DUO41 (Push Delivery), DUO26 (Push Delivery), DUO1 (Push Delivery), DUO27 (Push Delivery), DUO24 (Push Delivery), DUO3 (Push Delivery), DUO59 (Push Delivery), DUO57 (Push Delivery), DUO8 (Push Delivery), DUO5 (Push Delivery), DUO56 (Push Delivery), DUO39 (Push Delivery), DUO58 (Push Delivery), DUO55 (Push Delivery), DUO33 (Push Delivery), DUO29 (Push Delivery), DUO63 (Push Delivery), DUO64 (Push Delivery), DUO15 (Push Delivery), DUO32 (Push Delivery), DUO18 (Push Delivery), DUO46 (Push Delivery), DUO48 (Push Delivery), DUO25 (Push Delivery), DUO9 (Push Delivery), DUO54 (Push Delivery), DUO13 (Push Delivery), DUO22 (Push Delivery), DUO4 (Push Delivery), DUO14 (Push Delivery), DUO20 (Push Delivery), DUO6 (Push Delivery), DUO31 (Push Delivery), DUO44 (Push Delivery), DUO37 (Push Delivery), DUO60 (Push Delivery), DUO28 (Push Delivery), DUO7 (Push Delivery), DUO16 (Push Delivery), DUO17 (Push Delivery), DUO2 (Push Delivery), DUO65 (Push Delivery), DUO19 (Push Delivery), DUO36 (Push Delivery), DUO10 (Push Delivery), DUO52 (Push Delivery), DUO47 (Push Delivery), DUO61 (Push Delivery), DUO30 (Push Delivery), DUO12 (Push Delivery), DUO21 (Push Delivery), and DUO35 (Push Delivery).