Core Authentication Issues
Incident Report for Duo Security
Postmortem

From 16:15 to 17:56 UTC on December 16, 2017, the following Duo deployments experienced intermittent authentication timeouts affecting up to 5% of all requests: DUO8, DUO10, DUO21, DUO28, DUO39, DUO40, DUO49, DUO52.

Duo utilizes many premier cloud partners as part of our SaaS platform, including Amazon AWS. As part of normal operations, Amazon regularly performs maintenance to the underlying cloud infrastructure that powers the Duo service. This maintenance is expected to be non-impactful due to the multiple levels of redundancy built into the Duo platform, and has been validated as such numerous times over a multi-year period.

On December 16th at 16:00 UTC, AWS performed maintenance to the service that powers one of the database tiers of the Duo platform. Duo’s automated monitoring system alerted our engineering team to the beginning of this maintenance, and the team began monitoring to ensure that the maintenance completed successfully. During this window, the team observed numerous successful automated failovers between database instances, as expected. At 17:00 UTC, the team was alerted to intermittent authentication failures affecting between 1% and 5% of all requests, varying depending on the affected deployment.

After additional investigation, Duo’s engineering team identified the root cause as sporadic error responses from the AWS database service starting at 16:15 UTC. These errors were likely the result of maintenance being conducted against both primary and secondary failover instances resulting in a double-failover scenario. A rolling service restart of the affected deployments was performed at 17:21 in an effort to resolve these intermittent errors, and service was confirmed to be stabilized for all deployments except DUO21 at that time. Additional troubleshooting was performed against DUO21, and service was confirmed to be stabilized for this deployment at 17:56 UTC.

The Duo team will use data collected during this incident to influence future infrastructure related decisions regarding platform resilience. Specifically, we intend to determine why this unique double-failover situation left a small percentage of connections in a failed state, and continue to put systems and processes in place ensuring that all Duo services are resilient to failure modes of any kind. In the short term, we will work with AWS to ensure that maintenance affecting both the primary and secondary failover instances are not conducted in the same window.

Posted 12 months ago. Dec 18, 2017 - 12:46 EST

Resolved
Duo services across all deployments are now working as expected.
Posted 12 months ago. Dec 16, 2017 - 13:16 EST
Monitoring
We have taken action to stabilize service on affected Duo deployments and are monitoring to ensure all services are working as expected.
Posted 12 months ago. Dec 16, 2017 - 13:03 EST
Identified
We have identified an issue with logins to our Core Authentication service on some Duo deployments, and are working to correct the issue.
Posted 12 months ago. Dec 16, 2017 - 12:47 EST
This incident affected: DUO49 (Core Authentication Service, Admin Panel), DUO52 (Core Authentication Service, Admin Panel), DUO39 (Core Authentication Service, Admin Panel), DUO40 (Core Authentication Service, Admin Panel), DUO21 (Core Authentication Service, Admin Panel), DUO8 (Core Authentication Service, Admin Panel), DUO10 (Core Authentication Service, Admin Panel), and DUO28 (Core Authentication Service, Admin Panel).