DUO1 Deployment: Authentication Latency
Incident Report for Duo
Postmortem

Summary:

From 20:40 UTC to 22:04 UTC on January 10, 2019, performance on the DUO1 deployment degraded, resulting in increased authentication latency and intermittent request timeouts. Approximately 60% of authentication requests to DUO1 were slow to respond or timed out.

The root cause of this outage has been identified and engineering changes have been made to prevent similar issues going forward.

Details:

The Duo architecture has been designed with redundancy at each tier and is configured to automatically recover from failure without customer impact. The final stage in recovery is updating an internal map that specifies which load balancer nodes are active. A previous automated recovery event affecting DUO1 completed successfully but did not properly update this map.

At 20:27 UTC on January 10, 2019, a scheduled software update was deployed to DUO1. The automated software release process used the now out-of-date node map to reconfigure DUO1's load balancers. These incorrect configurations misrouted traffic, causing requests to fail or timeout. The percentage of misrouted requests increased as the release process continued modifying load balancers.

At 20:36 UTC, latency exceeded our monitoring thresholds and alerted the operations team to the degraded state. Initial triage implicated the software update as the most probable cause, and operations initiated a rollback at 20:42 UTC.

At 20:51 UTC, traffic was rerouted to servers running a previous software release as part of the rollback process. At 21:02 UTC, the Duo Status Page was updated to reflect data showing that the issue had been resolved.

Subsequently, data became available that showed signs of ongoing service degradation, so a new Status Page incident was opened to reflect the ongoing issue. After the system finished restoring load balancers to a known good configuration at 22:04 UTC, service returned to expected performance levels.

The engineering team has used data collected during this event to improve our release automation to detect errors in the load balancer configuration. Operational processes associated with this event have been reviewed and updated to incorporate lessons learned. With these improvements in place, DUO1 has since been updated to the latest production release of the Duo software. Additional infrastructure and software improvements are underway in order to provide further stability.

Posted 9 months ago. Jan 15, 2019 - 10:41 EST

Resolved
We have fully resolved the authentication latency issues on DUO1 and all services are now restored to normal operation.

We will attach a root-cause analysis (RCA) to this incident once our engineering team has finished its thorough investigation of the issues. The contents of our RCA will span both incidents reported today.

Please make sure to check back or subscribe to be notified when the RCA is posted.

UPDATE 11 January 2019 16:32 EST: Our Engineering Team is continuing their work to provide a detailed RCA. We will send customer communication updates on Monday.
Posted 9 months ago. Jan 10, 2019 - 17:58 EST
Monitoring
Our Engineering Team’s changes have been completed and have resulted in the issue being fixed.

We will continue to monitor the issue and will post any updates when the incident is considered fully resolved.

Please check back here or subscribe here for further updates.
Posted 9 months ago. Jan 10, 2019 - 17:24 EST
Investigating
We are currently investigating an ongoing issue causing authentication latency on our DUO1 deployment and are working to correct the issue as soon as possible.

Please check back here or subscribe to updates for any changes.
Posted 9 months ago. Jan 10, 2019 - 16:20 EST
This incident affected: DUO1 (Core Authentication Service).