DUO1 Authentication Issues Resolved Following Load Balancer Failover
Incident Report for Duo
Postmortem

Summary

From 15:20 UTC to 15:26 UTC on February 21, 2019, the DUO1 deployment experienced a load balancer failure resulting in all authentication requests to that deployment failing. The issue was identified and resolved via automated remediation.

Details

The DUO1 deployment’s load balancing tier is composed of several pairs of internally redundant load balancing hardware configurations. Each pair is composed of identically configured hardware in two separate locations in an active/standby setup.

At 15:20:21 UTC, one of DUO1’s active load balancers failed, preventing customers from connecting to DUO1. When this type of failure occurs, an automated process redirects customers to the standby load balancing hardware. The issue, which Duo detected immediately, was resolved automatically for all customers by 15:26:18 UTC. We will continue to explore additional infrastructure and software improvements to reduce failover time and customer impact moving forward.

Posted Feb 21, 2019 - 15:20 EST

Resolved
From 15:20 UTC to 15:26 UTC on February 21, 2019, the DUO1 deployment experienced a load balancer failure resulting in 100% of authentication requests failing to complete. This issue was identified and resolved via automated remediation.

We will attach a root-cause analysis (RCA) to this incident once our engineering team has finished its thorough investigation of the issue.
Posted Feb 21, 2019 - 11:19 EST
This incident affected: DUO1 (Core Authentication Service, Admin Panel).