Service degradation on some Duo deployments

Incident Report for Duo

Postmortem

Authentication and Admin Panel Degradation - Multiple Deployments Incident Report for Duo Security

From 01:51 to 03:05 UTC on Feb 10, 2017, the following Duo deployments experienced intermittent authentication timeouts: DUO2, DUO4, DUO6, DUO8, DUO10, DUO11, DUO12, DUO16, DUO20, DUO25, DUO35, DUO36, DUO37, DUO41, and DUO44. Additionally, Admin Panel and Admin API access on the aforementioned deployments was intermittently unavailable between 01:51 and 03:49 UTC.

Duo utilizes many premier cloud partners as part of our SaaS platform, including Amazon AWS. Per Amazon’s public status page (https://status.aws.amazon.com/), AWS began to experience a network issue specific to a single AWS availability zone at 00:52 UTC. This issue intermittently affected connectivity to infrastructure hosted within the affected availability zone. Because Duo’s platform spans multiple AWS regions and availability zones for redundancy, including this availability zone, a portion of our infrastructure was affected.

Duo’s monitoring system alerted our engineering team to issues when Duo deployments were first affected at 01:51 UTC. After identifying the impacted AWS availability zone, the engineering team began migrating affected infrastructure to other unaffected zones. Automatic cross zone failover is a feature of the affected Amazon services, but in this specific scenario the AWS automatic failover was not triggered. To ensure data integrity and avoid a complete authentication outage, the Duo team first conducted a manual failover on a test deployment. After successfully testing this process, the team triggered manual failovers across all remaining impacted deployments, stabilizing authentication service across all deployments at 03:05 UTC. The team then focused on restoring Admin Panel and Admin API access, completely restoring service at 03:49 UTC when Amazon resolved their network issue.

The Duo team will use data collected during this incident to influence future infrastructure related decisions regarding platform resilience. Specifically, we intend to determine why the automatic cross zone failover feature underperformed in this scenario and to put systems in place ensuring that all Duo services are resilient to failure modes of any kind.

Posted Feb 10, 2017 - 12:11 EST

Resolved

Admin panel access has now been restored for all customers. All services are operating as expected.

Posted Feb 09, 2017 - 23:40 EST

Update

Service is stabilized and authentications are processing as expected on affected deployments. Admin panel access is still limited for some customers, but we continue to work towards a full resolution of this issue.

Posted Feb 09, 2017 - 23:16 EST

Monitoring

We are seeing a significant increase in successful authentication requests for affected deployments. The admin panel may still be unavailable for some customers. We are continuing to work towards a full resolution of the issue.

Posted Feb 09, 2017 - 22:43 EST

Identified

We have confirmed that network connectivity issues one of our infrastructure providers is experiencing is causing a degradation of Duo services on some deployments. We are actively working to minimize and remove the impact to customers on affected deployments.

Posted Feb 09, 2017 - 21:36 EST

Monitoring

One of Duo's infrastructure providers is currently experiencing network connectivity issues impacting a subset of the resources that underpin many of Duo's isolated deployments. We are closely monitoring the situation to ensure the stability of Duo services.

Posted Feb 09, 2017 - 21:14 EST

This incident affected: DUO1 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO2 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO4 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO5 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO6 (Admin Panel, Core Authentication Service, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO7 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO8 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO10 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO11 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO12 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO13 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO14 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO16 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO18 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO19 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO20 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO21 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO23 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO24 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO25 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO28 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO31 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO32 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO33 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO36 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO37 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO41 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), DUO44 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery), and DUO35 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery).