Duo Authentication Failures in the Singapore region
Incident Report for Duo
Postmortem

Summary

On March 25, 2024, at around 8:42 ET, Duo's Engineering Team was alerted by monitoring that one of the endpoints was unreachable. The root cause was identified as misconfiguration in our routing infrastructure.

The issue was resolved on the same day by fixing the routing configuration.

Deployments Impacted

  • DUO68

Details

Duo SRE rolled out a new configuration to our applications and in the process of rolling out the changes some of the routing configurations were misconfigured. This caused the request to be not accepted by our load balancers. 

After discovering the change in routing configuration, we immediately fixed the issue and made sure traffic is being served in the deployment.

As a short term solution, engineers quickly fixed the routing configurations which were later ported back to our IaC solutions to be persisted across various environments. 

SRE has also taken up the task to improve monitoring around specific failure scenarios faster and to come up with fault tolerant routing mechanisms which will prevent such failures in the future.

Posted May 29, 2024 - 14:46 EDT

Resolved
We have resolved the issue that was causing authentication issues in the Singapore region and all services are now fully functional.
Posted May 24, 2024 - 08:57 EDT
Monitoring
We have identified the issue causing authentication failures in the Singapore region and have deployed a fix. We are currently monitoring the results of the change made.

Please check back here or subscribe to updates for any changes.
Posted May 24, 2024 - 08:29 EDT
Investigating
We are currently investigating an issue causing authentication failures in the Singapore region and are working to correct the issue as soon as possible. This is tied to the DUO68 deployment.

Please check back here or subscribe to updates for any changes.
Posted May 24, 2024 - 08:16 EDT
This incident affected: DUO68 (Core Authentication Service, SSO).