On June 17th, 2024 at 7:45 am PST, the SRE team was notified that several user IP addresses had been resolved to the wrong geographic location during authentication and in authentication logs.
The issue was resolved on the same day by 9:25 am PST.
2024-06-17 7:45 am PST Duo Site Reliability Engineering (SRE) is informed by Duo Customer Support (CS) that three customers are reporting issues with IP geolocation.
2024-06-17 8:18 am PST Incident gets triaged to the SRE team
2024-06-17 8:18 am PST SRE team starts investigation.
2024-06-17 8:30 am PST SRE team identifies the issue
2024-06-17 9:08 am PST Incident Channel gets created.
2024-06-17 9:13 am PST Status page updated to Issue Identified.
2024-06-17 9:25 am PST Issue Resolved and SRE team is testing internally.
2024-06-17 9:25 am Status page updated to Monitoring.
2024-06-17 10:15 am PST Email sent to the customers that have reported the issues to verify it is resolved on their end.
2024-06-17 12:06 PST Status page updated to Resolved.
The SRE team was notified that an IP address from the US was incorrectly identified as originating from the UK, resulting in the policy denying access to users. Upon investigation, it was discovered that the latest GeoIP database used during authentication was missing, causing it to default to an outdated database. The latest database was from June 14, 2024, but the environment’s authentication service reverted to a much older version of the database following a migration to a new Operating System on the night of June 14th, 2024.
After discovering the issue, the SRE team retrieved the latest database and deployed it across all nodes in the environment. This ensured that the application picked up the update and properly processed the geo location information, resolving the access denial issue.
The solution implemented today was an immediate remedy. The SRE team has made necessary changes for a more permanent resolution. Once tested in the Dev environment and proven to be working, the latest database will be included in the packages installed during node creation. It also will be periodically updated in the customer’s environment to prevent this issue from occurring again in the future.