On July 24, 2023, at around 20:20 EDT, Duo's Engineering Team was notified that a customer was seeing users’ iOS devices being blocked, despite having a policy to block only end-of-life iOS versions. The root cause was identified as Apple Rapid Security Response versions (e.g., “16.5.1 (c)”) being unknown in Duo’s version database, and incorrectly classified as end-of-life as a result.
The issue was resolved on July 25, 2023, by adding Apple Rapid Security Response versions as optional software versions in Duo’s version database.
2023-07-24 20:20 Duo Engineering is informed by Duo Customer Support that a customer is seeing iOS authentications blocked after the release of iOS 16.6. The customer has a “block if end-of-life” policy, but the impacted iOS versions are recent and not end-of-life.
2023-07-24 21:00 Many additional reports come in. Duo Customer Support works with customers to temporarily disable the “block if end-of-life” policy. Because there is a workaround, Duo Engineering commits to investigate in the morning.
2023-07-25 08:00 It is determined that Rapid Security Response iOS versions (predominantly “16.5.1 (c)”) are the affected versions. As a temporary stopgap, Duo Engineering marks iOS 16.5.1 as the “current” iOS version and 16.6 as an “optional” update. Users with a rapid security response version are no longer being blocked in error.
2023-07-25 08:49 Status page updated to Investigating.
2023-07-25 10:40 Duo Engineering determines that “16.5.1 (c)” and other rapid response versions are being reported by Duo Mobile but are not present in Duo’s version database, so the system cannot determine how out-of-date they are. The team manually adds these rapid security response versions to the database in their internal integration environment and performs manual verification testing.
2023-07-25 13:33 Status page updated to Identified.
2023-07-25 13:50 Duo Engineering adds the missing Rapid Security Response versions for iOS and macOS to the production version database and sets iOS 16.6 as the current version again. This change propagates to all systems within about 10 minutes, and the team monitors the logs to confirm that the erroneous blocks do not start again.
2023-07-25 15:22 Status page updated to Monitoring.
2023-07-25 16:04 Status page updated to Resolved.
Duo Mobile 4.42 added reporting for Rapid Security Response (RSR) version numbers. For example, it began reporting iOS version 16.5.1 (c) rather than 16.5.1 when that RSR is applied. The most recent release of Duo’s backend began ingesting these RSR versions and including them in authentication logs, endpoint records, etc.
Due to the timing of the release of different systems, Duo’s policy system did not yet have records for RSR versions like 16.5.1 (c). This did not cause any noticeable problems while 16.5.1 was the “current” version in Duo’s database — 16.5.1 (c) was unknown but was allowed because it was greater than the current version.
However, with the release of iOS 16.6, 16.5.1 (c) was still unknown but was now older than the current version. A blocking policy with a 0-day grace period trivially blocked these versions as potential end-of-life because the actual status could not be determined beyond being out-of-date.
As a short-term solution, Duo marked iOS 16.5.1 as current and 16.6 as optional. This stopped users from being blocked incorrectly (iOS 16.5.1 (c) was no longer older than the current version). But this workaround also meant that customers with a “less than latest” policy were temporarily not enforcing 16.6 as the required version.
The incident was resolved when Duo added iOS and macOS Rapid Security Response versions to its version database. These RSR versions were marked optional to avoid blocking sources (e.g., browser user agents) that do not report the RSR version.
This incident affected customers who have an iOS warning policy that would classify RSR versions as out-of-date, along with a 0-day blocking policy. It was reported by at least 15 customers.
The coming release of Duo’s backend software contains a more nuanced handling of RSR versions – if the user is using Duo Device Health App or Duo Mobile, RSR versions will be considered required rather than optional. Duo Engineering is also planning to improve the handling of unknown versions to make the system more robust. We will use adjacent versions to estimate how out-of-date the unknown version actually is.
Note: You can find your Duo deployment ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.