fix: fail over to other regions when Cloud rejects a connection with 403 - #1200
Open
xianshijing-lk wants to merge 1 commit into
Open
Conversation
LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths when a project is not allowed in the region the client geo-routed to. /settings/regions is deliberately left reachable so the client can discover its allowed regions and connect there. Room.connect excluded every NotAllowed error from region failover, and the validate response maps any status >= 400 to NotAllowed, so a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list. Key on the status rather than the server's message: that message is an unversioned human-readable string, and matching it would let a server-side copy edit break already-shipped clients. 401 stays terminal since no other region will accept the same token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xianshijing-lk
requested review from
cloudwebrtc and
hiroshihorie
as code owners
September 11, 2026 19:32
There was a problem hiding this comment.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of CLT-3325. Companion PRs: client-sdk-js#2097, client-sdk-swift.
Problem
LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths (
/rtc,/rtc/validate) when a project is not allowed in the region the client geo-routed to./settings/regionsis deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, "allow other APIs to pass because clients will permanently give up if they fail."Room.connectexcluded everyNotAllowederror from region failover:and the validate handler maps any status
>= 400toNotAllowed(signal_client.dart:209-215). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in.Fix
Extract the decision into
canFailOverToAnotherRegionand key it on the HTTP status:WebSocketExceptionand non-NotAllowedconnect errors → retry, unchanged. Unrelated error types → terminal, unchanged.Why status and not the error message
The server's body for this case is
"project not allowed in this region.", but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. That is especially bad on mobile, where a released app cannot be hot-patched.The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still —
/settings/regionsreturns 401 for it, and the existing handler already rethrows on that (room.dart:375-377).Cross-SDK status
This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so agents are unaffected.
Testing
test/support/region_failover_test.dartcovers the predicate directly.dart analyzeis clean on all three touched files.I could not execute the test suite locally — this repo requires Dart >= 3.10.0 and the toolchain on hand is 3.9.2, so
flutter testfails at dependency resolution before running anything. The test needs a CI run to confirm.Note, not fixed here
signal_client.dart:209-215classifies any status>= 400asNotAllowed, so 5xx responses from the validate endpoint are also treated as permission failures. That looks wrong independently of region pinning — a 5xx should probably be retryable — but it is out of scope for this change and behaviour there is unchanged.Open question
How often the RTC 403 actually fires for pinned projects has not been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in
prepareConnection()warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed.🤖 Generated with Claude Code