fix: fail over to other regions when Cloud rejects a connection with 403 - #2097
Merged
xianshijing-lk merged 1 commit intoSep 11, 2026
Conversation
LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths when a project is not allowed in the region the client geo-routed to. `/settings/regions` is deliberately left reachable so the client can discover its allowed regions and connect there. Room.connect excluded every NotAllowed error from region failover, which covers both 401 and 403, so a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list. Key on the status rather than the server's message: that message is an unversioned human-readable string, and matching it would let a server-side copy edit break already-shipped clients. 401 stays terminal since no other region will accept the same token, as does the 404 "room does not exist" case that is also reported as NotAllowed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 8780967 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
size-limit report 📦
|
1egoman
approved these changes
Sep 11, 2026
xianshijing-lk
deleted the
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over
branch
September 11, 2026 20:33
This was referenced Sep 11, 2026
hiroshihorie
added a commit
to livekit/client-sdk-flutter
that referenced
this pull request
Sep 14, 2026
…403 (#1200) Part of CLT-3325. Companion PRs: [client-sdk-js#2097](livekit/client-sdk-js#2097), [client-sdk-swift](https://github.com/livekit/client-sdk-swift). ## Problem LiveKit Cloud enforces project-level region pinning by returning **403 on the RTC paths** (`/rtc`, `/rtc/validate`) when a project is not allowed in the region the client geo-routed to. `/settings/regions` is deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, *"allow other APIs to pass because clients will permanently give up if they fail."* `Room.connect` excluded **every** `NotAllowed` error from region failover: ```dart (e is WebSocketException || (e is ConnectException && e.reason != ConnectionErrorReason.NotAllowed)) ``` and the validate handler maps any status `>= 400` to `NotAllowed` (`signal_client.dart:209-215`). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in. ## Fix Extract the decision into `canFailOverToAnotherRegion` and key it on the HTTP status: - **403 → retry other regions.** This is the region-pinning signal. - **401 → terminal.** No other region will accept a token this one rejected. - `WebSocketException` and non-`NotAllowed` connect errors → retry, unchanged. Unrelated error types → terminal, unchanged. ### Why status and not the error message The server's body for this case is `"project not allowed in this region."`, but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. That is especially bad on mobile, where a released app cannot be hot-patched. The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still — `/settings/regions` returns 401 for it, and the existing handler already rethrows on that (`room.dart:375-377`). ## Cross-SDK status This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so **agents are unaffected**. ## Testing `test/support/region_failover_test.dart` covers the predicate directly. `dart analyze` is clean on all three touched files. **I could not execute the test suite locally** — this repo requires Dart >= 3.10.0 and the toolchain on hand is 3.9.2, so `flutter test` fails at dependency resolution before running anything. The test needs a CI run to confirm. ## Note, not fixed here `signal_client.dart:209-215` classifies any status `>= 400` as `NotAllowed`, so 5xx responses from the validate endpoint are also treated as permission failures. That looks wrong independently of region pinning — a 5xx should probably be retryable — but it is out of scope for this change and behaviour there is unchanged. ## Open question How often the RTC 403 actually fires for pinned projects has **not** been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in `prepareConnection()` warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Reviewer note (hiroshi) Checked against the Cloud and OSS server code. The pin gate in the Cloud auth middleware matches `/rtc` and `/rtc/v1` only, and the OSS validate handler has no 403 path, so `/rtc/validate` answers 200 for a pinned project that geo-routed to a disallowed region. The 403 lands on the websocket upgrade, which Flutter already surfaces as `WebSocketException` and fails over on. This change therefore does not alter behaviour on today's Cloud. It is kept as parity with client-sdk-js#2097 and stays correct if the gate is ever widened to validate. Follow-ups in #1204, stacked on this branch: retry every listed region instead of one, treat a 5xx or failed validate request as retryable, and stop retried attempts from emitting a disconnect that races the next attempt's cleanup. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of CLT-3325. Companion PRs: client-sdk-flutter, client-sdk-swift.
Problem
LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths (
/rtc,/rtc/validate) when a project is not allowed in the region the client geo-routed to./settings/regionsis deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, "allow other APIs to pass because clients will permanently give up if they fail."Room.connectexcluded everyNotAllowederror from region failover:and
handleConnectionErrormaps both 401 and 403 tonotAllowed(SignalClient.ts:1290-1292). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in.Fix
Extract the decision into
canFailOverToAnotherRegionand key it on the HTTP status:NotAllowed.Why status and not the error message
The server's body for this case is
"project not allowed in this region.", but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. Browser clients can be refreshed; mobile ones cannot.The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still —
/settings/regionsreturns 401 for it, and the existing handler already bails on that (Room.ts:908-916).Cross-SDK status
This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so agents are unaffected.
Testing
src/room/errors.test.tscovers the predicate directly (7 assertions, passing). Typecheck is clean for the touched files — the two pre-existingtscerrors inSignalClient.ts/SignalClientStateMachine.tscome from the missingmachinadependency onmainand are untouched by this change.Open question
How often the RTC 403 actually fires for pinned projects has not been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in
prepareConnection()warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed.Separately worth pursuing server-side: a structured discriminator — ideally a header carrying the allowed region URL — would both remove the ambiguity and let clients skip the
/settings/regionsround trip entirely. It must stay advisory so clients can keep this status-based fallback for older servers.🤖 Generated with Claude Code