fix: try every Cloud region on initial connect and fail over on 5xx validate responses - #1204
Draft
hiroshihorie wants to merge 5 commits into
Draft
hiroshihorie wants to merge 5 commits into
hiroshihorie wants to merge 5 commits into
Conversation
LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths when a project is not allowed in the region the client geo-routed to. /settings/regions is deliberately left reachable so the client can discover its allowed regions and connect there. Room.connect excluded every NotAllowed error from region failover, and the validate response maps any status >= 400 to NotAllowed, so a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list. Key on the status rather than the server's message: that message is an unversioned human-readable string, and matching it would let a server-side copy edit break already-shipped clients. 401 stays terminal since no other region will accept the same token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lient-sdks-bail-on-region-pinning-403-insteadOf-failing-over
…on 5xx validate responses Room.connect retried a single region and then gave up. It now loops over the regions the server lists, the same as the reconnect path and the other SDKs. The validate response mapping treated every non-200 as NotAllowed, so a 5xx from a struggling node ended the connect instead of moving to the next region. Only 4xx is a verdict on the token or request now. A validate request that fails outright keeps the socket error for the same reason. Each retried attempt used to emit EngineDisconnectedEvent, which Room turned into a spurious RoomDisconnectedEvent and a cleanup racing the next attempt. The engine event is now emitted once, after the last attempt has failed. Adds sdkHttpClientOverride so tests can drive validate and the regions endpoint through package:http MockClient.
hiroshihorie
added a commit
that referenced
this pull request
Sep 14, 2026
…403 (#1200) Part of CLT-3325. Companion PRs: [client-sdk-js#2097](livekit/client-sdk-js#2097), [client-sdk-swift](https://github.com/livekit/client-sdk-swift). ## Problem LiveKit Cloud enforces project-level region pinning by returning **403 on the RTC paths** (`/rtc`, `/rtc/validate`) when a project is not allowed in the region the client geo-routed to. `/settings/regions` is deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, *"allow other APIs to pass because clients will permanently give up if they fail."* `Room.connect` excluded **every** `NotAllowed` error from region failover: ```dart (e is WebSocketException || (e is ConnectException && e.reason != ConnectionErrorReason.NotAllowed)) ``` and the validate handler maps any status `>= 400` to `NotAllowed` (`signal_client.dart:209-215`). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in. ## Fix Extract the decision into `canFailOverToAnotherRegion` and key it on the HTTP status: - **403 → retry other regions.** This is the region-pinning signal. - **401 → terminal.** No other region will accept a token this one rejected. - `WebSocketException` and non-`NotAllowed` connect errors → retry, unchanged. Unrelated error types → terminal, unchanged. ### Why status and not the error message The server's body for this case is `"project not allowed in this region."`, but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. That is especially bad on mobile, where a released app cannot be hot-patched. The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still — `/settings/regions` returns 401 for it, and the existing handler already rethrows on that (`room.dart:375-377`). ## Cross-SDK status This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so **agents are unaffected**. ## Testing `test/support/region_failover_test.dart` covers the predicate directly. `dart analyze` is clean on all three touched files. **I could not execute the test suite locally** — this repo requires Dart >= 3.10.0 and the toolchain on hand is 3.9.2, so `flutter test` fails at dependency resolution before running anything. The test needs a CI run to confirm. ## Note, not fixed here `signal_client.dart:209-215` classifies any status `>= 400` as `NotAllowed`, so 5xx responses from the validate endpoint are also treated as permission failures. That looks wrong independently of region pinning — a 5xx should probably be retryable — but it is out of scope for this change and behaviour there is unchanged. ## Open question How often the RTC 403 actually fires for pinned projects has **not** been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in `prepareConnection()` warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Reviewer note (hiroshi) Checked against the Cloud and OSS server code. The pin gate in the Cloud auth middleware matches `/rtc` and `/rtc/v1` only, and the OSS validate handler has no 403 path, so `/rtc/validate` answers 200 for a pinned project that geo-routed to a disallowed region. The 403 lands on the websocket upgrade, which Flutter already surfaces as `WebSocketException` and fails over on. This change therefore does not alter behaviour on today's Cloud. It is kept as parity with client-sdk-js#2097 and stays correct if the gate is ever widened to validate. Follow-ups in #1204, stacked on this branch: retry every listed region instead of one, treat a 5xx or failed validate request as retryable, and stop retried attempts from emitting a disconnect that races the next attempt's cleanup. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>
Base automatically changed from
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over
to
main
September 14, 2026 19:19
…er-all-regions # Conflicts: # lib/src/core/room.dart
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1200. Follow-up for the two Flutter-only gaps in initial-connect region failover.
Changes
Room.connecttries every region from/settings/regionsinstead of one. Same shape as the reconnect path and as JS, Swift, Android and Rust.NotAllowed. A 5xx now fails over, since it describes the node that answered, not the token. A validate request that fails outright keeps the socket error instead of replacing it with a transport error, which also fails over. 429 stays terminal, quota is project wide.EngineDisconnectedEvent. Room turned that into a spuriousRoomDisconnectedEventplus a_cleanUpracing the next attempt. The event is emitted once, after the last attempt fails.sdkHttpClientOverridetest hook so validate and the regions endpoint can be driven withpackage:httpMockClient.Server semantics this follows
Cloud auth middleware returns 401 and 403 for token, key and project state, 429 from the quota middleware, all project wide. 5xx comes from a key lookup or room store failure on that node, and the OSS validate returns 503 when node limits are reached. The regions endpoint is exempt from the pin gate and lists only regions where
/rtcwill pass.Tests
test/core/region_failover_loop_test.dart: all regions tried on 503, first accepting region wins with no disconnect event, 429 stops after the first attempt, failed validate request still fails over. Full suite green.Not run against Cloud. A multi-region failure cannot be produced on demand there.
🤖 Generated with Claude Code