Skip to content

fix: fail over to other regions when Cloud rejects a connection with 403 - #1200

Open
xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over
Open

fix: fail over to other regions when Cloud rejects a connection with 403#1200
xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/CLT-3325/some-client-sdks-bail-on-region-pinning-403-insteadOf-failing-over

Conversation

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Part of CLT-3325. Companion PRs: client-sdk-js#2097, client-sdk-swift.

Problem

LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths (/rtc, /rtc/validate) when a project is not allowed in the region the client geo-routed to. /settings/regions is deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, "allow other APIs to pass because clients will permanently give up if they fail."

Room.connect excluded every NotAllowed error from region failover:

(e is WebSocketException || (e is ConnectException && e.reason != ConnectionErrorReason.NotAllowed))

and the validate handler maps any status >= 400 to NotAllowed (signal_client.dart:209-215). So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list, and never reached the region it was allowed in.

Fix

Extract the decision into canFailOverToAnotherRegion and key it on the HTTP status:

  • 403 → retry other regions. This is the region-pinning signal.
  • 401 → terminal. No other region will accept a token this one rejected.
  • WebSocketException and non-NotAllowed connect errors → retry, unchanged. Unrelated error types → terminal, unchanged.

Why status and not the error message

The server's body for this case is "project not allowed in this region.", but that is an unversioned human-readable string. Matching it would mean five SDKs carrying identical literals forever, and a server-side copy edit — dropping the period, rewording — would silently break already-shipped clients. That is especially bad on mobile, where a released app cannot be hot-patched.

The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later. A genuinely bad token is capped tighter still — /settings/regions returns 401 for it, and the existing handler already rethrows on that (room.dart:375-377).

Cross-SDK status

This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so agents are unaffected.

Testing

test/support/region_failover_test.dart covers the predicate directly. dart analyze is clean on all three touched files.

I could not execute the test suite locally — this repo requires Dart >= 3.10.0 and the toolchain on hand is 3.9.2, so flutter test fails at dependency resolution before running anything. The test needs a CI run to confirm.

Note, not fixed here

signal_client.dart:209-215 classifies any status >= 400 as NotAllowed, so 5xx responses from the validate endpoint are also treated as permission failures. That looks wrong independently of region pinning — a 5xx should probably be retryable — but it is out of scope for this change and behaviour there is unchanged.

Open question

How often the RTC 403 actually fires for pinned projects has not been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in prepareConnection() warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed.

🤖 Generated with Claude Code

LiveKit Cloud enforces project-level region pinning by returning 403 on
the RTC paths when a project is not allowed in the region the client
geo-routed to. /settings/regions is deliberately left reachable so the
client can discover its allowed regions and connect there.

Room.connect excluded every NotAllowed error from region failover, and
the validate response maps any status >= 400 to NotAllowed, so a pinned
project that geo-routed to a disallowed region gave up before ever
fetching the region list.

Key on the status rather than the server's message: that message is an
unversioned human-readable string, and matching it would let a server-side
copy edit break already-shipped clients. 401 stays terminal since no other
region will accept the same token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant