Replace the home-relay watchdog with in-place relay failover - #6
Merged
Merged
Conversation
The watchdog's two steps were a nudge that did nothing on a stable host (`network_change()` only asks netmon to re-read interfaces, which reports "no change") and a rebuild that dropped every connection. Both are gone. `relay_failover::fail_over_home_relay` watches the home-relay status and, after 60 s without a connected home relay, removes the wedged relay from the endpoint's relay map. The relay-map change forces a full net report, which can only prefer a relay still in the map, so the endpoint homes on another configured relay while its identity, direct paths and established connections stay untouched. The removed relay is probed every 90 s and put back once it is connectable again, so a relay that answers HTTP but refuses relay connections cannot be re-selected and fail again in a loop. Custom relay sets must now hold at least two distinct relays, since one relay leaves nothing to fail over to. The startup probe fails only when no relay is reachable and warns per unreachable relay, so a client restarting during a relay outage still starts on the surviving relay. `rebuild_endpoint`, `EndpointFactory` and `RebuildableEndpoint` are removed with the rebuild step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HnozJMiCj3v5Xfdb9gB5S8
3 tasks
The custom relay count is taken after deduplication, so say "at least two distinct relays" everywhere the minimum is described; listing the same relay twice is not a valid two-relay configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HnozJMiCj3v5Xfdb9gB5S8
Review fixes in relay_failover: - The restore wait was a nested loop: while a removed relay stayed unconnectable, a newly wedged home relay was reported but never failed over. The loop is now a single select over the report timer, the failover timer, the per-relay restore deadlines and the status watcher, tracking every removed relay with its own next probe time (each still spaced RELAY_RESTORE_INTERVAL apart so a returning relay gets a fresh actor). The last relay left in the map is never removed; the endpoint is nudged instead, and a nudge re-inserts only a relay that is still in the map, never a removed one. - restore() reports success only after insert_relay ran; a connectable relay missing from the configured map is logged and left out. - fail_over_home_relay no longer indexes urls[0]: an empty custom relay set (constructible from outside the crate) stays pending instead of panicking. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HnozJMiCj3v5Xfdb9gB5S8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Endpoint::network_change()) never did anything on a stable host: netmon re-reads the interfaces, finds no change, and nothing reaches the relay actor. The 180 s rebuild dropped every connection.relay_watchdog,rebuild_endpoint,EndpointFactoryandRebuildableEndpointare gone.relay_failover::fail_over_home_relay): after 60 s without a connected home relay, remove the wedged relay from the endpoint's relay map. The map is a shared handle, so the removal is seen by net_report and the relay actor; a relay-map change forces a full report; the report can only prefer a relay still in the map; the endpoint homes there. Identity, direct paths and established connections are untouched. The removed relay is probed every 90 s and re-inserted once connectable, so a relay that answers HTTP but refuses relay connections cannot be re-selected in a loop. With no home relay selected at all, the first relay is re-inserted unchanged to force a report.relay::MIN_CUSTOM_RELAYS); one relay leaves nothing to fail over to. The default relay map is unaffected.Design and iroh 1.1.0 analysis: flexaccessdev/iroh-common-architecture#relay-failover (
relay-failover.md). First consumer: flexaccessdev/tunnel-rsrelay-failover, whose fully offline e2e suite exercises the failover with a relay that answers/pingbut refuses relay connections.Test plan
cargo clippy --all-targets --all-features -- -D warningscargo test --all-features(22 tests; six paused-clock tests for the failover loop)test-scripts/run_relay_failover_e2e.sh, all ten scenarios, twicetest-scripts/run_e2e.shagainst two local dev relays, relay-only and normal🤖 Generated with Claude Code
https://claude.ai/code/session_01HnozJMiCj3v5Xfdb9gB5S8