Bind without relays that fail the startup probe; test failover behind a direct path - #8
Merged
Merged
Conversation
The relay failover suite ran every scenario relay-only, so nothing showed
the failover acting while a direct path carried traffic, which is the
shape of the original incident (LAN clients kept working and masked it).
The harness now echoes every further stream on an authenticated
connection, and a client can hold its connection open (--hold-secs),
echoing on a fresh stream every few seconds and logging every path of
the connection with the selected one marked, like flextunnel's
connection-path status. It refuses mixed mode when built with the mdns
feature: on one host mDNS would hand the client the server's address and
the relays would never be exercised.
Phase D, server and held client with direct paths allowed:
D0 the held connection's paths are relay1 + direct, direct selected
D1 relay1 wedges; the failover still removes it and homes the server
on relay2 while the held client keeps echoing over its direct path;
the held connection ends up direct-only (iroh does not add a peer's
new home relay to an established connection); a fresh relay-only
client's path shows relay2
D2 a client with the same two-relay configuration redials during the
outage: it must come online, connect via relay2, and go direct again
D2 fails today: the redialing client's own net report prefers the wedged
relay (it answers probes fastest), its relay connection to it never
completes, iroh's online() needs a connected home relay, and
create_endpoint gives up after 10 s. The server's failover cannot help a
client that never comes online.
Every scenario that expects a specific home relay now asserts the relay
URL on the client's connection path, not just that an echo went through.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZhxxjP2TA93PHNr6p1BU8
A process starting while its preferred relay answers probes but refuses relay connections could never come online: iroh picks the home relay by probe latency, so it kept choosing the wedged relay, the relay connection failed forever, online() stayed pending, and create_endpoint gave up after 10 s. Every client restarting during such an outage failed to start although the other relay worked and the server had already failed over to it. This is what the watchdog restarts were working around, and DNS discovery cannot help: online() only watches the home relay status (verified with n0 DNS lookup and pkarr publishing enabled). probe_custom_relays now returns the relays that did not come online, and create_endpoint binds the endpoint with a relay map without them, so it homes on a relay that works. It returns a CreatedEndpoint carrying the relays left out; fail_over_home_relay takes them as already removed and its restore probe puts each one back once it is connectable again. The e2e scenario D2 (a client with the two-relay configuration redials after the server re-homed, relay1 still wedged) now passes: the client leaves relay1 out, connects via relay2, and moves onto a direct path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZhxxjP2TA93PHNr6p1BU8
Consumers must now take `CreatedEndpoint` from `create_endpoint` and hand its `relays_left_out` to `fail_over_home_relay`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZhxxjP2TA93PHNr6p1BU8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
online()waits only for a connected home relay, and iroh keeps choosing the wedged relay by latency). This is what the old watchdog restarts were working around. The startup probe now reports the relays that fail, the endpoint is bound without them, andfail_over_home_relayrestore-probes them every 90 s and puts them back once connectable. iroh DNS discovery was verified (source + empirical) not to fix this.create_endpointreturnsCreatedEndpoint { endpoint, relays_left_out };fail_over_home_relaytakesrelays_left_outas a third argument.run_relay_failover.shexercises the failover behind a live direct connection (mixed mode, built withoutmdns): the held client prints every connection path with its selected flag and RTT (like flextunnel's connection-path status), the server is asserted to re-home, a fresh relay-only client's path shows the new home relay, and a client redialing with the same two relays during the outage comes online through the new home relay and goes direct again (D2, the scenario that exposed the bug).Test plan
cargo testandcargo clippy --all-targets -- -D warnings, with and without--all-features./e2e/run_e2e.sh --local-relaysand--local-relays --relay-only./e2e/run_relay_failover.sh13/13 PASS (D2 fails without the fix)🤖 Generated with Claude Code
https://claude.ai/code/session_01RZhxxjP2TA93PHNr6p1BU8