diff --git a/AGENTS.md b/AGENTS.md index 664eaf3..a3ad264 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,4 +7,4 @@ - the desktop client (`flextunnel-desktop`) normally stores its config in the system keychain; set `FLEXTUNNEL_DEV_CONFIG=1` (or a file path) to store it as plaintext JSON instead, avoiding the macOS keychain access prompt on every unsigned rebuild. Development only — never set it for a real install (the auth secret key is stored unencrypted). - after rust changes that affect iOS (flextunnel-core or flextunnel-ffi, including the FFI config schema and `ios/flextunnel.h`), run `./build-ios.sh release` to rebuild `libflextunnel.xcframework` into `dist/ios/` (this script no longer writes into `../flextunnel-ios`). The iOS app links via its own Swift package (`../flextunnel-ios/Packages/Flextunnel`), which **defaults to the pinned GitHub release**, so it won't see local changes unless you build the app with `FLEXTUNNEL_LOCAL_XCFRAMEWORK=1` — that links this fresh `dist/ios` build through a committed symlink (set it for both `xcodegen generate` and `xcodebuild`, then clean-rebuild). This is an **extra step only needed when actively working on the iOS app side by side** (and only possible on macOS with Xcode + the iOS Rust targets); otherwise just skip it. - the iroh transport layer shared with tunnel-rs and ezvpn — relays and address lookup, the per-relay startup probe, relay auth tokens, relay self-hosting — is documented once in https://github.com/flexaccessdev/iroh-common-architecture. Do not duplicate it in this repo; update it there and link to it. -- that shared layer's code — `RelayConfig` and the relay probe, endpoint building and rebuild, the home-relay watchdog, the endpoint-bound auth transcript — lives in the `flexaccess-iroh` crate (`../flexaccess-iroh`, consumed by git tag). Fix it there, tag a release, and bump the tag here; never re-implement or fork a copy of it in this repo. Only flextunnel-specific pieces (ALPNs, the allowlist hook, the auth context, QUIC tuning) belong in `transport/` and `auth.rs`. +- that shared layer's code — `RelayConfig` and the relay probe, endpoint building (bound without the relays that fail the probe), the server's in-place home-relay failover, the endpoint-bound auth transcript — lives in the `flexaccess-iroh` crate (`../flexaccess-iroh`, consumed by git tag). Fix it there, tag a release, and bump the tag here; never re-implement or fork a copy of it in this repo. Only flextunnel-specific pieces (ALPNs, the allowlist hook, the auth context, QUIC tuning, and the client's rebuildable endpoint `ClientEndpoint` — the reconnect loop's escalation, which the crate does not carry) belong in `transport/` and `auth.rs`. diff --git a/Cargo.lock b/Cargo.lock index 76cec53..d73a56d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -181,7 +181,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -192,7 +192,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1452,7 +1452,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccc2776f0c61eca1ca32528f85548abd1a4be8fb53d1b21c013e4f18da1e7090" dependencies = [ "data-encoding", - "syn 1.0.109", + "syn 2.0.118", ] [[package]] @@ -1578,7 +1578,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -1776,7 +1776,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -1919,8 +1919,8 @@ dependencies = [ [[package]] name = "flexaccess-iroh" -version = "0.0.3" -source = "git+https://github.com/flexaccessdev/flexaccess-iroh?tag=v0.0.3#422dacee5b38a667bd6da2b709d28d86c8d7661f" +version = "0.0.8" +source = "git+https://github.com/flexaccessdev/flexaccess-iroh?tag=v0.0.8#e718f5d0b00f9837a7bea50114d77320f31f5197" dependencies = [ "anyhow", "base64", @@ -1945,7 +1945,7 @@ dependencies = [ [[package]] name = "flextunnel-cli" -version = "0.0.75" +version = "0.0.76" dependencies = [ "anyhow", "clap", @@ -1961,7 +1961,7 @@ dependencies = [ [[package]] name = "flextunnel-core" -version = "0.0.75" +version = "0.0.76" dependencies = [ "anyhow", "askama", @@ -1992,7 +1992,7 @@ dependencies = [ [[package]] name = "flextunnel-desktop" -version = "0.0.75" +version = "0.0.76" dependencies = [ "aes-gcm", "anyhow", @@ -2021,7 +2021,7 @@ dependencies = [ [[package]] name = "flextunnel-ffi" -version = "0.0.75" +version = "0.0.76" dependencies = [ "flextunnel-core", "iroh", @@ -2949,7 +2949,7 @@ dependencies = [ "js-sys", "log", "wasm-bindgen", - "windows-core 0.58.0", + "windows-core 0.62.2", ] [[package]] @@ -4097,7 +4097,7 @@ dependencies = [ "once_cell", "png 0.18.1", "thiserror 2.0.18", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4492,7 +4492,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4538,7 +4538,7 @@ version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" dependencies = [ - "proc-macro-crate 1.3.1", + "proc-macro-crate 2.0.2", "proc-macro2", "quote", "syn 2.0.118", @@ -6063,7 +6063,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.12.1", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6121,7 +6121,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6226,7 +6226,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5b55fb86dfd3a2f5f76ea78310a88f96c4ea21a3031f8d212443d56123fd0521" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6585,7 +6585,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -6851,7 +6851,7 @@ dependencies = [ "getrandom 0.4.3", "once_cell", "rustix 1.1.4", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -6873,7 +6873,7 @@ dependencies = [ "parking_lot", "rustix 1.1.4", "signal-hook", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -7407,7 +7407,7 @@ dependencies = [ "once_cell", "png 0.18.1", "thiserror 2.0.18", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -7445,7 +7445,7 @@ checksum = "f2f6fb2847f6742cd76af783a2a2c49e9375d0a111c7bef6f71cd9e738c72d6e" dependencies = [ "memoffset", "tempfile", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -8134,7 +8134,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 74252e1..0b44c87 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,7 +10,7 @@ default-members = [ ] [workspace.package] -version = "0.0.75" +version = "0.0.76" edition = "2024" description = "SOCKS5/HTTP-proxy-over-QUIC split tunnel via iroh P2P" @@ -31,7 +31,7 @@ hickory-resolver = { version = "0.26", default-features = false, features = ["to # Shared iroh transport layer (relay config + probe, endpoint building and # rebuild, home-relay watchdog, endpoint-bound auth transcript). `mdns` turns on # local-network discovery on every endpoint (compiled out on iOS by the crate). -flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.3", features = ["mdns"] } +flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.8", features = ["mdns"] } iroh = "1.1.0" libc = "0.2" log = "0.4" diff --git a/README.md b/README.md index c385b1c..eae274d 100644 --- a/README.md +++ b/README.md @@ -400,7 +400,7 @@ Client auth keypairs are generated with the standalone | `--default-config` | Load `~/.config/flextunnel/server.toml`. | | `--secret-file ` | Server identity key. | | `--authorized-keys-file ` | File of authorized client public keys, one `ed25519-pub:…` per line (optional trailing comment, ssh `authorized_keys` style). | -| `--relay-url ` | Custom relay URL(s) for failover (repeatable). Configuring custom relays disables n0 internet discovery: clients reach this server via relay hints, and outbound bridges attach the same hints when dialing peer servers. mDNS local discovery stays on. | +| `--relay-url ` | Custom relay URLs (repeatable; at least two distinct relays, since the server rides out a relay outage by moving onto another one). Configuring custom relays disables n0 internet discovery: clients reach this server via relay hints, and outbound bridges attach the same hints when dialing peer servers. mDNS local discovery stays on. | | `--relay-auth-token ` | Shared bearer token sent to every custom relay's WebSocket upgrade. Only valid with `--relay-url` (rejected with the default relays). | | `--quick` | Ephemeral one-off server: prompt for the client's EndpointId (shown by `client start --quick`) and natively allowlist it as the only allowed client — no auth keypair — then mint an in-memory identity, full-tunnel all traffic, print this server's EndpointId, and exit if the client doesn't connect within 5 minutes. Needs an interactive terminal. Takes no single-instance lock; nothing is persisted. Conflicts with `-c`/`--secret-file`/`--authorized-keys-file`. | @@ -413,7 +413,7 @@ Client auth keypairs are generated with the standalone | `--socks-port ` | Optional SOCKS5 listener port, e.g. `1080`. Binds `127.0.0.1` only. Disabled unless set. | | `--http-port ` | Optional HTTP proxy listener port (CONNECT + plain-HTTP forwarding). Binds `127.0.0.1` only. | | `--auth-key ` / `--auth-key-file ` | Client auth keypair (one required): the inline `ed25519-sec:…` secret, or the key file from `flexaccess-keys generate-auth-key`. | -| `--relay-url ` | Custom relay URL(s) for failover (repeatable). Configuring custom relays disables n0 internet discovery (the server is reached via relay hints); mDNS local discovery stays on. | +| `--relay-url ` | Custom relay URLs (repeatable; at least two distinct relays, the same set as the server). Configuring custom relays disables n0 internet discovery (the server is reached via relay hints); mDNS local discovery stays on. | | `--relay-auth-token ` | Shared bearer token sent to every custom relay's WebSocket upgrade. Only valid with `--relay-url` (rejected with the default relays). | | `--auto-reconnect` | Force auto-reconnect on (overrides `auto_reconnect = false` in the config). | | `--no-auto-reconnect` | Exit on the first disconnection instead of reconnecting. | @@ -620,11 +620,13 @@ Auto-reconnect is **enabled by default** (`auto_reconnect = true`); pass and only then fail with a network-unreachable reply. A **server** with custom relays watches its own home-relay registration: if it -has no connected home relay for 60s it re-checks the network, and if that -has not helped by 180s it rebuilds its endpoint in place (same server id) — -the in-process equivalent of a restart, so relay-only clients (the iOS app, -anything off the LAN) are not stranded until someone restarts the service. -See [`docs/architecture.md`](docs/architecture.md#relay-watchdog-server-custom-relays). +has no connected home relay for 60s and iroh has not re-homed it on its own, +it takes the wedged relay out of its relay map and homes on another configured +relay in place (same server id, same sockets, nothing dropped), so clients off +the LAN (the iOS app) are not stranded until someone restarts the service. The +relay is put back once it is connectable again. A custom relay set is +therefore at least two distinct relays. See +[`docs/architecture.md`](docs/architecture.md#relay-failover-server-custom-relays). ## Logging diff --git a/crates/flextunnel-cli/src/main.rs b/crates/flextunnel-cli/src/main.rs index b36b82a..80ba56a 100644 --- a/crates/flextunnel-cli/src/main.rs +++ b/crates/flextunnel-cli/src/main.rs @@ -13,7 +13,6 @@ use clap::{Parser, Subcommand}; use std::io::IsTerminal; use std::num::NonZeroU32; use std::path::PathBuf; -use std::pin::pin; use std::sync::Arc; use std::time::Duration; use tokio::sync::Notify; @@ -33,10 +32,9 @@ use flextunnel_core::proxy::{ BridgeUpstream, BridgeUpstreamConfig, DnsForwarder, ProxyServer, ProxyServerParams, RoutedSet, }; use flextunnel_core::secret::secret_to_endpoint_id; -use flextunnel_core::transport::endpoint::{ - EndpointAllowlists, RelayConfig, create_server_endpoint, server_rebuild_factory, -}; -use flextunnel_core::flexaccess_iroh::relay_watchdog::{self, RelayOutage}; +use flextunnel_core::transport::endpoint::{EndpointAllowlists, RelayConfig, create_server_endpoint}; +use flextunnel_core::flexaccess_iroh::endpoint::CreatedEndpoint; +use flextunnel_core::flexaccess_iroh::relay_failover::fail_over_home_relay; use flextunnel_core::{auth, config, secret}; #[derive(Parser)] @@ -617,28 +615,6 @@ struct QuickServer { /// relay/connection teardown must never leave the process unkillable. const SHUTDOWN_CLOSE_TIMEOUT: Duration = Duration::from_secs(5); -/// Pause between attempts to bind a replacement endpoint after the relay -/// watchdog retired the old one and the rebuild itself failed (e.g. no route -/// to bind on). The server has no endpoint at all during this wait, so it is -/// short — there is nothing to lose by trying again soon. -const REBUILD_RETRY: Duration = Duration::from_secs(30); - -/// Cap on the watchdog's rebuild deadline once consecutive rebuilt endpoints -/// keep failing to register on any home relay. -const REBUILD_DEADLINE_MAX: Duration = Duration::from_secs(30 * 60); - -/// The watchdog's rebuild deadline for the next serve pass, given how many -/// endpoints in a row never registered on a home relay: the usual -/// [`relay_watchdog::RELAY_OUTAGE_REBUILD`] after an endpoint that did -/// register, doubling per unregistered endpoint up to [`REBUILD_DEADLINE_MAX`] -/// (180s, 6m, 12m, 24m, 30m). Rebuilding while the relay itself is down -/// gains nothing and drops every LAN client, so it is done less and less -/// often; a relay that comes back resets the escalation. -fn rebuild_deadline(unregistered_endpoints: u32) -> Duration { - let factor = 1u32 << unregistered_endpoints.min(4); - (relay_watchdog::RELAY_OUTAGE_REBUILD * factor).min(REBUILD_DEADLINE_MAX) -} - /// Build the ephemeral `ServerConfig` for `server start --quick`: a full-tunnel /// routed set (`routed_domains = ["*"]`, `routed_cidrs = ["0.0.0.0/0", "::/0"]`) /// plus a freshly generated in-memory identity, returned *alongside* the config — @@ -823,16 +799,10 @@ async fn run_server( .map(|q| std::collections::HashSet::from([q.client_id])) .unwrap_or_default(), }; - let endpoint = create_server_endpoint(&relay_config, secret_key.clone(), allowlists.clone()) - .await - .context("Failed to create iroh endpoint")?; - // The relay watchdog's remedy of last resort: a fresh endpoint with the - // same identity and allowlists (see the serve loop below). - let rebuild = server_rebuild_factory(relay_config.clone(), secret_key, allowlists); - // Only a custom-relay server hangs its reachability on one home-relay - // registration (n0 discovery is off, clients dial by relay hint), so the - // watchdog is armed for custom relays only. - let relay_watchdog_armed = relay_config.is_custom(); + let CreatedEndpoint { endpoint, relays_left_out } = + create_server_endpoint(&relay_config, secret_key, allowlists) + .await + .context("Failed to create iroh endpoint")?; log::info!("flextunnel server Node ID: {}", endpoint.id()); match &quick { @@ -881,8 +851,8 @@ async fn run_server( // forever). `first_client` is `Some` exactly in quick mode; a normal server // parks here immediately, so the arm never fires. `notify_one` stores a // permit, so a client that connects before this future is first polled is - // not missed. Pinned outside the serve loop so it spans endpoint rebuilds. - let mut grace = pin!(async { + // not missed. + let grace = async { match &first_client { Some(notify) => { tokio::select! { @@ -892,106 +862,32 @@ async fn run_server( } None => std::future::pending::<()>().await, } - }); - // One signal listener for the whole serve loop: re-registering it per pass - // could drop a signal delivered while an endpoint is being rebuilt. - let mut shutdown = pin!(app::shutdown_signal()); - - /// How one pass of the serve loop ended. - enum Pass { - /// The server is done (clean or failed): close the endpoint and return. - Exit(Result<()>), - /// The relay watchdog gave up on the endpoint. - Rebuild(RelayOutage), - } - - // Serve loop. A pass serves on the current endpoint until the server ends, - // a shutdown signal arrives, the quick-mode grace expires, or — custom - // relays only — the relay watchdog reports the endpoint has lost its home - // relay for good. That last case is the in-process equivalent of the - // process restart known to fix it: close the wedged endpoint, bind a fresh - // one with the same identity, and serve again. The `ProxyServer` (its - // registries, blocklist, status state) carries over; the old endpoint's - // connections and bridge tasks end with it. - // - // A rebuild only helps when iroh's relay bookkeeping went stale. When the - // relay itself is unreachable the fresh endpoint never registers either, - // and rebuilding it again every few minutes would keep dropping the LAN - // clients that still work. So consecutive endpoints that never saw a home - // relay lengthen the watchdog's deadline (`rebuild_deadline`); one that - // did register resets the escalation. - let mut endpoint = endpoint; - let mut unregistered_endpoints: u32 = 0; - let res = loop { - let pass = { - let run = Arc::clone(&server).run(&endpoint); - let deadline = rebuild_deadline(unregistered_endpoints); - let outage = async { - if relay_watchdog_armed { - relay_watchdog::watch_home_relay(&endpoint, deadline).await - } else { - std::future::pending().await - } - }; - tokio::select! { - res = run => Pass::Exit(res.map_err(|e| anyhow::anyhow!("Server error: {e}"))), - sig = &mut shutdown => Pass::Exit(sig.map(|()| { - log::info!("Received shutdown signal, stopping server"); - })), - _ = &mut grace => { - log::warn!("Quick mode: no client connected within 5 minutes — exiting"); - Pass::Exit(Ok(())) - } - outage = outage => Pass::Rebuild(outage), - } - }; - let outage = match pass { - Pass::Exit(res) => break res, - Pass::Rebuild(outage) => outage, - }; + }; - unregistered_endpoints = if outage.relay_seen { 0 } else { unregistered_endpoints + 1 }; - log::error!( - "No connected home relay for {:.0}s despite a network re-check; rebuilding the \ - endpoint from scratch (server id stays {})", - outage.duration.as_secs_f64(), - endpoint.id() - ); - if unregistered_endpoints > 0 { - log::error!( - "{unregistered_endpoints} endpoint(s) in a row never registered on any home \ - relay; the relay itself is probably unreachable. If the rebuilt endpoint does \ - not register either, the next rebuild waits {}s", - rebuild_deadline(unregistered_endpoints).as_secs() - ); + // Serve until the server ends, a shutdown signal arrives, or the + // quick-mode grace expires. Alongside, with custom relays, the shared + // home-relay failover keeps the server dialable: a custom-relay server is + // reachable from off the LAN only through its home relay (n0 discovery is + // off, clients dial by relay hint), and if that relay is lost for a minute + // without iroh re-homing on its own, the failover moves the endpoint onto + // another configured relay in place. Nothing is torn down: the identity, + // the allowlists, the direct paths, the established connections and the + // bridge tasks all stay. Relays the startup probe could not connect (the + // endpoint was bound without them) are put back by the same failover once + // they are. With the default relays the failover future is pending + // forever. + let res = tokio::select! { + res = Arc::clone(&server).run(&endpoint) => { + res.map_err(|e| anyhow::anyhow!("Server error: {e}")) } - close_endpoint_or_exit(&endpoint).await; - endpoint = loop { - match rebuild().await { - Ok(fresh) => break fresh, - Err(e) => { - log::error!( - "Endpoint rebuild failed: {e:#}; retrying in {}s", - REBUILD_RETRY.as_secs() - ); - // Nothing is bound while waiting here, so both exits - // below return directly: there is no endpoint to close. - tokio::select! { - _ = tokio::time::sleep(REBUILD_RETRY) => {} - sig = &mut shutdown => { - sig?; - log::info!("Received shutdown signal, stopping server"); - return Ok(()); - } - _ = &mut grace => { - log::warn!("Quick mode: no client connected within 5 minutes — exiting"); - return Ok(()); - } - } - } - } - }; - log::warn!("Endpoint rebuilt; serving again as {}", endpoint.id()); + sig = app::shutdown_signal() => sig.map(|()| { + log::info!("Received shutdown signal, stopping server"); + }), + _ = grace => { + log::warn!("Quick mode: no client connected within 5 minutes — exiting"); + Ok(()) + } + () = fail_over_home_relay(&endpoint, &relay_config, &relays_left_out) => Ok(()), }; close_endpoint_or_exit(&endpoint).await; @@ -1025,17 +921,6 @@ mod tests { use super::*; use std::collections::HashMap; - #[test] - fn rebuild_deadline_doubles_per_unregistered_endpoint_up_to_the_cap() { - let base = relay_watchdog::RELAY_OUTAGE_REBUILD; - assert_eq!(rebuild_deadline(0), base); - assert_eq!(rebuild_deadline(1), base * 2); - assert_eq!(rebuild_deadline(2), base * 4); - assert_eq!(rebuild_deadline(3), base * 8); - assert_eq!(rebuild_deadline(4), REBUILD_DEADLINE_MAX); - assert_eq!(rebuild_deadline(50), REBUILD_DEADLINE_MAX); - } - fn forwarder(suffix: &str) -> DnsForwarder { let mut m = HashMap::new(); m.insert(suffix.to_string(), vec!["10.0.0.53".to_string()]); diff --git a/crates/flextunnel-core/src/lib.rs b/crates/flextunnel-core/src/lib.rs index 0e6e1df..1f4786e 100644 --- a/crates/flextunnel-core/src/lib.rs +++ b/crates/flextunnel-core/src/lib.rs @@ -18,8 +18,8 @@ pub use iroh; // Same for the shared FlexAccess key crate (`ed25519-sec:` / `ed25519-pub:` // tokens, key files, authorized-keys parsing) that `auth` builds on, and the -// shared iroh transport crate (relay config, endpoint rebuild, relay -// watchdog) that `transport` builds on. +// shared iroh transport crate (relay config and probe, endpoint building, +// the server's in-place home-relay failover) that `transport` builds on. pub use flexaccess_iroh; pub use flexaccess_keys; diff --git a/crates/flextunnel-core/src/proxy/bridge.rs b/crates/flextunnel-core/src/proxy/bridge.rs index 23406c1..615ed2d 100644 --- a/crates/flextunnel-core/src/proxy/bridge.rs +++ b/crates/flextunnel-core/src/proxy/bridge.rs @@ -106,9 +106,7 @@ impl BridgeUpstream { /// only when `endpoint` closes underneath it, failing each retry. pub async fn run(self: Arc, endpoint: Endpoint) { let name = &self.config.name; - // A previous run (on a since-closed endpoint, see the server's relay - // watchdog rebuild) may have been aborted while connected; its stale - // connection must not read as live until this run establishes its own. + // Nothing reads as live until this run establishes its own connection. *self.conn.lock().expect("bridge conn lock") = None; let mut attempt: u32 = 0; loop { diff --git a/crates/flextunnel-core/src/proxy/server.rs b/crates/flextunnel-core/src/proxy/server.rs index f2dc022..08e8b62 100644 --- a/crates/flextunnel-core/src/proxy/server.rs +++ b/crates/flextunnel-core/src/proxy/server.rs @@ -307,12 +307,7 @@ impl ProxyServer { } /// Accept connections until the endpoint closes or the server self-blocks. - /// - /// May be called again on a *fresh* endpoint after the previous one was - /// closed (the relay watchdog's rebuild): the registries and blocklist - /// carry over, the old endpoint's connection handlers end as its - /// connections close, and its bridge tasks are aborted when the previous - /// `run` future is dropped. + /// The bridge tasks are owned by this future and end with it. pub async fn run(self: Arc, endpoint: &Endpoint) -> ProxyResult<()> { // Maintain the outbound bridge upstreams for the life of this run. The // bridging side dials out on this same server endpoint, so the TLS diff --git a/crates/flextunnel-core/src/transport/endpoint.rs b/crates/flextunnel-core/src/transport/endpoint.rs index 4047446..66b1df5 100644 --- a/crates/flextunnel-core/src/transport/endpoint.rs +++ b/crates/flextunnel-core/src/transport/endpoint.rs @@ -1,15 +1,18 @@ //! flextunnel's endpoints: what this program layers onto the shared //! [`flexaccess_iroh::endpoint`] builder — its three ALPNs, the native -//! per-ALPN allowlist hook, and the client/server identity rules. Relay -//! configuration, the per-relay startup probe, the creation-vs-rebuild -//! policy, and the rebuildable endpoint handle all come from the shared crate; -//! the server's secret-key file is [`crate::secret`]'s. +//! per-ALPN allowlist hook, the client/server identity rules, and the +//! client's rebuildable endpoint handle ([`ClientEndpoint`], the reconnect +//! loop's escalation for a wedged endpoint). Relay configuration, the +//! per-relay startup probe, the bind-and-come-online policy, and the server's +//! in-place home-relay failover come from the shared crate; the server's +//! secret-key file is [`crate::secret`]'s. use crate::transport::{ALPN, BRIDGE_ALPN, QUICK_ALPN, build_quic_transport_config}; -use anyhow::Result; +use anyhow::{Context, Result}; use flexaccess_iroh::endpoint::{ - EndpointOptions, create_endpoint, endpoint_builder, rebuild_endpoint, + CreatedEndpoint, EndpointOptions, create_endpoint, endpoint_builder, }; +use futures::future::BoxFuture; use iroh::{ Endpoint, EndpointId, SecretKey, endpoint::{ @@ -19,8 +22,8 @@ use iroh::{ }; use std::collections::HashSet; use std::sync::Arc; +use std::time::Duration; -pub use flexaccess_iroh::endpoint::{EndpointFactory, RebuildableEndpoint as ClientEndpoint}; pub use flexaccess_iroh::relay::{RELAY_CONNECT_TIMEOUT, RelayConfig}; /// QUIC application close code sent when a connection is rejected by an @@ -118,9 +121,7 @@ fn base_builder(relay_config: &RelayConfig, publish_address: bool) -> Result Result { +) -> Result { create_endpoint(relay_config, server_builder(relay_config, secret, allowlists)?).await } -/// The rebuild recipe for the server endpoint, used when the relay watchdog -/// gives up on the current one. Same identity and allowlists as the original, -/// so the server's id — what clients dial — never changes. Tolerant rebuild -/// policy (see [`rebuild_endpoint`]): no relay probe, and the online wait may -/// fail — the watchdog trips again if the relays stay unreachable, with a -/// lengthening deadline so a dead relay does not churn the endpoint every few -/// minutes (see the CLI's serve loop). -pub fn server_rebuild_factory( - relay_config: RelayConfig, - secret: SecretKey, - allowlists: EndpointAllowlists, -) -> EndpointFactory { - Arc::new(move || { - let relay_config = relay_config.clone(); - let secret = secret.clone(); - let allowlists = allowlists.clone(); - Box::pin(async move { - rebuild_endpoint(server_builder(&relay_config, secret, allowlists)?).await - }) - }) -} - /// A client endpoint builder. A client never publishes its address (it only /// dials out), even with a quick-mode secret bound as its identity. fn client_builder(relay_config: &RelayConfig, secret: Option) -> Result { @@ -184,7 +166,9 @@ fn client_builder(relay_config: &RelayConfig, secret: Option) -> Resu } /// Create a client endpoint (ephemeral identity) that can rebuild itself -/// mid-session. Strict first-creation policy, tolerant rebuilds. +/// mid-session. The first creation probes every custom relay and must come +/// online (see [`create_endpoint`]); rebuilds are tolerant (see +/// [`rebuild_client_endpoint`]). pub async fn create_client_endpoint(relay_config: &RelayConfig) -> Result { create_client(relay_config, None).await } @@ -204,14 +188,233 @@ pub async fn create_quick_client_endpoint( } async fn create_client(relay_config: &RelayConfig, secret: Option) -> Result { - let endpoint = create_endpoint(relay_config, client_builder(relay_config, secret.clone())?).await?; + // A relay that failed the startup probe stays out of this endpoint's relay + // map for its lifetime: a client only dials, runs no failover, and + // reaches the server through the relay hints it attaches, so a rebuilt + // endpoint (below) goes back to the full configured set instead. + let CreatedEndpoint { endpoint, .. } = + create_endpoint(relay_config, client_builder(relay_config, secret.clone())?).await?; let factory: EndpointFactory = { let relay_config = relay_config.clone(); Arc::new(move || { let relay_config = relay_config.clone(); let secret = secret.clone(); - Box::pin(async move { rebuild_endpoint(client_builder(&relay_config, secret)?).await }) + Box::pin(async move { + rebuild_client_endpoint(client_builder(&relay_config, secret)?).await + }) }) }; Ok(ClientEndpoint::from_parts(endpoint, factory)) } + +/// Mid-session replacement of a client endpoint, the recipe behind an +/// [`EndpointFactory`]. Differs from [`create_endpoint`] deliberately: +/// +/// - **No per-relay probe.** At creation the probe validates the +/// configuration; during an outage it would only delay the reconnect it is +/// part of. +/// - **The online wait is tolerated failing.** A fresh endpoint is no worse +/// than the wedged one it replaces — a client only dials, so its relay +/// hints and mDNS can still reach the server without a home relay — and the +/// reconnect loop that asked for the rebuild escalates again if the relays +/// stay unreachable. +async fn rebuild_client_endpoint(builder: EndpointBuilder) -> Result { + let endpoint = builder.bind().await.context("Failed to create iroh endpoint")?; + if tokio::time::timeout(RELAY_CONNECT_TIMEOUT, endpoint.online()) + .await + .is_err() + { + log::warn!( + "Rebuilt endpoint has no connected home relay after {}s; continuing (relay hints \ + and local discovery may still reach the server)", + RELAY_CONNECT_TIMEOUT.as_secs() + ); + } + Ok(endpoint) +} + +/// Recipe producing a fresh, fully bound endpoint — how a [`ClientEndpoint`] +/// replaces itself mid-session. +pub type EndpointFactory = Arc BoxFuture<'static, Result> + Send + Sync>; + +/// Bound wait on the old endpoint's graceful close during a rebuild. The close +/// runs as its own task and is never cancelled (dropping a bound endpoint +/// without `close()` is fatal under panic=abort); the bound only keeps the +/// reconnect loop from stalling behind it, letting a slow close finish in the +/// background. +const REBUILD_CLOSE_TIMEOUT: Duration = Duration::from_secs(5); + +/// A client endpoint handle that can be **rebuilt** from scratch mid-session. +/// +/// `Endpoint::network_change()` re-binds dead UDP transports, but a wedged +/// endpoint can be broken beyond what a rebind repairs: a relay link lost to a +/// ping timeout that never re-establishes, stale cached paths for the server, +/// dead discovery state. A process restart always recovers because it builds a +/// brand-new endpoint; [`Self::rebuild`] gives the reconnect loop that same +/// remedy in-process — fresh sockets, fresh relay connections, fresh discovery +/// — without dropping anything else the process holds (the bound proxy +/// listeners, port forwards, the status socket). +/// +/// The handle is `Clone` and shared: the reconnect loop escalates to +/// [`Self::rebuild`] after repeated failures, while the embedder logs +/// [`Self::id`] and [`Self::close`]s whatever endpoint is current at teardown. +/// Client-only: the server's endpoint is bound once and kept, its home relay +/// moved in place by the shared failover. +#[derive(Clone)] +pub struct ClientEndpoint { + /// The live endpoint, swapped by [`Self::rebuild`]. Std lock: accessors + /// clone the handle out synchronously and never hold it across an await. + current: Arc>, + factory: EndpointFactory, + /// Serializes [`Self::rebuild`]'s build-and-swap: the handle is shared, + /// and two callers noticing the same outage must not each build an + /// endpoint and have the second discard (and close) the first's good one. + rebuilding: Arc>, +} + +/// The installed endpoint plus how many rebuilds produced it, so a rebuild +/// caller can tell whether one already happened while it waited its turn. +struct Current { + generation: u64, + endpoint: Endpoint, +} + +impl ClientEndpoint { + /// Wrap a bound endpoint with the recipe that rebuilds it. + pub fn from_parts(endpoint: Endpoint, factory: EndpointFactory) -> Self { + Self { + current: Arc::new(std::sync::RwLock::new(Current { + generation: 0, + endpoint, + })), + factory, + rebuilding: Arc::new(tokio::sync::Mutex::new(())), + } + } + + /// A clone of the current endpoint handle. Take it fresh per use: a handle + /// held across a [`Self::rebuild`] keeps pointing at the old, closed + /// endpoint. + pub fn endpoint(&self) -> Endpoint { + self.current.read().expect("endpoint lock").endpoint.clone() + } + + /// The current endpoint id. Changes on rebuild for an ephemeral identity; + /// stable when the factory binds a fixed secret (quick mode). + pub fn id(&self) -> EndpointId { + self.endpoint().id() + } + + fn generation(&self) -> u64 { + self.current.read().expect("endpoint lock").generation + } + + /// Swap in a freshly built endpoint and close the old one. On error the + /// current endpoint stays in place, so the caller can simply retry with it. + /// + /// Concurrent calls coalesce: a caller that arrives while another rebuild + /// is in flight waits for it and, if it installed a fresh endpoint, + /// returns `Ok` without building another — its trigger was the same dead + /// endpoint, and [`Self::endpoint`] now yields the replacement. Only if + /// the in-flight rebuild failed does the waiter build one itself. + pub async fn rebuild(&self) -> Result<()> { + let seen = self.generation(); + let old = { + let _serialized = self.rebuilding.lock().await; + if self.generation() != seen { + log::info!( + "Endpoint already rebuilt by a concurrent caller; endpoint id: {}", + self.id() + ); + return Ok(()); + } + let fresh = (self.factory)().await?; + let mut current = self.current.write().expect("endpoint lock"); + current.generation += 1; + std::mem::replace(&mut current.endpoint, fresh) + }; + // Graceful close on its own task: bounded wait here, but the task is + // never cancelled (see [`REBUILD_CLOSE_TIMEOUT`]). + let mut close = tokio::task::spawn(async move { old.close().await }); + if tokio::time::timeout(REBUILD_CLOSE_TIMEOUT, &mut close) + .await + .is_err() + { + log::warn!("Old endpoint's close is slow; leaving it to finish in the background"); + } + log::info!("Endpoint rebuilt; endpoint id: {}", self.id()); + Ok(()) + } + + /// Close the current endpoint gracefully (session teardown). + pub async fn close(&self) { + self.endpoint().close().await; + } +} + +#[cfg(test)] +mod tests { + use super::*; + use iroh::{RelayMode, endpoint::presets}; + + // Hermetic: loopback-only endpoints, no relays, no discovery. + fn loopback() -> EndpointBuilder { + Endpoint::builder(presets::Empty) + .relay_mode(RelayMode::Disabled) + .crypto_provider(Arc::new(rustls::crypto::ring::default_provider())) + } + + #[tokio::test] + async fn client_endpoint_swaps_and_closes_the_old_one() { + let first = loopback().bind().await.unwrap(); + let first_id = first.id(); + let handle = ClientEndpoint::from_parts( + first.clone(), + Arc::new(|| Box::pin(async { loopback().bind().await.map_err(Into::into) })), + ); + assert_eq!(handle.id(), first_id); + + handle.rebuild().await.unwrap(); + assert_ne!(handle.id(), first_id, "an ephemeral rebuild gets a new id"); + assert!(first.is_closed(), "the replaced endpoint is closed"); + handle.close().await; + assert!(handle.endpoint().is_closed()); + } + + #[tokio::test] + async fn concurrent_rebuilds_coalesce_into_one() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + let builds = Arc::new(AtomicUsize::new(0)); + let first = loopback().bind().await.unwrap(); + let handle = ClientEndpoint::from_parts(first.clone(), { + let builds = builds.clone(); + Arc::new(move || { + builds.fetch_add(1, Ordering::SeqCst); + Box::pin(async { + // Hold the build long enough for the second caller to + // queue behind it. + tokio::time::sleep(Duration::from_millis(200)).await; + loopback().bind().await.map_err(Into::into) + }) + }) + }); + + let a = handle.clone(); + let b = handle.clone(); + let (ra, rb) = tokio::join!(a.rebuild(), async { + tokio::time::sleep(Duration::from_millis(50)).await; + b.rebuild().await + }); + ra.unwrap(); + rb.unwrap(); + assert_eq!(builds.load(Ordering::SeqCst), 1, "the second caller joined the first"); + assert!(first.is_closed()); + assert!(!handle.endpoint().is_closed(), "the one fresh endpoint is live"); + + // A rebuild after the coalesced one is a new outage: it builds again. + handle.rebuild().await.unwrap(); + assert_eq!(builds.load(Ordering::SeqCst), 2); + handle.close().await; + } +} diff --git a/docs/architecture.md b/docs/architecture.md index c9fdb7e..253849a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -44,7 +44,7 @@ server. | `secret.rs` | iroh identity key commands (`generate-iroh-key`/`show-iroh-id`); client auth keypairs are generated with the standalone `flexaccess-keys` CLI | | `error.rs` | `ProxyError` (`Network`/`Config`/`Signaling`/`AuthenticationFailed`/`ConnectionLost`) + `is_recoverable()` | | `transport/mod.rs` | QUIC transport config, ALPN, heartbeat/liveness timing | -| `transport/endpoint.rs` | flextunnel's layer over the shared [flexaccess-iroh](https://github.com/flexaccessdev/flexaccess-iroh) endpoint builder: the three ALPNs, client/server identity rules, native per-ALPN allowlist hook (`AllowlistHook` over `EndpointAllowlists`: bridges + quick clients). `RelayConfig`, relay-mode-dependent n0 discovery, the per-relay startup probe, the create-vs-rebuild policy, and the rebuildable client endpoint are the shared crate's | +| `transport/endpoint.rs` | flextunnel's layer over the shared [flexaccess-iroh](https://github.com/flexaccessdev/flexaccess-iroh) endpoint builder: the three ALPNs, client/server identity rules, native per-ALPN allowlist hook (`AllowlistHook` over `EndpointAllowlists`: bridges + quick clients). `RelayConfig`, relay-mode-dependent n0 discovery, the per-relay startup probe (the endpoint is bound without the relays that fail it), and the server's in-place home-relay failover are the shared crate's; the rebuildable client endpoint (`ClientEndpoint`, the reconnect loop's escalation) is flextunnel's own | | `transport/paths.rs` | connection-path snapshot (direct/relay) + on-demand custom-relay `/healthz` health | | `proxy/signaling.rs` | length-prefixed `Hello`/`HelloResponse`, control frames, per-stream `Target` codec, `REP_*` codes | | `proxy/socks5.rs` | client-side RFC 1928: method negotiation + `CONNECT` parsing + replies | @@ -245,49 +245,49 @@ Implemented in `ProxyClient::run` / `handle_failure`: (`TUNNEL_RECOVERY_HOLD`), deploy-style connection holding — and only then fail with a network-unreachable reply. -## Relay watchdog (server, custom relays) +## Relay failover (server, custom relays) Implemented once for every FlexAccess program in the shared [flexaccess-iroh](https://github.com/flexaccessdev/flexaccess-iroh) crate -(`relay_watchdog`), driven here by the serve loop in the CLI's `run_server`. A custom-relay server is dialable from off the LAN only -while it is **registered on its home relay** (n0 discovery is off; clients dial -by relay hint, and a relay forwards Initials only to endpoints connected to it). -iroh has been seen to silently lose its home relay for good after a routine -relay reconnect: no dial retries, no warnings, no registration on any relay — -the server stops being reachable through the relays until the process restarts, -while LAN clients that find it over mDNS keep working and hide the outage (the -mac desktop reconnects, the iOS app times out). Observed on v1.0.3; no fix for -it is called out through v1.1.0, so the watchdog stays. - -The watchdog observes `Endpoint::home_relay_status()` and escalates like the -client's reconnect loop: - -1. no connected home relay for `RELAY_OUTAGE_NUDGE` (60s) → log a warning and - call `Endpoint::network_change()` (forces a fresh net report and relay - re-selection — enough when only the bookkeeping went stale); -2. still none at `RELAY_OUTAGE_REBUILD` (180s from the outage start) → the - serve loop closes the endpoint, binds a fresh one with the **same identity - and allowlists** (`server_rebuild_factory`: no per-relay probe, online-wait - tolerated failing), and calls `ProxyServer::run` again on it. The - `ProxyServer` — registries, blocklist, status state — carries over; the old - endpoint's connections end with it, and its bridge tasks are aborted with - the previous `run` future (they are owned by a `JoinSet` per run). A failed - rebuild is retried every `REBUILD_RETRY` (30s). - -A rebuild only helps when iroh's bookkeeping went stale; when the relay itself -is unreachable the fresh endpoint never registers either, and rebuilding again -every three minutes would keep dropping the LAN clients that still work. The -watchdog therefore reports whether the endpoint held a home relay at any point -(`RelayOutage::relay_seen`), and the serve loop doubles the rebuild deadline -for each consecutive endpoint that never did (`rebuild_deadline`: 180s, 6m, -12m, 24m, then capped at `REBUILD_DEADLINE_MAX`, 30m). An endpoint that -registers resets the escalation to the usual 180s. The 60s nudge is unaffected. +(`relay_failover::fail_over_home_relay`), run here by the CLI's `run_server` +alongside `ProxyServer::run`; the design is documented once in +[iroh-common-architecture/relay-failover.md](https://github.com/flexaccessdev/iroh-common-architecture/blob/main/relay-failover.md). +A custom-relay server is dialable from off the LAN only while it is +**registered on its home relay** (n0 discovery is off; clients dial by relay +hint, and a relay forwards Initials only to endpoints connected to it). iroh +re-homes on its own when a relay is really down, but not when the relay keeps +answering net-report probes while relay connections to it fail (the shape seen +on v1.0.3 behind Cloudflare tunnels, unchanged in v1.1.0): net_report keeps +preferring it, the relay connection never re-establishes, and the server is +registered nowhere until the process restarts, while LAN clients that find it +over mDNS keep working and hide the outage (the mac desktop reconnects, the iOS +app times out). + +The failover watches `Endpoint::home_relay_status()`. After 60 s +(`RELAY_OUTAGE_FAILOVER`) without a connected home relay it takes the wedged +relay **out of the endpoint's relay map**; the forced net report can only +prefer a relay still in the map, so the endpoint homes on another configured +relay **in place**: same server id, same allowlists, same sockets, same direct +paths, same established connections and bridge tasks. Nothing is rebuilt. The +removed relay is probed every 90 s (`RELAY_RESTORE_INTERVAL`) and put back once +it is connectable again. Relays that failed the **startup** probe are handled +the same way: the endpoint is bound without them +(`CreatedEndpoint::relays_left_out`) and the failover restores them, so a +server or client that starts during such an outage still comes online on the +relay that works. A reconnect at any point resets the outage clock. Non-home relays are connected on demand and dropped after a minute idle, which is normal and never -counts as an outage. With the default relays the watchdog is not armed: +counts as an outage. With the default relays the failover is pending forever: reachability there rests on n0 publishing/resolution, not on one relay -registration. +registration. A custom relay set must therefore hold at least two distinct +relays, which `RelayConfig` enforces at startup. + +Until flexaccess-iroh v0.0.7 this was a watchdog that nudged +`Endpoint::network_change()` (a no-op on a stable host) and then rebuilt the +endpoint, dropping every connection; both are gone. The client's own endpoint +rebuild (above) is unrelated and stays: it answers a wedged *client* endpoint +after a network change, and it is flextunnel's code, not the crate's. On every exit path both `run_server` and `run_client` call `endpoint.close().await` before the `Endpoint` drops; skipping it makes iroh tear @@ -345,10 +345,9 @@ defenses. | `HEARTBEAT_INTERVAL` | 10s | `transport/mod.rs` | | `LIVENESS_WINDOW` | 33s | `transport/mod.rs` | | `RELAY_CONNECT_TIMEOUT` (`endpoint.online()`) | 10s | flexaccess-iroh `relay` | -| `RELAY_OUTAGE_NUDGE` (server relay watchdog) | 60s | flexaccess-iroh `relay_watchdog` | -| `RELAY_OUTAGE_REBUILD` (server relay watchdog, default deadline) | 180s | flexaccess-iroh `relay_watchdog` | -| `REBUILD_DEADLINE_MAX` (server relay watchdog, escalated deadline cap) | 30m | `flextunnel-cli/src/main.rs` | -| `REBUILD_RETRY` (server endpoint rebuild) | 30s | `flextunnel-cli/src/main.rs` | +| `RELAY_OUTAGE_FAILOVER` (server home-relay failover) | 60s | flexaccess-iroh `relay_failover` | +| `RELAY_RESTORE_INTERVAL` (restore probe for a relay taken out of the map) | 90s | flexaccess-iroh `relay_failover` | +| `REBUILD_CLOSE_TIMEOUT` (client endpoint rebuild, old endpoint close) | 5s | `transport/endpoint.rs` | | `CONNECT_TIMEOUT` (client server connect) | 30s | `proxy/client.rs` | | `HANDSHAKE_TIMEOUT` | 10s | `proxy/client.rs`, `proxy/server.rs`, `proxy/bridge.rs` | | `LOCAL_HANDSHAKE_TIMEOUT` | 10s | `proxy/client.rs` |