Retry every client connection failure; make long outages cheap

Cargo.toml

@@ -10,7 +10,7 @@ default-members = [
 ]
 
 [workspace.package]
-version = "0.0.78"
+version = "0.0.79"
 edition = "2024"
 description = "SOCKS5/HTTP-proxy-over-QUIC split tunnel via iroh P2P"
 

README.md

@@ -416,8 +416,8 @@ Client auth keypairs are generated with the standalone
 | `--relay-url <URL>` | Custom relay URLs (repeatable; at least two distinct relays, the same set as the server). Configuring custom relays disables n0 internet discovery (the server is reached via relay hints); mDNS local discovery stays on. |
 | `--relay-auth-token <TOKEN>` | Shared bearer token sent to every custom relay's WebSocket upgrade. Only valid with `--relay-url` (rejected with the default relays). |
 | `--auto-reconnect` | Force auto-reconnect on (overrides `auto_reconnect = false` in the config). |
-| `--no-auto-reconnect` | Exit on the first disconnection instead of reconnecting. |
-| `--max-reconnect-attempts <N>` | Cap reconnect attempts between successful connections (unlimited if unset). |
+| `--no-auto-reconnect` | Exit on the first failed connection attempt or drop instead of retrying. |
+| `--max-reconnect-attempts <N>` | Cap consecutive retries before giving up (unlimited if unset). |
 | `--quick` | Self-contained ephemeral session (pairs with `server start --quick`): ignore any saved config, print this client's EndpointId (enter it at the quick server's prompt — that allowlist entry is the credential; no auth keypair), prompt for the server EndpointId, then run the live control panel in this terminal. Needs an interactive terminal. Takes no lock and opens no control socket; quitting the panel disconnects. Nothing is persisted. Conflicts with `-c`/`--auth-key(-file)`. |
 
 `flextunnel client start` needs at least one flag — run with no arguments and it
@@ -616,13 +616,21 @@ Auto-reconnect is **enabled by default** (`auto_reconnect = true`); pass
 `--no-auto-reconnect` (or set `auto_reconnect = false`) to disable it, and
 `--auto-reconnect` to force it on over a config that disabled it.
 
-- The **first** connection must succeed. If it fails — bad node id, wrong
-  relay, server down, or a rejected token — the client **exits immediately**
-  rather than retrying blindly.
-- Once connected at least once, a transient drop triggers reconnection with
-  **exponential backoff + jitter** (1s → 60s), indefinitely, unless
-  `--max-reconnect-attempts` caps it or auto-reconnect is disabled.
-- A permanent error (auth/config) never retries.
+- A failed connection attempt — the **first one included** — or a lost
+  connection is retried with **exponential backoff + jitter** (1s doubling to
+  5 min), indefinitely, unless `--max-reconnect-attempts` caps it or
+  auto-reconnect is disabled. A server that is down, or not up yet, is the
+  ordinary case, not a reason to exit: the client waits it out and connects
+  when the server appears.
+- A long outage is cheap to sit through: once the backoff reaches its cap the
+  client makes one bounded connect attempt every five minutes. Repeated
+  failures escalate to rebuilding the iroh endpoint from scratch after the
+  third one, and then at most every 30 minutes for as long as the outage
+  lasts (see [`docs/architecture.md`](docs/architecture.md#reconnect-policy-client)).
+- A permanent error (a rejected key, a malformed config) never retries; that
+  is the only kind of error the client exits on.
+- The control panel shows the outage's progress: failed attempts so far, the
+  last error, and when the next attempt is due.
 - The local proxy listeners stay bound across reconnects. Off-list targets keep
   connecting directly; on-list requests are held for the reconnect (up to 45s)
   and only then fail with a network-unreachable reply.

client.toml.example

@@ -47,8 +47,8 @@ auth_key_file = "~/.config/flextunnel/client.key"
 # custom relay_urls (rejected with the default iroh relays).
 # relay_urls = ["https://relay.example"]
 # relay_auth_token = "shared-relay-secret"
-# auto_reconnect = true                      # default: true. set false to exit on first disconnect
-# max_reconnect_attempts = 10                # default: unlimited (omit for no cap)
+# auto_reconnect = true                      # default: true. set false to exit on the first failed attempt or drop
+# max_reconnect_attempts = 10                # default: unlimited (omit for no cap). retries before giving up
 
 # The split-tunnel routed set (the "tunnel set") is configured on the server and
 # pushed to this client during the handshake — there is no client-side routed-set

crates/flextunnel-cli/src/client_session.rs

@@ -20,7 +20,7 @@ use flextunnel_core::forwards::{
 };
 use flextunnel_core::config::ForwardConfig;
 use flextunnel_core::iroh::SecretKey;
-use flextunnel_core::proxy::{ClientAuth, ClientConfig, ProxyClient, reserved};
+use flextunnel_core::proxy::{ClientAuth, ClientConfig, ProxyClient, ReconnectStatus, reserved};
 use flextunnel_core::transport::endpoint::{
     ClientEndpoint, RelayConfig, create_client_endpoint, create_quick_client_endpoint,
 };
@@ -259,7 +259,7 @@ async fn build_session(
         http_addr,
         ever_connected: false,
         connected_since: None,
-        last_error: None,
+        reconnect: ReconnectStatus::default(),
         disabled_reasons: HashMap::new(),
     };
 
@@ -346,6 +346,7 @@ async fn drive_session(
                     fwd_mgr.apply(&forwards);
                 }
                 state.observe_connection(routes.lock().map(|r| r.connected).unwrap_or(false));
+                state.reconnect = client.reconnect_status();
                 // The reconnect loop rebuilds the endpoint after repeated
                 // failures, which changes the (ephemeral) node id — keep the
                 // status display current.
@@ -448,7 +449,9 @@ struct SessionState {
     http_addr: Option<SocketAddr>,
     ever_connected: bool,
     connected_since: Option<Instant>,
-    last_error: Option<String>,
+    /// The core's reconnect progress (failed attempts, last error, next
+    /// attempt), polled by the ticker; all defaults while connected.
+    reconnect: ReconnectStatus,
     /// Bind-failure reasons of forwards switched off by the ticker, keyed by
     /// forward id, shown next to their rows for the rest of the session.
     disabled_reasons: HashMap<String, String>,
@@ -494,7 +497,12 @@ impl SessionState {
             socks_addr: self.socks_addr,
             http_addr: self.http_addr,
             status_page_host: reserved::STATUS_HOST.to_string(),
-            last_error: self.last_error.clone(),
+            failed_attempts: self.reconnect.failed_attempts,
+            next_attempt_secs: self
+                .reconnect
+                .next_attempt_at
+                .map(|at| at.saturating_duration_since(Instant::now()).as_secs()),
+            last_error: self.reconnect.last_error.clone(),
             routes: wire_routes(routes),
             forwards: forwards
                 .iter()

crates/flextunnel-cli/src/ipc.rs

@@ -121,6 +121,13 @@ pub struct StatusSnapshot {
     pub http_addr: Option<SocketAddr>,
     /// Reserved host that is always tunneled to the server's status page.
     pub status_page_host: String,
+    /// Consecutive failed connection attempts in the current outage (0 while
+    /// connected).
+    pub failed_attempts: u32,
+    /// Seconds until the next connection attempt is due while backing off;
+    /// `Some(0)` while an attempt is in progress, `None` when none is pending.
+    pub next_attempt_secs: Option<u64>,
+    /// What the last connection attempt failed with, while the tunnel is down.
     pub last_error: Option<String>,
     pub routes: WireRoutes,
     pub forwards: Vec<ForwardRow>,
@@ -599,6 +606,8 @@ mod tests {
             socks_addr: Some("127.0.0.1:1080".parse().unwrap()),
             http_addr: None,
             status_page_host: "flextunnel.internal".into(),
+            failed_attempts: 0,
+            next_attempt_secs: None,
             last_error: None,
             routes: WireRoutes {
                 domains: vec!["*.internal".into()],

crates/flextunnel-cli/src/main.rs

@@ -175,10 +175,10 @@ enum ClientAction {
         /// Force auto-reconnect on (overrides `auto_reconnect = false` in the config).
         #[arg(long, conflicts_with = "no_auto_reconnect")]
         auto_reconnect: bool,
-        /// Disable auto-reconnect (exit on the first disconnection).
+        /// Disable auto-reconnect (exit on the first failed connection attempt or drop).
         #[arg(long, conflicts_with = "auto_reconnect")]
         no_auto_reconnect: bool,
-        /// Cap on reconnect attempts between successful connections (unlimited if unset).
+        /// Cap on consecutive retries before giving up (unlimited if unset).
         #[arg(long)]
         max_reconnect_attempts: Option<NonZeroU32>,
         /// Ignore any saved config, print this client's EndpointId (enter it on

crates/flextunnel-cli/src/tui/view.rs

@@ -105,6 +105,17 @@ fn header_lines(s: &StatusSnapshot) -> Vec<Line<'static>> {
     if let Some(secs) = s.connected_secs {
         first.push(Span::styled(format!("  for {}", format_uptime(secs)), DIM));
     }
+    // While down, say how far the retry loop has got and when it tries again,
+    // so a backoff step of minutes reads as waiting, not stuck.
+    if s.failed_attempts > 0 {
+        let next = match s.next_attempt_secs {
+            Some(secs) if secs > 0 => format!("next in {}", format_uptime(secs)),
+            _ => "trying now".to_string(),
+        };
+        let n = s.failed_attempts;
+        let s_ = if n == 1 { "" } else { "s" };
+        first.push(Span::styled(format!("  {n} failed attempt{s_}, {next}"), DIM));
+    }
 
     let proxy = |name: &str, addr: Option<std::net::SocketAddr>| match addr {
         Some(addr) => Span::raw(format!("{name} {addr}")),

crates/flextunnel-core/src/config.rs

@@ -150,9 +150,11 @@ pub struct ClientConfig {
     /// Shared bearer token sent to every custom relay's WebSocket upgrade. Only
     /// valid with custom `relay_urls`; rejected with the default iroh relays.
     pub relay_auth_token: Option<String>,
-    /// Reconnect with backoff on a transient drop (default true).
+    /// Retry failed connection attempts and lost connections with backoff
+    /// (default true; `false` exits on the first failure of either kind).
     pub auto_reconnect: Option<bool>,
-    /// Cap on reconnect attempts between successful connections.
+    /// Cap on consecutive retries before the client gives up (default
+    /// unlimited).
     pub max_reconnect_attempts: Option<NonZeroU32>,
     /// Server-direct port forwards (`[[forwards]]` tables). Config-file only —
     /// there is no CLI flag.

crates/flextunnel-core/src/proxy/bridge.rs

@@ -8,16 +8,17 @@
 //!
 //! [`BRIDGE_ALPN`]: crate::transport::BRIDGE_ALPN
 //!
-//! The connect/auth/heartbeat machinery mirrors [`super::client`], with two
-//! deliberate differences in reconnect policy: a bridge retries **forever** (no
-//! fail-fast first connect, no attempt cap). The peer server may simply not be
-//! up yet, and a server daemon must not exit — or stop serving its other
-//! routes — because a peer is down. While the upstream is down, matching
-//! streams fail with host-unreachable (see `route_to_bridge` in
-//! [`super::server`]). And a bridge never escalates to the client's endpoint
-//! rebuild: it dials on the **server's own endpoint**, which is also accepting
-//! inbound clients on its persistent identity — rebuilding it to unwedge one
-//! upstream would sever every connected client.
+//! The connect/auth/heartbeat machinery mirrors [`super::client`], including
+//! its backoff series (shared `calculate_backoff`), with two deliberate
+//! differences in reconnect policy: a bridge retries **forever** — no
+//! auto-reconnect switch, no retry cap. The peer server may simply not be up
+//! yet, and a server daemon must not exit — or stop serving its other routes —
+//! because a peer is down. While the upstream is down, matching streams fail
+//! with host-unreachable (see `route_to_bridge` in [`super::server`]). And a
+//! bridge never escalates to the client's endpoint rebuild: it dials on the
+//! **server's own endpoint**, which is also accepting inbound clients on its
+//! persistent identity — rebuilding it to unwedge one upstream would sever
+//! every connected client.
 
 use crate::error::{ProxyError, ProxyResult};
 use crate::proxy::client::{calculate_backoff, client_heartbeat_loop, connect_with_timeout};