tunnel-client: wedges after sustained uplink flap — instant-fail loop, then a hung connect attempt with no timeout; never self-recovers without manual reconnect

[waleedkadous]

Observed (2026-08-08, live)

Host uplink flapped ~22:02–22:04Z (independently corroborated by another workspace's monitors; link flaky since Aug 4). Afterward, with network and cloud both verified healthy (DNS resolves, https://cloud.codevos.ai answers 307 in ~420ms, reference host 200), the tunnel client:

1. Cycled disconnected → connecting → disconnected with the transition completing in ~4ms — an instant local failure, not a real network attempt (real attempts earlier in the day took ~500ms and succeeded).
2. At 22:06:44Z entered connecting and never transitioned again — the attempt hung with no connect timeout for 5+ minutes.
3. Recovered instantly (connected in 600ms with existing credentials) when an empty-body POST /api/tunnel/connect ran resetCircuitBreaker() + connectTunnel() (which builds a fresh client).

Net effect for the owner: "the tower keeps disconnecting from cloud" — every uplink blip risks a permanent cloud outage until someone manually reconnects.

Code observations (lib/tunnel-client.ts)

• handleAuthError('invalid_api_key') → auth_failed → never retries (deliberate breaker). Everything that checks state === 'auth_failed' stays parked forever. If a proxy/CDN error page during a blip is ever (mis)classified as an auth error, that's a permanent stop on a transient condition. Worth verifying how close-reason parsing behaves on a Cloudflare 5xx/HTML body.
• Transient path (scheduleReconnect + calculateBackoff) looks correct in isolation — but the observed 4ms failure loop and the final hung connecting state show some path where doConnect fails synchronously (or never completes) without a watchdog.

Asks

1. Connect-attempt watchdog: a connecting state older than N seconds must be torn down and rescheduled — an attempt that can hang forever defeats all backoff logic.
2. Self-healing breaker: auth_failed should half-open (retry at a long interval, e.g. 10–15 min) rather than stop forever; a genuinely revoked key just fails again cheaply.
3. Log the failure reason on each transition (Tunnel: connecting → disconnected (reason)) — today's log has no reasons, which made this diagnosis archaeology.
4. Consider rebuilding the client object after K consecutive instant failures (the manual recovery worked by replacing the client, suggesting wedged internal state).

Relation to #1370

#1370 covers the two same-day full deregistrations (config deleted, server-side DELETE) — a different signature, still unattributed (owner attests no dashboard click). This issue is the stays-down defect; #1370 is the credentials-vanish defect. The source-attribution logging asked there would also disambiguate here.

[waleedkadous]

On it! Working on a fix now.

[waleedkadous]

Completion stats — bugfix-1372

Fixed in PR #1373, merged 2026-08-09T00:38:00Z (78909017, regular merge).

Root cause

doConnect() created the WebSocket with no handshakeTimeout and no watchdog, so neither the handshake nor the auth-response wait had a deadline. A half-open path left by the uplink flap (stale NAT/conntrack entry) accepts the TCP connection and then goes silent — no error, no close, and the heartbeat is only armed after connected. Every recovery path was gated on leaving connecting:

Path	Why it couldn't help
scheduleReconnect()	only reached from socket events — a silent hang emits none
startHeartbeat()	armed only after connected
connect()	early-returns when state === 'connecting'
resetCircuitBreaker()	only acts on auth_failed

That is why POST /api/tunnel/connect recovered it instantly: connectTunnel() builds a brand-new TunnelClient. Recovery came from replacing the object, not from resetCircuitBreaker().

Asks

#	Ask	Status
1	Connect-attempt watchdog	Done — CONNECT_TIMEOUT_MS = 20s, covers both silent-hang phases
2	Self-healing auth breaker	Done — AUTH_RETRY_INTERVAL_MS = 15min half-open
3	Log the failure reason	Done — reason threaded into every transition and the tower log
4	Rebuild client after K instant failures	Not implemented — see below

Ask #4 was deliberately skipped. doConnect() already builds a fresh WebSocket per attempt and cleanup() nulls the h2 server/session/stream. The only state carried across attempts was the stuck connecting flag, which the watchdog dissolves; a rebuild path would be a redundant second recovery mechanism.

The issue's auth-misclassification hypothesis does not hold. auth_failed is reachable only from an explicit {type:'auth_error', reason:'invalid_api_key'} JSON frame. A Cloudflare 5xx/HTML body fails JSON.parse and routes to the transient retry path; a failed upgrade never opens the socket. Ask #2 is hardening against a future misclassification, not the fix for this incident — the watchdog is.

The ~4ms instant-fail cycles are consistent with the kernel refusing connect() outright (ENETUNREACH/EHOSTUNREACH, no default route, DNS cached). That path worked as designed and was not the wedge.

Effort

Metric	Value
Source changed	210 insertions / 31 deletions across 2 files
Test code added	446 lines across 2 files
Regression tests	16, every one mutation-verified
Commits (excl. porch)	7
CMAP rounds	4 (12 lane-runs)
Suite at merge	4579 passing, 0 failing

Review findings — the part worth keeping

Four real defects across four CMAP rounds. Three of the four were introduced by the fix itself, not present in the original code:

Round	Defect	Origin
1	late auth_ok resurrects a torn-down socket	the fix (watchdog made it reachable)
2	three unsanitized relay-text paths into the tower log, incl. an unbounded raw-payload echo	the fix (round-1 patch, half-applied)
2	auth alarm repeating every 15 min forever	the fix (half-open breaker)
3	new URL() outside the try/catch — reintroduced this bug's own wedge class	the fix (round-1 hardening)

Two recurring habits behind them, both worth naming:

1. Fixing the instance, not the class. Rounds 1→2 on sanitization and 1→3 on synchronous-throw guarding: each time the exact thing pointed at was patched without sweeping for siblings. lessons-critical.md already carries "grep the whole repo before claiming all fixed" — it applies within a single file, not just across codev/ and codev-skeleton/.
2. Comments asserting coverage the code lacks. The round-3 defect hid behind a comment claiming "malformed URL" coverage that the code did not have. A comment is a claim and needs the same scrutiny as an assertion.

A 115-LOC "minimal fix" to reconnection logic had a considerably larger blast radius than that phrase suggests. Solo review would have shipped all four.

Known limitation

The tower UI cloud chip briefly flickers auth failed → connecting… → auth failed every 15 minutes while the breaker half-opens. Inherent to half-opening, revoked-key case only. Ruled acceptable at the PR gate.