Reconnect SSE stream after fatal EventSource errors

[selfcontained]

Problem

EventSource only auto-reconnects after transient failures. A fatal one — a non-200 response or a wrong content-type, e.g. loading the page while the server is mid-restart — puts it in CLOSED permanently, per spec.

use-sse.ts didn't handle that case:

• onerror only called recordSSEReconnect().
• The dead instance stayed in eventSourceRef, so openSSE() early-returned forever.
• Only a tab hide/show or an authState change cleared the ref — a visible desktop tab never self-healed.

Impact: silent loss of all realtime updates (agent status, terminal-state banner, media/review refreshes, injection-hold badge) while the app looks perfectly healthy.

Fix

In onerror, branch on readyState:

• CONNECTING — the browser is retrying on its own, leave it alone.
• CLOSED — the instance is dead: drop it from the ref and reopen on a capped backoff (1s, doubling to 30s).

The backoff resets when the tab is foregrounded, and when the stream delivers an event on a connection that has already lasted 10s (STABLE_CONNECTION_MS).

That stability gate matters more than it looks. The server writes the snapshot unconditionally the moment it accepts a connection (events-routes.ts:42-48), so a delivered event only proves the connect succeeded — not that the stream is healthy. Resetting on it made a no-backoff loop reachable by the very scenario this PR targets: the browser's own transient retry lands on a half-started server that accepts and sends a snapshot, the stream dies, and the next retry hits the 502 window → CLOSED, but the delay was already back at the 1s floor. A server restarting in a loop would pin every open tab near the floor, so the cap never engaged in the case it exists for — and each cycle carried a full agent-list re-render plus the invalidation burst.

The reset stays keyed on a delivered event rather than a bare timer, so a hung proxy that holds the socket open without sending anything still can't clear the backoff. Tradeoff: a connection that stays healthy but silent for its whole lifetime fails with an elevated delay (capped at 30s, and cleared on foreground). Documented in a comment.

No heartbeat watchdog. The idea suggested one as optional; it isn't implementable client-side here. The server's keepalives are SSE comments (: keepalive), which EventSource never surfaces to onmessage — a liveness timer couldn't tell a hung proxy from a genuinely idle stream. That would need a real data-event heartbeat from the server; out of scope.

No "realtime is down" indicator. Deliberate for this PR — silent self-healing beats "stale until you reload", and a one-second blip during a routine restart shouldn't paint anything. Filed as a follow-up, with the sharper argument that the settings health dots currently read ok in exactly this scenario (the API is up; only the stream is dead).

Validation

Unit — 8 tests in use-sse.test.ts drive a fake EventSource with fake timers: fatal error reopens, transient error doesn't, backoff doubles, a stable connection's delivery resets it, a flapping server's connect-time snapshot does not, hidden-tab with a pending retry doesn't reconnect behind the user's back and recovers exactly once on foreground, and unmount stops retrying. Verified non-vacuous: 3 fail against the pre-fix hook, and the flapping test fails if the stability gate is set to 0.

Live, in a browser — against an isolated dev stack:

• Fatal path (/api/v1/events → 500 + text/html): retry deltas 1.1s / 2.2s / 4.1s / 8.1s / 16.0s, the capped doubling, with no tab switch.
• Flapping path (alternating 200 + snapshot and 502): our backoff escalated 1.1 → 2.0 → 4.0 → 8.0 → 16.2 → 30.1 → 30.1s to the cap, cycle time growing 4.2s → 33.3s. Pre-fix this stayed pinned near the floor.
• End-to-end: renamed an agent server-side while the stream was dead → UI did not update (the bug's symptom, reproduced); un-blocked the endpoint → UI picked up the new name 6.6s later, no reload, no tab switch.

Test harness fix — this vitest config sets neither globals: true nor setupFiles, so React Testing Library's auto-cleanup never registered and every rendered hook stayed mounted for the rest of the file. Without the explicit cleanup(), the new hidden-tab test sees 7 connections instead of 2 — six leaked hooks answering the same visibilitychange.

Checks — pnpm run check 0 errors · finalize:web builds · web unit 712 passed (55 files) · E2E 178 passed / 12 skipped (terminal-live, needs tmux) · CI green.

Structure

The connection and its backoff are one state machine, so they share one lifetime: source, reconnectDelayMs, reconnectTimer, and connectStartedAt are all effect-scoped locals torn down together. Nothing here is read during render, so none of it needs a ref — the EventSource was demoted from one.

The backoff reset lives with the rest of the backoff mutation (noteStreamActivity, next to cancelReconnect/scheduleReconnect) and is wrapped at the onmessage assignment site, so handleSSEMessage stays pure event dispatch and depends only on queryClient/jotaiStore. That keeps the dispatch half free to move out later.

connectionAliveSince is stamped on open — which fires on every establishment, including the browser's own internal retries after a transient drop. Those never re-run openSSE, so stamping only there would measure the age of the EventSource instance rather than of the connection, and a connect-time snapshot delivered by an internal retry would clear the backoff. The openSSE stamp is kept as the pre-open fallback: generous by the establishment latency, but never 0 — a zero would make the staleness check trivially true.

Review

Four persona reviews (frontend-ux, backend-security, and two architecture passes on different models); 13 items, all resolved.

• frontend-ux and backend-security independently identified the flapping-reset hole and converged on the same gate. Backend-security separately verified there's no post-logout reconnect window and no rate limiter or lockout on /api/v1/events, so 401 retries are benign, and judged jitter not worth adding at this scale. Frontend-ux found the RTL cleanup leak and the missing hidden-tab test.
• architecture confirmed no restructure belongs in this PR and took the remaining items as placement/naming: the reset's location, the split lifetimes, the connectStartedAt name, and the constant's rationale. The second architecture pass found the connection-age gap above, plus a defensive asymmetry in onerror where the identity check guarded the null-out but not scheduleReconnect(). Two things recorded rather than built — splitting event dispatch out of useSSE (brain idea split-use-sse-event-dispatch), and an explicit don't on a shared backoff primitive, since the three reconnect implementations here are different policies rather than instances of one (use-terminal is linear/ref-keyed/UI-driving on a WebSocket; use-release-stream doesn't reconnect at all).

Note on PR #918

This is a fresh re-land of #918, which was opened and closed without merging on 2026-08-09. That PR had no review comments or objections — Brad confirmed it was simply stopped, not rejected. Re-validated from scratch against current main.

🤖 Generated with Claude Code