Reconnect SSE stream after fatal EventSource errors - #918
Closed
selfcontained wants to merge 1 commit into
Closed
Conversation
EventSource only auto-reconnects after transient failures. A fatal one — non-200 or a wrong content-type, e.g. hitting the server mid-restart — moves it to CLOSED permanently. `onerror` only recorded a metric, and the dead instance stayed in `eventSourceRef`, so `openSSE` early-returned forever. Only a tab hide/show or an authState change cleared the ref, so a visible desktop tab silently lost every realtime update — agent status, terminal-state banner, media/review refreshes, injection-hold badge — while the app looked healthy. On a CLOSED readyState we now drop the dead instance and reopen on a capped backoff (1s doubling to 30s), reset when the stream delivers again (the server sends a snapshot on every connect) and when the tab is foregrounded. CONNECTING is left alone — that is the browser's own retry. No heartbeat watchdog: the server's keepalives are SSE comments, which EventSource never surfaces to `onmessage`, so a client-side liveness timer can't distinguish a hung proxy from an idle stream. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
EventSourceonly auto-reconnects after transient failures. A fatal one — a non-200 response or a wrong content-type, e.g. loading the page while the server is mid-restart — puts it inCLOSEDpermanently, per spec.use-sse.tsdidn't handle that case:onerroronly calledrecordSSEReconnect().eventSourceRef, soopenSSE()early-returned forever.authStatechange cleared the ref — a visible desktop tab never self-healed.Impact: silent loss of all realtime updates (agent status, terminal-state banner, media/review refreshes, injection-hold badge) while the app looks perfectly healthy.
Fix
In
onerror, branch onreadyState:CONNECTING— the browser is retrying on its own, leave it alone.CLOSED— the instance is dead: drop it from the ref and reopen on a capped backoff (1s, doubling to 30s).The backoff resets when the stream delivers an event (the server sends a
snapshoton every connect, so that's a clean success signal) and when the tab is foregrounded.No heartbeat watchdog. The idea suggested one as optional; it isn't implementable client-side here. The server's keepalives are SSE comments (
: keepalive), whichEventSourcenever surfaces toonmessage— a liveness timer couldn't tell a hung proxy from a genuinely idle stream. That would need a real data-event heartbeat from the server; out of scope for this fix.Validation
Unit — 5 new tests in
use-sse.test.tsdrive a fakeEventSourcewith fake timers: fatal error reopens, transient error doesn't, backoff doubles, delivery resets the backoff, unmount stops retrying. 3 of the 5 fail against the pre-fix hook (verified by stashing the change).Live, in a browser — intercepted
/api/v1/eventswith a 500 to force the fatal path:+0.3s, +1.4s, +3.5s, +7.7s, +15.8s— the capped doubling, reconnecting on its own with no tab switch.Checks —
pnpm run check0 errors ·finalize:webbuilds · web unit 710 passed · server unit 2503 passed · E2E 178 passed / 12 skipped (terminal-live, needs tmux).🤖 Generated with Claude Code