Uh oh!
There was an error while loading. Please reload this page.
[world-vercel] Recover from wedged HTTP/2 events connections - #3370
Conversation
undici never retires an HTTP/2 session whose stream timed out ("We do not
destroy the socket as we can continue using the session" in client-h2.js),
and it consults neither `keepAliveTimeout` nor `headersTimeout` on the H2
path. When a connection's flow stops delivering while the TCP connection
stays established, every event request multiplexed onto that session times
out for as long as the compute instance lives.
Two changes, both in the events transport:
- The shared events dispatcher is now owned by a recycler. Three consecutive
transport failures (`UND_ERR_INFO`, headers/body timeouts) retire the pool
and the next request builds a fresh one. Connect/DNS/abort errors don't
count: a new pool hits the same wall.
- `WORKFLOW_H2_MULTIPLEX=0` now sets `allowH2: false` instead of only
skipping the multiplexing interceptor, so the kill switch actually takes
the events path off HTTP/2.
Covered by a black-hole harness: an h2 origin behind a TCP proxy that drops
bytes for one flow. It asserts the old behavior (12/12 requests fail, one TCP
flow, no recovery) and the new one (three failures, then a rebuilt pool that
stays healthy).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>🦋 Changeset detectedLatest commit: 130faeb The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 128734ms → this run 121569ms (Δ -7165ms, -6%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
karthikscale3
left a comment
There was a problem hiding this comment.
AI Review: Approving. The implementation is well designed and the black-hole reproduction provides strong coverage of the observed production failure. The two inline notes are non-blocking follow-ups.
| @@ -455,6 +467,7 @@ export async function instrumentedFetch( | |||
| throw error; | |||
There was a problem hiding this comment.
AI Review: Non-blocking caveat — this reports success as soon as fetch() returns a Response, which is normally after headers arrive and before the body is consumed. The V4 paths later consume response.arrayBuffer() or stream response.body, so a connection that black-holes mid-body can raise UND_ERR_BODY_TIMEOUT after this success has already reset the recycler, and that later error never reaches onTransportOutcome. The fix is correct for the observed no-response failure, but the advertised body-timeout coverage is incomplete; consider reporting the outcome after body consumption or narrowing that guarantee.
| // Re-resolved per request, exactly as fetchV4 does — caching the | ||
| // dispatcher across requests would pin the caller to the retired pool. | ||
| const dispatcher = recycler.get(); | ||
| const error = await attempt(dispatcher); |
There was a problem hiding this comment.
AI Review: Non-blocking test suggestion — this excellent black-hole test drives requests sequentially. In production, multiplexed requests can complete concurrently; successes from a healthy connection may interleave with failures from a poisoned one and continually reset the consecutive-failure counter. A concurrent/interleaved test would document whether that degraded multi-connection case is intentionally tolerated or should also trigger recycling.
Problem
A compute instance can end up with an HTTP/2 connection to the events API that stays established but delivers nothing. Every event request multiplexed onto that session then times out, for the rest of the instance's life. Observed in production as
UND_ERR_INFO(HTTP/2: "stream timeout after ...") withdnsDuration/tlsDuration/netDurationall zero, repeating on the retry ladder: no new connection is ever attempted.undici (7.28.0) does not recover from this on its own:
lib/dispatcher/client-h2.js). The H1 path destroys the socket, which is why this was self-healing before the events transport moved to H2.keepAliveTimeoutis never read on the H2 path, so the agent's 10s idle timeout does not retire the session.headersTimeoutis unused on H2 as well; onlybodyTimeoutarms the stream timer.session.ping()cannot detect it either: Node's callback only fires on ACK, with no timeout of its own.Better connection reuse in recent versions concentrates traffic onto fewer sessions, so one poisoned session now affects far more requests than it used to.
Changes
Both scoped to the events transport, which is the only H2 agent with multiplexing on.
1. The shared events dispatcher is recycled after repeated transport failures.
createDispatcherRecyclerowns the singleton. Three consecutive failures (UND_ERR_INFO, headers timeout, body timeout) retire the pool and the next request builds a fresh one. Details worth knowing:GOAWAY/reset: one failure and done) from the wedged-session case, which is unbounded. RoutineGOAWAYbursts don't trigger a rebuild.close()d after a delay rather than destroyed, so a request that already resolved it isn't turned into a step retry byClientClosedError. Adestroy()backstop follows in caseclose()never settles. Both timers are unref'd.instrumentedFetchgained anonTransportOutcomehook to report this. An HTTP error status is not a transport failure: the origin answered.2.
WORKFLOW_H2_MULTIPLEX=0now disables HTTP/2 for the events path. It previously only skipped the multiplexing interceptor and leftallowH2: truewithpipelining: 100, which is not a mitigation for an H2 transport fault. Against a black-holed loopback origin, that configuration still failed one request in eight indefinitely;allowH2: falseevicted the socket on the first timeout and recovered on the next request.Testing
http-client.test.tsgained a black-hole harness: a real h2 origin behind anode:netTCP proxy that keeps both sockets open and drops all bytes for one chosen flow. It asserts three things.WORKFLOW_H2_MULTIPLEX=0: the origin sees onlyHTTP/1.1, failures stay below the request count, and new connections are made.Plus accounting tests (consecutive-only counting, ignoring dispatchers it no longer owns, which error codes qualify, self-referential cause chains) and an
events-v4test that a v4 request whosefetchrejects the way a wedged session does actually reaches the recycler, so the wiring can't silently regress.Follow-ups, not in this PR
keepAliveTimeoutandheadersTimeouton the H2 path, and add a client-side PING timeout.allowH2: true(multiplexing off, since appends aren't idempotent). They could get the same accounting; left out to keep this scoped to the path where the failure was observed.