Skip to content

fix(headless): keep provider proxy upstream on HTTP/1.1 - #1658

Merged
Astro-Han merged 5 commits into
mainfrom
fix/headless-proxy-h2-serialization
Jul 30, 2026
Merged

fix(headless): keep provider proxy upstream on HTTP/1.1#1658
Astro-Han merged 5 commits into
mainfrom
fix/headless-proxy-h2-serialization

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

All proxied harness cells (opencode / kimi-code / codex arms, and any Maka arm configured through the proxy) share one Node process, and the proxy forwarded upstream requests with the global fetch. Provider gateways (api.z.ai, api.kimi.com) negotiate HTTP/2 via ALPN, and once one h2 session existed, every concurrent cell's streaming completion serialized behind the full generation streams of the others (root cause below). The queueing happened between proxy receipt and upstream dispatch, so telemetry recorded it as provider first-token latency.

Measured impact in existing runs (same key, same windows, direct-connect arms unaffected at ~1.5–4.5 s):

runproxied arm first-token waitshare of wall clock
TB 2.1 GLM-5.2 Maka vs OpenCode (3 rounds)p50 12.5 s, mean 32.8 s, max 628 s57.8% of the OpenCode arm
TB 2.1 K3 Maka vs Kimi Code v11p50 13.7 s, mean 35.1 s, max 1,023 s

The fix routes upstream fetches through an explicit undici Agent with allowH2 disabled (HTTP/1.1 pools dial parallel connections), injectable per route so tests can trust a local CA. Switching to the npm undici@8.8.0 fetch also independently removes the gate; forcing h1 keeps the proxy safe regardless of which undici build serves the fetch.

Existing A/B timing conclusions drawn from proxied arms need re-examination; report corrections are tracked separately.

Root cause

Node 26.5.0 bundles undici 8.7.0. Its fetch wraps every non-empty request body into an async iterable (lib/web/fetch/index.js), and its h2 client refuses to multiplex stream/async-iterable bodies on a session with requests in flight (lib/dispatcher/client-h2.jsbusy(): bodyLength !== 0 && (isStream || isAsyncIterable || isFormDataLike) → busy), because such bodies cannot be retried after a mid-flight session error. The queued request stays in the h2 client's queue — the pool never dials a second connection. Net effect: every streaming POST to the same origin from one process runs strictly one-at-a-time. undici 8.8.0 removed this gate and multiplexes up to maxConcurrentStreams.

Differential confirmation (local ALPN-h2 upstream, zero network): staggered POSTs via built-in fetch serialize on one session; staggered GETs (empty body) multiplex on the same session; staggered POSTs via npm undici 8.8.0 with h2 enabled multiplex. Against the real provider: 4 staggered streaming requests + 1 tiny request from one process serialized strictly (tiny request waited 30.8 s); the same schedule from separate processes ran fully parallel (3.65 s).

Verification

  • npm run test --workspace @maka/headless — 1432 tests, 0 failures.
  • New regression test drives two concurrent held-open SSE streams through the proxy against a local ALPN-h2 upstream. The httpVersion === '1.1' assertion is the timing-independent lock; the upstream additionally refuses to end either stream until both have arrived, so the undici ≤ 8.7 staggered-dispatch path deadlocks instead of passing by luck. Mutation check: flipping allowH2: true in the built dispatcher makes exactly this test fail.
  • End-to-end against the real provider through the fixed proxy in one process: 4 concurrent long streams got response headers in 3.7–4.0 s in parallel, and a fifth request injected mid-stream got headers in 5.5 s (previously 30.8 s under the same schedule).
  • Independent external review (Codex) confirmed the root-cause chain against undici 8.7/8.8 sources and Node 26.5.0's embedded snapshot, found no P0/P1, and prompted the comment narrowing in the follow-up commit.
  • npm run format / npm run lint clean.

Observability

responseHeadersMs starts at proxy receipt, so dispatcher connection-queue time was indistinguishable from upstream wait — the interval where the serialization hid for two benchmark campaigns. The new upstreamStartMs telemetry field stamps the moment undici begins writing the request to an upstream connection (per-request composed dispatcher wrapping onRequestStart); responseHeadersMs − upstreamStartMs is now pure upstream wait. Live check through the fixed proxy: upstreamStartMs 231–403 ms across 5 concurrent requests while headers took 1.9–4.7 s — under the old defect this field would have read 10–30 s and exposed the queue on day one.

Review focus

  • Known bypasses left as-is: Codex OAuth token refresh uses global fetch (single refresh under a credential-store lease; not a completion path), and the Copilot model probe is a bodyless GET that does not hit the gate.
  • proxy.close() intentionally does not close the module-level shared dispatcher; idle h1 sockets are unref'd and do not hold the process open.

Post-review hardening

Two independent optimality reviews (Codex CLI and a separate Claude subagent, same criteria: first principles / Occam / test lock enumeration / refactor-to-optimal) both reported no P0/P1 and recommended merging. Accepted findings landed as the last two commits:

  • onRequestStart delegation now binds this to the wrapper (.call(this, ...)), removing a latent dependency on which side of the Object.create prototype split undici handler callbacks write their state to.
  • A deterministic injected-clock test locks upstreamStartMs to dispatcher queue-exit semantics: one upstream connection holds the first stream while the second request provably enters the pool queue (compose-counter gate, no wall-clock waits), then exact equalities distinguish queue-exit (5000) from a pre-dispatch stamp (0), a response-headers stamp (6000), or a dropped stamp (undefined). Mutation-verified.

Recorded as accepted residual risk (both reviews concur): the default-singleton fallback path (?? defaultUpstreamDispatcher) has no direct test — the injected-dispatcher mutation lock plus Agent's own allowH2: false default keep the invariant — and the inline test PEM pair is a throwaway localhost self-signed keypair, not a secret.

Provider gateways negotiate HTTP/2 via ALPN, and undici's h2 client runs
one request at a time per origin connection. All proxied cells in a
harness run share one Node process, so their streaming completions
serialized behind each other's full generation streams: every request
waited for all earlier in-flight streams before its response headers
arrived, and the wait was recorded as provider first-token latency
(observed p50 ~13s, tails past 600s, versus ~1.5-4.5s on direct
per-process connections).
Route upstream fetches through an explicit undici Agent with allowH2
disabled, injectable per route so tests can trust a local CA. The
regression test drives two concurrent held-open streams through the
proxy against an ALPN-h2 upstream and asserts both arrive as HTTP/1.1
in parallel; with h2 re-enabled the second request deadlocks behind the
first and the test fails.
responseHeadersMs starts at proxy receipt, so dispatcher connection-queue
time was indistinguishable from upstream wait — the interval where the h2
serialization hid. upstreamStartMs stamps the moment undici begins
writing the request to an upstream connection via a per-request composed
dispatcher, making the split observable.
Delegate onRequestStart with this bound to the wrapper so undici handler
state written across callbacks always lands on a single object instead of
relying on undici's internal read/write ordering across the prototype
split.
Deterministic injected-clock test: one upstream connection holds the first
stream while the second request provably enters the pool queue, then exact
equalities pin upstreamStartMs to when the request leaves the queue. A
stamp taken before dispatch, at response headers, or dropped entirely all
fail. Mutation-verified (pre-dispatch stamp fails with 0 !== 5000).
@Astro-Han
Astro-Han merged commit 7a222d2 into mainJul 30, 2026
3 checks passed
@Astro-Han
Astro-Han deleted the fix/headless-proxy-h2-serialization branch July 30, 2026 13:25
Astro-Han added a commit that referenced this pull request Jul 30, 2026
requestDrain() flips in-memory #state to 'draining' synchronously, but
persisting that to the registration file is the async
writeHostRegistration I/O in #closeResources, while #start() had already
persisted 'recovering' before entering the factory. The fixed sleep(50)
bet that I/O landed in time, which lost on loaded CI runners and read
back stale 'recovering' (the #1658 main CI failure).
Poll the registration file for 'draining' instead of sleeping a fixed
gap. #closeResources writes 'draining' first, and the file stays
'draining' while the factory is suspended, because removeHostRegistration
runs only after the blocked compositionStartup await later in
#closeResources.
A handshake connection does not work here: #closeResources calls
server.close() right after the registration write, so on loaded CI the
socket stops accepting before connectRuntimeHost finishes its
resolveStorageRoot, mkdir, and registration read, returning 'unavailable'
instead of 'draining' (the #1660 CI failure).
Verified with a 300ms writeHostRegistration delay: the old sleep(50) form
fails with the same 'recovering' seen on main, while the polling form
passes.
Astro-Han added a commit that referenced this pull request Jul 30, 2026
… on CI
The "drain requested before factory completion begins drain before
recovery exactly once" test failed the test job on main after #1658
landed (run 30544963454), even though #1658 only touches
packages/headless. The flaky test was introduced in #1359.
The flake came from asserting on the registration file mid-flight.
requestDrain() flips in-memory #state to 'draining' synchronously, but
persisting that to the registration file is the async writeHostRegistration
I/O in #closeResources, while #start() had already persisted 'recovering'
before entering the factory. The fixed sleep(50) bet that I/O landed in
time, which lost on loaded CI runners and read back stale 'recovering'.
That assertion tested a non-contract. No production caller reads
HostRegistration.state; clients learn draining from the handshake
(#admitHandshake returns kind:'draining' from #shutdownRequested), and
connectResolvedRuntimeHost reads the registration only for rootId,
hostEpoch, and the endpoint. The invariant the test name claims, "begins
drain before recovery exactly once", is already proven by the final
lifecycle array ['factory-return', 'begin-drain', 'recover', 'close'] with
begin-drain counted once. The in-memory draining state is covered
elsewhere (candidate.host.state === 'draining').
Drop the sleep, the readHostRegistration call, and the state assertion.
The test still verifies, while the factory is suspended, that startup has
not settled, that no lifecycle event has fired, and that the host still
holds the interactive root owner lock, then checks the full lifecycle
ordering after release. A handshake-based replacement was tried first but
also raced: #closeResources calls server.close() right after the
registration write, so on loaded CI the socket stops accepting before
connectRuntimeHost connects, returning 'unavailable' instead of 'draining'
(the first push of #1660).
Refs #1658 (main CI failure, not a regression from that PR).
Astro-Han added a commit that referenced this pull request Jul 30, 2026
… on CI (#1660)
The "drain requested before factory completion begins drain before
recovery exactly once" test failed the test job on main after #1658
landed (run 30544963454), even though #1658 only touches
packages/headless. The flaky test was introduced in #1359.
The flake came from asserting on the registration file mid-flight.
requestDrain() flips in-memory #state to 'draining' synchronously, but
persisting that to the registration file is the async writeHostRegistration
I/O in #closeResources, while #start() had already persisted 'recovering'
before entering the factory. The fixed sleep(50) bet that I/O landed in
time, which lost on loaded CI runners and read back stale 'recovering'.
That assertion tested a non-contract. No production caller reads
HostRegistration.state; clients learn draining from the handshake
(#admitHandshake returns kind:'draining' from #shutdownRequested), and
connectResolvedRuntimeHost reads the registration only for rootId,
hostEpoch, and the endpoint. The invariant the test name claims, "begins
drain before recovery exactly once", is already proven by the final
lifecycle array ['factory-return', 'begin-drain', 'recover', 'close'] with
begin-drain counted once. The in-memory draining state is covered
elsewhere (candidate.host.state === 'draining').
Drop the sleep, the readHostRegistration call, and the state assertion.
The test still verifies, while the factory is suspended, that startup has
not settled, that no lifecycle event has fired, and that the host still
holds the interactive root owner lock, then checks the full lifecycle
ordering after release. A handshake-based replacement was tried first but
also raced: #closeResources calls server.close() right after the
registration write, so on loaded CI the socket stops accepting before
connectRuntimeHost connects, returning 'unavailable' instead of 'draining'
(the first push of #1660).
Refs #1658 (main CI failure, not a regression from that PR).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han