Uh oh!
There was an error while loading. Please reload this page.
fix(spec): bound the DTS pass's heap ceiling to the build container's memory - #12684
Conversation
… memory The DTS pass declared --max-old-space-size=12288 inside an 8 GB Vercel build container. A ceiling above the container's memory does not permit a bigger build: V8 defers major GCs below its ceiling, so the process is SIGKILLed at the container limit before V8 ever reaches it -- exit 137, no diagnostic. Measured inside a cgroup capped at 8192 MB (peak anonymous RSS, whole process tree): 12288 -> 7290 MB; 6144 -> 5794 MB; 5120 -> 5328 MB; 4096 -> ERR_WORKER_OUT_OF_MEMORY. Declaration output is byte-identical across every completing ceiling (122 files, one sha256 over all of them). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKWDdUJ2XNRESVVWUvcpnh
📓 Docs Drift Check
What this run could not see
Coarse fallback — 126 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): |
os-zhuang
left a comment
There was a problem hiding this comment.
PM verification record (dispatching seat for #12677; this seat shares the PR's author identity, so this is a COMMENT review — the queue does not require an approval for non-governed surfaces, and packages/spec build config is not governed).
Read the full diff and the measurement chain:
- The fix is the minimal correct shape: one ceiling number brought under the container it runs in, the rationale + measurement table recorded at the exact place the next author would edit the number, and a patch changeset. No source, schema, or output changes — declaration tree byte-identical across completing ceilings (
37cf1007189f945cat both 6144 and 5120). - The harness was validated in both directions (positive + negative control reproducing the exit-137/oom_kill=1 signature) before any number was trusted, and the "faithful container" emulation (cgroup cap + private mount namespace lying about
/proc/meminfo) is the right instrument — a cap alone doesn't change what tools believe they have. - Card lever 1 (turbo concurrency) ruled out by measurement (2-task linear graph; the killed process was alone), which the card explicitly allowed for.
- Residual risk correctly severed into #12683 (the
next build~7.6 GB single-process peak) rather than ridden into this fix — that one needs a maintainer decision.
Landing via the normal queue. Post-merge verification stays as written: the data-dpl-id on objectstack.ai must move off dpl_2nfWjGjSwZjakVEmUG1kBWD6697r.
Generated by Claude Code
⛔ merge queue 构建失败 — 先分诊,再决定要不要重排队列构建 33057527457 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集), 失败的 job(日志抽取,best effort):
跨 PR 相同签名(24h,按失败测试文件聚合):
历史信号:
分诊清单:
Generated by Claude Code · merge-queue-triage workflow (#4859) |
Uh oh!
There was an error while loading. Please reload this page.
os-zhuang
commented
Aug 27, 2026
Merge-queue ejection triage — the failure is not this PR's; spending the one re-run.
Generated by Claude Code |
Fixes#12677
What was wrong
packages/spec's build ran its DTS pass underNODE_OPTIONS="--max-old-space-size=12288"— a 12 GB heap ceiling inside an ~8 GB build container.A heap ceiling is a promise to V8 that the memory is there: below it, V8 defers major GCs and lets the resident set grow. A ceiling above the container's memory therefore does not permit a bigger build — it converts a recoverable JS heap error into a kernel SIGKILL. The process meets the container limit long before V8 ever considers the ceiling reached, and
exit 137carries no diagnostic. That is precisely the signature in the card's Vercel log, on precisely the process it names.The change
One number, plus its reasoning and a changeset:
packages/spec/package.json— the DTS pass's ceiling,12288to6144.packages/spec/tsup.config.ts— the rationale recorded next to the DTS split it governs, with the measurements below, so the next author who hitsERR_WORKER_OUT_OF_MEMORYdoes not "fix" it by raising the number past what the container has..changeset/spec-dts-heap-ceiling-fits-container.md.No source, no schema, no build output changes.
Measured — method
Peak anonymous RSS of the whole process tree, sampled from a cgroup-v1
memory.stattotal_rssat 150 ms with per-process attribution recorded at each new peak. The harness was validated in both directions before any number below was trusted: a positive control (600 MB allocation under an 8 GB cap completes, measured 589 MB) and a negative control (2 GB allocation under a 1 GB cap reproducesexit 137withoom_kill=1).The container emulation is two halves, because a cgroup cap alone is not a container — it bounds what a process may use without changing what it believes it has. So the runs marked "faithful" add a private mount namespace whose
/proc/meminforeports the container's size, and tools that size themselves from reported memory then see 8 GB rather than this host's 16 GB.Measured — before / after
Spec's DTS pass, in a faithful 8192 MB container:
−1311 MB (−19%) on the phase that was being killed, at no wall-time cost.
The cap sweep that chose 6144 (cgroup cap only, so the numbers run slightly higher):
ERR_WORKER_OUT_OF_MEMORY6144 is chosen as the largest ceiling whose worst case still fits: V8 cannot exceed it, and this pass's non-heap overhead measured ~250 MB, so the bound is ~6.4 GB inside an 8 GB container — while leaving 1.5x growth room above the 4096 floor before the pass reds. Below the ceiling it now fails loud (
ERR_WORKER_OUT_OF_MEMORY, exit 1) instead of being SIGKILLed, which is what the card asked for.Whole pipeline,
pnpm turbo run build --filter=@objectstack/docs --forcefrom a cleaneddist/.next, faithful emulation, with this fix: 2 successful, 2 total, 5m02s, peak 8135 MB.Output identity — this buys headroom, not a different build
Every completing ceiling emitted a byte-identical declaration tree: 122 files, compared as one sha256 over all of them,
37cf1007189f945cat both 6144 and 5120. Declarations were deleted before each run so a hash could not be satisfied by leftovers (the DTS pass runs withclean: false).check-dts-emittedreports 34/34 declared declaration files present.Lever 1 (cap turbo concurrency) — ruled out by measurement, not skipped
pnpm turbo run build --filter=@objectstack/docs --dry=jsonreports 2 tasks:@objectstack/spec#buildand@objectstack/docs#build. The docs app's only workspace dependency is spec, so the two run sequentially. There is no parallel fan-out for--concurrencyto cap, and the killed process was alone in the container when it died — capping concurrency would have changed nothing. The build command is versioned inapps/docs/vercel.json, so this was checkable in-repo; no dashboard-only configuration was involved in the command itself.Lever 3 (name the regression)
No commit lands in the 16:52–16:57 window. The last change to spec's type surface before the failure boundary is
daae7aa3at 16:13 on 2026-08-25 — "declare the search and data.clone route response contracts" — which adds schemas and so grows the DTS type graph. Offered as the most plausible straw, not proven: the honest reading is that the configuration was one growth-step from this outage the whole time, sitting at ~7.0 GB with under 1 GB of spare in the container. Which commit crossed the line matters much less than that the ceiling was set above the container at all.Residual risk, filed separately
This PR fixes the phase that was failing. It does not lower the pipeline's peak, which is owned by the other phase:
next buildholds ~7.6 GB in a single process, and neither--max-old-space-size(Turbopack is Rust; its memory is outside V8) norexperimental.cpus(the workers measure 100–430 MB each) bounds it. That phase has been building successfully on Vercel at this size for weeks, so it is a standing risk rather than a live failure — but it is now the largest consumer in the pipeline and nothing in this repo bounds it. Measured and filed as #12683, with the options laid out; it needs a maintainer decision, not a rider on this fix.Verification
Gates run at
a0a64684on a clean tree — 30 derived families, all green, pluscheck:nul-bytesand@objectstack/spec typecheck. Two produced no reading rather than a failure and are recorded as such:check-dev-prereqsreports an unmet precondition (this worktree built only spec and docs, not all 67 packages), andscripts/pm/check-half-states.mjsexits 3 stating "no reading at all" without a GitHub token. Repository-wide lint is left to CI.After merge, per the card:
must report an id other than
dpl_2nfWjGjSwZjakVEmUG1kBWD6697r.Refs: #12333 (measurement card) · epic #12243 · follow-up #12683
Generated by Claude Code
Generated by Claude Code