Skip to content

fix(spec): bound the DTS pass's heap ceiling to the build container's memory - #12684

Merged
os-zhuang merged 1 commit into
mainfrom
claude/issue-12677-docs-build-oom
Aug 27, 2026
Merged

fix(spec): bound the DTS pass's heap ceiling to the build container's memory#12684
os-zhuang merged 1 commit into
mainfrom
claude/issue-12677-docs-build-oom

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Fixes#12677

What was wrong

packages/spec's build ran its DTS pass under NODE_OPTIONS="--max-old-space-size=12288" — a 12 GB heap ceiling inside an ~8 GB build container.

A heap ceiling is a promise to V8 that the memory is there: below it, V8 defers major GCs and lets the resident set grow. A ceiling above the container's memory therefore does not permit a bigger build — it converts a recoverable JS heap error into a kernel SIGKILL. The process meets the container limit long before V8 ever considers the ceiling reached, and exit 137 carries no diagnostic. That is precisely the signature in the card's Vercel log, on precisely the process it names.

The change

One number, plus its reasoning and a changeset:

  • packages/spec/package.json — the DTS pass's ceiling, 12288 to 6144.
  • packages/spec/tsup.config.ts — the rationale recorded next to the DTS split it governs, with the measurements below, so the next author who hits ERR_WORKER_OUT_OF_MEMORY does not "fix" it by raising the number past what the container has.
  • .changeset/spec-dts-heap-ceiling-fits-container.md.

No source, no schema, no build output changes.

Measured — method

Peak anonymous RSS of the whole process tree, sampled from a cgroup-v1 memory.stattotal_rss at 150 ms with per-process attribution recorded at each new peak. The harness was validated in both directions before any number below was trusted: a positive control (600 MB allocation under an 8 GB cap completes, measured 589 MB) and a negative control (2 GB allocation under a 1 GB cap reproduces exit 137 with oom_kill=1).

The container emulation is two halves, because a cgroup cap alone is not a container — it bounds what a process may use without changing what it believes it has. So the runs marked "faithful" add a private mount namespace whose /proc/meminfo reports the container's size, and tools that size themselves from reported memory then see 8 GB rather than this host's 16 GB.

Measured — before / after

Spec's DTS pass, in a faithful 8192 MB container:

ceilingresultpeak RSSwall
12288 (before)ok6965 MB126s
6144 (after)ok5654 MB124s

−1311 MB (−19%) on the phase that was being killed, at no wall-time cost.

The cap sweep that chose 6144 (cgroup cap only, so the numbers run slightly higher):

ceilingresultpeak RSSwall
12288ok7290 MB132s
6144ok5794 MB134s
5120ok5328 MB148s
4096ERR_WORKER_OUT_OF_MEMORY113s

6144 is chosen as the largest ceiling whose worst case still fits: V8 cannot exceed it, and this pass's non-heap overhead measured ~250 MB, so the bound is ~6.4 GB inside an 8 GB container — while leaving 1.5x growth room above the 4096 floor before the pass reds. Below the ceiling it now fails loud (ERR_WORKER_OUT_OF_MEMORY, exit 1) instead of being SIGKILLed, which is what the card asked for.

Whole pipeline, pnpm turbo run build --filter=@objectstack/docs --force from a cleaned dist/.next, faithful emulation, with this fix: 2 successful, 2 total, 5m02s, peak 8135 MB.

Output identity — this buys headroom, not a different build

Every completing ceiling emitted a byte-identical declaration tree: 122 files, compared as one sha256 over all of them, 37cf1007189f945c at both 6144 and 5120. Declarations were deleted before each run so a hash could not be satisfied by leftovers (the DTS pass runs with clean: false). check-dts-emitted reports 34/34 declared declaration files present.

Lever 1 (cap turbo concurrency) — ruled out by measurement, not skipped

pnpm turbo run build --filter=@objectstack/docs --dry=json reports 2 tasks: @objectstack/spec#build and @objectstack/docs#build. The docs app's only workspace dependency is spec, so the two run sequentially. There is no parallel fan-out for --concurrency to cap, and the killed process was alone in the container when it died — capping concurrency would have changed nothing. The build command is versioned in apps/docs/vercel.json, so this was checkable in-repo; no dashboard-only configuration was involved in the command itself.

Lever 3 (name the regression)

No commit lands in the 16:52–16:57 window. The last change to spec's type surface before the failure boundary is daae7aa3 at 16:13 on 2026-08-25 — "declare the search and data.clone route response contracts" — which adds schemas and so grows the DTS type graph. Offered as the most plausible straw, not proven: the honest reading is that the configuration was one growth-step from this outage the whole time, sitting at ~7.0 GB with under 1 GB of spare in the container. Which commit crossed the line matters much less than that the ceiling was set above the container at all.

Residual risk, filed separately

This PR fixes the phase that was failing. It does not lower the pipeline's peak, which is owned by the other phase: next build holds ~7.6 GB in a single process, and neither --max-old-space-size (Turbopack is Rust; its memory is outside V8) nor experimental.cpus (the workers measure 100–430 MB each) bounds it. That phase has been building successfully on Vercel at this size for weeks, so it is a standing risk rather than a live failure — but it is now the largest consumer in the pipeline and nothing in this repo bounds it. Measured and filed as #12683, with the options laid out; it needs a maintainer decision, not a rider on this fix.

Verification

Gates run at a0a64684 on a clean tree — 30 derived families, all green, plus check:nul-bytes and @objectstack/spec typecheck. Two produced no reading rather than a failure and are recorded as such: check-dev-prereqs reports an unmet precondition (this worktree built only spec and docs, not all 67 packages), and scripts/pm/check-half-states.mjs exits 3 stating "no reading at all" without a GitHub token. Repository-wide lint is left to CI.

After merge, per the card:

curl -s https://objectstack.ai/ | grep -o 'data-dpl-id="[^"]*"'

must report an id other than dpl_2nfWjGjSwZjakVEmUG1kBWD6697r.

Refs: #12333 (measurement card) · epic #12243 · follow-up #12683

Generated by Claude Code


Generated by Claude Code

… memory
The DTS pass declared --max-old-space-size=12288 inside an 8 GB Vercel build
container. A ceiling above the container's memory does not permit a bigger
build: V8 defers major GCs below its ceiling, so the process is SIGKILLed at
the container limit before V8 ever reaches it -- exit 137, no diagnostic.
Measured inside a cgroup capped at 8192 MB (peak anonymous RSS, whole process
tree): 12288 -> 7290 MB; 6144 -> 5794 MB; 5120 -> 5328 MB; 4096 ->
ERR_WORKER_OUT_OF_MEMORY. Declaration output is byte-identical across every
completing ceiling (122 files, one sha256 over all of them).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKWDdUJ2XNRESVVWUvcpnh
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

⚠️1 changed file(s) yielded no anchor (packages/spec/tsup.config.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files. Nothing else in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 1 changed package(s)).

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/spec/tsup.config.ts) — pages documenting those are invisible to this run
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 126 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 07e64656593afb7c1915f5c97ea2e1587ef3b003packageMentionDocs.

@github-actionsgithub-actionsBot added dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation tooling labels Aug 27, 2026

@os-zhuangos-zhuang left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PM verification record (dispatching seat for #12677; this seat shares the PR's author identity, so this is a COMMENT review — the queue does not require an approval for non-governed surfaces, and packages/spec build config is not governed).

Read the full diff and the measurement chain:

  • The fix is the minimal correct shape: one ceiling number brought under the container it runs in, the rationale + measurement table recorded at the exact place the next author would edit the number, and a patch changeset. No source, schema, or output changes — declaration tree byte-identical across completing ceilings (37cf1007189f945c at both 6144 and 5120).
  • The harness was validated in both directions (positive + negative control reproducing the exit-137/oom_kill=1 signature) before any number was trusted, and the "faithful container" emulation (cgroup cap + private mount namespace lying about /proc/meminfo) is the right instrument — a cap alone doesn't change what tools believe they have.
  • Card lever 1 (turbo concurrency) ruled out by measurement (2-task linear graph; the killed process was alone), which the card explicitly allowed for.
  • Residual risk correctly severed into #12683 (the next build ~7.6 GB single-process peak) rather than ridden into this fix — that one needs a maintainer decision.

Landing via the normal queue. Post-merge verification stays as written: the data-dpl-id on objectstack.ai must move off dpl_2nfWjGjSwZjakVEmUG1kBWD6697r.


Generated by Claude Code

@os-zhuang
os-zhuang marked this pull request as ready for review August 27, 2026 08:55
@os-zhuang
os-zhuang enabled auto-merge August 27, 2026 08:55
@os-zhuang
os-zhuang added this pull request to the merge queueAug 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 33057527457 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Test Core (2/6) — 失败步骤: Run this shard's tests

    @objectstack/metadata-fs:test: FAIL test/watch-dot-root.test.ts > FileSystemRepository watcher — dot-rooted watch root (#7150) > sees an external edit when the root is under a dot-directory
    ↳ 失败原因: @objectstack/metadata-fs:test: AssertionError: expected 'delete' to be 'update' // Object.is equality
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

  • test/watch-dot-root.test.ts — 24h 窗口内只有本 PR 撞到过,暂不汇总(再有一个不同 PR 撞到就会自动开汇总 issue)。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 1 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 27, 2026
@os-zhuangClaude

Copy link
Copy Markdown
ContributorAuthor

Merge-queue ejection triage — the failure is not this PR's; spending the one re-run.


Generated by Claude Code

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependenciesPull requests that update a dependency filedocumentationImprovements or additions to documentationsize/stooling

Projects

None yet

2 participants

@os-zhuang@claude