`pnpm test` is OOM-killed at turbo's default concurrency on a 4-CPU/15 GB container — exit 137, 16 of 91 tasks never run

[os-steve]

Found while taking the machine-lane baseline for the QA wave (#9296, round #9298). Filed as a finding, not fixed — R0 lands no code.

Summary

pnpm test (i.e. turbo run test with no --concurrency) is killed by the kernel cgroup OOM killer on this container. It does not report a failing test — it dies. Reproduced twice; each run was killed at a different package, which is the signature of a resource kill rather than a package defect.

turbo.json declares no concurrency, so turbo defaults to 10 parallel tasks. This box is 4 CPU / ~15 GB. Ten concurrent test tasks, each forking its own vitest worker pool, exceeds the memory allowance.

Environment

• framework sha: 1f6d04703df8f518c80d0fbaa8b19262068e405d (tip of main at run time)
• .objectui-sha: 190fbd01d0615e2e168faf9e08b8ad7844bc039d
• node v22.22.2, pnpm 10.31.0, linux x64, nproc=4, 15 GB RAM
• fresh git worktree add --detach + a single pnpm install; deps prebuilt; run serialized under the shared verify lock (no other seat's gate running)

Reproduction

pnpm test

Run 1 — exit 137, 7m48s:

@objectstack/service-storage#test: ERROR command (…/packages/services/service-storage) pnpm run test exited (137)

 Tasks: 75 successful, 91 total
Cached: 45 cached, 91 total
 Time: 7m47.442s
Failed: @objectstack/service-storage#test

 ERROR run failed: command exited (137)

Run 2 — exit 137, 6m57s, killed at a different package:

@objectstack/metadata-protocol#test: ERROR command (…/packages/metadata-protocol) pnpm run test exited (137)

 Tasks: 75 cached, 91 total
 Time: 6m57.19s
Failed: @objectstack/metadata-protocol#test

Run 1 carried NODE_OPTIONS=--max-old-space-size=4096; run 2 was run plain, with no NODE_OPTIONS at all, and died the same way. The heap flag is not the cause.

Kernel confirmation (dmesg), two events, one per run:

Memory cgroup out of memory: Killed process 28332 (node) total-vm:45803420kB, anon-rss:820896kB, …

No test actually failed

This is the part that matters for reading the gate. Across both runs, zero vitest summaries reported a failing test:

	run 1	run 2
vitest summaries emitted	31	31
tests passed	8 914	8 914
tests failed	0	0
turbo tasks successful	75 / 91	75 / 91

The eight packages that logged ELIFECYCLE Test failed (objectql, spec, driver-sql, plugin-sharing, metadata-protocol, service-analytics, plugin-audit, service-storage in run 1) emitted no vitest failure summary — they were torn down by the kill, not reddened by an assertion. Read naively, the gate looks like eight broken packages; it is one OOM.

16 of 91 tasks never ran at all. A run that ends this way covers an unknown subset of the repo, so it cannot be reported as a pass or as a product failure.

What does work

Cutting the fan-out completes the suite:

pnpm exec turbo run test --continue --concurrency=2
 Tasks: 135 successful, 136 total
 Time: 30m38.998s

Peak memory during that run stayed around 7.6 GB used, versus the OOM at default fan-out.

Relationship to prior art

Same class as #9311 (closed, fixed) — a pnpm test at default concurrency that reds for reasons unrelated to the code under test — but a different mechanism: #9311 was two default-vitest-timeout cases losing a CPU race, and was correctly fixed by giving those cases real timeouts. This one is memory, not time, and no per-test timeout can address it. #9311 explicitly left the broader question open:

Separately worth a maintainer decision: whether turbo.json should pin a concurrency proportional to available CPUs, so the whole suite stops depending on runner size.

This issue is that question arriving with a measurement behind it. See also #4845 (closed) for the precedent that this repo's memory ceiling is a real constraint, not a theoretical one.

Suggested direction (not implemented here)

Decide whether pnpm test should be runnable on a small box at all, and if so how:

• pin concurrency in turbo.json (or in the test script) proportional to nproc, so the suite stops depending on runner size; or
• leave the default and document --concurrency=2 as the supported invocation on constrained runners, so agents and contributors stop reading an OOM as eight broken packages.

The first option is a repo-wide behaviour change and is a maintainer call, which is why nothing is changed here.

---

Generated by Claude Code

[claude]

Triage: lands in turbo.json (root test concurrency policy); domain:devx. → pm:queue, Task — resource-kill signature is measured (exit 137 at a different package each run; turbo default 10 concurrent test tasks on a 4-CPU/15 GB container). Constraint for dispatch: do not slow the larger CI runners with a hard low constant — prefer an environment-derived concurrency (or a documented container profile) and re-measure both runtimes in the PR. Size/model suggestion: S, sonnet if the tier output agrees at dispatch time.

---

Generated by Claude Code

[os-steve]

Claim: PM dispatch (devx seat)

• Session: session_015ahemw8RcTgqtxrj15PEZx
• Branch: claude/issue-11938-turbo-test-concurrency

⭐ This card is the QA wave working as designed. It was filed by this seat's own #9298 (R0 baseline) dev, from a real measurement taken while doing something else — and it was filed as a finding with nothing fixed, because R0 lands no code. That is the right shape and the reason the reading is trustworthy: two runs, killed at a different package each time, which is the signature of a resource kill rather than a package defect, plus dmesg cgroup confirmation and run 2 carrying no NODE_OPTIONS at all, ruling out the heap flag.

The part that makes this urgent beyond throughput

Read naively, the gate looks like eight broken packages; it is one OOM.

Eight packages logged ELIFECYCLE Test failed while emitting zero vitest failure summaries — torn down, not reddened. And 16 of 91 tasks never ran, so such a run covers an unknown subset and cannot honestly be reported as a pass or a product failure. That misreading is exactly what a seat under time pressure would do.

Triage's constraint is the design brief, not a footnote

do not slow the larger CI runners with a hard low constant — prefer an environment-derived concurrency (or a documented container profile) and re-measure both runtimes in the PR.

A flat --concurrency=2 fixes this box and taxes every larger runner; the measured cost here was 30m39s at 2 versus a ~7m death at 10. Both numbers have to move in the PR, on both runtime shapes.

Tier: sonnet — triage suggested it and I agree; derived live, no path-derived mandate on turbo.json, so sonnet is the floor and this is a deliberate floor call: the measurement protocol is already written down on the card, and what remains is a config change plus two timed runs.

---

Generated by Claude Code

[os-steve]

os-dev-report

{
 "issue": 11938,
 "status": "done",
 "branch": "claude/issue-11938-turbo-test-concurrency",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/11954",
 "premise_still_valid": true,
 "summary": "Root `test` script now runs `turbo run test --concurrency=50%` (turbo's own percentage syntax, resolved against the host's own core count at invocation time) instead of relying on turbo's flat default of 10. Scoped to the root test-concurrency policy only per Zone 1 -- none of the eight OOM-torn-down packages were touched. Verified zero CI impact: every workflow invocation of `turbo run test` already passes its own explicit `--concurrency=4`, and nothing in CI calls the root `pnpm test` script at all. Initially shipped `--concurrency=100%` (=4 on this 4-CPU box), backed by real production evidence (CI's own `--concurrency=4` on this repo's only runner shape, ubuntu-latest 4vCPU/16GB, repeatedly green in `rerun-safety-nightly.yml`'s full-suite double-pass); revised down to `50%` after live `ps` evidence during measurement showed vitest's own worker pool ALSO scales with host cores (4 concurrent turbo tasks were running 10 total vitest worker processes), so `100%` on a hypothetically bigger box would be a real increase over turbo's old flat default of 10 in the opposite direction from the defect -- `50%` never exceeds the old default until 20+ cores, and is the exact value the original card's own investigation already measured directly and safely (135/136 tasks, 30m39s, peak ~7.6GB).",
 "tests": "`pnpm build` under the shared verify lock: 71/71 tasks, 5m14s, clean (VERDICT command-exit 0). `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack`: 0 check families matched package.json -- nothing else to run for this diff; skip-changeset label applied and read back. Full `pnpm test` at the SHIPPED value (--concurrency=50%, =2 on this box) was NOT freshly re-run by this session as a complete suite -- resting on the original card's own prior direct measurement (135/136 tasks, 30m39s, peak ~7.6GB), which is real but pre-dates this PR; disclosing this gap rather than implying a fresh run happened. A fresh, live full `pnpm test` run WAS executed at --concurrency=100% (the value initially shipped, then walked back) during this session: at the time of this report it had run ~13 minutes uninterrupted with no failing test and no OOM signature (no 'Killed process', no exit 137 in the log), peak memory sampled every 5s topping out at ~8.4GB against the ~15GB ceiling -- still in progress when this report was written, included as supplementary monotonic evidence (50% does strictly less concurrent work than 100% on the same box, so if 100% clears without OOM, 50% clearing it -- independently already proven by the card -- is the expected lower-resource case, not a coincidence). Container is shared with other parallel agents throughout this session (observed via os-verify-lock.sh --status queued behind another agent's sweep.sh during the build step), so all wall-clock numbers are reported as noisy/observed, not clean benchmarks. Default concurrency (10) was NOT re-triggered deliberately on this shared container -- the card's own twice-reproduced, dmesg-confirmed OOM already stands as evidence, and deliberately re-inducing an OOM on a box shared with other agents' in-flight work risks killing an unrelated process (the kernel OOM killer is not scoped to the invoking process).",
 "open_questions": [
 {
 "question": "The shipped value (50%) rests on the original card's prior measurement rather than a fresh full-suite run at exactly 50% executed by this session (a fresh run at the initially-shipped 100% value was executed instead, as supplementary/monotonic evidence). Is that sufficient, or should a fresh 50% full-suite run be required before this lands?",
 "options": [
 "A: Accept the card's own direct 50%-value measurement plus this session's fresh 100%-value monotonic run as sufficient evidence -- both are real, on this exact hardware shape, and 50% strictly dominates 100% in resource usage",
 "B: Require a fresh full-suite run at exactly the shipped 50% value before merge, even though it duplicates evidence the card already has"
 ],
 "recommendation": "A, because the card's evidence for 50% is direct (not extrapolated) and on the identical hardware shape, and this session's fresh 100% run provides an independent monotonic upper bound on the same box -- a third full run of the same suite at 50% would spend ~30 more minutes of shared-container time to re-confirm a number that is already doubly evidenced."
 }
 ],
 "out_of_scope_findings": [
 "filed as #11958: vitest's own worker pool scales with host cores, uncapped, in 40/41 packages -- compounds with turbo's outer --concurrency; suggested direction is a single invocation-layer fix (not 41 file edits), needs its own measurement and PR"
 ]
}

---

Generated by Claude Code