Skip to content

test(cli): record the measured cause of this suite's cost in the vitest config header - #10258

Merged
os-elon merged 1 commit into
mainfrom
claude/issue-10152-cli-suite-cost
Aug 20, 2026
Merged

test(cli): record the measured cause of this suite's cost in the vitest config header#10258
os-elon merged 1 commit into
mainfrom
claude/issue-10152-cli-suite-cost

Conversation

@os-elon

Copy link
Copy Markdown
Collaborator

Fixes#10152

Comment-only change to packages/cli/vitest.config.ts. No test block was added; the executable config is byte-identical.

What was measured

One machine (4 cores), warm build, npx vitest run --maxWorkers=2 per package, vitest 4.1.10, tree at 04096f1. Gate union re-run on the final commit b1ff122.

Both normalisers, because per-file cost alone cannot separate "expensive suite" from "more tests per file":

packagefilesteststests/filewalls/files/test
@objectstack/cli137149810.9495.81s3.6190.3310
@objectstack/spec4151104526.6325.31s0.7840.0295
@objectstack/service-automation8399111.957.97s0.6980.0585
@objectstack/driver-turso39100325.738.50s0.9870.0384
@objectstack/client2331413.723.75s1.0330.0756
@objectstack/example-showcase2134216.331.15s1.4830.0911

The null hypothesis — "cli simply carries more tests per file, so per-file cost is the wrong normaliser" — is falsified, and in the direction that strengthens the finding: cli has the lowest tests-per-file of the six, so its 2.4–5.2× per-file cost becomes 3.6–11× per test.

Where the cost is not

cli 495.81s (transform 29.47s, setup 0ms, import 192.91s, tests 774.95s, environment 17ms)
spec 325.31s (transform 13.79s, setup 0ms, import 63.90s, tests 456.36s, environment 49ms)
svc-auto 57.97s (transform 14.54s, setup 0ms, import 96.78s, tests 4.73s, environment 9ms)
turso 38.50s (transform 16.85s, setup 0ms, import 65.98s, tests 3.56s, environment 4ms)
client 23.75s (transform 18.31s, setup 0ms, import 39.72s, tests 2.25s, environment 3ms)

Per file, cli's import cost is 1.41s (mid-band — client 1.73s, driver-turso 1.69s) and its transform cost 0.215s (the lowest of the six). The wide dependency closure in package.json is not what the test files import. setup is 0ms everywhere.

Where it is

Test-body work, concentrated rather than uniform: median file 0.03s, 105 of 137 files under 2s, top 20 files = 87.7% of the wall. The 20 files that spawn the real CLI as a subprocess (bin/run-dev.js through tsx, against a mkdtemp project) are 56.1% of the file wall (300.1s) while carrying 177 of 1498 tests.

Each spawn re-executes the CLI's module graph in a cold process — so the standing per-file module-graph theory is the right mechanism, relocated out of vitest's worker where neither its transform cache nor its module registry can reach it. Floor per spawn, printing nothing but a version:

tsx bin/run-dev.js --version 6.5-6.8s (the source entry the e2e files use)
node bin/run.js --version 2.9-3.2s (the built entry)
node -e 0 0.031s (process floor)

Two levers measured and rejected here

  • test: { maxWorkers: 4 } — 2→4 workers on an idle box is 495.81s → 337.13s, but CPU is flat (user+sys 1291.3s → 1256.2s) and per-file wall inflates (sum 535.1s → 748.3s; longest file 74.3s → 104.1s). The box is saturated: that is packing, not work. In CI the box is not this package's — ci.yml runs turbo run test --concurrency=4 — so pinning a worker count here spends cores belonging to whatever else lands on the shard. Worker allocation is a shard property, decided in ci.yml.
  • NODE_COMPILE_CACHE — 6.98 / 6.24 / 6.41 / 6.18s cached vs 6.39 / 6.72s uncached: inside noise, for 42MB of cache. The per-spawn cost is module-graph execution, not compilation.

So the work is real and nothing contained in this package removes it without changing what the e2e tests assert. Swapping the spawns to the built entry would halve per-spawn boot and is exactly the source-vs-dist trade scripts/check-test-source-alias.mjs exists to refuse.

Scope

scripts/partition-test-shards.mjs, ci.yml and the shard count are untouched by design — #10149's A/B choice is parked behind this measurement and is not prejudged here. #10149 is not addressed by this PR and remains open; the reading that bears on it is in the report comment on #10152.

Verification

b1ff122 · all captured by redirecting to a file before reading, never through a pipe:

check:slot-lookup exit=0 baseline key set verified against 04096f1: no files added.
check-affected-docs exit=0
check:test-source-alias exit=0 72 packages with tests scanned; 43 published subpath(s) resolved through every alias table.
check:nul-bytes exit=0 scanned 6075 text file(s); no raw ASCII control bytes.
check:cross-package-test-inputs exit=0 All 60 self-test cases passed.
check:docs-image-tag exit=0
check:override-consistency exit=0
check:route-envelope exit=0
check-changeset-fixed exit=0
check-osv-exemptions exit=0
cli typecheck (tsc --noEmit) exit=0

node scripts/pm/dispatch-gates.mjs (no paths — it derives its own change set) named only check:slot-lookup and check-affected-docs for this diff; the rest of the list above is the dispatch's lead list, run anyway. No family had to be added.

Suite itself: Test Files 137 passed (137) · Tests 1498 passed (1498) at both --maxWorkers=2 and --maxWorkers=4.

skip-changeset: the diff is one comment block in a test-config file. Nothing lands in src/, so nothing publishes.

Generated by Claude Code


Generated by Claude Code

…st config header
`@objectstack/cli`'s suite is the largest item on the Test Core critical path
(548.6s / 474.4s in two merge_group runs). Nobody had measured why, and the
standing theory for suite cost in this repo — per-file module-graph
re-execution, the proxy partition-test-shards.mjs weights by — predicted a
`test` block (pool / isolate) would be the lever.
Measured: it is not. Per file this package's import cost (1.41s) is mid-band
and its transform cost (0.215s) the lowest of six packages measured; `setup`
is 0ms everywhere. The cost is test-body work, concentrated in the 20 files
that spawn the real CLI as a subprocess — 56.1% of the file wall while
carrying 177 of 1498 tests — where each spawn re-executes the CLI's module
graph in a cold process at ~6.6s through tsx.
Normalising per test rather than per file makes the outlier larger, not
smaller: this package has the lowest tests-per-file of the six, so 2.4-5.2x
per file becomes 3.6-11x per test.
Comment-only. The config keeps no `test` block, and the header now records the
two levers that were measured and rejected (`maxWorkers`, NODE_COMPILE_CACHE)
so the next person does not re-derive them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185KTgmREZA4sY5pnRbubXj
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 1 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/cli/vitest.config.ts) — pages documenting those are invisible to this run

Coarse fallback — 23 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 318f96ae5cc25d04a2650a6ef6d4c2b930619b96packageMentionDocs.

@os-elonos-elon added skip-changeset PR has no user-facing published change; bypasses the changeset gate and removed tests labels Aug 20, 2026 — with Claude
@os-elon
os-elon marked this pull request as ready for review August 20, 2026 15:17
@os-elon
os-elon added this pull request to the merge queueAug 20, 2026
Merged via the queue into main with commit ad492e7Aug 20, 2026
31 checks passed
@os-elon
os-elon deleted the claude/issue-10152-cli-suite-cost branch August 20, 2026 15:32
qq9340100 pushed a commit that referenced this pull request Aug 20, 2026
Test Core's per-package durations are contended wall clock, not package
cost: `--concurrency=4` on a 4-vCPU runner puts up to four suites on four
cores. Measured like-for-like against the same suites run alone on an idle
4-core box (#10152 / PR #10258), neither long pole is inherently over
#4859's 420s line -- cli 548.6s contended vs 337.13s alone, spec 496.4s vs
325.31s. What is over the line is the co-tenancy.
Six is the smallest shard count that puts `@objectstack/spec` -- 415 of the
workspace's ~2360 test files, and indivisible because sharding is BY
PACKAGE -- alone in its bin (415/389/389/389/389/388). Past six its shard
cannot improve, only the others can.
`--concurrency=4` is deliberately unchanged: narrower shards may want a
lower value, but moving both at once makes the next measurement unreadable.
The two gate self-tests that anchor on ci.yml's literal text move with it.
The `grow the matrix` anchor gains the full matrix including its closing
bracket -- the dogfood job also spells `shard: [1, 2, 3]` and
`String.replace` takes the first match, so the short anchor would have
silently started mutating dogfood's matrix instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C5fGRTUHXFevsEdMEFDadM
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/sskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] @objectstack/cli's test suite costs ~2.8× the per-test-file norm, and on its own exceeds #4859's Test Core shard budget

2 participants

@os-elon@claude