Uh oh!
There was an error while loading. Please reload this page.
test(cli): record the measured cause of this suite's cost in the vitest config header - #10258
Merged
Merged
Conversation
…st config header `@objectstack/cli`'s suite is the largest item on the Test Core critical path (548.6s / 474.4s in two merge_group runs). Nobody had measured why, and the standing theory for suite cost in this repo — per-file module-graph re-execution, the proxy partition-test-shards.mjs weights by — predicted a `test` block (pool / isolate) would be the lever. Measured: it is not. Per file this package's import cost (1.41s) is mid-band and its transform cost (0.215s) the lowest of six packages measured; `setup` is 0ms everywhere. The cost is test-body work, concentrated in the 20 files that spawn the real CLI as a subprocess — 56.1% of the file wall while carrying 177 of 1498 tests — where each spawn re-executes the CLI's module graph in a cold process at ~6.6s through tsx. Normalising per test rather than per file makes the outlier larger, not smaller: this package has the lowest tests-per-file of the six, so 2.4-5.2x per file becomes 3.6-11x per test. Comment-only. The config keeps no `test` block, and the header now records the two levers that were measured and rejected (`maxWorkers`, NODE_COMPILE_CACHE) so the next person does not re-derive them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185KTgmREZA4sY5pnRbubXj
Contributor
📓 Docs Drift CheckNothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 1 changed package(s)), so this run has no opinion about the docs. What this run could not see
Coarse fallback — 23 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): |
os-elon
marked this pull request as ready for review
August 20, 2026 15:17
Uh oh!
There was an error while loading. Please reload this page.
qq9340100 pushed a commit
that referenced
this pull request
Aug 20, 2026
Test Core's per-package durations are contended wall clock, not package cost: `--concurrency=4` on a 4-vCPU runner puts up to four suites on four cores. Measured like-for-like against the same suites run alone on an idle 4-core box (#10152 / PR #10258), neither long pole is inherently over #4859's 420s line -- cli 548.6s contended vs 337.13s alone, spec 496.4s vs 325.31s. What is over the line is the co-tenancy. Six is the smallest shard count that puts `@objectstack/spec` -- 415 of the workspace's ~2360 test files, and indivisible because sharding is BY PACKAGE -- alone in its bin (415/389/389/389/389/388). Past six its shard cannot improve, only the others can. `--concurrency=4` is deliberately unchanged: narrower shards may want a lower value, but moving both at once makes the next measurement unreadable. The two gate self-tests that anchor on ci.yml's literal text move with it. The `grow the matrix` anchor gains the full matrix including its closing bracket -- the dogfood job also spells `shard: [1, 2, 3]` and `String.replace` takes the first match, so the short anchor would have silently started mutating dogfood's matrix instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5fGRTUHXFevsEdMEFDadM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes#10152
Comment-only change to
packages/cli/vitest.config.ts. Notestblock was added; the executable config is byte-identical.What was measured
One machine (4 cores), warm build,
npx vitest run --maxWorkers=2per package, vitest 4.1.10, tree at04096f1. Gate union re-run on the final commitb1ff122.Both normalisers, because per-file cost alone cannot separate "expensive suite" from "more tests per file":
@objectstack/cli@objectstack/spec@objectstack/service-automation@objectstack/driver-turso@objectstack/client@objectstack/example-showcaseThe null hypothesis — "
clisimply carries more tests per file, so per-file cost is the wrong normaliser" — is falsified, and in the direction that strengthens the finding:clihas the lowest tests-per-file of the six, so its 2.4–5.2× per-file cost becomes 3.6–11× per test.Where the cost is not
Per file,
cli's import cost is 1.41s (mid-band —client1.73s,driver-turso1.69s) and its transform cost 0.215s (the lowest of the six). The wide dependency closure inpackage.jsonis not what the test files import.setupis 0ms everywhere.Where it is
Test-body work, concentrated rather than uniform: median file 0.03s, 105 of 137 files under 2s, top 20 files = 87.7% of the wall. The 20 files that spawn the real CLI as a subprocess (
bin/run-dev.jsthroughtsx, against amkdtempproject) are 56.1% of the file wall (300.1s) while carrying 177 of 1498 tests.Each spawn re-executes the CLI's module graph in a cold process — so the standing per-file module-graph theory is the right mechanism, relocated out of vitest's worker where neither its transform cache nor its module registry can reach it. Floor per spawn, printing nothing but a version:
Two levers measured and rejected here
test: { maxWorkers: 4 }— 2→4 workers on an idle box is 495.81s → 337.13s, but CPU is flat (user+sys 1291.3s → 1256.2s) and per-file wall inflates (sum 535.1s → 748.3s; longest file 74.3s → 104.1s). The box is saturated: that is packing, not work. In CI the box is not this package's —ci.ymlrunsturbo run test --concurrency=4— so pinning a worker count here spends cores belonging to whatever else lands on the shard. Worker allocation is a shard property, decided inci.yml.NODE_COMPILE_CACHE— 6.98 / 6.24 / 6.41 / 6.18s cached vs 6.39 / 6.72s uncached: inside noise, for 42MB of cache. The per-spawn cost is module-graph execution, not compilation.So the work is real and nothing contained in this package removes it without changing what the e2e tests assert. Swapping the spawns to the built entry would halve per-spawn boot and is exactly the source-vs-dist trade
scripts/check-test-source-alias.mjsexists to refuse.Scope
scripts/partition-test-shards.mjs,ci.ymland the shard count are untouched by design — #10149's A/B choice is parked behind this measurement and is not prejudged here. #10149 is not addressed by this PR and remains open; the reading that bears on it is in the report comment on #10152.Verification
b1ff122· all captured by redirecting to a file before reading, never through a pipe:node scripts/pm/dispatch-gates.mjs(no paths — it derives its own change set) named onlycheck:slot-lookupandcheck-affected-docsfor this diff; the rest of the list above is the dispatch's lead list, run anyway. No family had to be added.Suite itself:
Test Files 137 passed (137) · Tests 1498 passed (1498)at both--maxWorkers=2and--maxWorkers=4.skip-changeset: the diff is one comment block in a test-config file. Nothing lands insrc/, so nothing publishes.Generated by Claude Code
Generated by Claude Code