Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions packages/cli/vitest.config.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -38,6 +38,82 @@
// the gate's walk. That is one package, `@objectstack/observability`, which is
// already in this package's ledger entry — so the required set is unchanged in
// both directions.
//
// ## WHY THERE IS STILL NO `test` BLOCK — the suite cost, measured (#10152)
//
// This package's suite was the largest single item on the Test Core critical
// path (548.6s / 474.4s in two `merge_group` runs), and the standing theory for
// suite cost in this repo — per-file module-graph re-execution under isolation,
// the proxy `scripts/partition-test-shards.mjs` weights by — predicted the
// cause would be import surface, which would have made a `test` block (`pool`,
// `isolate`) the lever. Measured on 2026-08-20, it is NOT that, and the
// measurement is recorded here because this file is where the next person
// looking for a lever will arrive.
//
// One machine, 4 cores, warm build, `npx vitest run --maxWorkers=2` per
// package, vitest 4.1.10. Both normalisers, because per-FILE cost alone cannot
// tell "expensive suite" from "more tests per file":
//
// package files tests tests/file wall s/file s/test
// @objectstack/cli 137 1498 10.9 495.81s 3.619 0.3310
// @objectstack/spec 415 11045 26.6 325.31s 0.784 0.0295
// …/service-automation 83 991 11.9 57.97s 0.698 0.0585
// …/driver-turso 39 1003 25.7 38.50s 0.987 0.0384
// @objectstack/client 23 314 13.7 23.75s 1.033 0.0756
// …/example-showcase 21 342 16.3 31.15s 1.483 0.0911
//
// Normalising per TEST makes this package look worse, not better: it has the
// LOWEST tests-per-file of the six, so its 2.4-5.2x per-file cost becomes
// 3.6-11x per test. "It just has more tests per file" is falsified.
//
// The `Duration` split says where the cost is NOT:
//
// cli 495.81s (transform 29.47s, setup 0ms, import 192.91s, tests 774.95s)
// spec 325.31s (transform 13.79s, setup 0ms, import 63.90s, tests 456.36s)
// svc-auto 57.97s (transform 14.54s, setup 0ms, import 96.78s, tests 4.73s)
// turso 38.50s (transform 16.85s, setup 0ms, import 65.98s, tests 3.56s)
// client 23.75s (transform 18.31s, setup 0ms, import 39.72s, tests 2.25s)
//
// Per file this package's import cost is 1.41s and its transform cost 0.215s —
// MID-BAND and LOWEST respectively (client 1.73s/file import, turso 1.69s).
// The wide dependency closure in `package.json` is not what the test files
// import. `setup` is 0ms everywhere, so setupFiles cost is not it either.
//
// The cost is test-body work, and it is concentrated, not uniform: median file
// 0.03s, 105 of 137 files under 2s, top 20 files = 87.7% of the wall. The 20
// files that spawn the real CLI as a subprocess (`bin/run-dev.js` through
// `tsx`, against a `mkdtemp` project) are 56.1% of the file wall (300.1s) while
// carrying 177 of 1498 tests. Each spawn re-executes the CLI's module graph in
// a COLD process, which is the standing theory after all — relocated out of
// vitest's worker, where neither its transform cache nor its module registry
// can reach it. Floor per spawn, doing nothing but printing a version:
//
// tsx bin/run-dev.js --version 6.5-6.8s (the source entry these use)
// node bin/run.js --version 2.9-3.2s (the built entry)
// node -e 0 0.031s (process floor)
//
// ⚠️ Two levers were measured and both are rejected HERE, on this evidence:
//
// `test: { maxWorkers: 4 }` — real but not ours to take. 2->4 workers on an
// idle box is 495.81s -> 337.13s, but CPU is flat (user+sys 1291.3s ->
// 1256.2s) and per-file wall INFLATES (sum 535.1s -> 748.3s; longest file
// 74.3s -> 104.1s): the box is saturated, so this is packing, not work. In CI
// the box is not this package's — `ci.yml` runs `turbo run test
// --concurrency=4` — so pinning a worker count here spends cores belonging to
// whatever else lands on the shard. Worker allocation is a property of the
// shard, decided in `ci.yml`, not of this config (#10149).
//
// `NODE_COMPILE_CACHE` for the spawned processes — measured 6.98/6.24/6.41/
// 6.18s cached vs 6.39/6.72s uncached, i.e. inside noise for 42MB of cache.
// The per-spawn cost is module-graph EXECUTION, not compilation.
//
// So the work is real, the price is fair, and nothing contained in this package
// removes it without changing what the e2e tests assert. Swapping the spawns to
// the built entry would halve per-spawn boot and is exactly the source-vs-dist
// trade `scripts/check-test-source-alias.mjs` exists to refuse — see the note
// above on why a test that passes GREEN against a stale artifact is the
// dangerous outcome. Before adding a `test` block for speed, re-measure: if
// `tests` is still the dominant term, the block is not the lever.
import { defineConfig } from 'vitest/config';
import path from 'path';

Expand Down
Loading