[finding] Per-package test durations read out of Test Core logs are contended wall-clock, not per-package cost — `turbo run test --concurrency=4` puts up to 4 suites on one 4-vCPU runner

[os-elon]

Observational finding, split out of the #10152 measurement so it is not buried inside a card about one package. Filed unassigned, no labels — for triage to grade. Nothing in the tree is broken. This is about what a number means before anyone builds on it.

The mechanism

ci.yml's shard step runs, verbatim:

node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/test-core.log" --stall-minutes 10 \
 --report-dir "$RUNNER_TEMP/stall-reports" -- \
 pnpm turbo run test $FILTERS --concurrency=4

So a Test Core shard runs up to four packages' suites concurrently on one ubuntu-latest runner (4 vCPU), and each of those is an independent vitest run that sizes its own worker pool from availableParallelism(). The duration printed inside any one turbo task group is therefore that package's wall-clock while sharing a 4-core box with up to three other suites — it is not that package's cost, and how much it is inflated depends on which packages the partitioner happened to co-locate.

Measured size of the confound

Same tree (04096f1), six packages, each run alone on an idle 4-core box at --maxWorkers=2, against the CI figures those same packages produced in merge_group runs (the numbers in #10152's table and in scripts/partition-test-shards.mjs's header):

package	CI log	clean local	ratio
@objectstack/service-automation	118.9s	57.97s	2.05×
@objectstack/spec	496.4s	325.31s	1.53×
@objectstack/client	34.7s	23.75s	1.46×
@objectstack/driver-turso	53.5s	38.50s	1.39×
@objectstack/cli	548.6s	495.81s	1.11×
@objectstack/example-showcase	21.6s	31.15s	0.69×

A ~3× spread in the confound alone. Caveat stated rather than hidden: the local leg is itself handicapped at --maxWorkers=2, so these ratios understate the gap for the big packages and are unreliable for the small ones — the spread is the finding, not any single row. The cleanest single point is @objectstack/cli, measured on the same box at both worker counts: 495.81s at 2 workers, 337.13s at 4, against 548.6s in CI — i.e. CI is 1.63× the same package's clean cost when the package is given the whole box.

Why it matters

#10149 §6's second suggested route is to replace partition-test-shards.mjs's test-file count weight with measured per-package durations taken from recent main runs. Those durations carry exactly this confound, and the loop closes on itself: the weights would encode which packages happened to share a runner, and the weights then decide who shares a runner next time. A package that got co-located with three heavy suites is recorded as heavy, gets placed alone, measures light, and oscillates. That is a feedback loop, not a measurement.

Secondary consequence: the "the proxy tracks within roughly +-20% across the big packages" precision claimed in partition-test-shards.mjs's header is derived from this same source, so the ±20% band is not supportable at that resolution. (The @objectstack/cli outlier itself survives clean re-measurement — see the report on #10152 — so this does not overturn that finding; it bounds how finely the other rows can be read.)

What a contention-free source would look like

Not proposing an implementation — noting that the options differ a lot in cost:

• Run the weight-measurement leg with --concurrency=1, on main only, on a schedule rather than per-merge.
• Have each vitest write its own machine-readable duration (--reporter=json) and record CPU (user+sys) alongside wall — CPU is far less contention-sensitive than wall and is what a packing weight actually wants.
• Or keep test-file count and accept it as a coarse proxy, which is what it already claims to be.

Refs

• merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测) #10149 — the merge-queue critical-path card; its §6 route 2 is what this bears on. Not addressed here and remains open.
• [finding] @objectstack/cli's test suite costs ~2.8× the per-test-file norm, and on its own exceeds #4859's Test Core shard budget #10152 — the @objectstack/cli suite-cost card this was split out of; its report comment carries the full six-package measurement.
• .github/workflows/ci.yml — the --concurrency=4 line quoted above
• scripts/partition-test-shards.mjs — the weight function and the measured table in its header

Generated by Claude Code