ci: shard Test Core six ways to break the runner contention

[qq9340100]

Fixes#10227

Implements the maintainer's option C ruling of 2026-08-20 (recorded on the card): attack the runner contention by raising the Test Core shard count, with a like-for-like measurement on real GitHub 4-vCPU runners as a mandatory pre-step. Option B (rebasing the threshold) is rejected on that ruling; scripts/partition-test-shards.mjs semantics are untouched and there are no manual per-package exemption lists.

Result: the measurement clears the line.@objectstack/cli drops from 464.37s to 302.21s, and @objectstack/spec — alone on its own runner — finishes its whole shard in 283s, cold Turbo cache included. Numbers and method below.

The change

test job matrix [1, 2, 3] → [1, 2, 3, 4, 5, 6], plus the counts mechanically coupled to it and checked statically by check:shard-attestation: the job name:, the --shard N/M argument, the attestation --total, the artifact name, and the gate's declared leg.

--concurrency=4 is unchanged. Narrower shards may well want a lower value, but moving both knobs in one run makes the resulting measurement unreadable. That is a follow-up, not this PR.

Why six, not the four or five the card's arithmetic suggested. Sharding is BY PACKAGE, so no shard can finish faster than its single heaviest package, and the weight function is test-file count. @objectstack/spec carries 415 of the workspace's ~2360 test files. At five shards or fewer the partitioner must co-schedule it; at six it is alone in its bin. Bins from partition-test-shards.mjs against the full 76-package list, unchanged script:

shards	bins (test-file weight)	@objectstack/spec	@objectstack/cli
3	787 / 786 / 786	with 22 others	with 23 others
4	590 / 590 / 590 / 589	with 15 others	with 23 others
5	472 ×4 / 471	with 9 others	with 15 others
6	415 / 389 ×4 / 388	alone	with 13 others
7	415 / 324 ×6	alone	with 11 others

Six is the smallest count that isolates the heaviest indivisible suite. Past six, spec's shard cannot improve, only the others can.

A second effect that only shows up on the affected-set path: on a small affected set — the common PR — six shards give cli a runner to itself. Verified locally against a three-package affected set: shard 1 spec, shard 2 cli, shard 3 client, shards 4–6 empty and short-circuiting on the existing [ ! -s ] guard.

Pre-step: the measurement

The instrument named in the dispatch does not work as stated, and that is finding #1. A .github/**-only PR has an affected set of exactly 0 packages — measured locally against this branch with turbo ls --affected from merge-base(origin/main, HEAD):

affected count (ci.yml-only diff): 0
affected count (global-dep diff): 77

So an ordinary run of this PR schedules six shards, each of which prints "No packages on this shard — nothing to test", attests, and measures nothing. (.github/workflows/ci.yml is in the core paths-filter, so the job does start; it simply has no work.)

This PR therefore carried a temporary measurement scaffold — one comment appended to tsup.config.ts, a turbo globalDependencies entry — which made the affected set the full 77 packages and invalidated every task hash. The scaffold has been reverted; the diff under review is three files. The scaffold's effect is confirmed in the run itself: Cached: 0 cached, 68 total on cli's shard.

Baseline — 3 shards, real runners

push-to-main run 32392712980, full package list, ubuntu-latest 4-vCPU:

shard	fixed overhead	Run this shard's tests	job total
1/3	57s	398s	481s
2/3	65s	341s	431s
3/3	61s	645s	726s

Measured — 6 shards, real runners

Run 32396909164, full 77-package list, cold Turbo cache, all six shards green:

shard	fixed overhead	Run this shard's tests	contents
1/6	194s	283s	@objectstack/specalone — 1 of 1 pkg, 11059 tests
2/6	48s	474s
3/6	54s	586s
4/6	50s	614s	@objectstack/cli + 13 — 14 of 14 pkgs, 7129 tests
5/6	59s	511s
6/6	346s	536s

The number that decides it

The step totals above include a cold dependency build, because the scaffold invalidates every hash — on shard 4/6, 312s elapsed before cli's suite even started. So the step column is an upper bound, not the steady-state shard time. The metric that is immune to that, and that isolates exactly the variable this card is about, is the per-package vitest Duration:

3 shards, run 32392712980, shard 3/3 — @objectstack/cli
Duration 464.37s (transform 47.25s, setup 0ms, import 260.48s, tests 1102.42s)
6 shards, run 32396909164, shard 4/6 — @objectstack/cli
Duration 302.21s (transform 24.63s, setup 0ms, import 168.42s, tests 716.55s)
Tasks: 68 successful, 68 total
Cached: 0 cached, 68 total

Identical work both legs — 137 files, 1500 tests — and wall clock falls 35%, from 464.37s to 302.21s, now 118s under the 420s line. The CPU total falls with it, ~1410 CPU-s to ~909 CPU-s. That second number is the point: the same tests do not become cheaper, they stop paying for cache-thrash and context switching against three co-tenant suites. It is a direct confirmation that the durations this job's logs print are contention, not cost — the card's central claim, now measured on the runners rather than on a dev box.

spec's side is settled even more simply: alone on shard 1/6 it finished the entire step, cold build included, in 283s against 496.4s contended.

This closes both of the card's confidence gaps. Gap 1 (dev box vs real runner): cli costs ~1410 CPU-s on a runner against 1291 CPU-s on the dev box in #10152, so runner cores are only ~9% slower — the margin survives the environment change. Gap 2 (per-shard fixed overhead): at the median it is ~50-60s, far too small to eat the margin.

⚠ But gap 2 has a tail, and it is a real cost of this change

Fixed overhead is contended, not constant. Checkout repository across the six shards of one run: 155s, 14s, 21s, 13s, 13s, 309s. Same workflow, same commit, same runner label — a 24x spread, because six shards plus three dogfood legs plus Build Core, Build Docs, Console Pin Gate and Temporal Conformance all pull fetch-depth: 0 simultaneously. Two shards paid 194s and 346s of overhead, straight off the budget. The 3-shard baseline was tight by comparison (57s / 65s / 61s).

That is filed separately as #10383 with the numbers — it is out of this card's region and wants its own measurement across several runs before anyone changes the checkout.

Coupled changes

scripts/check-shard-attestation.mjs and scripts/check-required-contexts.mjs each carry self-test fixtures that mutate ci.yml by string replacement and assert the mutation matched rather than silently judging the pristine file. They did their job: both went red on this change and named the dead anchor. Three anchors moved with the matrix.

One is worth naming, because it would have rotted quietly. The grow the matrix fixture anchored on shard: [1, 2, 3] — which the dogfood job also spells. With the test matrix now six wide, String.replace would have taken dogfood's match first: still red, still for a plausible-looking reason, and no longer the assertion written there. The anchor now carries the full matrix including its closing bracket, with a comment saying why.

Verification

• All six Test Core shards green on run 32396909164, and the aggregate Test Core gate concluded success counting the six shard-attest-test-N-of-6 credentials — so the coupled --total / --leg / artifact-name edits are proven on a real run, not only by the static drift guard.
• Gates re-derived at the final commit 3ce316f8 with node scripts/pm/dispatch-gates.mjs (no args, three-dot semantics against merge base 0bfbeb70c) and run there. Their own verdict lines:
  • check-node-version: OK (29 setup-node step(s) across 26 workflow(s), all on Node 22).
  • ✓ check-shard-attestation: 2 aggregate gate(s) count 3 declared leg(s) across 3 attesting job(s).
  • check-workflow-status-functions: OK (scanned 26 workflow file(s), 45 job(s) …)
  • OK: 12 package(s) read outside themselves, all declared, and turbo.json hashes every declared glob.
  • check-nul-bytes: OK (scanned 6115 text file(s) … no raw ASCII control bytes).
  • check:required-contexts green (full registry printed).
• pnpm lint green repo-wide.
• .github/** and scripts/** only, no published package changes, so skip-changeset is applied.

Notes for the neighbours

• Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 — this PR's region is the test job's shard matrix, the coupled attestation counts, and the --concurrency=4 comment. It does not touch the concurrency: group block or any Turbo cache restore/save step. The chosen shard count is 6, which multiplies that card's per-shard cache-key namespaces sixfold; stated on that issue when this PR opened. Note that shards 4–6 have no main-seeded namespace yet and will fall back through the shard-agnostic Linux-turbo-test- restore key until a main push seeds them.
• CI flake: @objectstack/example-showcase#test fails on mass sys_* Find operation failed + Acquire connection error: operation timed out — hit a merge-queue build and a PR run within 2h on unrelated prose-only diffs #10293 (pool-timeout flake) — shard membership is re-derived by the partitioner, so the flaky suite's co-tenants change. Six shards mean fewer concurrent suites per runner, which reduces the resource pressure that timeout is most likely reacting to; but this PR neither fixes nor tests that, and the flake should be re-observed against the new placement.
• Card confidence gap 4 — whether the 420s line should be 420s at all — remains an open question on the card. The ruling explicitly left it as a recorded question and this PR does not touch it.

Cost, stated plainly

Per-shard fixed overhead is paid once per shard and the merge queue multiplies it by the speculative build count. Six shards trade roughly 1.7x the billed runner-minutes for a shorter critical path. This buys latency, not cheaper CI, and that trade is written into the ci.yml comment so the next reader does not have to rediscover it.

---

Generated by Claude Code