fix(devx): one path convention for the shard package list, and the measurement ranking the Test Core imbalance hypotheses

[os-elon]

Fixes#10056

Part of #10149 — the measurement half only. The weight function, the partitioner and the shard count are unchanged, on purpose: the measurement says re-weighting alone cannot reach that card's acceptance threshold, and picking a direction anyway would be guessing. The open question is spelled out at the bottom for the maintainer.

Gates below were run at d80ca88, the head of this branch.

---

1. The measurement (this was the first deliverable)

Three competing explanations were on the table. Ranked against real merge_group data, not the two-run sample the card was filed on.

Test Core (2/3) = 3 seconds is not an empty bin and not a weight-0 cascade

This was the datapoint asked to discriminate the hypotheses. From the job log of that leg (run 32335265453):

check-test-completeness: OK (24 package(s), 14551 test(s) declared and all 14551 accounted for).
Tasks: 79 successful, 79 total
Cached: 79 cached, 79 total
Time: 866ms >>> FULL TURBO

A fully populated shard — 24 packages, 14,551 tests — replaying from a warm Turbo cache. It also cannot be a sibling-failure artifact: the matrix is fail-fast: false, and that job published its attestation.

H1 (the heterogeneous-path mechanism) cannot fire in a queue build

ci.yml's "Compute this shard's package set" computes SCM_BASE only when github.event_name == 'pull_request'. On merge_group it is empty, so the else branch runs pnpm exec turbo ls --output=json and --union-into is never invoked. No absolute path is appended in a queue build, and turbo ls on this tree emits 0 absolute paths of 77. The mechanism is real — it is just real on pull_request runs, which is not where the symptom was measured.

The partitioner and LPT are exonerated

Reproducing the exact input a queue build hands the script (full turbo ls, run from the repo root):

shard 1/3: 22/76 packages, weight 783 (all bins: 783/783/782)
shard 2/3: 25/76 packages, weight 783 (all bins: 783/783/782)
shard 3/3: 29/76 packages, weight 782 (all bins: 783/783/782)

A one-file spread across 2348 — 0.13%. The card compared a 3.4× duration spread against LPT's ≤4/3 bound, but that bound is over weights, and on weights it is met with enormous margin. There is no bucketing defect.

H2 (proxy decay) is supported, and localised

Per-package durations read out of the turbo task groups in queue-build logs, against each package's weight in the same tree:

package	weight (test files)	measured	s/file
@objectstack/cli	135	548.6s / 474.4s	3.5–4.1
@objectstack/spec	414	496.4s	1.20
@objectstack/service-automation	83	118.9s	1.43
@objectstack/driver-turso	39	53.5s	1.37
@objectstack/client	24	34.7s	1.45
@objectstack/example-showcase	21	21.6s	1.03

The proxy holds to roughly ±20% across the big packages and is off by ~2.8× on @objectstack/cli, which LPT places in bin 3 — the shard that was the long pole in every cold run sampled.

H3 (cache/runner variance) is real, large, and separate

Run this shard's tests, with the Turbo cache state of each leg:

run	shard	cached / total	time
32233301407	1	28 / 79	9m22s
	2	0 / 85	10m08s
	3	31 / 79	14m48s
32335265453	1	81 / 83	4m48s
	2	79 / 79	0m03s
32352993803	1	31 / 79	8m18s
	2	55 / 59	4m53s
	3	33 / 83	16m27s

The apparent "the bottleneck moves between runs" signature is a warm-vs-cold comparison, not a placement change: in run 32335265453 shards 1 and 2 were at 98–100% cache hits. In the two runs where shards ran cold, shard 3 was the long pole both times. Mechanism, reported and not touched here: this job's Turbo cache key is namespaced per shard — it interpolates matrix.shard into the key alongside github.job — and only main push runs write it, so each shard's cache ages independently.

Ranking: H2 > H3 ≫ H1. H2 explains which shard is the long pole; H3 explains why a single build's spread cannot be read as placement; H1 is falsified for the environment the symptom lives in.

2. What this PR changes (#10056)

turbo ls emits every packages.items[].path repo-relative; --union-into appended absolute ones, so one array could carry two conventions — and the obvious way to read such a document, join(REPO_ROOT, it.path), is correct for every entry turbo wrote and garbage for exactly the appended ones.

Both halves of the chain, since the document and its reader each owned a piece:

• the union now appends repo-relative paths, so the document carries one rule;
• partition-test-shards.mjs resolves it.path against the repo root explicitly instead of inheriting process.cwd(). path.resolve leaves an already-absolute entry alone, so a document written by the old union step still resolves where it always did.

The card left the choice open between three normalisations; this takes the third (consumer resolves explicitly) and the first (writer matches turbo), because the third alone leaves the document heterogeneous for the future consumer the card warned about, and the first alone leaves the cwd fragility it measured.

Verification

Placement is unchanged — the entry-gate consequence the card warned about does not fire. Against the full turbo ls payload this tree produces, all three shards keep byte-identical package lists and the bins stay 783/783/782.

Reverse-verified rather than asserted. With the resolution base ablated back to process.cwd() and the ablation confirmed on disk, run from /:

shard 1/3: 76/76 packages, weight 0 (all bins: 0/0/0)

Every package in bin 1, shards 2 and 3 empty, exit code 0 — the silent failure mode, worse at full scale than the single-package measurement on the card showed. Restored, from the same cwd: shard 1/3: 22/76 packages, weight 783 (all bins: 783/783/782). The new self-test case fails on the ablated build (path: resolution moved with the cwd) and passes on the restored one.

Self-tests extended on both sides of the cross-script invariant: the reader pins repo-relative resolution, absolute passthrough, and cwd-independence of the resulting weight; the writer pins the appended convention end-to-end through the real unionInto() on the fixture that first measured the divergence.

3. Gates

Re-derived from the actual changed files with node scripts/pm/dispatch-gates.mjs (no paths passed), which narrowed the dispatch's lead list — ci.yml is untouched, so the four workflow-shaped families it named are no longer implicated. All run at d80ca88:

gate	verdict line
node scripts/check-cross-package-test-inputs.mjs	OK: 12 package(s) read outside themselves, all declared, and turbo.json hashes every declared glob.
check-cross-package-test-inputs --self-test	All 60 self-test cases passed.
partition-test-shards --self-test	partition-test-shards: self-test OK
check-nul-bytes	check-nul-bytes: OK (scanned 6066 text file(s) ... no raw ASCII control bytes)
check-shard-attestation	✓ check-shard-attestation: 2 aggregate gate(s) count 3 declared leg(s) across 3 attesting job(s).
check-required-contexts	✓ check-required-contexts: 6 required context name(s) pinned across 2 workflow(s)

skip-changeset: root scripts/ only, nothing published changes.

4. The open question, for the maintainer

Sharding is by package, so a shard can never finish faster than its single heaviest package. Two packages already exceed #4859's Test Core 最慢分片 ≤ ~7min on their own: @objectstack/spec at 496s (8m16s) and @objectstack/cli at 548s (9m09s). Measured consequence — in run 32352993803, Test Core shard 1 took 8m17.77s wall and spec's own suite was 8m16.4s of it. The shard is that one package.

So the route this dispatch flagged as most likely — keep LPT, replace the weight input with measured per-package duration — would improve balance and still not reach the threshold. Its best case is a makespan of ~9m09s, the cost of @objectstack/cli alone, against a ≤7min target.

Three directions, none taken here:

1. Split those two suites below package granularity (file-level --shard within spec and cli only, package-level everywhere else). Reaches the threshold; the partitioner header records why blanket file-level sharding was rejected, and a two-package exception is a new mechanism with its own failure modes.
2. Duration-weighted LPT anyway, and move ci: merge queue 吞吐优化——Test Core / Dogfood 各 2→3 分片 + 队列失败自动分诊评论 #4859's threshold to what package-level sharding can actually deliver (~9–10 min). Cheap and honest, but it re-baselines the acceptance criterion rather than meeting it.
3. Reduce the cost of @objectstack/cli's suite — at 3.5–4.1 s/file against a ~1.4 s/file norm it is the outlier, not just the biggest.

Which of these is wanted is a call about the acceptance target itself, so it is left to the maintainer.

Generated by Claude Code