ci: shard the coverage lane 4 ways and make a skipped Codecov upload loud

[os-support-ai]

Fixes#5403

Both halves live in the same test-coverage job, so this is one PR: shard the coverage lane 4 ways with a blob-report merge, and stop the Codecov upload from disappearing in silence.

Read this first — the card's numbers were stale, and the re-measurement moved the conclusion in the other direction

#5403 was written while the coverage job was failing 100% of the time, and its unblock comment predicted that #5402 would have made the unsharded arm faster than the 39 min 05 s measured before it. Re-measured on aa949ba2 (current main, the first commit carrying #5402's fix), both arms on the same commit and the same runner class, elapsed from each job's own started_at/completed_at:

arm	round 1 (run 32387182880)	round 2 (run 32387778786)
slowest shard job	11 min 25 s	10 min 52 s
the merge job	~1 min 15 s (13 s of it the merge itself)	—
sharded lane, wall clock	~12 min 40 s	~12 min 07 s
unsharded, one runner	39 min 51 s tests / 40 min 19 s job	39 min 29 s tests / 40 min 00 s job *

* the second unsharded sample is from run 32388915404, not from round 2 — round 2's own unsharded leg was still running when this was written, and rounds 1 and 4 are two independent samples of the same arm on the same tree.

The gap did not narrow. 39 min 51 s and 39 min 29 s are the old 39 min 05 s back within run-to-run noise — #5402 removed ~40 s from a ~2,400 s run, which is what its own numbers said it would. And measured at the level the cap acts on, the job took 40 min 19 s, so timeout-minutes: 40 would have killed it roughly twenty seconds before the suite finished — and the second sample landed on 40 min 00 s exactly: the fifth firing of a timeout that has already fired four times, and a job killed at 40 minutes uploads nothing. Unsharded, this lane no longer reliably reaches its own upload step. Sharded it is a third of that, with 28 minutes of headroom under the same cap.

⚠️ The same run also demonstrates the thing that makes the number hard to see: its coverage artifact upload failed with if-no-files-found: error, because the suite is red on this commit and vitest had already deleted coverage/. See the findings at the bottom.

Two more measurements point the same way, and neither is in the card:

• The push lane's own coverage job on this very commit was cancelled at 22 min 20 s, by the next merge (cancel-in-progress keyed on github.ref) — run 32385114700, job Test (coverage), Upload coverage to Codecov conclusion skipped. A third silent path to "Codecov got nothing".
• Over the last 30 consecutive push/main intervals (2026-08-19T15:31Z .. 2026-08-20T15:38Z) the median gap between merges is 25.2 minutes. A ~13-minute lane survives to completion in 20 of 30 of those windows; a ~39-minute one in 12 of 30.

Half 1 — four shards, and exactly one place a report is produced

test-coverage becomes a 4-way matrix that writes .vitest-reports/blob-N-4.json (the shard index is in the file name, so four artifacts merge into one directory without colliding — measured, not assumed). coverage-report downloads all four, refuses to continue unless all four arrived, merges them with --merge-reports, and uploads that one complete report.

The shard legs neutralise coverage.thresholds; the merge does not. Without that, sharding would evaluate whole-suite thresholds five times per push — once per shard over a quarter of the suite, plus once on the merged report. Those four extra evaluations are assertions this lane has never made and can only produce a false red. With it, the threshold gate runs exactly once, over the complete report, as it did unsharded.

⚠️ That override is not here because shards were seen to fail thresholds. That was the prediction, and CI refuted it: a shard leg run with the thresholds left alone passes (PROTO shard 1/4 WITHOUT the threshold override, conclusion=success, in both rounds). The v8 provider reports only files a run actually loaded, so one shard's percentage lands near the whole suite's. The override stays for the reason above, restated honestly, rather than the one that motivated writing it.

Equivalence — measured, and it holds inside the noise floor

Both arms on the same commit, compared file-by-file (coverage-final.json vs coverage-final.json), plus a second sharded round so the delta can be read against run-to-run noise rather than against zero:

	files	onlyUnsharded	onlyMerged	statements	delta	files with fewer covered statements
merged vs unsharded (same commit)	1288 / 1288	0	0	55516 vs 55521 of 84749	+5 statements, +0.0059 pp	1
merged vs merged (two identical sharded rounds)	1288 / 1288	0	0	55521 vs 55518 of 84749	-3 statements, -0.0035 pp	2

The acceptance criteria #5403 names are met: same file count both ways, onlyUnsharded=0, onlyMerged=0, and a delta of the same order as the noise floor — the card measured 3 statements between two identical sharded runs, and this measurement reproduces exactly 3. The merged report is not identical to the unsharded one; it is equivalent to within the variation two identical sharded runs already show, which is the strongest statement this comparison can support. Note the sign: the merged report has more covered statements than the unsharded one, not fewer.

The card's own figures were 1287 files and -0.0437 pp on an earlier commit; the file count moved by one because the tree moved (this is aa949ba2, not the commit #5395 measured).

⚠️ Both arms here are red runs — the suite fails on this commit (#5421) — so the instrument's unsharded leg needed --coverage.reportOnFailure=true to keep its report from being deleted before it could be captured. Both arms therefore describe the same suite in the same state, which is what makes them comparable; it is not the shipped recipe, and the shipped recipe is unchanged.

Evidence: runs 32388915404 (both arms) and 32391890554 (second sharded round); the comparison is run 32393342485, whose two artifact names carry the numbers above — the Actions log CDN, artifact bodies and a workflow-token branch push are all unreachable from the measuring session, and artifact metadata is not.

Half 2 — a skipped upload is now loud, ablated in both directions on real CI

The upload runs only when all four shards are green and all four blobs arrived. A final step then runs on every path and states the outcome — ::error annotation plus a job-summary block — and fails the job when Codecov did not receive a report.

That last part is the load-bearing one, and it was ablated rather than argued. Two jobs, identical except that one has the verdict step deleted, both driven by a genuinely failed needs:

job	download / merge / upload	verdict step	job conclusion
with the verdict step	all skipped	failure	failure — says Codecov got nothing
verdict step deleted	all skipped	—	success

The second row is the defect in one line: a job whose every step was skipped is a green check sitting next to a Codecov that received nothing. Reproduced in runs 32387182880, 32387778786 and 32388816770.

fail_ci_if_error flips false to true for the same reason: with false, an upload that errors reports success, and the verdict step would then truthfully report a lie. The cost is that a Codecov outage paints this lane red — nothing blocks on it, so the cost is attention, which is the point.

The reportOnFailure decision routed here by #5402: A, leave the default false — vitest.config.mts is untouched

Sharding resolves this half on its own, which the card suspected and #5402's comment explained. The artefact that survives a red run is now the blob, not coverage/: each shard uploads .vitest-reports/blob-N-4.json with if: always(), and a blob carries both the failing results and that runner's raw coverage. So a red coverage run is diagnosable from artifacts today — strictly more so than reportOnFailure: true would make it, since a blob can be merged locally into the full report.

Setting true would add a partial coverage/ directory on a red runner that nothing collects, and one more thing that could be uploaded by mistake. The two are one decision, as the card says: with the upload gated on all-four-green, reportOnFailure: true would upload nothing on red anyway.

⚠️ One honest cost: the throwaway measuring instrument did need --coverage.reportOnFailure=true on its unsharded leg, because on a red suite the unsharded arm deletes coverage/ before it can be captured. That is a measurement need, met with a CLI flag, not a reason to change the committed default.

Reverse-verification: predicted, then observed

leg	predicted	observed
shard leg without the threshold override	red on thresholds	green — refuted; rationale rewritten to match
red-arm merge with the verdict step	red, and says why	red, verdict step failure
red-arm merge with the verdict step deleted	green (the bug)	green, every step skipped
local gates after the ci.yml edit	dependabot-merge-gate red on 4 unclassified names	red exactly there, plus two more the prediction missed (see below)
merged report still judged by thresholds	red under --coverage.thresholds.statements=99	red — but not conclusive: the merged blobs carry a failing shard, so both legs go red for that reason. Claim now rests on the code path, and says so.
merged vs unsharded coverage	equal file sets, delta inside the noise floor	confirmed: 1288/1288, 0/0, +0.0059 pp against a -0.0035 pp noise floor

Files outside the card's declared surface — both gate-forced, declared on the issue before this PR

The card scoped this to ci.yml (plus vitest.config.mts if needed). Renaming and adding a job in ci.yml turns three repo gates red on its own:

• scripts/dependabot-merge-gate.mjs — these checks run on every pull request but the gate does not classify them: Test (coverage shard 1/4) ... 4/4. The four new names are classified in NOT_A_GATE (push-only, skipped on a PR by design). REQUIRED_CONTEXTS is untouched — nothing is added to or removed from the gating set.
• content/docs/guide/ci-cd-pipeline.md — its ci.yml job table is pinned to ci.yml's jobs, in both directions, and its "Appears as" column to each job's name:. Updated, together with the two other places on that page that name this job.

A fourth failure came from the same run and is fixed inside ci.yml: the first draft counted blobs with a stderr-suppressing ls, which merge-queue-reporting.test.ts rejects repo-wide as the fail-CLOSED spelling #3723 removed.

Local gates after the amendment: scripts/ — 58 files, 1553 tests, all passing; check:control-bytes OK.

Two findings this run turned up, filed separately, not fixed here

• The suite is still red under v8 coverage on main after #5402 — shard 3/4 fails deterministically and the unsharded run leaves no coverage/ at all #5421 — the suite is still red under v8 coverage on main after ci.ymlTest (coverage) has failed 100% of the time since 2026-08-16 — one test times out under v8 coverage, and Codecov has received nothing since #5402. Shard 3/4 fails deterministically (three rounds, same commit), and the unsharded arm of the same commit produced no coverage/ directory at all, which is ci.ymlTest (coverage) has failed 100% of the time since 2026-08-16 — one test times out under v8 coverage, and Codecov has received nothing since #5402's own deletion mechanism. So the red is the suite, not the sharding: both arms fail on aa949ba2. ⛔ The failing test is not identified there — the Actions log CDN is unreachable from this session — and the job links are in the issue for anyone who can open one.
• ci.yml's concurrency cancels the push-lane coverage job mid-run, so Codecov gets nothing on a merge that is followed by another merge #5422 — ci.yml's concurrency cancels the push-lane coverage job mid-run. On this very commit it was cancelled at 22 min 20 s and the upload was skipped. With a 25.2-minute median merge interval, a ~40-minute job survives 12 of 30 windows and a ~13-minute one survives 20 of 30 — so this PR shrinks the exposure without removing the mechanism, and the fix is a decision (own concurrency group / no cancel / accept) rather than a patch.

Deliberately unchanged

• timeout-minutes: 40. Raising or lowering a cap is a separate decision.
• No test deleted, skipped, quarantined or given a raised timeout.
• The coverage thresholds: same one evaluation per push, over the complete report.
• changeset-release.yml — that is changeset-release.yml: drop the duplicate pnpm test, and stop concurrency discarding release runs (95 of 200 never executed) #5404's.

The measuring instrument lives on claude/issue-5403-proto and is not part of this PR.

---

Generated by Claude Code