Skip to content

Shard Test (coverage) 4 ways and stop the Codecov upload being skipped by its implicit success() #5403

Description

@os-support-ai

Blocked-by: #5402

Ruled by the maintainer on 2026-08-20 (#5393, verbatim 「接受你的所有建议,现在开始处理。」). Item 2 of 5. Both halves live in the same test-coverage job in ci.yml, so they are one card and one PR.

Half 1 — shard it

Measured on the same commit by #5395:

measured
unsharded (today)39 min 05 s
4-way sharded + blob merge12 min 04 s (3.24×)
extra runner minutes+9.2%
the blob-report merge ci.yml declines on cost grounds13 seconds

ci.yml's comment declines sharding because it would need a blob-report merge and "nothing blocks on this job". That trade was priced when the job was shorter; the merge costs 13 seconds and buys back 27 minutes.

Equivalence is the acceptance test, and it has a measured baseline: 1287 files both ways, 55432 vs 55395 of 84590 statements, onlyUnsharded=0, onlyMerged=0, delta −0.0437 pp — against a noise floor of 3 statements between two identical sharded runs. Reproduce that, do not assume it.

⚠️ The job also has timeout-minutes: 40, which has already fired 4 times ("The job has exceeded the maximum execution time of 40m0s"). At ~12 minutes the cap stops being load-bearing. ⛔ Do not raise it as part of this card — if sharding works the margin is no longer tight, and raising a cap is a separate decision.

Half 2 — the upload cannot stay silent

Upload coverage to Codecov has no if:, so it carries the implicit success() and is skipped whenever the test step fails — 311 of 373 coverage jobs, and 120 of the 121 most recent. "Codecov received nothing" and "Codecov received an unchanged report" are indistinguishable from outside, which is how four dark days went unnoticed.

Make the failure visible. ⛔ Do not make the upload unconditional in a way that publishes a partial or empty report as if it were complete — a wrong coverage number is worse than a missing one. The requirement is that a skipped upload is loud, not that it always uploads.

⚠️#5402 also found that on a failing suite the live unsharded recipe leaves no coverage/ directory at all (reproduced 2 of 2), while the blob-report path produced a complete 1287-file report every time. If sharding changes that, say so — it may resolve this half on its own, and that is a finding worth recording either way.

⛔ Constraints

Evidence and method: #5395. Ruling: #5393.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions