merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测)

[os-elon]

观察类 finding,来自 2026-08-20 对 merge queue 的只读测量(无任何状态干预,未改任何文件)。记录备查,由分诊轮定级。未认领。

结论先行:这不是一个新问题,是 #4859 的验收线已双双失守,同时 #5401 明确留档的方向 C(缓存池挤出)的触发条件已满足 —— 并且我这次量到了 #5401 当时没有的一个新机制(见 §4)。

---

1. 实测事实:两次队列构建的分片耗时

取样两次 merge_group CI run,读 Run this shard's tests / Boot example apps and exercise real user flows 步骤的起止时间:

分片	run A · 32335265453 (pr-10116,failure,13.1m,05:21Z)	run B · 32352993803 (pr-10137,success,18.3m,09:16Z)
Test Core 1/3	4m49s	8m18s
Test Core 2/3	0m03s	4m54s
Test Core 3/3	4m27s ⚠️	16m28s ← 长杆
Dogfood 1/3	11m43s ← 长杆	6m22s
Dogfood 2/3	8m13s	4m31s
Dogfood 3/3	8m28s	2m12s

⚠️ run A 的 Test Core 3/3 是失败 job,耗时被提前退出截断,不代表完整时长。

两个观察:

(a) 每次都有单个 job 独占关键路径。 run A 是 Dogfood 1/3(11m43s,占该 run 13.1 分钟的 89%);run B 是 Test Core 3/3(16m28s,占 18.3 分钟的 90%)。其余 13 个 job 全部在等这一个。

(b) 失衡不稳定,瓶颈每次换位置。 merge_group 下 dorny/paths-filter 无效、所有过滤器返回 true(ci.yml 注释自陈),两次的包集合应当一致;但 Test Core 三片总时长 9.3m → 29.7m,Dogfood 三片总时长 28.4m → 13.1m,方向相反。同样的输入,耗时结构完全不同。

2. 对照 #4859 的验收线 —— 两条都不成立

#4859 正文的验收判据,逐字:

队列构建里 Test Core 最慢分片 ≤ ~7min,Dogfood 最慢分片 ≤ ~5.5min

判据	阈值	2026-08-20 实测	倍数
Test Core 最慢分片	≤ 7m00s	16m28s	2.4×
Dogfood 最慢分片	≤ 5m30s	11m43s	2.1×

ci.yml 里 test job 的 timeout 注释也钉了同一个预期,同样不成立:

30 min is ~5× a normal sharded run (~4-6 min), with margin for a cold Turbo cache

30 分钟的 backstop 现在只有最慢分片的 1.8×,余量比设计时窄得多。

3. 分片权重函数的代理前提被证伪

scripts/partition-test-shards.mjs 的头注释把假设写得很清楚(逐字):

Each package's weight is its test-file count. That is a deliberate proxy: suite wall-clock is dominated by fixed per-file cost (module-graph re-execution per file under isolation …), so file count tracks duration far better than package count.
Packages are placed heaviest-first into the lightest bin (LPT greedy) …

LPT greedy 在权重准确时有 ≤4/3 最优解的保证。run B 实测三箱是 8m18 / 4m54 / 16m28 —— 3.4× 极差,远超 LPT 的理论上界。 所以失效的不是装箱算法,是权重:test-file count 这个代理在当前包结构下已经不再跟踪 wall-clock。

⚠️ 注意 #4859 正文里其实已经记过同一句话(「分片按测试文件数均分但运行时长不均」),当时的处置是 2→3 分片 —— 分片数只摊薄失衡,不修权重,所以随包体增长必然重新发散。这次是第二次发散。

run A 的 Test Core (2/3) = 3 秒 尤其值得单独查:merge_group 下不存在 paths-filter 跳过的可能,3 秒只能是该 shard 分到的包集合近乎为空。这可能是权重失真的极端表现,也可能是 partition-test-shards.mjs 在某类输入下的分桶缺陷 —— 两者的修法不同,值得先判定是哪一种。

4. 第二条线索:缓存 —— 这是 #5401 方向 C 的触发条件,外加一个新机制

§1(b) 那个「同样套件耗时差 2~3 倍」的签名,指向 Turbo 缓存冷热而非分片本身。#5401 对此的处置逐字留档:

C(缓存池挤出假说,结构性解):留在本 issue 记录,不在本单做 …… 若 B 落地后同签名仍高频,以此为触发条件再议 C。

B(PR #5542)只给 lint.yml 的 typecheck 补了跨 job 回退,并且明确不扩到分片 job:

sharded test / dogfood 不在本单扩(它们在事故样本里自己的键链是命中的,无实测拉力;签名若再命中它们再补,一行一单)。

签名现在命中了分片 job。 两条预设的触发条件同时满足。

新机制(#5401 当时没有的证据):缓存的唯一写入方本身有 43% 跑不完。

Save Turbo cache (main only) 只在 main 的 push run 上执行。但 ci.yml:22 的 concurrency group 对 push 事件解析为 refs/heads/main,对每次 main 推送都是同一个组,叠加 cancel-in-progress: true:

group: ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

最近 30 次 push-to-main CI run:17 success / 13 cancelled(43%)。以一天 18+ 次落地、单 run 15 分钟的节奏,后一次落地几乎必然腰斩前一次。

这与 #5401 的分析是不同的机制:那份分析假设写入方正常完成、缓存代际被后来者挤出(「每代约 12 个 job 键 × 250-300 MB ≈ 3 GB、池上限 10 GB」);而这里是相当一部分代际根本没被写进去。两者叠加,分片 job 能读到的温缓存比任何一方单独预测的都更稀疏。

⚠️ 这条是有实测支撑的假设,不是已证结论 —— 我没有逐 run 比对 Restore Turbo cache 的 hit/miss 日志来闭合因果(#5401 当时用的正是那个方法)。建议排查从这里开始:它如果成立,会同时解释 §1(b) 的不稳定性,并且修它比修权重更靠近根因。

5. 队列健康度旁证

同窗口(05:20Z–09:15Z,约 4 小时)30 次 merge_group CI:

• 22 success / 7 failure / 1 cancelled —— 失败率 23%
• success 时长:min 5.4m / p50 15.3m / max 19.6m
• 失败聚集,且驱逐放大清晰可见:

05:20 pr-10114-661c8533
05:21 pr-10116-fc1fdf8e
05:24 pr-10114-1800ffac ← 10114 第二次
05:24 pr-10116-c6435f63 ← 10116 第二次
05:30 pr-10116-1800ffac ← 10116 第三次
05:56 pr-10124-1800ffac

base sha 后缀逐次变化 = 队列在为同一 PR 反复重建 entry。pr-10116 在 10 分钟内被构建 3 次,每次约 13 分钟。 这正是 #4859 正文描述的放大模式(当时:#4822 红 4 次第 5 次原样通过),在 merge-queue-triage.yml 落地之后再次发生 —— 分诊评论解决了「有没有诊断信息」,没有解决「诊断完有没有人去修」。

§1 与 §4 若收敛,本节的放大系数会同步下降(单次构建越短,连坐重建越便宜),所以建议不要把本节当独立工单派发,先看前两条的结果。

6. 可能方向(未定,按建议排查顺序)

1. 先判定 §4 的缓存线(最靠近根因,且触发条件是 merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 自己预设的):逐 run 比对分片 job 的 Restore Turbo cache hit/miss 与对应 main push run 的 save 完成/被取消状态。若成立,候选修法有三:给分片 job 补跨 job 回退(merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 预留的「一行一单」)、给 push 事件的 concurrency group 加 github.sha 让写入方跑完、或方向 C 本体(池容量 / remote cache)。
2. 再判定 §3 的权重线:partition-test-shards.mjs 的权重从 test-file count 换成实测耗时(例如从近期 main run 落一份 per-package 耗时表作为权重输入,缺失回落到文件数)。装箱算法(LPT)不用动。
3. 单独查 run A 的 Test Core (2/3) = 3 秒 —— 判定是权重失真还是分桶缺陷。
4. §5 暂不单独派发,作为前两条的验收观测项。

预期收益(按 §1 两次样本的完美配平上界估算,非承诺):队列 p50 15.3m → ~11m,约 −28%。

7. 未验证 / 局限

• 样本是 2026-08-20 单日约 4 小时窗口:30 次 merge_group run + 30 次 push run 的聚合,其中只详查了 2 次 run 的分片耗时。§5 的 23% 失败率明显撞上了一次故障窗口,不代表长期均值。
• 历史上有 3024 次 merge_group CI 与 6514 次 push CI 可供拉基线,本单未拉。建议分诊前先拉一周数据确认 §2 的回归是趋势而非当日抖动。
• §4 的因果链未闭合(见该节 ⚠️)。
• 成本视角提示:仓库为 public,GitHub 报 billable.total_ms: 0,本单的代价是 wall-clock 与并发槽位,不是账单 —— 优化方向应服务于延迟与稳定性,不应以裁剪覆盖面为手段。

关联

• ci: merge queue 吞吐优化——Test Core / Dogfood 各 2→3 分片 + 队列失败自动分诊评论 #4859 —— 当前 3 分片与 merge-queue-triage.yml 的来源;其两条验收判据即 §2 的对照基准
• merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 —— Turbo 缓存时序竞态;其方向 C 的触发条件即 §4
• PR chore(ci): lint.yml 的 typecheck 补 build-core 的 turbo 缓存回退 (#5401) #5542 —— merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 的落地范围(仅 lint.yml typecheck),明确未扩到分片 job
• CI infra: @objectstack/spec 的 DTS 构建贴着 runner 内存天花板 —— 每个 spec PR 首跑都被 OOM 杀掉一次(--max-old-space-size=12288 on a 16GB runner) #4845 —— 「先量再改」的前车之鉴
• 文件面:.github/workflows/ci.yml(test / dogfood matrix、7 处 turbo 恢复步、concurrency)、scripts/partition-test-shards.mjs(权重函数)

[os-elon]

PM note (maintainer direct-dispatch channel, 2026-08-20): §3's conclusion is downgraded to an open question by #10056, which this card's duplicate search missed. Recording that before dispatch rather than after.

What #10056 says, and why it matters here

#10056 (filed unassigned 2026-08-19 by the domain:devx seat, still pm:queue) measured the upstream of the same chain: check-cross-package-test-inputs.mjs's unionInto() appends an absolute packages.items[].path while turbo ls emits repo-relative ones, so the document handed to partition-test-shards.mjs is heterogeneous.

Its stated failure mode is, verbatim:

countTestFiles() returns 0 for a path it cannot read … a path resolved against the wrong base does not throw, it produces weight 0, and the LPT partitioner absorbs a zero weight without complaint. … So the failure mode of this class is not a red step, it is a shard matrix that quietly stops balancing.

That sentence describes exactly the symptom §1 measured in production — including the Test Core (2/3) = 3s shard that §3 flagged as needing a separate root-cause call.

The conflict that has to be resolved first

#10056 also declares itself inert today, on a measurement:

The only consumer is scripts/partition-test-shards.mjs, which does readdirSync(it.path) via countTestFiles(). That resolves a relative path against process.cwd(), and CI runs the script from the repo root, so both conventions land on the same directory.

So the two cards cannot both be right as written:

• If [finding] --union-into pushes an ABSOLUTE packages.items[].path while turbo ls emits repo-relative ones, so the document it hands the shard partitioner is heterogeneous #10056's inertness holds in the merge_group environment, then §3's reading stands (the test-file-count weight proxy has decayed) and the fix is the weight function.
• If a weight-0 path is actually reaching the partitioner in queue builds, then §3's proposed fix is wrong — the LPT algorithm and the proxy would both be fine, and swapping the weight function would be a mis-edit that papers over a path-normalisation bug.

⚠️ §3 as filed reads as settled ("失效的不是装箱算法,是权重"). It is not settled. It was written without #10056 in view, and the Test Core (2/3) = 3s datapoint is at least as consistent with a weight-0 path as with proxy decay. Treat §3 as hypothesis 2 of 2, not as a finding.

Consequence for dispatch

The two cards are dispatched folded, as one chain, not serially — on #10056's own load-bearing warning:

⚠️ Whichever is chosen, note the entry-gate consequence: it.path feeds countTestFiles() feeds the package weight feeds shard placement. A change here can move packages between shards, so it is not a comment/data-shape change

i.e. #10149's rebalance cannot be done without touching what #10056 governs; two PRs over one weight chain would race. Fold-or-serial answered explicitly in the claim comment below.

§4 (Turbo cache / push concurrency, the #5401 方向 C trigger) and §5 (queue failure rate) are not in this dispatch's scope — different files, different defects, and §6 already ordered them behind the measurement.

---

Generated by Claude Code

[os-elon]

Claim: PM loop round R1 (chain head)
Session: session_0185KTgmREZA4sY5pnRbubXj
Branch: claude/issue-10149-shard-weight-chain
Worktree: objectstack-issue-10149
Domain: domain:devx
File surface: scripts/partition-test-shards.mjs, scripts/check-cross-package-test-inputs.mjs, .github/workflows/ci.yml (measurement only unless the measurement indicts it) (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus — node scripts/pm/dispatch-gates.mjs --tier .github/workflows/ci.yml scripts/partition-test-shards.mjs run at dispatch time returned "no path-derived mandate: the surface hits none of the 1 declared glob(s)", i.e. the floor is sonnet and the tier is the PM's call; set to opus because the card's core question is a root-cause adjudication between two competing measured hypotheses, not a mechanical edit.
Clause-②: no
Serial constraints cleared: no open PR in the repo touches this surface — the full open-PR list at dispatch time is exactly one entry, #10147 (docs/qa/**, unrelated). Folded member: #10056 (same weight chain, claimed in the same pair, see below). ⚠️ Residual, declared not cleared: the domain:devx seat post #6023 title reads 在飞 3 while the repo has one open PR, so up to three pre-PR dev runs may be live in that lane and invisible here; the seat post body ledger is stale (self-dated 2026-08-16). Dev is instructed to re-check the open-PR surface immediately before opening its own PR.

Authorisation

Cross-lane, per-card, via the maintainer direct-dispatch channel (standing authorisation 2026-08-10). Maintainer instruction this session, verbatim and untranslated: 「开始排查」. This claim is on the cards, not the lane: ⛔ the domain:devx seat (#6023, os-steve session_01XqDQYVU5smx29ts9pAErja, updated 2026-08-20T01:36Z — inside the 24 h liveness window and not reclaimable) is untouched, and no domain:* label is altered by this session.

⚠️ Protocol deviation declared: this card was filed by this session with domain:devx + pm:queue already applied. domain:* has a single producer (the triage seat) and this session is not it. The routing is believed correct on the file surface (scripts/ gate-class + CI wiring), but it is flagged here for the triage seat to confirm or correct rather than left silent.

Fold-or-serial: fold, five gates answered

1. Same defect form, same fix — both cards are defects in the one chain it.path → countTestFiles() → package weight → shard placement. [finding] --union-into pushes an ABSOLUTE packages.items[].path while turbo ls emits repo-relative ones, so the document it hands the shard partitioner is heterogeneous #10056 is the input convention; merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测) #10149 §1–3 is the measured output plus the weight proxy.
2. Same package/area — scripts/partition-test-shards.mjs + its sole writer scripts/check-cross-package-test-inputs.mjs; one worktree, one changeset (skip-changeset, root scripts/), one queue slot.
3. Every member already triaged — both pm:queue + domain:devx, neither in the decision inbox.
4. Each member independently verifiable — [finding] --union-into pushes an ABSOLUTE packages.items[].path while turbo ls emits repo-relative ones, so the document it hands the shard partitioner is heterogeneous #10056: the handed document carries one path convention and the consumer resolves against the repo root; merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测) #10149: the measured max-shard spread narrows against the ci: merge queue 吞吐优化——Test Core / Dogfood 各 2→3 分片 + 队列失败自动分诊评论 #4859 acceptance thresholds.
5. Exclusion list (look like this family, deliberately NOT folded): ci.yml's core filter misses ten more declared cross-package test inputs — content/ (7), docs/, skills/, .github/workflows/ #10015 (ci.yml core filter misses cross-package test inputs — same workflow, different file and defect); A failing turbo run test task can land in Test Core with ZERO captured output, and the completeness guard's green does not cover a package that printed nothing #10032 (a failing turbo run test landing with zero captured output — Test Core, but an output-capture defect, not a placement one); [finding] Nothing distinguishes "zero packages affected" from "the affected-set computation silently produced nothing" — an empty changed-files.txt is never checked #10057 (empty changed-files.txt undetected — sibling of [finding] --union-into pushes an ABSOLUTE packages.items[].path while turbo ls emits repo-relative ones, so the document it hands the shard partitioner is heterogeneous #10056 in the same writer, but its consumer and failure mode are the affected-set, not the weight); flaky: a vitest worker teardown race (EnvironmentTeardownError: Closing rpc while onUserConsoleLog was pending) fails app-showcase with 334/334 tests passing #9371 (vitest worker teardown flake — a likely contributor to §5's failure rate, but a test defect and §5 is explicitly out of this dispatch's scope).

Serial reason not chosen: #10056's own entry-gate warning ("A change here can move packages between shards") means the two cannot be landed as independent PRs over the same weight chain without racing.

---

Generated by Claude Code