Test Core merge-queue legs restore a cache HIT and still replay only ~1/85 tasks, while main-push legs replay ~50/83 — #5401's queue tax is unpaid for Test Core, and none of the three known mechanisms explain it

[qq9340100]

Unclaimed finding, no pm:* / domain:* labels — left for triage to rate and route. Produced while verifying #10228's two stated mechanisms (both measured false, see the report comment there); this is the signature that survived the falsification.

The measurement

Same day (2026-08-20), same workflow, same shard, Test Core:

event	run / job	shard	Cached:	wall
merge_group (pr-10291)	job 96486448748	3	1 / 85	19m15s
merge_group (pr-10276)	job 96493089308	2	1 / 79	12m50s
push main (9a1ed7a0)	job 96502967114	3	55 / 83	10m44s
push main (9a1ed7a0)	job 96502967129	1	44 / 79	—

Merge-queue legs — the ones that gate every merge — replay ~1% of their task graph. Main-push legs on the same shards replay 55-66%.

Why the three known mechanisms do NOT explain it

All three were measured and ruled out (method and n in the #10228 report comment):

1. Not the writer being cancelled (Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 mechanism one). Save Turbo cache (main only) carries if: always(), and GitHub runs always() steps on a cancelled job. Over 151 Test Core legs on main-push runs, 138 saves concluded success; among the 88 legs whose job was cancelled, 82 (93%) still wrote the cache. Individually verifiable: job 96493855475 was cancelled at 5m29s and its Save Turbo cache (main only) step still concluded success.
2. Not shard-namespace aging (Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 mechanism two). Across 105 merge_group consumer legs, the freshest completed own-shard main-push save was p50 5.0 min / p90 9.1 min / max 15.6 min old. No shard is ever starved. (The cross-shard fallback key ${{ runner.os }}-turbo-${{ github.job }}- that Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 proposes adding is already present — ci.yml:226 for Test Core, ci.yml:905 for Dogfood.)
3. Not a restore MISS — this is what distinguishes it from merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401. In merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 the lint typecheck step literally missed (0 秒, MISS). Here every leg hits: over 59 sampled legs the Restore Turbo cache step ran p50 17s (push) / 18s (merge_group), min 3.0s, and 0 of 59 were ≤1s (MISS-shaped). A real multi-hundred-MB entry is downloaded every time.

So: the cache is written, is fresh, and is restored — and the merge-queue leg still cannot use it. Cache age explains none of the spread: Pearson r(own-shard save age, leg duration) = −0.05 (n=105); within-shard −0.18 / +0.07 / −0.24. A leg starting 0.0 min after a completed shard-3 save still took 20.1 min at 1/85 cached.

What that leaves

The restored entry's task hashes do not match the merge-queue tree, near-universally, while they match ~60% for a main-push tree restoring the same namespace. Candidate causes, none verified:

• the queue tree is main + one or more stacked PRs, so the batch's combined diff invalidates far more of the graph than a single landed commit does;
• something in the hash inputs varies with the ref/event rather than with content (turbo.json globalDependencies is only tsconfig.json + tsup.config.ts, so that is not it on its face);
• lockfile / ^build cascade: test is dependsOn: ["^build"], so one invalidated root package takes the whole closure with it.

Closing this needs a per-task hash diff (turbo run test --dry=json on a queue tree vs the main tree whose entry it restored), which is a different investigation from #10228's.

Why it is worth a card

• It is the actual cost behind the symptom Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 and merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测) #10149 §1 were chasing, and it is entirely unaddressed: merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 was closed by PR chore(ci): lint.yml 的 typecheck 补 build-core 的 turbo 缓存回退 (#5401) #5542, which extended fallback keys to lint.yml's typecheck only, explicitly not to the sharded jobs.
• It is paid on every merge-queue entry, i.e. on the critical path of every merge, at ~18 merges/day.
• ⛔ It is not fixable by adding fallback restore-keys. actions/cache/restore restores exactly one entry — the first restore-keys prefix that matches — and Test Core's ${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}- already matches on every run. A build-core key appended below it would be dead configuration, never reached. (That is why the chore(ci): lint.yml 的 typecheck 补 build-core 的 turbo 缓存回退 (#5401) #5542 precedent transfers to lint.yml but not here: typecheck's own namespace could be empty, so its build-core key could actually fire.)

Related

• merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401 — the closed parent; same queue-tax family, but its mechanism (restore MISS from a timing race) is measurably not this one
• PR chore(ci): lint.yml 的 typecheck 补 build-core 的 turbo 缓存回退 (#5401) #5542 — merge_group 条目在前一次 main 合并后 ~4 分钟内入队时永远吃不到 Turbo 缓存 —— 连续合并每条多付数分钟冷构建(实测) #5401's landed scope: lint.yml typecheck only
• Test Core 的 Turbo 缓存有两条独立的失效机制,#5401 只覆盖了其中零条 —— 写入方 43% 被取消,且缓存键按 shard 分命名空间 #10228 — where this was found; its two mechanisms are falsified in the report comment
• merge queue 关键路径回归:Test Core 最慢分片 16m28s、Dogfood 11m43s —— #4859 的两条验收判据均已不成立,且 #5401 方向 C 的触发条件已满足(实测) #10149 §1 — the "bottleneck changes shard every run" signature this partly explains
• File surface: .github/workflows/ci.yml (Turbo restore/save steps), turbo.json

---

Generated by Claude Code

[qq9340100]

Triage: lands in .github/workflows/ci.yml (Turbo restore/save steps) + turbo.json hash-input analysis ⇒ domain:devx · pm:queue · type Task. Rationale: measured, per-merge critical-path cost (~18 merges/day pay a near-total cache replay on every queue leg — the natural experiment d7283250 ran the same shard as 1/85 cached on merge_group and 55/83 on push); the falsification of the two prior mechanisms is recorded on #10228, so this card starts from a clean, closed causal chain. The stated next step is the right instrument: per-task hash diff via turbo run test --dry=json on a queue tree vs the main tree whose cache entry it restored — an investigation with a bounded deliverable (name the varying hash input), not an open-ended optimization. ⛔ Note for the implementing seat: fallback restore-keys are measured dead config here (the shard prefix always matches first) — do not re-walk the #5542 route. Serial note: ci.yml's Test Core regions are hot this hour (PR #10376 changes the shard matrix; its landing changes shard membership and cache namespaces) — claim after #10376 lands and re-derive the namespaces on the merged ref.

Size/model suggestion: M, mode:subagent, opus — measurement-first with one design-shaped conclusion.

---

Generated by Claude Code