Skip to content

[finding] Merge-queue check-set parity for tree-global ratchets — measure which lint.yml jobs actually run on merge_group, then close the gap the stale-ledger outage rode through #12211

Description

@os-litant

Blocked-by: #12343

Filed by the spec PM seat (session_01NDGG54XF5gbTLdQzCtnaVV) after the 2026-08-25 stale-ledger outage (~70 min of fleet-wide red; incident cards: the p0 card, its two sibling filings, fix PRs #12186/#12192). Dedup: the incident cards asked triage to consider "the class fix" but no dedicated card exists; #4897 (closed) is the historical precedent that first wired merge_group into workflows.

⚠️ STATUS (PM seat #6023, 2026-08-25).Step 1 is done and it is a NEGATIVE result: step 2's premise is false. PR #12338 (9fffd327c7, Part of — deliberately not Fixes) landed the measurement. No tree-global ratchet is missing merge_group coverage, and adding a trigger anywhere would not have prevented the outage. Exactly four workflows carry merge_group as an on: trigger — ci.yml, governed-surface-guard.yml, lint.yml, spec-liveness-check.yml — all six required contexts live in ci.yml + lint.yml, and no gate job or step inside them is event-gated. The two contradictory observations are resolved below. The residual is two repository SETTINGS facts, both maintainer-only, now in the decision box as #12343. This card is blocked on that ruling, not on any further measurement.

The mechanism this card closes

The gate that went red (check:exported-any-returns) is a tree-global ratchet: it judges the whole tree's consistency (ledger vs built dist), not the diff. For that gate class, PR-level green is a snapshot — a PR branched before a new gate lands (or before main's state changes) carries stale green. The only place that sees the REAL merge state is the merge queue. The outage happened because the closing PR (#12062, branched before the gate-adding #12115) passed the queue with an inconsistent tree and landed it on main.

Contradictory observations — which is why step 1 is a measurement, not a fix

Both cannot be true of a static check set. Possibilities to distinguish: the job runs on merge_group but #12062's queue run predates a required-status association; the kicks came from a different shared failure; queue batching semantics. ⛔ Do not wire anything until this is measured.

⚠️ RESOLVED by PR #12338 — and one of the two was not an observation.

Steps

  1. Measure: enumerate which jobs/checks actually execute on merge_group events today (workflow triggers + a real queue run's check list), and diff that against the set of tree-global ratchets that run at PR level (consumer gates, spec-liveness family, the shrink-only ledgers). Also pull the queue's own logs for the 12:26–13:05 mass-dequeue to name what failed. — ✅ DONE, PR ci(lint): record the merge-queue check-set measurement — the queue already runs every tree-global ratchet #12338.
  2. Close the gap: any tree-global ratchet missing from the queue's set gets merge_group coverage (the merge queue 已启用但零 workflow 声明 merge_group —— 队列里一个检查都不跑,进队列的 PR 只会空转并锁死自己的分支 #4897 pattern). After that, a fix(client): bind the three verifiable methods of the unannotated return-type erasure population (#11925) #12062-shaped race is KICKED in the queue instead of landing red on main. — ⛔ PREMISE FALSE: none is missing. The gap is settings; see [Decision] The merge queue is already at check-set parity — the 2026-08-25 outage rode a commit that never produced a merge_group build. Two settings toggles, or neither? #12343.
  3. Explicitly out of scope: making ratchets diff-scoped (blinds them — their whole point is global invariants), and auto-re-running checks on all open PRs when a gate lands (expensive; queue parity covers the same hole at one chokepoint).

Why this is the structural layer

Detection was fast (11 min) and the repair was one line; most of the outage was the defect being ABLE to land, plus the fix queuing behind the outage's own wreckage. Queue parity removes the first; the second is a process convention (p0 fix-forward queue-jump / maintainer break-glass merge) that belongs to the PM/skills text, not to CI — noted for the maintainer separately.

⭐ That closing paragraph is now load-bearing on the decision: option A in #12343 (require the queue for main) removes the first half and lengthens the second, because a p0 fix-forward could no longer be hand-merged past the wreckage.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions