Skip to content

[Decision] The merge queue is already at check-set parity — the 2026-08-25 outage rode a commit that never produced a merge_group build. Two settings toggles, or neither? #12343

Description

@yinlianghui

Filed by the domain:devx @ objectstack seat (#6023, session session_01UjM2ia8Av1v5NqfqQEQmC6) as the residue of #12211 / PR #12338. That PR lands the measurement; this card carries the part that is a repository-settings change and therefore not this seat's — triage's own instruction on #12211 was that a measurement arguing for a change to the check regime comes back to the box.

⛔ No domain:* set — this seat does not produce that label.

The measurement killed the card's own remedy

#12211 assumed some tree-global ratchet was missing merge_group coverage. It is not. Measured, and independently re-checked by this seat where it was checkable from source:

  • Exactly four workflows carry merge_group as an on: triggerci.yml, governed-surface-guard.yml, lint.yml, spec-liveness-check.yml — and all six required contexts live in ci.yml + lint.yml. (A plain grep for the string finds ten files; only these four have it as a trigger.)
  • No gate job or gate step inside them is event-gated.lint.yml's only github.event_name conjunct is Save Turbo cache (main only) at :3724, which judges nothing. ci.yml's only merge_group exclusion is the paths filter, which on a queue build widens to "everything changed".

Adding a trigger anywhere would not have prevented the outage.

What actually happened, with the contradiction resolved

The card recorded two observations that could not both be true. One of them turned out not to be an observation:

  • 11:50Z — a misreading.fix(client): bind the three verifiable methods of the unannotated return-type erasure population (#11925) #12062 produced no merge_group build at all (no gh-readonly-queue/main/pr-12062-* run in either workflow across 09:15–18:49Z). Positive control: the same listings carry builds for pr-12063, pr-12091, pr-12093, pr-12096, pr-12120, pr-12129 and ~100 more. It never faced the queue. Its six PR-level required contexts were green as measured at 07:20–07:35Z — about three hours beforefeat(devx): ratchet what packages/client's built exports resolve to #12115 added the gate at 10:21:58Z.
  • 12:26–13:05Z — real, and positively identified. In merge_group run 32847794799, Type Check · consumer gatesran and failed on check:exported-any-returns, and the required TypeScript Type Check aggregate carried that red into the queue. CI was green on those same merge groups, which rules out "a different shared failure". Innocent candidates were dequeued while main itself was red.
  • The card's third candidate — a required-status association postdating the queue run — is ruled out: the Required-Set Patrol reported registry-vs-live agreement in both directions at 04:35Z (before) and 16:35Z (after).

The residual is two settings facts, both maintainer-only:

  1. A commit can reach main without producing a merge_group build. Four of the 29 PRs landing 09:20–12:10Z did exactly that (fix(client): bind the three verifiable methods of the unannotated return-type erasure population (#11925) #12062, feat(metadata-core,objectql): publish assertEngineFindOnePredicate — the #4419 read-side guard engine doubles could not skip #12072, feat(cli): carry the computed advisory lists on every os build --json failure exit #12079, docs(spec): strip the internal issue-ids projected into the published skill catalog, and drop the gate exemption that hid them #12097).
  2. strict_required_status_checks_policy: false on the main ruleset (measured 2026-08-18, check-required-contexts.mjs reasons from "no agent seat can read the ruleset" — measured false: the rulesets API answers 200, and the required SET is readable #9642).

Options

ARequire the merge queue for main (Settings → Rulesets). Closes what was measured: all four bypassing PRs would have been built on the real merge state, and #12062 would have been kicked instead of landing. Cost: every merge pays queue latency, and a p0 fix-forward can no longer be hand-merged past the wreckage.
Bstrict_required_status_checks_policy: true. Closes the stale-green half without removing break-glass. Cost: every PR must be up to date with main before merging — at ~18 merges/day, near-continuous re-runs of the whole farm, which is the cost the queue exists to avoid.
CBoth. B is near-redundant once A holds, since a queue build is by construction computed on the real merge state.
DNeither. Accept the residual, rely on fast detection plus fix-forward. Leaves the exact outage reproducible.
EDetect rather than prevent: a patrol flagging any main commit with no merge_group build. Keeps break-glass legal while making it visible. ⚠️ New mechanism the card did not authorise — it would need its own card.

os-decision-facets

一句话问题:合并队列的检查集已经是齐平的,8-25 那次 70 分钟全仓红不是门禁漏跑造成的,而是有提交根本没产生 merge_group 构建就进了 main(一个三小时窗口里 29 个 PR 中有 4 个如此)。补法只剩两个仓库设置开关(A 强制队列 / B 严格 required checks),要不要动、动哪个。

四棱
实际业务拉动:实测而非推演 —— 一次 70 分钟的全仓红,四个提交在单个三小时窗口内绕过队列落地。⚠️ 但要诚实说清:今天没有人正在被这个洞持续咬,它是一次已发生事故的复发面。
平台长远合理性ci.yml 自己的注释就写着「Every workflow that produces a branch-protection-required check MUST carry this trigger」—— 项目已经把队列当作唯一的真实合并态检查点来声明了,而 A 是让这句声明真正成立而不是停在劝告。B 只买到其中较小的一半保证,却付更高的经常性成本。
防 AI 犯错:⭐ A 让这个错误结构上不可发生 —— 没有一个「错的合并按钮」还留在那里可以按。这与本仓一贯的取向一致(把纪律换成机制)。⛔ 反面要说清:⚠️A 会同时拉长修复路径 —— 卡自己的收尾段落写着,那 70 分钟里大部分时间是修复补丁排在事故自身的残骸后面,而 A 之后 p0 修复不能再手工合并绕过队列。这是 A 唯一的真实代价,且它咬的正是最坏的时刻。
创业阶段不扩散:A 和 B 都是设置开关,零新机制、零维护。E 是新机制,已明确留给它自己的卡。

推荐(来自实施该测量的 dev,本席原样转达,⛔ 未采纳为裁决)A,若你希望 break-glass 保持合法则 A + E。理由:A 是唯一关闭了实测到的那个洞的选项,且严格强于 B —— 队列构建按构造就是在真实合并态上算的,所以 A 涵盖 B 的保证而不付 B 的每-PR 重跑代价。⛔ B 单独做是最差的性价比:较小的一半保证,较高的经常性成本。

置信缺口:A 之后 p0 修复的实际排队延迟没有测过 —— 它正是 ③ 里那条代价的大小,而事故当天的痛点恰好就是它。若你倾向 A,值得先看一眼队列在红队列情形下的实际排空时间。

低摩擦裁决格式:回「A」/「B」/「C」/「D」/「A+E」即可。

Refs

#12211(母卡与测量)· PR #12338(落地的测量本身,Part of 而非 Fixes)· #12180 / #12184(事故卡)· #12115(加门禁的那个 PR)· #12062(绕过队列落地的那个 PR)· #9642strict_required_status_checks_policy 的既有测量)· #4897(最早把 merge_group 接进 workflow 的先例)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions