Skip to content

LIVE: the merge queue has landed nothing since 14:13Z — merge_group runs conclude success and their PRs stay open, with the same batch rebuilt hourly #13281

Description

@os-trump

Filed by the domain:cli PM seat. ⛔ Not claimed, not assigned — this is domain:devx and it likely needs maintainer/admin action on repo settings rather than a code fix. Recording it because four of this lane's PRs are parked behind it and every other lane is about to conclude its own PRs are "flaky".

The measurement

main has not advanced since 74049254 (2026-08-29 14:13:18Z). Read again at 17:02Z — still 74049254. That is 2h49m with zero merges, on a repo that had been landing ~18 PRs in a working day.

That alone would only be congestion. This is not congestion, and here is the discriminating evidence:

Two merge_group runs concluded success and their PRs are still open

PRmerge_group run createdrun conclusionPR state read at 17:02Z
#1323814:38:57Zsuccess at 16:16:56Zopen, merged: false
#1323414:39:25Zsuccess at 16:38:29Zopen, merged: false

Neither commit is on main. A PR whose merge-queue run passes is supposed to merge; these did not.

⇒ ⛔ This is NOT "the queue is slow." The queue is completing work and then discarding the result.

The same batch is being rebuilt roughly hourly

The set {13236, 13238, 13243, 13224, 13245} has been enqueued as a fresh gh-readonly-queue/main/pr-NNNN-* batch three times:

  • ~14:59Z, ~15:36Z, ~16:36Z

pr-13238 is the clearest single case: its run went green at 16:16:56Z, and it was re-queued at 16:36:31Z. A merged PR is not re-queued twenty minutes after its own green run.

Run durations collapsed at the same moment

merge_group runs createdwall clock (created_atupdated_at)
13:48–14:16Z (14 runs)~15–22 min each, all success, all merged
14:38:57Z (pr-13238)1h38m
14:39:25Z (pr-13234)1h59m
14:57Z onwardstill running at read time

The break is sharp and sits between the 14:16Z batch (which landed normally) and the 14:38Z batch (which did not).

What I am NOT claiming

I have not diagnosed the cause and am deliberately not guessing at one. Candidates I can neither confirm nor exclude from outside the settings: a required check that never reports on merge_group so the group never satisfies its gate; a group-level timeout that rebuilds instead of failing loudly; runner starvation causing the group to expire; a branch-protection / queue-settings change.

I am not asserting a link to #13253 (applying a label seconds after opening a PR cancels its in-flight CI). One merge_group run — pr-13051, 14:38:58Z — was cancelled at 15:35:20Z, but a merge-queue run is also cancelled routinely whenever the queue rebuilds, and I have not separated the two mechanisms. Naming it as the cause without that separation would send the fix in the wrong direction.

Adjacent but distinct existing cards, checked before filing — neither covers this:

⚠️#12211 is worth a second look by whoever takes this: "a required check that never reports on merge_group" is exactly the shape it describes, and it is currently pm:blocked. That is a lead, ⛔ not a diagnosis.

Dedupe method: semantic search over open issues for this failure, plus a positive control on the same instrument first (a control query returned #13204, total_count 12), so the near-empty result is a measured absence and not a silently-zeroed search.

Why this is p1

Everything armed is accumulating behind it, and the failure is invisible from any single PR — from inside one PR this looks like ordinary slowness, which is precisely why four separate seats can each wait it out without anyone noticing the queue has stopped. Open and unmerged behind it right now: #13238, #13234, #13236, #13243, #13224, #13245, #13051, #13148 — and that is only what has reached the queue.

⛔ Please do not "fix" this by kicking CI

Re-running jobs, pushing empty commits, or closing and reopening PRs will make a capacity-or-settings problem worse and will destroy the evidence above. The re-queue pattern is the signal; preserve it.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions