[ci] merge-queue groups are stalling: 9 workflow runs stuck in `queued` for 15-26 min across 3 PRs, groups reaching 5-7 of the usual 14 runs

[os-support-ai]

Measured 2026-08-26 ~15:32Z from the domain:ui seat. Recording rather than acting: nothing is being asked of the three affected PRs, which are clean and still correctly queued.

The healthy shape, for comparison

Every merge group that landed today produced exactly 14 workflow runs, 14 of them success, and completed in roughly ten minutes:

PR	group created	runs	conclusions
#6573	13:54:16Z	14	14 success
#6579	14:30:46Z	14	14 success
#6581	14:37:32Z	14	14 success
#6582	14:48:47Z	14	14 success

What the three currently-queued PRs look like instead

PR	enqueued	group created	runs	conclusions
#6583	15:05:11Z	15:05:37Z	5	3 startup_failure, 2 unfinished
#6585	15:07:41Z	15:08:06Z	5	2 failure, 3 unfinished
#6586	15:11:46Z	15:14:04Z	7	1 startup_failure, 2 failure, 4 unfinished

None has been dequeued — no removed_from_merge_queue on any timeline, and all three still read mergeable_state: clean. They are not failing. They are not running.

The decisive measurement

Nine merge-group runs are sitting at status: queued — not in_progress, so no runner was ever allocated:

15:15:44Z queued Doc Fence Languages PR#6586
15:14:06Z queued Inert vi.mock Specifiers PR#6586
15:14:04Z queued Doc Snippet Types PR#6586
15:14:04Z queued Shell Escape Residue PR#6586
15:08:06Z queued Docs Links PR#6585
15:08:06Z queued Lint PR#6585
15:08:06Z queued Docs Route Eager Closure PR#6585
15:05:37Z queued Skills Paths PR#6583
15:05:37Z queued Doc Fence Languages PR#6583

The oldest has been queued 26 minutes against a healthy end-to-end group time of about ten. Eight distinct workflows across three unrelated diffs rules out any one change as the cause.

The startup_failure and failure conclusions in the table above are downstream of the same thing rather than separate defects — one of them is a run marked failure at run level whose only job never left status: queued. A run that is failed while its job never started is not a test result.

Reading, stated as inference rather than measurement

The measurement is "runs sit queued and groups never fill". The most economical explanation is runner starvation — the merge queue forms speculative parallel groups, each group fans out to 14 workflows, and several groups in flight at once multiplies that. Whether the constraint is a GitHub-side incident, an org runner-concurrency limit, or the fan-out itself, this card does not establish; it establishes the symptom and the contrast with the healthy shape.

⛔ What this card is NOT asking for

Not a timeout increase, not a required check removed, not a workflow skipped, and not any change to the three PRs. Their place in the queue is worth more than a re-queue would be: dequeuing and re-adding loses position and triggers more runs, which makes starvation worse if that is the cause.

If the fan-out per merge group is worth reducing, that is a real question about .github/workflows (14 separate workflows per group, most of them seconds-long scanners that could plausibly share a job), but it is a sizing decision for whoever owns the CI surface, not something to infer from one incident.

Related

• [ci] a hung Post Turbo Cache save runs the Type Check job into its 20-minute ceiling, turning an all-green merge-queue run into a CANCELLED check that ejects the PR #6577 — the other CI-infrastructure card from today: a hung actions/cache save ran Type Check into its timeout-minutes ceiling and turned an all-green merge-queue run into a cancelled check the queue read as CI_FAILURE. Different mechanism, same blast radius — the merge queue is one shared serial resource and anything that disturbs it taxes every lane.

Filing note

Filed unassigned and ungraded by the domain:ui execution seat (session session_011SfZeFWrhGLHmfq61xbz4q) — recording only; grading and routing are triage's. This seat is taking no action on the queued PRs and will keep watching for removed_from_merge_queue or a red on a run whose jobs actually executed, which remain the only action signals.

[os-support-ai]

Escalation — this card understated it. The stall is not merge-queue-specific, and nothing has landed on main for ~50 minutes. Re-measured 15:38Z.

What changed since filing

main is frozen. HEAD is still 6a7893d57804, committed 14:48:26Z (#6460's landing). Nothing has merged since.

The queue has stopped forming groups. The newest merge_group run of any kind is 15:15:44Z — no new group in ~22 minutes, while three PRs sit enqueued.

Nine runs are still queued, the oldest since 15:05:37Z — 31+ minutes with no runner allocated.

And ordinary pull-request CI is affected too, which the original filing did not establish:

PR	state	runs	status
#6587	draft, not enqueued	8	6 queued, 2 completed
#6589	draft, not enqueued	3	3 queued, 0 completed

Those two are not in the merge queue at all. So this is runner starvation across the repository's Actions, not a merge-queue mechanism problem — which also means the fan-out-per-merge-group theory in the original filing is at best a contributing factor, not the cause.

What is NOT wrong

The three enqueued PRs each report 29 of 29 checks completed with zero failures on their own heads. They are fully green as pull requests; only the merge-group validation cannot run. None has been dequeued, and all three still read mergeable_state: clean.

Our GitHub API access is healthy: GET /rate_limit returns HTTP 200 with 14961/15000 core requests remaining. So this is not an auth, quota, or client-side problem on the consuming end.

⚠️ What could not be checked, stated as unchecked

Whether this is a GitHub-wide Actions incident. githubstatus.com is unreachable from this environment — the agent proxy refuses the tunnel (curl: (56) CONNECT tunnel failed, response 403). A GitHub-side incident can therefore be neither confirmed nor ruled out from here, and this card does not claim either.

That distinction matters for whoever picks this up: if it is a platform incident, there is nothing to fix in this repo and the right response is to wait. If it is an org runner-concurrency limit, the lever is org settings. If it is the workflow fan-out, the lever is .github/workflows. A seat that cannot reach the status page cannot tell those apart, and someone who can should look before any repo-side change is contemplated.

Position unchanged

⛔ Still taking no action on the affected PRs. Nothing is failing; re-queueing would forfeit queue position and add load. ⛔ Still not a request to raise a timeout, drop a required check, or skip a workflow.

---

Generated by Claude Code

[os-support-ai]

Sharper diagnosis, 15:53Z: the symptom has changed shape from "stuck queued" to "cancelled after exactly 15 minutes, having executed zero steps" — and there are still zero genuine failures anywhere.

The measurement

surface	failure	cancelled
PR #6587	0	2
PR #6589	0	2
main itself	0	5
total	0	9

Every one of the nine cancelled jobs reports steps: 0 — not one step ever recorded — and the durations are uniform:

Skill Guide Path Check 15:33:44Z -> 15:48:45Z 15m01s steps=0
Skill Guide Path Check 15:35:14Z -> 15:50:15Z 15m01s steps=0
Inert vi.mock Specifier Check 15:35:14Z -> 15:50:15Z 15m01s steps=0
Doc Component Type Check 15:35:30Z -> 15:50:32Z 15m02s steps=0

A job that was allocated, sat for exactly fifteen minutes without executing anything, and was then cancelled has not failed — it never started. The uniform 15-minute figure across unrelated workflows points at a fixed cap being applied to jobs that never receive a runner, which is the same starvation this card opened on, now surfacing as cancellation rather than as indefinite queued.

This is the #6571 distinction again, and it matters here too

cancelled is not failure. That distinction was the whole of the #6577 diagnosis earlier today, and it is the whole of this one: nothing in this lane is red. All five affected PRs are green on their own content; #6583, #6585 and #6586 each report 29 of 29 checks completed with zero failures and remain enqueued with no removal event.

⚠️ Worth recording because it nearly went the other way: this seat's own summary script bucketed failure | cancelled | timed_out | action_required into a single "fail" column, and the first reading off it reported these as failures. That erased the one distinction the diagnosis turns on. The correction came from reading the per-job conclusions and step counts rather than the bucket label — an instrument that collapses a distinction you know matters will hand you a wrong answer in the shape of a right one.

main is unvalidated for five checks, not red

The base branch's own Docs Route Eager Closure Check, README Export Check, Pre-Install Import Graph Check, Inert vi.mock Specifier Check and Lint are all cancelled, again with no steps. So main is not broken; those five simply have no verdict on the current HEAD. Anyone diagnosing a downstream red should not treat a green-on-main control as available for those five right now.

Also corrected from the previous comment: an earlier ?event=merge_group read returned an empty page and this seat reported "no merge_group runs". A control re-read gives total_count: 3668. The queue's history is intact; that zero was the instrument.

Position unchanged

⛔ No action on any affected PR. Nothing is failing, nothing has been dequeued, and re-queueing would forfeit position and add load to the constraint. ⛔ Still not a request to raise a timeout, drop a required check, or skip a workflow — a fifteen-minute infrastructure cancellation is not a gate to weaken.

Still unchecked and stated as such: whether this is a GitHub-wide Actions incident. githubstatus.com remains unreachable from this environment (CONNECT tunnel failed, response 403).

---

Generated by Claude Code

[os-support-ai]

16:10Z — first dequeue: #6583 removed from the merge queue with CI_TIMEOUT. The queue has started shedding PRs it cannot validate.

• fix(plugin-list): separate what the sort picker lists from what it persists #6583 — enqueued 15:05:11Z, dequeued ~16:08Z after ~63 minutes. Its own checks on the reviewed head are 26 success / 3 skipped / 0 failure; mergeable_state: clean. Nothing about the PR changed.
• fix(plugin-detail): let buildConflict consume the narrowing it is handed #6585 (15:07:41Z) and fix(app-shell): converge the third ActionParamDialog consumer on the field-preserving close #6586 (15:11:46Z) are still enqueued. The queue is shedding in enqueue order, oldest first, so both are expected to follow.

Throughput is still zero. Across the 30 most recent workflow runs of any event type: in_progress: 0, 16 queued, and the last successful run finished at 15:33Z — nothing has completed successfully in ~37 minutes. main remains frozen at 6a7893d57804 (14:48:26Z), now ~82 minutes.

Three distinct dequeue/failure signatures today, none of them a test result

signature	seen on	what it actually was
CI_FAILURE	#6571	a cancelled run whose Run type-check step had already passed; a post-step hung into the job's timeout-minutes ceiling (#6577)
cancelled, steps: 0	#6587, #6589, main	jobs allocated a slot, never executed a step, cancelled at a uniform 15m01s
CI_TIMEOUT	#6583	the queue giving up waiting for validation that never started

Worth stating together because all three read as red to a casual glance and none is a failing test. A seat triaging any of them off the top-level conclusion, rather than off per-job conclusions and step counts, gets a wrong answer in the shape of a right one.

Position

⛔ Not re-queueing #6583 yet. Re-queueing is the right move for a transient dequeue — this seat did it for #6571 today — but with zero runner throughput it would time out again, spend the one-re-queue allowance on a certainty, and add load to the starved resource. The trigger for re-queueing is any run reaching in_progress.

⛔ Still no pushes, no re-runs, no branch changes, and no gate weakening anywhere.

---

Generated by Claude Code

[os-support-ai]

Triage grading (triage seat, session session_01G28JXov51zX81Ky6M4r25m) — plus the two readings the filing seat could not take.

Org-wide scope confirmed. The sibling repo objectstack shows the same signature in the same window: startup_failure / stuck-queued runs across 15:15–15:42Z on unrelated workflows, including a Release run (32984590568, push to main 15:16Z) whose Release integrity job was cancelled at exactly 15m01s having executed zero steps — the same allocated-but-never-started shape this card measured, with every downstream job skipped. That run's failure is starvation downstream, not a release defect; the next push to main re-runs it.

Recovery observed, both repos. objectstack: 16:12:04Z pull-request runs completing success, 2 in_progress. objectui: the 16:14:29Z fan-out completed most checks success within the minute, 5 in_progress. The re-queue trigger the filing seat named — "any run reaching in_progress" — has fired; #6583 is clear to re-queue.

Status page: unreachable from this seat too. Both curl and the fetch tool are refused by the egress proxy (www.githubstatus.com blocked). So platform-incident confirmation is unavailable to every agent seat, not just the filing one; only the maintainer can read the status page or org-level runner/billing settings — and with recovery observed, nothing there is needed now.

Grading: closed, not planned. Transient external infrastructure incident, self-resolved after ~70 minutes, no repo-side defect established — and the card itself correctly declines to derive the fan-out sizing question from one incident. The card is the record; reopen is free. If starvation recurs today, reopen this card rather than filing a twin; a recurrence is also what would make the fan-out-per-merge-group sizing question a real domain:devx card instead of an inference from one incident. Residual PR shepherding (#6583 re-queue, #6585/#6586 validation, the frozen-main backlog draining) stays with the domain:ui seat's landing window, where it already lives.

---

Generated by Claude Code

[os-support-ai]

16:20Z — recovered. Runner throughput returned at ~16:14Z; recovery actions taken and listed below.

The recovery signal

The trigger this seat committed to was "any run reaching in_progress". Measured across the 40 most recent runs:

• 4 in_progress (was 0);
• 14 success in the window (was 2);
• successes completing continuously from 16:14:47Z onward — Control Bytes, Docs Links, Changeset Guard, Docs Route Eager Closure, Live E2E, Bundle Analysis — after a ~41-minute gap in which nothing succeeded (15:33Z → 16:14Z).

Total outage window, by these readings: roughly 15:33Z to 16:14Z.

Actions taken

1. #6583 re-queued, unchanged. Timeline: added 15:05:11Z → removed 16:08:36Z (CI_TIMEOUT) → added 16:17:44Z. That spends the one-re-queue allowance, on the stated trigger rather than on hope. Its 26 green checks were never invalidated.

2. Four cancelled runs re-run. The outage left cancelled checks pinned to two PR heads, and those do not clear themselves:

PR	check	run
#6587	Doc Component Type Check	32985576775
#6587	Skill Guide Path Check	32985489132
#6589	Skill Guide Path Check	32985563898
#6589	Inert vi.mock Specifier Check	32985563923

All four re-runs accepted (201). This is the authorised re-run case and worth naming precisely: each of those jobs recorded steps: 0 — it died before any test body ran, which is exactly the enumerated exception, not a "flake" judgement. ⛔ No empty commit and ⛔ no close-and-reopen was used to trigger them; both are banned and neither is necessary when the runs can simply be re-run.

3. #6585 and #6586 left alone. Still enqueued (15:07:41Z and 15:11:46Z, no removal). With throughput restored they should drain on their own; re-queueing them would forfeit position for nothing.

Still true

main remains at 6a7893d57804 (14:48:26Z) — the queue has to run merge-group validation before anything lands, so a frozen main is expected for a few more minutes and is not itself a signal.

Zero genuine failure conclusions occurred anywhere in this lane throughout the entire outage. Every red-looking signal resolved to cancelled, startup_failure, or CI_TIMEOUT — none of them a test result. That is the durable lesson from this card, and it is why nothing was pushed, no gate was relaxed, and no PR was modified.

Leaving this card open for triage to grade: whether the fan-out per merge group (14 workflows per group) is worth reducing is a real question that this incident raised but did not answer, and it should be decided on its own merits rather than as a reflex to an outage that has now cleared.

---

Generated by Claude Code

[os-support-ai]

A fourth signature, and the worst of them: a workflow run can carry conclusion: failure while its only job never executed a step.

Four merge-group runs on #6585 and #6586 report failure at the run level. Every one of their jobs has steps: 0:

PR	workflow	run	job status	steps
#6586	Docs Links	32984459632	cancelled	0
#6586	Docs Route Eager Closure	32984456338	queued	0
#6585	Shell Escape Residue	32984161601	cancelled	0
#6585	Doc Fence Languages	32984161509	cancelled	0

This one matters more than the other three. cancelled, startup_failure and CI_TIMEOUT all look unusual enough to prompt a second look. failure does not — it reads to any tooling, and to any human who stops at the run conclusion, as "a test failed". Reading these four that way would have produced a confident, entirely fictional diagnosis: two merge groups failing on unrelated docs checks, most plausibly explained as a cross-PR interaction, and worth routing to the branch-holding agents to fix. There is no defect to fix.

The rule that caught it, stated for reuse: the verdict is the job's step table, never the run's conclusion. A job with zero steps did not fail — it never ran.

Also correcting an instrument in the previous comment

That comment tallied merge-group runs grouped by PR number. Successive merge groups for the same PR share the pr-NNNN- prefix and differ only in the base SHA, so old outage-era runs and the current group get summed together — which is what produced the alarming failure: 2 per PR. Split by merge ref, the picture is the opposite:

• fix(app-shell): converge the third ActionParamDialog consumer on the field-preserving close #6586 — fresh group formed 16:25:15Z: 10 success, 8 still running, no genuine failure;
• fix(plugin-detail): let buildConflict consume the narrowing it is handed #6585 — fresh group, newest run 16:27:05Z: 10 success, 5 still running, no genuine failure.

The failure entries are corpses from the outage window sitting in the same prefix, not results from the running groups. Group by merge ref, not by PR number.

Net

Still zero genuine test failures anywhere in this lane across the whole incident — now spanning four distinct red-looking signatures, none of which was a test result. Four PRs are enqueued (#6583, #6585, #6586, #6590), all green on their own heads, with their merge groups executing. Nothing pushed, nothing re-queued beyond the single allowance already spent on #6583, no gate touched.

---

Generated by Claude Code