Uh oh!
There was an error while loading. Please reload this page.
fix(bugbot-gate): an unclaimed head means two things, and the author says which (backend#2586) - #369
fix(bugbot-gate): an unclaimed head means two things, and the author says which (backend#2586)#369LukasWodka wants to merge 2 commits into
Conversation
…says which (backend#2586) The UNCLAIMED report told every reader the same thing: that Bugbot "drops PRs" and that "no one here can fix" it. Re-measuring says that is true of exactly one class -- a PR a Bot authored -- and false of every other, where an absent review is unprecedented and the reader should re-run. Measured 2026-08-27, 662 PRs across 20 repos (plus a 2026-07-01 sweep for the bot rows), anchored on the head COMMIT rather than PR creation: PR author head commit author n got a Cursor Bugbot check human human 594 yes (all) human release-train[bot] 43 yes, 43 of 43 Bot (App) human 3 yes, 3 of 3 Bot (App, sync) release-train[bot] 20 NO, 0 of 20 Bot (dependabot) either 15 NO, 0 of 15 So the discriminator is the PR's AUTHOR, and the head commit's author does not matter. backend#2114 closed saying "no discriminator survives the data"; one does. The measurement the previous comment carried is retracted in place rather than deleted: .github#349/#350/#352/#353 and e2e-test-agent#273 were cited as never reviewed, and every one of them carried a COMPLETED Cursor Bugbot run on every commit it ever had -- this gate's own `bugbot / review` concluded SUCCESS on their heads. Timing latency from PR creation while the head had moved is the likely cause of the original reading. No exit code changes. UNCLAIMED still exits 0 for every author kind, so whether the HUMAN case should block stays an open decision on backend#2586 rather than being smuggled in here. - author_kind derives the branch from GitHub's answer; an unmeasured actor type and a deleted account are "cannot tell", never folded into either neighbour - query_lacks_author_kind refuses the run if the query stops asking for author.__typename, so the discriminator cannot go inert while green
…nd#2586) The first commit said the discriminator is the PR's author and that the head commit's author does not matter. Its own table contradicts that: 3 of 3 Bot-opened release-train promotion PRs, whose head COMMIT was human-authored, WERE reviewed. Neither single field decides. What fits all three sweeps -- and no simpler rule does -- is that a PR is reviewed when SOME identity on it has a Cursor seat, author or head-commit author, with dependabot's never reviewed either way (0 of 15, including 2 with human head commits). So the three sweeps are now recorded separately, with their own windows and repo sets, instead of merged into one tidy table; the human count is corrected to 635 of 635 non-draft (637 human-authored, 2 of them drafts); and the Bot paragraph names the measured exception instead of asserting that no Bot-opened PR is ever reviewed. The branch still keys on the author, which is the field that identifies the class with no remedy -- but it now says what was measured rather than rounding it off.
LukasWodka
commented
Aug 27, 2026
This PR names
The body carries
So the options on this PR were: write It is advisory, re-measured just now — It clears itself when |
LukasWodka
commented
Aug 27, 2026
The red Measured just now: The gate parses its non-closing vocabulary out of This PR uses
No action needed on this branch. |
saqlainsyed007
left a comment
There was a problem hiding this comment.
Reviewed at 517913b4. The code is approve-quality and I verified it rather than trusting the writeup: it reworks the org-wide bugbot-gate.py so the UNCLAIMED report retracts the earlier (measured-false) "Bugbot drops PRs" claim and instead branches three ways on author identity — bot / human / cannot-tell — via a new author_kind(), adds author { __typename login } to the query, and adds query_lacks_author_kind() as a self-guard (exit 2 if the discriminator field ever disappears), mirroring the existing totalCount guard. The exit code for UNCLAIMED is deliberately left at 0, so no consumer's block/no-block behaviour changes. Tests hold the bar — selftest 109 assertions, mutation suite 47/47 caught with 0 stale, fixtures independent of the module constants, and the report-prose is the behaviour here since the exit code is constant. No findings.
Not approving this pass only because CI is red: set-status / closing-ref fails. The diff doesn't touch the closing-ref gate — the body ends with Part of tracebloc/backend#2586 · Part of tracebloc/backend#2284, and Part of is a MENTIONED (non-closing) keyword, so the gate reddens by design. This PR is genuinely partial, so you can't honestly add Closes — it's the gate's known partial-PR tension (closing-ref-gate.py:45-48), a policy call rather than anything to fix in this diff. Get that check resolved (a legitimately-closed ref if one applies, or the documented exemption path) and I'll approve — the code itself is ready.
LukasWodka
commented
Aug 28, 2026
@saqlainsyed007 — It was stale rather than a policy tension. The gate reads its keyword vocabulary from (all four on Same thing cleared No action needed from me on the diff; over to you for the approval you'd conditioned on this. |
LukasWodka
commented
Aug 28, 2026
@saqlainsyed007 — the
The passing run's own output is So the partial-PR tension you named is genuinely resolved rather than worked around — no false Two caveats so the green reads honestly:
No code changed since your review, so nothing to re-read. |
LukasWodka
commented
Aug 28, 2026
@saqlainsyed007 — the Your read of the mechanism was right, and it turned out to be already fixed rather than a policy call. The three stale runs on this head were re-queued after that landed. All four So no |
One absence, two opposite meanings
UNCLAIMEDsays the same thing to every reader today: Bugbot "drops PRs", "no onehere can fix" it. That is true of one class of PR and false of the rest, and the
run cannot tell them apart without reading who opened the PR. So it reads the
author.
The measurement
Three sweeps, kept separate because their windows and repo sets differ — merging
them into one tidy table is how the first commit on this branch overclaimed, and
the second commit fixes it.
A. 2026-08-24..27, 20 repos, all 662 open+closed PRs
Cursor BugbotcheckB. 2026-07-01.., 10 repos, Bot-authored only
C. 2026-08-18.., 7 repos — human-authored PRs whose head commit was authored by
tracebloc-release-train[bot]: 43 of 43 reviewed.The rule that fits all of it
A PR is reviewed when some identity on it has a Cursor seat — the author or
the head commit's author — and dependabot's are never reviewed either way.
Two simpler readings do not survive:
backend#2114closed COMPLETED saying "no discriminator survives the data".Identity does, even though no single field does.
What is retracted, and why in place
#356recorded, as the justification for the tolerance: "6 of 9 never got acheck at all — .github#349 (57 min), #350 (55), #352 (40), #353 (37), #354 (32),
e2e-test-agent#273 (2h+)", and "requiring this context while failing on
UNCLAIMED would block roughly two thirds of all PRs".
Re-measured per commit, every one of those rows is false:
Every commit those PRs ever had carried a COMPLETED Bugbot run, and this gate's
own
bugbot / reviewcontext concluded SUCCESS on their heads — so the gate sawthe reviews it was said to have missed.
#349has exactly one commit, nobugbot runcomment, and its Cursor check suite was created 3s after the commit. Thelikely cause of the original reading is timing latency from PR creation while the
head had since moved.
Retracted in the file rather than deleted, because that paragraph is the stated
reason the tolerance exists.
What this does NOT change
No exit code moves.
UNCLAIMEDstill exits 0 for every author kind, and theselftest pins that for all three. Whether the human case should block — now that
the honest estimate of the blast radius is 0 of 635 non-draft rather than "two
thirds" — is a decision for @LukasWodka on backend#2586, deliberately not smuggled
in here.
What changes is the sentence the next reader acts on:
dependabot), re-running cannot help because Cursor attributes
bugbot runtothe author and answers with a seat refusal, and the remedy is a seated human
author (backend#2590, fix(2590): the sync opens its PRs as a human, so Bugbot reviews them #348). It also names the measured exception — 3 of 3
Bot-opened PRs with a human head commit were reviewed — rather than
overclaiming.
because it would be the first measured instance.
(
author: null) and an unmeasured actor type both land here rather than beingfolded into a neighbour that would then say something confident and wrong.
Fail-closed, and derived rather than restated
author_kindreads GitHub'sauthor.__typename; no decision restates the tablesabove (rule 1). And because a guard that reads a field can be disarmed by editing
the query instead of the guard,
query_lacks_author_kindrefuses the whole runif the query stops asking for
author.__typename— the same shape as the existingconnections_missing_totalcountstartup refusal, one field over. Its limits arewritten next to it: it checks that some
authorselection asks for__typename, and does not model nesting.Test plan
make check— green, on both commits (lint + ruff + shellcheck + house-rulesbugbot-gate-selftest.py— 109 assertions, all passed (was 82). The new oneshold the report, because with the exit code unchanged the report is the
behaviour: the three paragraphs, their distinctness, both cross-checks (the bot
paragraph must not reach a human author and vice versa),
author_kindin fiveinput shapes, and both directions of the query guard.
bugbot-gate-mutations.py— 47 mutations: 0 stale, 0 uncaught, 7 new, eachcaught by the assertion named for it:
a Bot author is a bot,the two kinds are DIFFERENT answersa User author is human,the two kinds are DIFFERENT answersan unmeasured actor type is 'cannot tell',main: an unreadable author kind says soan UNCLAIMED human-authored head reads as anomalousan UNCLAIMED human-authored head reads as anomalousauthor.__typenamea query that asks only for the login is caughtThe mutations edit
scripts/bugbot-gate.pyitself and re-run the real suite(rule 9) — no second copy of the rule lives in the harness.
Still open on backend#2586, not fixed here
UNCLAIMEDblock on human-authored PRs? One line, once decided.patch-level ones auto-merge in
backend. Cursor-side: a dashboard change or anexplicit accepted-risk note.
bugbot / reviewbe required anywhere? It is required on no branchin the org (re-measured today), so it is advice everywhere it is called a gate.
Part of tracebloc/backend#2586 · Part of tracebloc/backend#2284