Uh oh!
There was an error while loading. Please reload this page.
Auto-apply high-confidence frontend-triage results - #6439
Conversation
b7f1425 to
1095882Comparefrontend-triage produces a root-cause analysis and fix plan and records it as
Bugzilla actions, but nothing applied them: a human had to click Apply in the
hackbot UI. Close that loop for the results the agent is confident about.
`AgentSpec` gains `auto_apply_confidence` — the `findings.confidence` levels whose
actions may be applied unattended — and frontend-triage opts in at `{"high"}` only,
so widening the policy later is an edit to that set rather than to the applier.
Because `confidence` is parsed out of the agent's free-form JSON block, it is
compared case- and whitespace-insensitively, and a run reporting nothing usable
never qualifies.
Confidence gates the agent's judgement, not its reach. An action's params are
model output and the apply step dispatches them against the runtime's *global*
handler registry — which can create bugs, attach files and write to Phabricator —
so restricting which tools the agent was given does not restrict what its
recorded actions reach. `auto_apply_guard` bounds that: for triage, one comment
and one add-only `keywords`/`severity` change on the bug the run was asked about.
Anything else holds the whole run, since the comment explains the field change and
the two are coalesced into one PUT.
The agent is now told what its rating causes, because it was being asked to
self-report a control input without knowing it was one.
Medium and low results are unchanged: still recorded, still visible in the UI,
still appliable by hand. So are all the other agents.1095882 to
d0e2d83Compareevgenyrp
commented
Aug 6, 2026
I'm adding Slack support here, so I hope you'll be able to use it instead of emails when it's landed #6523 |
d0e2d83 to
58b42a6Comparemsujaws
commented
Aug 7, 2026
Thanks! I have removed the Slack notification commit from this PR since your approach will be much better. |
| return isinstance(value, int) or (isinstance(value, str) and value.isdigit()) | ||
| def _field_change(field: str, value: Any) -> str | None: |
There was a problem hiding this comment.
Nice work on this!
One thing that looks left open is the values themselves. _field_change accepts any non-empty string for severity and any keyword for keywords. Doesn't seem to fire today for triage but may be worth bounding both to known lists in case that changes.
- for severity https://bugzilla.mozilla.org/rest/field/bug/bug_severity has the legal values
- for keywords https://bugzilla.mozilla.org/describekeywords.cgi
There was a problem hiding this comment.
I think the instruction for the Bugzilla action would better fit in the agent's system prompt.
| run for review. See `auto_apply_confidence` in | ||
| `services/hackbot-api/app/agents.py` and `frontend_triage_guard` in | ||
| `services/hackbot-api/app/auto_apply.py`. |
There was a problem hiding this comment.
The agent doesn’t have access to that. And even if it has access, it might not be the best way.
| ) | ||
| def _should_auto_apply( |
There was a problem hiding this comment.
Why not do that in the agent itself?
| return isinstance(value, int) or (isinstance(value, str) and value.isdigit()) | ||
| def _field_change(field: str, value: Any) -> str | None: |
There was a problem hiding this comment.
We could do this validation in an action hook for now. I'm planing to do more generic validation through pydantic. Once that in place, we could drop the or simplify the hook if needed.
frontend-triageproduces a root-cause analysis and fix plan and records it as Bugzilla actions — but nothing applied them, so every triage needed a human to click Apply in the hackbot UI.This closes that loop for the results the agent is confident about.
Auto-apply, gated on confidence
AgentSpecgainsauto_apply_confidence: the set offindings.confidencelevels whose actions may be applied unattended, layered on top of the existingauto_apply_actionsswitch.frontend-triageopts in at{"high"}only, so widening the policy later — to also acceptmedium— is an edit to that set rather than to the applier.The decision is a pure predicate over
(spec, run)so the policy is testable on its own, and it fails closed: a run whose findings carry no usable confidence never qualifies. Becauseconfidenceis parsed out of the agent's free-form JSON block, it's compared case- and whitespace-insensitively, so"High"doesn't silently mean "never apply".Medium and low results are unchanged from today: still recorded, still visible in the UI, still appliable by hand. So are all the other agents — there's a test asserting that.
Bounding reach, not just judgement
Confidence gates how sure the agent is, not how far its actions can go. An action's params are model output, and the apply step dispatches them against the runtime's global handler registry — which can create bugs, attach files and write to Phabricator — so restricting which tools the agent was given does not restrict what its recorded actions reach.
auto_apply_guardbounds that separately. For triage it allows one comment and one add-onlykeywords/severitychange, both on the bug the run was asked about. Anything else holds the whole run rather than applying the parts that pass, since the comment explains the field change and the two are coalesced into a single PUT.Tell the agent what its confidence rating now causes
confidenceused to be advisory metadata a human read off the run. Now it's a control input that decides whether a comment reaches a real bug unreviewed — and the agent was being asked to self-report it without being told what it does, which invites grading on a curve.Testing
118 passedinservices/hackbot-api. The 3 failures intest_actions_applier.py(comment.is_markdownmissing from the expectations) and the 8 errors intest_list_runs_api.py/test_create_run_api.py(fixture 'client' not found) are pre-existing on master — verified by running master's suite unmodified in a clean worktree, which gives80 passed, 4 failed, 8 errors. This branch's failures are a strict subset: it rewrites the fourth,test_succeeded_opted_in_agent_records_and_applies.Deploying
No new configuration.
BUGZILLA_API_KEYis already set on the service and working (hackbot@mozilla.tldhas been applying comments), so nothing new is needed for the apply path.Follow-up
Firefox :: New Tab Pagebugs from staff — land this first, so a triggered run auto-applies.