hooks: gate the confident wrong diagnosis, and the bare retraction by EdbertChan · Pull Request #319 · EdbertChan/catstack

EdbertChan · 2026-09-09T20:41:52Z

Summary

An eleven-hour job was declared dead on the strength of a report that never
tracked it. A capacity query showed zero slots in use.

That absence went out in bold as a settled cause: "It's a zombie, not slow."
Ten workers were in fact running at 96% CPU.

No hook objected. The evidence gate only knew hedge words, so the confident
claim passed the bar the tentative one would have failed.

hedge-runs-prove-it now also blocks a diagnosis of live system state that
ships with no instrument-level proof beside it.

wrong-check-reflect now fires on a bare "I was wrong", which is how that
claim was retracted a message later.

Review Claim

Two Stop hooks widen: an unhedged live-state diagnosis with no same-message
proof blocks, and a bare first-person retraction routes to /reflect. All
existing suppression guards on both detectors keep working.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Both detectors fail open and both gates are additive: every input that was
silent before this diff is still silent unless it matches one of the new
shapes. decide() short-circuits on stop_hook_active before either new
path runs, so a rewrite turn cannot loop. The diagnosis gate reads nothing
off disk — it inspects the outgoing message only — so it has no new failure
mode on an unreadable transcript, and it is proven silent on ten negative
fixtures covering fences, quotes, backticks, blockquotes, hypotheticals and
past-tense narration.

Assumptions: published under the skill's headless clause. This invariant
is proposed, not human-confirmed.

Slice Rationale

One incident, one message pair, two adjacent holes in the same evidence
layer: the claim that should have been gated, and the concession that should
have triggered a reflect. Splitting them would leave either half landing
without the transcript that motivates it. Both are regex-and-fixture changes
inside engine/hooks/, one review unit, no install or wiring change — both
hooks are already installed for Claude.

Extending hedge-runs-prove-it rather than adding a sibling hook follows
cat-mode's "prefer extending an existing durable mechanism over adding a
new one-off." The alternatives were checked and rejected on evidence, not
preference: prove-it-ship-gate matches done/shipped/live/proven, and its
CLAIM_RE and LIVE_NOUN_RE both return no match on this text.
diu-stop's CAUSAL_CLOSER_RE wants "because" or "the cause is", and its
paragraph loop skips any paragraph containing an inline backtick — run
against the real claim, with and without its backticks, it returns None
both ways.

The rule already existed in prose with nothing implementing it, in
corpus/skills/cat-mode/SKILL.md under ## Verify:

Unhedged root-cause or fix claims about live system behavior need
instrument-level proof in the same message, or UNVERIFIED:. The gate
is the claim type ("this is why it's slow," "this is the bug"), not a
hedge word.

Grounding. Four independent conditions must hold before the diagnosis
gate blocks — a diagnosis shape, a runtime noun within 200 characters, no
instrument evidence anywhere in the message, and no hedge or conditional in
the clause before it. That conservatism is deliberate: a checker's effective
false-positive rate, not its recall, decides whether developers leave it
turned on (Sadowski et al., "Lessons from Building Static Analysis Tools at
Google," CACM 61(4), 2018 —
https://cacm.acm.org/research/lessons-from-building-static-analysis-tools-at-google/).
The inversion itself — that an unhedged claim is the riskier one — is stated
here as this repo's own rule from the incident, with no known prior art
claimed for it.

Overlap, disclosed. Open PR #309 (vacuous-gate-fix) adds a structural
retraction layer to the same wrong-check-reflect/detect.py. It does not
cover this case: run against the real concession, its find_admission
returns None, because the sentence names no prior statement for its
PRIOR_STATEMENT_RE to find. The two changes are complementary and will
want a trivial merge in NEGATIVE_RES.

Non-goals

No change to HEDGE_RE. An unhedged claim has no hedge word by
construction; widening that pattern would be the wrong lever and would
raise false positives on the hedge path.
No change to prove-it-ship-gate or diu-stop.
No new hook, no install.sh or settings.json change. Both hooks are
already linked and wired.
No agent_id guard added. auto-pr and frustration-watchdog carry one
because they act on the whole session's behalf, which a subagent must not
do; this hook only blocks the offending reply and tells that same agent to
go get evidence, which is as correct inside a subagent as outside it.
Cursor and Codex entrypoints for wrong-check-reflect are untouched; they
pick up the widened detector for free.

Test Plan

Fail-before, on the unmodified detectors (git stash of the two
detect.py files, tests and fixtures in place):

===== hedge-runs-prove-it BEFORE =====
FAIL: test_blocks_each_diagnosis_fixture (label='zombie-vs-slow root cause read off a capacity projection, no process table (real session)')
FAIL: test_blocks_each_diagnosis_fixture (label='same claim with the status word unbackticked')
FAIL: test_blocks_each_diagnosis_fixture (label="that's the bug, asserted about a worker")
FAIL: test_blocks_each_diagnosis_fixture (label='root cause named for a queue that has not been instrumented')
FAIL: test_blocks_each_diagnosis_fixture (label='process called hung with no second sample')
FAIL: test_blocks_each_diagnosis_fixture (label="this is why it's slow, asserted about a pool")
FAIL: test_hook_blocks_zombie_claim_with_exit_2
FAIL: test_blocks_diagnosis_with_no_transcript_at_all
FAIL: test_diagnosis_gate_ignores_an_unreadable_transcript
ERROR: test_blocks_zombie_claim_even_though_the_turn_ran_a_tool
ERROR: test_no_hit_when_the_process_table_ships_in_the_same_message
Ran 17 tests
FAILED (failures=9, errors=2)

===== wrong-check-reflect BEFORE =====
FAIL: test_hit_bare_i_was_wrong
FAIL: test_hit_bare_i_was_wrong_conceding_a_live_diagnosis
FAIL: test_hit_i_got_that_wrong
Ran 51 tests
FAILED (failures=3)

Pass-after, same tests, detectors restored:

===== AFTER: hedge-runs-prove-it =====
Ran 17 tests in 0.005s
OK
===== AFTER: wrong-check-reflect =====
Ran 51 tests in 0.014s
OK

Every CI gate, run locally on this commit:

$ bash scripts/run_all_tests.sh        -> exit 0, 1043 tests, 34 suites
$ python3 scripts/check_hook_test_coverage.py
check_hook_test_coverage: OK (23 hook(s) checked)
$ python3 scripts/check_no_tracked_local_artifacts.py
check_no_tracked_local_artifacts: OK
$ python3 scripts/check_skills_three_harnesses.py
ok      skills three-harness check
$ python3 scripts/check_ecosystem_boundaries.py
ok      ecosystem boundaries
$ python3 scripts/check_skill_file_refs.py
ok      skill file refs
$ python3 scripts/check_skill_test_coverage.py --base <merge-base> --head HEAD
ok      skill test coverage
$ python3 scripts/check_skill_test_debt_no_growth.py
ok      skill test debt allowlist did not grow
$ python3 scripts/check_skill_trigger_mechanism.py
ok      skill trigger mechanism
$ python3 scripts/check_dora_baseline.py
check_dora_baseline: OK (baseline present)
$ python3 scripts/check_no_dated_provenance.py --base origin/main
ok      no dated provenance
$ python3 scripts/check_no_new_comments.py --base origin/main
ok      no new comments
$ ruff check . --select E9,F
All checks passed!
$ shellcheck install.sh
(exit 0)
$ python3 engine/skills/make-pr/scripts/preflight.py --base origin/main
unit    engine-runtime: 8 file(s)
gate    check_hook_test_coverage: OK (1 hook(s) checked)   [hedge-runs-prove-it]
gate    check_hook_test_coverage: OK (1 hook(s) checked)   [wrong-check-reflect]
ok      preflight passed
$ node engine/skills/draft-pr/scripts/lint-diff-atomicity.mjs --base origin/main
Diff atomicity validation passed.

Re-run:

python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v
python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests -v
bash scripts/run_all_tests.sh
python3 scripts/check_hook_test_coverage.py
python3 engine/skills/make-pr/scripts/preflight.py --base origin/main

Revert Plan

Safe to revert? Yes
Revert command: git revert 72ef794
Post-revert steps: None. No install, symlink, or settings.json change; the
hooks resolve detect.py through the existing symlink, so a revert takes
effect on the next turn with no re-install.
Data migration? No

🤖 Generated with Claude Code

https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8

Note

Medium Risk
Stop-hook behavior widens on assistant replies (more blocks and /reflect prompts); regex-heavy paths add false-positive risk, though guards and fail-open semantics are preserved.

Overview
Extends two Claude Stop hooks so confident wrong live-state claims get gated and bare retractions trigger reflection.

hedge-runs-prove-it now has a second bar besides hedged code/repo claims: unhedged diagnoses of live runtime behavior (e.g. “it's a zombie”, “that's the bug”, “the root cause is…”) near a runtime noun block unless the same message includes instrument-level proof (fenced output, file:line, pid/exit code, /proc, or UNVERIFIED:). Unlike the hedge path, running Read/Bash/etc. in the turn does not clear this gate. Hedge detection also treats hedges anywhere inside double/backtick/single-quoted spans as citations (with apostrophe-aware single quotes), not only when the hedge is the first character after the opener.

wrong-check-reflect now matches bare first-person admissions like “I was wrong” / “I got that wrong”, with added negatives for hypotheticals (“if/unless I was wrong”) and reported speech (“they said I was wrong”); fence/quote/backtick stripping unchanged.

Docs and fixture-driven tests cover diagnosis fire/silent cases, quoted hedges, and the new admission shapes.

^{Reviewed by Cursor Bugbot for commit 98c297b. Bugbot is set up for automated code reviews on this repo. Configure here.}

A capacity projection reported zero slots in use for a pool whose task still read `running`, and that absence was written out as a settled root cause -- "it's a zombie, not slow" -- while ten workers sat in R state at 96% CPU. Nothing stopped it. `hedge-runs-prove-it` only looked for hedge words, so the confident claim, which carries no signal that a check is outstanding, passed the gate the hedged one would have failed. `prove-it-ship-gate` only covers done/shipped/live. `diu-stop`'s causal closer wants "because" or "the cause is", and skips any paragraph holding an inline backtick. hedge-runs-prove-it now also blocks a diagnosis of live system state -- "it's a zombie", "that's the bug", "the root cause is X", "the worker is hung", "this is why it's slow" -- near a runtime noun with no instrument-level proof in the same message. Having run a tool this turn does not clear it; a projection that omits a field is not proof the state is absent. A fenced block of real output, a file:line, a pid, an exit code, a /proc path, or an explicit UNVERIFIED: prefix does. Four conditions must hold before it blocks, because a Stop hook's effective false-positive rate decides whether it survives at all (Sadowski et al., "Lessons from Building Static Analysis Tools at Google," CACM 61(4), 2018). wrong-check-reflect now fires on a bare "I was wrong". The retraction after a false claim is often the shortest sentence in the turn, and requiring it to name the check it retracts let the plainest concession through. The hypothetical, reported-speech, product-blame, third-person, quote, backtick and fence guards all still hold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8

cursor · 2026-09-09T20:42:00Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_fd9859e0-858c-4951-a3e7-77959d4e9a05)

…oken A reply describing this hook to the user, quoting its own trigger words in prose, got blocked by it. `code_hedges()` tested the single character before the match, so a hedge was exempt only when it opened the quote: "should work" passed and "that should work" fired, in double quotes and backticks alike. Quote runs are now found up front and the match is skipped when it falls inside one. The runs are tested by containment and never stripped: CODE_NOUN_RE counts a backticked name as a code noun, and stripping the spans silences an existing positive fixture whose only code nouns are `claude.hook.json` and `Edit|Write|Bash` inside backticks. A single quote opens a run only when it is not an apostrophe -- no letter or digit on the inside edge -- so "it's", "don't" and "the workers' pool" keep their quote as a letter and a hedge beside one still fires. The old adjacent-character check is kept as a fallback so an unbalanced opening quote stays exempt exactly as before. The same three patterns now clean the diagnosis path too, which previously handled double quotes and backticks but not single ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8

cursor · 2026-09-09T20:53:17Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_f63e75e9-8950-4fc3-b4b5-c308b4b944f3)

EdbertChan merged commit 7194f6e into main Sep 9, 2026
4 checks passed

Navigation Menu

hooks: gate the confident wrong diagnosis, and the bare retraction - #319

hooks: gate the confident wrong diagnosis, and the bare retraction#319
EdbertChan merged 2 commits into
mainfrom
reflect/unhedged-diagnosis-claim-20260909

EdbertChan commented Sep 9, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot commented Sep 9, 2026

Uh oh!

cursor Bot commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

EdbertChan commented Sep 9, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Test Plan

Revert Plan

Uh oh!

cursor Bot commented Sep 9, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot commented Sep 9, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

EdbertChan commented Sep 9, 2026 •

edited by cursor Bot

Loading