From 72ef79480d88c66c8dc5a32809cb651789857689 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 13:40:05 -0700 Subject: [PATCH 1/2] hooks: gate the confident wrong diagnosis, and the bare retraction A capacity projection reported zero slots in use for a pool whose task still read `running`, and that absence was written out as a settled root cause -- "it's a zombie, not slow" -- while ten workers sat in R state at 96% CPU. Nothing stopped it. `hedge-runs-prove-it` only looked for hedge words, so the confident claim, which carries no signal that a check is outstanding, passed the gate the hedged one would have failed. `prove-it-ship-gate` only covers done/shipped/live. `diu-stop`'s causal closer wants "because" or "the cause is", and skips any paragraph holding an inline backtick. hedge-runs-prove-it now also blocks a diagnosis of live system state -- "it's a zombie", "that's the bug", "the root cause is X", "the worker is hung", "this is why it's slow" -- near a runtime noun with no instrument-level proof in the same message. Having run a tool this turn does not clear it; a projection that omits a field is not proof the state is absent. A fenced block of real output, a file:line, a pid, an exit code, a /proc path, or an explicit UNVERIFIED: prefix does. Four conditions must hold before it blocks, because a Stop hook's effective false-positive rate decides whether it survives at all (Sadowski et al., "Lessons from Building Static Analysis Tools at Google," CACM 61(4), 2018). wrong-check-reflect now fires on a bare "I was wrong". The retraction after a false claim is often the shortest sentence in the turn, and requiring it to name the check it retracts let the plainest concession through. The hypothetical, reported-speech, product-blame, third-person, quote, backtick and fence guards all still hold. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- engine/hooks/hedge-runs-prove-it/README.md | 39 ++++++ engine/hooks/hedge-runs-prove-it/detect.py | 122 ++++++++++++++++-- .../tests/fixtures/diagnosis_fires.json | 32 +++++ .../tests/fixtures/diagnosis_silent.json | 52 ++++++++ .../hedge-runs-prove-it/tests/test_hooks.py | 65 +++++++++- engine/hooks/wrong-check-reflect/README.md | 10 +- engine/hooks/wrong-check-reflect/detect.py | 18 +++ .../wrong-check-reflect/tests/test_hooks.py | 51 ++++++++ 8 files changed, 373 insertions(+), 16 deletions(-) create mode 100644 engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json create mode 100644 engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json diff --git a/engine/hooks/hedge-runs-prove-it/README.md b/engine/hooks/hedge-runs-prove-it/README.md index be7add03..bf61d954 100644 --- a/engine/hooks/hedge-runs-prove-it/README.md +++ b/engine/hooks/hedge-runs-prove-it/README.md @@ -14,6 +14,43 @@ sentence carries a cannot-verify reason ("cannot verify: no network", "would need the live token"), when the hedge is quoted, or when the hedge is about something that is not code or state (a company's motive). +## The unhedged half + +A hedge announces its own missing check. A confident diagnosis does not, so +it is the more dangerous shape and it used to pass this gate untouched. The +hook also blocks a reply that asserts live system state -- "it's a zombie", +"that's the bug", "the root cause is X", "the worker is hung", "this is why +it's slow" -- within 200 characters of a runtime noun (process, pid, worker, +task, pool, queue, slot, workflow, socket, lock, CPU, disk) and carries no +instrument-level proof in the same message. + +Running a tool in the turn does not clear this one; a projection that omits +a field is not proof the state is absent. Only same-message proof clears it: +a fenced block of real output, a `file:line`, a pid, an exit code, a +`/proc/` path, or an explicit `UNVERIFIED:` prefix. + +Stays silent on a diagnosis inside a fence, a double-quoted span, a backtick +span, a markdown blockquote, a hypothetical ("if it's a zombie, ..."), and +past-tense narration of an old incident ("the task was a zombie"). + +Mechanical half of `corpus/skills/cat-mode/SKILL.md`'s Verify rule: +"Unhedged root-cause or fix claims about live system behavior need +instrument-level proof in the same message, or `UNVERIFIED:`." Four +independent conditions must hold before it blocks, because a Stop hook's +effective false-positive rate is what decides whether anyone keeps it on +(Sadowski et al., "Lessons from Building Static Analysis Tools at Google," +CACM 61(4), 2018). + +`diu-stop` has a narrower causal closer of its own ("the cause is", +"because"). It stays silent on the copula shape, and its paragraph check +skips any paragraph containing an inline backtick, so it is not the lever +for this class. + +No `agent_id` guard: `auto-pr` and `frustration-watchdog` carry one because +they act on the whole session's behalf, which a subagent must not do. This +hook only blocks the offending reply and tells that same agent to go get +evidence -- as correct inside a subagent as outside it. + Mechanical half of the evidence rules in `engine/CLAUDE.core.md` ("never claim ... without evidence in the SAME message"; `UNVERIFIED:` is the escape hatch, not a free pass). Fail-open on parse or read errors; @@ -26,6 +63,8 @@ escape hatch, not a free pass). Fail-open on parse or read errors; - `claude.hook.json` / `install_claude_hook.py` -- settings.json merge (idempotent). - `tests/fixtures/hedges_{fires,silent}.json` -- sanitized real replies and the rule's own shapes. +- `tests/fixtures/diagnosis_{fires,silent}.json` -- unhedged diagnosis + claims, including the same claim shipped with its process table. - `tests/test_hooks.py` Tests: `python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v` diff --git a/engine/hooks/hedge-runs-prove-it/detect.py b/engine/hooks/hedge-runs-prove-it/detect.py index 863281f0..3f39d659 100644 --- a/engine/hooks/hedge-runs-prove-it/detect.py +++ b/engine/hooks/hedge-runs-prove-it/detect.py @@ -1,14 +1,27 @@ -"""hedge-runs-prove-it: a hedge about code or repo state is a prompt to verify. - -"I think", "I believe", "probably", "should work", "presumably", or an -`UNVERIFIED:` prefix next to a code noun (a path, a backticked name, test, -CI, build, bug, fix, script, hook, PR, merge, branch, commit) means the -agent has a check it has not run. The reply passes only when the turn ran -a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:` clause -says why it cannot be verified ("cannot verify: no network"). Hedges about -things that are not code or state (a company's motive, a filing date) -are out of scope. Judgment stays with the model; this file matches -shapes and fails open. +"""hedge-runs-prove-it: an unrun check about code or repo state is a prompt to verify. + +Two shapes, two bars. + +A hedge -- "I think", "I believe", "probably", "should work", "presumably", +or an `UNVERIFIED:` prefix next to a code noun (a path, a backticked name, +test, CI, build, bug, fix, script, hook, PR, merge, branch, commit) -- +means the agent has a check it has not run. The reply passes only when the +turn ran a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:` +clause says why it cannot be verified ("cannot verify: no network"). + +An unhedged diagnosis -- "it's a zombie", "that's the bug", "the root cause +is X", "the worker is hung", "this is why it's slow" -- asserts live system +state with no hedge word at all, so the hedge bar never sees it. Confidence +is the more dangerous shape, not the safer one: the claim carries no signal +that a check is outstanding. Running a tool in the turn does not clear it, +because a projection that omits a field is not proof the state is absent. +Only instrument-level proof in the same message clears it: pasted output, a +`file:line`, a pid, an exit code, or an explicit `UNVERIFIED:` prefix. + +Hedges about things that are not code or state (a company's motive, a +filing date) are out of scope, as are diagnoses inside a fence, a quote, a +blockquote, or a hypothetical. Judgment stays with the model; this file +matches shapes and fails open. """ from __future__ import annotations @@ -37,12 +50,63 @@ VERIFY_TOOLS = {"Bash", "Read", "Grep", "Glob", "Monitor", "WebFetch"} QUOTED_BEFORE = ('"', "'", "`") +DIAGNOSIS_RE = re.compile( + r"(?:\b(?:it'?s|that'?s|this\s+is|they'?re|the\s+[\w-]+(?:\s+[\w-]+)?\s+is)\s+" + r"(?:just\s+|simply\s+|actually\s+|basically\s+|clearly\s+|a\s+|an\s+|the\s+)*" + r"(?:zombie|hung|hanging|stuck|wedged|deadlocked|dead(?!\s+code)|crashed|" + r"leaking|thrashing|starved|orphaned|frozen|spinning|silently\s+failing|" + r"corrupt(?:ed)?|misconfigured)\b)|" + r"(?:\b(?:that'?s|this\s+is|here'?s)\s+(?:the|your|our|my)\s+" + r"(?:bug|root\s+cause|cause|culprit|problem|issue|failure)\b)|" + r"(?:\bthe\s+(?:root\s+)?cause\s+is\b)|" + r"(?:\bthe\s+reason\s+is\b)|" + r"(?:\bwhat'?s\s+(?:happening|going\s+on)\s+is\b)|" + r"(?:\b(?:that|this)(?:'?s|\s+is)\s+why\s+(?:it|its|it'?s|the|they|that|this)\b)", + re.IGNORECASE, +) +RUNTIME_NOUN_RE = re.compile( + r"\b(?:process(?:es)?|pids?|threads?|workers?|tasks?|jobs?|runs?|runners?|" + r"daemons?|services?|servers?|hosts?|containers?|pods?|nodes?|pools?|" + r"queues?|slots?|workflows?|sessions?|agents?|sockets?|ports?|" + r"connections?|locks?|loops?|requests?|cpu|memory|disk|cache|database|db|" + r"quer(?:y|ies)|logs?|streams?|builds?|tests?|suites?|CI|pipelines?|" + r"hooks?|scripts?|commands?|replay)\b", + re.IGNORECASE, +) +INSTRUMENT_EVIDENCE_RE = re.compile( + r"```|\bUNVERIFIED:|/proc/\d+|" + r"\b[\w./-]+\.[A-Za-z]{1,6}:\d+\b|" + r"\bpids?\b\s*[:=#]?\s*\d+|\bMainPID\b|" + r"\bexit\s+(?:code|status)\b|\bexit[_-]?code\b", + re.IGNORECASE, +) +DIAGNOSIS_LOOKBACK_RE = re.compile( + r"\b(?:if|unless|whether|suppose|assuming|in case|maybe|perhaps|" + r"might\s+be|could\s+be|may\s+be|not\s+sure|unclear)\b[^.!?\n]*$", + re.IGNORECASE, +) +LOOKBACK = 120 +FENCE_RE = re.compile(r"```.*?```", re.DOTALL) +BLOCKQUOTE_RE = re.compile(r"(?m)^\s*>.*$") +DOUBLE_QUOTE_RE = re.compile(r'"[^"]*"', re.DOTALL) +BACKTICK_RE = re.compile(r"`[^`]*`", re.DOTALL) + MESSAGE = ( "hedge-runs-prove-it: this reply hedges about code or repo state ({hedge}) and " "the turn ran no verification (no Bash / Read / Grep). Run prove-it now: verify in " "this turn, or write `UNVERIFIED: -- cannot verify: `." ) +DIAGNOSIS_MESSAGE = ( + "hedge-runs-prove-it: this reply asserts a diagnosis about live system state " + "({claim}) with no instrument-level proof in the same message. An unhedged " + "root-cause claim is the dangerous shape, not the hedged one -- nothing in it " + "signals an outstanding check -- and a projection that omits a field is not " + "proof the state is absent. Attach the instrument output here: pasted ps / " + "strace / /proc output, a live query's real result, a pid, an exit code, or a " + "file:line. Otherwise prefix the claim with `UNVERIFIED:`." +) + def _sentence_after(text: str, start: int) -> str: end = len(text) @@ -68,6 +132,29 @@ def code_hedges(text: str) -> list[str]: return hits +def _diagnosis_text(text: str) -> str: + cleaned = FENCE_RE.sub(" ", text or "") + cleaned = BLOCKQUOTE_RE.sub(" ", cleaned) + cleaned = DOUBLE_QUOTE_RE.sub(" ", cleaned) + return BACKTICK_RE.sub(" ", cleaned) + + +def diagnosis_claims(text: str) -> list[str]: + """Unhedged diagnoses of live system state carrying no same-message proof.""" + if not text or INSTRUMENT_EVIDENCE_RE.search(text): + return [] + cleaned = _diagnosis_text(text) + hits: list[str] = [] + for match in DIAGNOSIS_RE.finditer(cleaned): + window = cleaned[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY] + if not RUNTIME_NOUN_RE.search(window): + continue + if DIAGNOSIS_LOOKBACK_RE.search(cleaned[max(0, match.start() - LOOKBACK): match.start()]): + continue + hits.append(match.group(0)) + return hits + + def _text_content(data: dict) -> str: message = data.get("message") content = message.get("content") if isinstance(message, dict) else data.get("content") @@ -117,7 +204,17 @@ def verified_this_turn(lines: list[dict]) -> bool: return False +def _diagnosis_feedback(message: str) -> str | None: + claims = diagnosis_claims(message) + if not claims: + return None + return DIAGNOSIS_MESSAGE.format(claim=", ".join(f'"{c}"' for c in claims[:3])) + + def decide_from_lines(message: str, lines: list[dict]) -> str | None: + diagnosis = _diagnosis_feedback(message) + if diagnosis: + return diagnosis hedges = code_hedges(message) if not hedges: return None @@ -131,6 +228,9 @@ def decide(payload: dict) -> str | None: if payload.get("stop_hook_active"): return None message = payload.get("last_assistant_message") or "" + diagnosis = _diagnosis_feedback(message) + if diagnosis: + return diagnosis if not code_hedges(message): return None transcript_path = payload.get("transcript_path") or payload.get("transcriptPath") or "" diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json new file mode 100644 index 00000000..27cfda30 --- /dev/null +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json @@ -0,0 +1,32 @@ +[ + { + "label": "zombie-vs-slow root cause read off a capacity projection, no process table (real session)", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the\ntask claims `running`. Checking whether the replay process actually exists:", + "verified": true + }, + { + "label": "same claim with the status word unbackticked", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the task claims running. Checking whether the replay process actually exists:", + "verified": true + }, + { + "label": "that's the bug, asserted about a worker", + "reply": "That's the bug: the worker never re-registers its slot after a restart.", + "verified": true + }, + { + "label": "root cause named for a queue that has not been instrumented", + "reply": "The root cause is the poll loop holding the lock while the queue drains.", + "verified": false + }, + { + "label": "process called hung with no second sample", + "reply": "The replay process is hung on the socket read.", + "verified": true + }, + { + "label": "this is why it's slow, asserted about a pool", + "reply": "This is why it's slow -- the pool serializes every task through one worker.", + "verified": false + } +] diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json new file mode 100644 index 00000000..57965200 --- /dev/null +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json @@ -0,0 +1,52 @@ +[ + { + "label": "same zombie claim, but the process table ships in the same message", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use.\n\n```\nUID PID %CPU STAT ELAPSED CMD\nme 4412 96.2 R 11:19:03 node replay-worker.js\n```", + "verified": true + }, + { + "label": "hypothetical diagnosis, still on the way to the check", + "reply": "If it's a zombie, the pool would still show the slot free, so I am reading the process table next.", + "verified": false + }, + { + "label": "quoting the user's own diagnosis back", + "reply": "You said \"it's a zombie, not slow\" -- reading the process table now.", + "verified": false + }, + { + "label": "the claim shape shown inside a fence, not asserted", + "reply": "The detector fires on this shape:\n\n```\nIt's a zombie, not slow. The pool shows zero slots.\n```\n", + "verified": false + }, + { + "label": "diagnosis explicitly prefixed UNVERIFIED", + "reply": "UNVERIFIED: it's a zombie, not slow -- the pool shows zero slots but I have not looked at the process table.", + "verified": true + }, + { + "label": "past incident narrated, not a live claim", + "reply": "Yesterday the pool showed zero slots and the task was a zombie; that run was killed by hand.", + "verified": false + }, + { + "label": "describing the rule itself in backticks", + "reply": "The gate fires when a reply says `it's a zombie` with no process-table output in the same message.", + "verified": false + }, + { + "label": "user's line pasted as a blockquote", + "reply": "> It's a zombie, not slow. The pool shows zero slots.\n\nThat is the line to check, so I am pulling the process table.", + "verified": false + }, + { + "label": "ordinary because-clause about the agent's own sequencing", + "reply": "That's why I ran the test suite before touching the worker pool.", + "verified": false + }, + { + "label": "diagnosis carrying a file:line instead of pasted output", + "reply": "That's the bug: the worker slot is released before the task row is updated, at packages/pool/slots.py:212.", + "verified": false + } +] diff --git a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py index 8be71333..5d7775ae 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py +++ b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py @@ -3,10 +3,13 @@ Run: python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v -Fixtures in tests/fixtures/ are sanitized replies from one real session -("presumably moving it to catstack" with no check run) plus the shapes the -rule names. Each fixture carries `verified`: whether the turn ran a -verification tool. +Fixtures in tests/fixtures/ are sanitized replies from real sessions +("presumably moving it to catstack" with no check run; "it's a zombie, not +slow" asserted off a capacity projection while ten workers were at 96% CPU) +plus the shapes the rule names. Each fixture carries `verified`: whether the +turn ran a verification tool. The diagnosis fixtures set it true on purpose +-- an unhedged root-cause claim is not cleared by having run a tool, only by +instrument-level proof in the same message. """ from __future__ import annotations @@ -113,5 +116,59 @@ def test_fails_open_on_garbage_stdin(self): self.assertEqual(err.getvalue(), "") +class TestBlocksUnhedgedDiagnosis(unittest.TestCase): + def test_blocks_each_diagnosis_fixture(self): + for case in load("diagnosis_fires.json"): + with self.subTest(label=case["label"]): + self.assertIsNotNone( + detect.decide_from_lines(case["reply"], turn_lines(case["verified"])) + ) + + def test_blocks_zombie_claim_even_though_the_turn_ran_a_tool(self): + case = load("diagnosis_fires.json")[0] + self.assertTrue(detect.diagnosis_claims(case["reply"])) + self.assertEqual(detect.code_hedges(case["reply"]), []) + self.assertIsNotNone(detect.decide_from_lines(case["reply"], turn_lines(True))) + + def test_hook_blocks_zombie_claim_with_exit_2(self): + case = load("diagnosis_fires.json")[0] + path = transcript_file(turn_lines(True)) + try: + code, err = run_hook({"last_assistant_message": case["reply"], "transcript_path": path}) + finally: + os.unlink(path) + self.assertEqual(code, 2) + self.assertIn("instrument-level proof", err) + self.assertIn("zombie", err) + + def test_blocks_diagnosis_with_no_transcript_at_all(self): + case = load("diagnosis_fires.json")[2] + self.assertIsNotNone(detect.decide({"last_assistant_message": case["reply"]})) + + +class TestAllowsProvenOrQuotedDiagnosis(unittest.TestCase): + def test_allows_each_silent_diagnosis_fixture(self): + for case in load("diagnosis_silent.json"): + with self.subTest(label=case["label"]): + self.assertIsNone( + detect.decide_from_lines(case["reply"], turn_lines(case["verified"])) + ) + + def test_no_hit_when_the_process_table_ships_in_the_same_message(self): + self.assertEqual(detect.diagnosis_claims(load("diagnosis_silent.json")[0]["reply"]), []) + + def test_allows_diagnosis_when_stop_hook_active(self): + self.assertIsNone(detect.decide({ + "last_assistant_message": load("diagnosis_fires.json")[0]["reply"], + "stop_hook_active": True, + })) + + def test_diagnosis_gate_ignores_an_unreadable_transcript(self): + self.assertIsNotNone(detect.decide({ + "last_assistant_message": load("diagnosis_fires.json")[0]["reply"], + "transcript_path": "/nonexistent/x.jsonl", + })) + + if __name__ == "__main__": unittest.main() diff --git a/engine/hooks/wrong-check-reflect/README.md b/engine/hooks/wrong-check-reflect/README.md index bf877c47..7a71bf15 100644 --- a/engine/hooks/wrong-check-reflect/README.md +++ b/engine/hooks/wrong-check-reflect/README.md @@ -4,7 +4,15 @@ When the assistant admits a prior check/claim was wrong ("Good catch — my earlier check was wrong", "You're right, I misread the file", "I incorrectly assumed…", "the file I cited was a duplicate", "My mistake — I misread it", "I misread the front matter on that skill"), inject a -`/reflect` follow-up. Finish the live correction first. Fail-open. +`/reflect` follow-up. + +A bare "I was wrong" counts, with no named check after it. The retraction +that follows a false claim is often the shortest sentence in the turn, and +requiring it to name the check it retracts let the plainest concession +through. The hypothetical ("if I was wrong about this…"), reported-speech +("the reviewer said I was wrong"), product-blame ("the test was wrong"), +third-person, quote, backtick and fence guards all still hold, so only an +admission asserted in the agent's own voice fires. Finish the live correction first. Fail-open. Once per transcript. Skip if the user already said `/reflect`. Not word-count (`diu-stop`). Not token_audit thrash (`reflect-on-thrash`). diff --git a/engine/hooks/wrong-check-reflect/detect.py b/engine/hooks/wrong-check-reflect/detect.py index c41aa795..465d1d98 100644 --- a/engine/hooks/wrong-check-reflect/detect.py +++ b/engine/hooks/wrong-check-reflect/detect.py @@ -1,5 +1,12 @@ """Detect first-person “my earlier check was wrong” admissions. +A bare “I was wrong” counts. The retraction that follows a false claim is +often the shortest sentence in the turn, and the earlier requirement that it +name the check it retracts let the plainest concession through. The +hypothetical, product-blame, reported-speech, quote and fence guards below +still hold, so only a first-person admission asserted in the agent's own +voice fires. + Assistant text only. Fail-open: parse/IO errors mean no hit. Once per transcript. Skip if the user already asked /reflect. """ @@ -48,6 +55,9 @@ r"(?i)\bi\s+(was\s+wrong|got\s+it\s+wrong)\s+(about|on)\s+" r"(the\s+)?(file|path|source|check|assumption)\b" ), + re.compile( + r"(?i)\bi\s+(?:was\s+wrong|got\s+(?:it|that|this)\s+wrong)\b" + ), re.compile( r"(?i)\bi\s+(read|got|took|marked|logged|noted)\s+(that|this|it)\s+wrong\s+" r"in\s+my\s+(earlier|previous|prior)\s+\w+" @@ -87,6 +97,14 @@ re.compile(r"(?i)\bthe\s+(test|ui|build|product|code)\s+was\s+wrong\b"), re.compile(r"(?i)\bif\b.{0,40}\bmy\s+mistake\b"), re.compile(r"(?i)\bif\s+i\s+misread\s+(it|that|this|the)\b"), + re.compile( + r"(?i)\b(?:if|unless|whether|in\s+case|suppose|assuming)\s+i\s+" + r"(?:was|were)\s+wrong\b" + ), + re.compile( + r"(?i)\b(?:says?|said|thinks?|thought|claims?|claimed|argued|insisted|" + r"told\s+me|telling\s+me)\s+(?:that\s+)?i\s+(?:was|were)\s+wrong\b" + ), ] FOLLOWUP = ( diff --git a/engine/hooks/wrong-check-reflect/tests/test_hooks.py b/engine/hooks/wrong-check-reflect/tests/test_hooks.py index b27d876a..d6bf2dbd 100644 --- a/engine/hooks/wrong-check-reflect/tests/test_hooks.py +++ b/engine/hooks/wrong-check-reflect/tests/test_hooks.py @@ -191,6 +191,57 @@ def test_no_hit_normal_correction_language(self): ) ) + def test_hit_bare_i_was_wrong(self): + match = detect.find_admission("I was wrong. The pool never tracked that slot.") + self.assertIsNotNone(match) + self.assertIn("i was wrong", match.lower()) + + def test_hit_bare_i_was_wrong_conceding_a_live_diagnosis(self): + self.assertIsNotNone(detect.find_admission( + "I was wrong - it **is** genuinely computing. 10 workers in R state " + "at ~96% CPU." + )) + + def test_hit_i_got_that_wrong(self): + self.assertIsNotNone( + detect.find_admission("I got that wrong -- the worker was live the whole time.") + ) + + def test_no_hit_hypothetical_bare_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "If I was wrong about this, then the pool would show a free slot." + )) + + def test_no_hit_unless_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "Unless I was wrong about the ordering, the queue drains first." + )) + + def test_no_hit_reported_speech_someone_said_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "The reviewer said I was wrong, but the diff shows the guard is present." + )) + + def test_no_hit_third_person_was_wrong(self): + self.assertIsNone(detect.find_admission("He was wrong about the pool, not me.")) + + def test_no_hit_bare_i_was_wrong_inside_code_fence(self): + self.assertIsNone(detect.find_admission( + "Here is the shape:\n```\nI was wrong - it is genuinely computing.\n```\n" + "That is what fires." + )) + + def test_no_hit_bare_i_was_wrong_quoted(self): + self.assertIsNone(detect.find_admission( + 'The hook catches replies like "I was wrong" when they are asserted, ' + "not cited." + )) + + def test_no_hit_bare_i_was_wrong_backticked(self): + self.assertIsNone(detect.find_admission( + "The regex looks for `I was wrong` in assistant text." + )) + def test_no_hit_hypothetical_reversed_word_order(self): self.assertIsNone( detect.find_admission( From 98c297b327e90bc2844e413c744b7f1d7c097994 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 13:52:14 -0700 Subject: [PATCH 2/2] hedge-runs-prove-it: exempt the whole quote run, not just its first token A reply describing this hook to the user, quoting its own trigger words in prose, got blocked by it. `code_hedges()` tested the single character before the match, so a hedge was exempt only when it opened the quote: "should work" passed and "that should work" fired, in double quotes and backticks alike. Quote runs are now found up front and the match is skipped when it falls inside one. The runs are tested by containment and never stripped: CODE_NOUN_RE counts a backticked name as a code noun, and stripping the spans silences an existing positive fixture whose only code nouns are `claude.hook.json` and `Edit|Write|Bash` inside backticks. A single quote opens a run only when it is not an apostrophe -- no letter or digit on the inside edge -- so "it's", "don't" and "the workers' pool" keep their quote as a letter and a hedge beside one still fires. The old adjacent-character check is kept as a fallback so an unbalanced opening quote stays exempt exactly as before. The same three patterns now clean the diagnosis path too, which previously handled double quotes and backticks but not single ones. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- engine/hooks/hedge-runs-prove-it/README.md | 17 ++++++- engine/hooks/hedge-runs-prove-it/detect.py | 40 ++++++++++++++-- .../tests/fixtures/hedges_fires.json | 10 ++++ .../tests/fixtures/hedges_silent.json | 20 ++++++++ .../hedge-runs-prove-it/tests/test_hooks.py | 48 +++++++++++++++++++ 5 files changed, 130 insertions(+), 5 deletions(-) diff --git a/engine/hooks/hedge-runs-prove-it/README.md b/engine/hooks/hedge-runs-prove-it/README.md index bf61d954..27e7154b 100644 --- a/engine/hooks/hedge-runs-prove-it/README.md +++ b/engine/hooks/hedge-runs-prove-it/README.md @@ -14,6 +14,21 @@ sentence carries a cannot-verify reason ("cannot verify: no network", "would need the live token"), when the hedge is quoted, or when the hedge is about something that is not code or state (a company's motive). +## What counts as quoted + +A hedge anywhere inside a double-quoted, backticked or single-quoted run is +someone citing the word, not claiming it. The exemption covers the whole +run, not just its first token, so a reply explaining this hook to the user +-- `The gate fires on "that should work"` -- stays silent. + +Spans are tested by containment and never stripped out. `CODE_NOUN_RE` +counts a backticked name as a code noun, so deleting the span would take +the proximity signal with it and silence real hedges. + +A single quote opens a span only when it is not an apostrophe: `it's`, +`don't` and `the workers' pool` keep their quote as a letter, and a hedge +beside one still fires. + ## The unhedged half A hedge announces its own missing check. A confident diagnosis does not, so @@ -62,7 +77,7 @@ escape hatch, not a free pass). Fail-open on parse or read errors; - `claude_stop_check.py` -- Claude Stop entrypoint. - `claude.hook.json` / `install_claude_hook.py` -- settings.json merge (idempotent). - `tests/fixtures/hedges_{fires,silent}.json` -- sanitized real replies and - the rule's own shapes. + the rule's own shapes, including a hedge quoted mid-span. - `tests/fixtures/diagnosis_{fires,silent}.json` -- unhedged diagnosis claims, including the same claim shipped with its process table. - `tests/test_hooks.py` diff --git a/engine/hooks/hedge-runs-prove-it/detect.py b/engine/hooks/hedge-runs-prove-it/detect.py index 3f39d659..69f35f70 100644 --- a/engine/hooks/hedge-runs-prove-it/detect.py +++ b/engine/hooks/hedge-runs-prove-it/detect.py @@ -19,7 +19,9 @@ `file:line`, a pid, an exit code, or an explicit `UNVERIFIED:` prefix. Hedges about things that are not code or state (a company's motive, a -filing date) are out of scope, as are diagnoses inside a fence, a quote, a +filing date) are out of scope, and so is either shape quoted rather than +claimed -- anywhere inside a double-quoted, backticked or single-quoted run, +not merely as its first token -- along with diagnoses inside a fence, a blockquote, or a hypothetical. Judgment stays with the model; this file matches shapes and fails open. """ @@ -90,6 +92,8 @@ BLOCKQUOTE_RE = re.compile(r"(?m)^\s*>.*$") DOUBLE_QUOTE_RE = re.compile(r'"[^"]*"', re.DOTALL) BACKTICK_RE = re.compile(r"`[^`]*`", re.DOTALL) +SINGLE_QUOTE_RE = re.compile(r"(? str: return text[start:end] +def quoted_spans(text: str) -> list[tuple[int, int]]: + """Character ranges of double-quoted, backticked and single-quoted runs. + + A single quote only opens a span when it is not an apostrophe: "it's", + "don't" and "the workers' pool" all keep their quote as a letter, so a + hedge beside one is not exempt on that basis. + """ + spans: list[tuple[int, int]] = [] + for pattern in QUOTE_SPAN_RES: + spans.extend((m.start(), m.end()) for m in pattern.finditer(text or "")) + return spans + + +def _inside_quoted_span(position: int, spans: list[tuple[int, int]]) -> bool: + return any(start <= position < end for start, end in spans) + + def code_hedges(text: str) -> list[str]: - """Hedge phrases near a code noun that lack a cannot-verify reason.""" + """Hedge phrases near a code noun that lack a cannot-verify reason. + + A hedge anywhere inside a quoted run is someone citing the word, not + claiming it, so the whole span is exempt rather than only its first + token. The span is tested by containment and never stripped: a + backticked name is itself a code noun, so removing the span would take + the proximity signal with it. + """ hits: list[str] = [] + spans = quoted_spans(text or "") for match in HEDGE_RE.finditer(text or ""): + if _inside_quoted_span(match.start(), spans): + continue if match.start() and text[match.start() - 1] in QUOTED_BEFORE: continue window = text[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY] @@ -135,8 +166,9 @@ def code_hedges(text: str) -> list[str]: def _diagnosis_text(text: str) -> str: cleaned = FENCE_RE.sub(" ", text or "") cleaned = BLOCKQUOTE_RE.sub(" ", cleaned) - cleaned = DOUBLE_QUOTE_RE.sub(" ", cleaned) - return BACKTICK_RE.sub(" ", cleaned) + for pattern in QUOTE_SPAN_RES: + cleaned = pattern.sub(" ", cleaned) + return cleaned def diagnosis_claims(text: str) -> list[str]: diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json index 1caafd09..0c842d0b 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json @@ -18,5 +18,15 @@ "label": "probably about a merge state, nothing run", "reply": "#229 is probably merged by now since #228 landed and Mergify chains them.", "verified": false + }, + { + "label": "unquoted mid-sentence hedge, no quote run anywhere near it", + "reply": "I might send you the matcher change near a PR; it should work for the heredoc case in claude.hook.json too.", + "verified": false + }, + { + "label": "apostrophes are letters, not quote delimiters, so the hedge is bare", + "reply": "It's probably still on the branch and I don't think the build ran.", + "verified": false } ] diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json index a51c254b..30705d8e 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json @@ -23,5 +23,25 @@ "label": "quoted hedge phrase, describing the rule itself", "reply": "The hook fires on a reply containing \"I think\" or `UNVERIFIED:` with no verification command in the same turn.", "verified": false + }, + { + "label": "hedge quoted mid-span, not as the quote's first token (real session)", + "reply": "I might send you \"that should work\" near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "same mid-span hedge inside backticks", + "reply": "I might send you `that should work` near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "same mid-span hedge inside single quotes", + "reply": "I might send you 'that should work' near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "describing the hook's own triggers to the user, every one of them quoted", + "reply": "The gate fires on `I think`, \"that should work\", or a bare `UNVERIFIED:` next to a code noun in the same PR reply.", + "verified": false } ] diff --git a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py index 5d7775ae..8f8c6ad4 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py +++ b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py @@ -116,6 +116,54 @@ def test_fails_open_on_garbage_stdin(self): self.assertEqual(err.getvalue(), "") +class TestQuotedHedgeSpans(unittest.TestCase): + """A hedge cited inside a quote run is exempt wherever it sits in the run, + not only when it is the run's first token.""" + + def test_blocks_unquoted_mid_sentence_hedge(self): + self.assertEqual( + detect.code_hedges( + "I might send you the matcher change near a PR; it should work " + "for the heredoc case in claude.hook.json too." + ), + ["should work"], + ) + + def test_no_hit_hedge_quoted_past_the_first_token(self): + for quote in ('"that should work"', "`that should work`", "'that should work'"): + with self.subTest(quote=quote): + self.assertEqual( + detect.code_hedges(f"I might send you {quote} near a PR."), [] + ) + + def test_no_hit_when_describing_the_hooks_own_triggers(self): + self.assertEqual( + detect.code_hedges( + 'The gate fires on `I think`, "that should work", or a bare ' + "`UNVERIFIED:` next to a code noun in the same PR reply." + ), + [], + ) + + def test_blocks_hedge_when_apostrophes_are_the_only_single_quotes(self): + self.assertEqual( + detect.code_hedges("It's probably still on the branch and I don't think the build ran."), + ["probably"], + ) + + def test_no_hit_apostrophe_does_not_open_a_span_over_a_possessive(self): + self.assertEqual( + detect.code_hedges("The workers' pool probably still holds the stale commit."), + ["probably"], + ) + + def test_no_hit_after_an_unbalanced_opening_quote(self): + self.assertEqual(detect.code_hedges('He said "should work for the test suite'), []) + + def test_quoted_spans_ignores_apostrophes(self): + self.assertEqual(detect.quoted_spans("it's, don't, the workers' pool"), []) + + class TestBlocksUnhedgedDiagnosis(unittest.TestCase): def test_blocks_each_diagnosis_fixture(self): for case in load("diagnosis_fires.json"):