diff --git a/engine/hooks/hedge-runs-prove-it/README.md b/engine/hooks/hedge-runs-prove-it/README.md index be7add03..27e7154b 100644 --- a/engine/hooks/hedge-runs-prove-it/README.md +++ b/engine/hooks/hedge-runs-prove-it/README.md @@ -14,6 +14,58 @@ sentence carries a cannot-verify reason ("cannot verify: no network", "would need the live token"), when the hedge is quoted, or when the hedge is about something that is not code or state (a company's motive). +## What counts as quoted + +A hedge anywhere inside a double-quoted, backticked or single-quoted run is +someone citing the word, not claiming it. The exemption covers the whole +run, not just its first token, so a reply explaining this hook to the user +-- `The gate fires on "that should work"` -- stays silent. + +Spans are tested by containment and never stripped out. `CODE_NOUN_RE` +counts a backticked name as a code noun, so deleting the span would take +the proximity signal with it and silence real hedges. + +A single quote opens a span only when it is not an apostrophe: `it's`, +`don't` and `the workers' pool` keep their quote as a letter, and a hedge +beside one still fires. + +## The unhedged half + +A hedge announces its own missing check. A confident diagnosis does not, so +it is the more dangerous shape and it used to pass this gate untouched. The +hook also blocks a reply that asserts live system state -- "it's a zombie", +"that's the bug", "the root cause is X", "the worker is hung", "this is why +it's slow" -- within 200 characters of a runtime noun (process, pid, worker, +task, pool, queue, slot, workflow, socket, lock, CPU, disk) and carries no +instrument-level proof in the same message. + +Running a tool in the turn does not clear this one; a projection that omits +a field is not proof the state is absent. Only same-message proof clears it: +a fenced block of real output, a `file:line`, a pid, an exit code, a +`/proc/` path, or an explicit `UNVERIFIED:` prefix. + +Stays silent on a diagnosis inside a fence, a double-quoted span, a backtick +span, a markdown blockquote, a hypothetical ("if it's a zombie, ..."), and +past-tense narration of an old incident ("the task was a zombie"). + +Mechanical half of `corpus/skills/cat-mode/SKILL.md`'s Verify rule: +"Unhedged root-cause or fix claims about live system behavior need +instrument-level proof in the same message, or `UNVERIFIED:`." Four +independent conditions must hold before it blocks, because a Stop hook's +effective false-positive rate is what decides whether anyone keeps it on +(Sadowski et al., "Lessons from Building Static Analysis Tools at Google," +CACM 61(4), 2018). + +`diu-stop` has a narrower causal closer of its own ("the cause is", +"because"). It stays silent on the copula shape, and its paragraph check +skips any paragraph containing an inline backtick, so it is not the lever +for this class. + +No `agent_id` guard: `auto-pr` and `frustration-watchdog` carry one because +they act on the whole session's behalf, which a subagent must not do. This +hook only blocks the offending reply and tells that same agent to go get +evidence -- as correct inside a subagent as outside it. + Mechanical half of the evidence rules in `engine/CLAUDE.core.md` ("never claim ... without evidence in the SAME message"; `UNVERIFIED:` is the escape hatch, not a free pass). Fail-open on parse or read errors; @@ -25,7 +77,9 @@ escape hatch, not a free pass). Fail-open on parse or read errors; - `claude_stop_check.py` -- Claude Stop entrypoint. - `claude.hook.json` / `install_claude_hook.py` -- settings.json merge (idempotent). - `tests/fixtures/hedges_{fires,silent}.json` -- sanitized real replies and - the rule's own shapes. + the rule's own shapes, including a hedge quoted mid-span. +- `tests/fixtures/diagnosis_{fires,silent}.json` -- unhedged diagnosis + claims, including the same claim shipped with its process table. - `tests/test_hooks.py` Tests: `python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v` diff --git a/engine/hooks/hedge-runs-prove-it/detect.py b/engine/hooks/hedge-runs-prove-it/detect.py index 863281f0..69f35f70 100644 --- a/engine/hooks/hedge-runs-prove-it/detect.py +++ b/engine/hooks/hedge-runs-prove-it/detect.py @@ -1,14 +1,29 @@ -"""hedge-runs-prove-it: a hedge about code or repo state is a prompt to verify. - -"I think", "I believe", "probably", "should work", "presumably", or an -`UNVERIFIED:` prefix next to a code noun (a path, a backticked name, test, -CI, build, bug, fix, script, hook, PR, merge, branch, commit) means the -agent has a check it has not run. The reply passes only when the turn ran -a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:` clause -says why it cannot be verified ("cannot verify: no network"). Hedges about -things that are not code or state (a company's motive, a filing date) -are out of scope. Judgment stays with the model; this file matches -shapes and fails open. +"""hedge-runs-prove-it: an unrun check about code or repo state is a prompt to verify. + +Two shapes, two bars. + +A hedge -- "I think", "I believe", "probably", "should work", "presumably", +or an `UNVERIFIED:` prefix next to a code noun (a path, a backticked name, +test, CI, build, bug, fix, script, hook, PR, merge, branch, commit) -- +means the agent has a check it has not run. The reply passes only when the +turn ran a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:` +clause says why it cannot be verified ("cannot verify: no network"). + +An unhedged diagnosis -- "it's a zombie", "that's the bug", "the root cause +is X", "the worker is hung", "this is why it's slow" -- asserts live system +state with no hedge word at all, so the hedge bar never sees it. Confidence +is the more dangerous shape, not the safer one: the claim carries no signal +that a check is outstanding. Running a tool in the turn does not clear it, +because a projection that omits a field is not proof the state is absent. +Only instrument-level proof in the same message clears it: pasted output, a +`file:line`, a pid, an exit code, or an explicit `UNVERIFIED:` prefix. + +Hedges about things that are not code or state (a company's motive, a +filing date) are out of scope, and so is either shape quoted rather than +claimed -- anywhere inside a double-quoted, backticked or single-quoted run, +not merely as its first token -- along with diagnoses inside a fence, a +blockquote, or a hypothetical. Judgment stays with the model; this file +matches shapes and fails open. """ from __future__ import annotations @@ -37,12 +52,65 @@ VERIFY_TOOLS = {"Bash", "Read", "Grep", "Glob", "Monitor", "WebFetch"} QUOTED_BEFORE = ('"', "'", "`") +DIAGNOSIS_RE = re.compile( + r"(?:\b(?:it'?s|that'?s|this\s+is|they'?re|the\s+[\w-]+(?:\s+[\w-]+)?\s+is)\s+" + r"(?:just\s+|simply\s+|actually\s+|basically\s+|clearly\s+|a\s+|an\s+|the\s+)*" + r"(?:zombie|hung|hanging|stuck|wedged|deadlocked|dead(?!\s+code)|crashed|" + r"leaking|thrashing|starved|orphaned|frozen|spinning|silently\s+failing|" + r"corrupt(?:ed)?|misconfigured)\b)|" + r"(?:\b(?:that'?s|this\s+is|here'?s)\s+(?:the|your|our|my)\s+" + r"(?:bug|root\s+cause|cause|culprit|problem|issue|failure)\b)|" + r"(?:\bthe\s+(?:root\s+)?cause\s+is\b)|" + r"(?:\bthe\s+reason\s+is\b)|" + r"(?:\bwhat'?s\s+(?:happening|going\s+on)\s+is\b)|" + r"(?:\b(?:that|this)(?:'?s|\s+is)\s+why\s+(?:it|its|it'?s|the|they|that|this)\b)", + re.IGNORECASE, +) +RUNTIME_NOUN_RE = re.compile( + r"\b(?:process(?:es)?|pids?|threads?|workers?|tasks?|jobs?|runs?|runners?|" + r"daemons?|services?|servers?|hosts?|containers?|pods?|nodes?|pools?|" + r"queues?|slots?|workflows?|sessions?|agents?|sockets?|ports?|" + r"connections?|locks?|loops?|requests?|cpu|memory|disk|cache|database|db|" + r"quer(?:y|ies)|logs?|streams?|builds?|tests?|suites?|CI|pipelines?|" + r"hooks?|scripts?|commands?|replay)\b", + re.IGNORECASE, +) +INSTRUMENT_EVIDENCE_RE = re.compile( + r"```|\bUNVERIFIED:|/proc/\d+|" + r"\b[\w./-]+\.[A-Za-z]{1,6}:\d+\b|" + r"\bpids?\b\s*[:=#]?\s*\d+|\bMainPID\b|" + r"\bexit\s+(?:code|status)\b|\bexit[_-]?code\b", + re.IGNORECASE, +) +DIAGNOSIS_LOOKBACK_RE = re.compile( + r"\b(?:if|unless|whether|suppose|assuming|in case|maybe|perhaps|" + r"might\s+be|could\s+be|may\s+be|not\s+sure|unclear)\b[^.!?\n]*$", + re.IGNORECASE, +) +LOOKBACK = 120 +FENCE_RE = re.compile(r"```.*?```", re.DOTALL) +BLOCKQUOTE_RE = re.compile(r"(?m)^\s*>.*$") +DOUBLE_QUOTE_RE = re.compile(r'"[^"]*"', re.DOTALL) +BACKTICK_RE = re.compile(r"`[^`]*`", re.DOTALL) +SINGLE_QUOTE_RE = re.compile(r"(? -- cannot verify: `." ) +DIAGNOSIS_MESSAGE = ( + "hedge-runs-prove-it: this reply asserts a diagnosis about live system state " + "({claim}) with no instrument-level proof in the same message. An unhedged " + "root-cause claim is the dangerous shape, not the hedged one -- nothing in it " + "signals an outstanding check -- and a projection that omits a field is not " + "proof the state is absent. Attach the instrument output here: pasted ps / " + "strace / /proc output, a live query's real result, a pid, an exit code, or a " + "file:line. Otherwise prefix the claim with `UNVERIFIED:`." +) + def _sentence_after(text: str, start: int) -> str: end = len(text) @@ -53,10 +121,37 @@ def _sentence_after(text: str, start: int) -> str: return text[start:end] +def quoted_spans(text: str) -> list[tuple[int, int]]: + """Character ranges of double-quoted, backticked and single-quoted runs. + + A single quote only opens a span when it is not an apostrophe: "it's", + "don't" and "the workers' pool" all keep their quote as a letter, so a + hedge beside one is not exempt on that basis. + """ + spans: list[tuple[int, int]] = [] + for pattern in QUOTE_SPAN_RES: + spans.extend((m.start(), m.end()) for m in pattern.finditer(text or "")) + return spans + + +def _inside_quoted_span(position: int, spans: list[tuple[int, int]]) -> bool: + return any(start <= position < end for start, end in spans) + + def code_hedges(text: str) -> list[str]: - """Hedge phrases near a code noun that lack a cannot-verify reason.""" + """Hedge phrases near a code noun that lack a cannot-verify reason. + + A hedge anywhere inside a quoted run is someone citing the word, not + claiming it, so the whole span is exempt rather than only its first + token. The span is tested by containment and never stripped: a + backticked name is itself a code noun, so removing the span would take + the proximity signal with it. + """ hits: list[str] = [] + spans = quoted_spans(text or "") for match in HEDGE_RE.finditer(text or ""): + if _inside_quoted_span(match.start(), spans): + continue if match.start() and text[match.start() - 1] in QUOTED_BEFORE: continue window = text[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY] @@ -68,6 +163,30 @@ def code_hedges(text: str) -> list[str]: return hits +def _diagnosis_text(text: str) -> str: + cleaned = FENCE_RE.sub(" ", text or "") + cleaned = BLOCKQUOTE_RE.sub(" ", cleaned) + for pattern in QUOTE_SPAN_RES: + cleaned = pattern.sub(" ", cleaned) + return cleaned + + +def diagnosis_claims(text: str) -> list[str]: + """Unhedged diagnoses of live system state carrying no same-message proof.""" + if not text or INSTRUMENT_EVIDENCE_RE.search(text): + return [] + cleaned = _diagnosis_text(text) + hits: list[str] = [] + for match in DIAGNOSIS_RE.finditer(cleaned): + window = cleaned[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY] + if not RUNTIME_NOUN_RE.search(window): + continue + if DIAGNOSIS_LOOKBACK_RE.search(cleaned[max(0, match.start() - LOOKBACK): match.start()]): + continue + hits.append(match.group(0)) + return hits + + def _text_content(data: dict) -> str: message = data.get("message") content = message.get("content") if isinstance(message, dict) else data.get("content") @@ -117,7 +236,17 @@ def verified_this_turn(lines: list[dict]) -> bool: return False +def _diagnosis_feedback(message: str) -> str | None: + claims = diagnosis_claims(message) + if not claims: + return None + return DIAGNOSIS_MESSAGE.format(claim=", ".join(f'"{c}"' for c in claims[:3])) + + def decide_from_lines(message: str, lines: list[dict]) -> str | None: + diagnosis = _diagnosis_feedback(message) + if diagnosis: + return diagnosis hedges = code_hedges(message) if not hedges: return None @@ -131,6 +260,9 @@ def decide(payload: dict) -> str | None: if payload.get("stop_hook_active"): return None message = payload.get("last_assistant_message") or "" + diagnosis = _diagnosis_feedback(message) + if diagnosis: + return diagnosis if not code_hedges(message): return None transcript_path = payload.get("transcript_path") or payload.get("transcriptPath") or "" diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json new file mode 100644 index 00000000..27cfda30 --- /dev/null +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json @@ -0,0 +1,32 @@ +[ + { + "label": "zombie-vs-slow root cause read off a capacity projection, no process table (real session)", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the\ntask claims `running`. Checking whether the replay process actually exists:", + "verified": true + }, + { + "label": "same claim with the status word unbackticked", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the task claims running. Checking whether the replay process actually exists:", + "verified": true + }, + { + "label": "that's the bug, asserted about a worker", + "reply": "That's the bug: the worker never re-registers its slot after a restart.", + "verified": true + }, + { + "label": "root cause named for a queue that has not been instrumented", + "reply": "The root cause is the poll loop holding the lock while the queue drains.", + "verified": false + }, + { + "label": "process called hung with no second sample", + "reply": "The replay process is hung on the socket read.", + "verified": true + }, + { + "label": "this is why it's slow, asserted about a pool", + "reply": "This is why it's slow -- the pool serializes every task through one worker.", + "verified": false + } +] diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json new file mode 100644 index 00000000..57965200 --- /dev/null +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json @@ -0,0 +1,52 @@ +[ + { + "label": "same zombie claim, but the process table ships in the same message", + "reply": "**It's a zombie, not slow.** The pool shows zero slots in use.\n\n```\nUID PID %CPU STAT ELAPSED CMD\nme 4412 96.2 R 11:19:03 node replay-worker.js\n```", + "verified": true + }, + { + "label": "hypothetical diagnosis, still on the way to the check", + "reply": "If it's a zombie, the pool would still show the slot free, so I am reading the process table next.", + "verified": false + }, + { + "label": "quoting the user's own diagnosis back", + "reply": "You said \"it's a zombie, not slow\" -- reading the process table now.", + "verified": false + }, + { + "label": "the claim shape shown inside a fence, not asserted", + "reply": "The detector fires on this shape:\n\n```\nIt's a zombie, not slow. The pool shows zero slots.\n```\n", + "verified": false + }, + { + "label": "diagnosis explicitly prefixed UNVERIFIED", + "reply": "UNVERIFIED: it's a zombie, not slow -- the pool shows zero slots but I have not looked at the process table.", + "verified": true + }, + { + "label": "past incident narrated, not a live claim", + "reply": "Yesterday the pool showed zero slots and the task was a zombie; that run was killed by hand.", + "verified": false + }, + { + "label": "describing the rule itself in backticks", + "reply": "The gate fires when a reply says `it's a zombie` with no process-table output in the same message.", + "verified": false + }, + { + "label": "user's line pasted as a blockquote", + "reply": "> It's a zombie, not slow. The pool shows zero slots.\n\nThat is the line to check, so I am pulling the process table.", + "verified": false + }, + { + "label": "ordinary because-clause about the agent's own sequencing", + "reply": "That's why I ran the test suite before touching the worker pool.", + "verified": false + }, + { + "label": "diagnosis carrying a file:line instead of pasted output", + "reply": "That's the bug: the worker slot is released before the task row is updated, at packages/pool/slots.py:212.", + "verified": false + } +] diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json index 1caafd09..0c842d0b 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json @@ -18,5 +18,15 @@ "label": "probably about a merge state, nothing run", "reply": "#229 is probably merged by now since #228 landed and Mergify chains them.", "verified": false + }, + { + "label": "unquoted mid-sentence hedge, no quote run anywhere near it", + "reply": "I might send you the matcher change near a PR; it should work for the heredoc case in claude.hook.json too.", + "verified": false + }, + { + "label": "apostrophes are letters, not quote delimiters, so the hedge is bare", + "reply": "It's probably still on the branch and I don't think the build ran.", + "verified": false } ] diff --git a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json index a51c254b..30705d8e 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json +++ b/engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_silent.json @@ -23,5 +23,25 @@ "label": "quoted hedge phrase, describing the rule itself", "reply": "The hook fires on a reply containing \"I think\" or `UNVERIFIED:` with no verification command in the same turn.", "verified": false + }, + { + "label": "hedge quoted mid-span, not as the quote's first token (real session)", + "reply": "I might send you \"that should work\" near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "same mid-span hedge inside backticks", + "reply": "I might send you `that should work` near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "same mid-span hedge inside single quotes", + "reply": "I might send you 'that should work' near a PR to show what the gate catches.", + "verified": false + }, + { + "label": "describing the hook's own triggers to the user, every one of them quoted", + "reply": "The gate fires on `I think`, \"that should work\", or a bare `UNVERIFIED:` next to a code noun in the same PR reply.", + "verified": false } ] diff --git a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py index 8be71333..8f8c6ad4 100644 --- a/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py +++ b/engine/hooks/hedge-runs-prove-it/tests/test_hooks.py @@ -3,10 +3,13 @@ Run: python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v -Fixtures in tests/fixtures/ are sanitized replies from one real session -("presumably moving it to catstack" with no check run) plus the shapes the -rule names. Each fixture carries `verified`: whether the turn ran a -verification tool. +Fixtures in tests/fixtures/ are sanitized replies from real sessions +("presumably moving it to catstack" with no check run; "it's a zombie, not +slow" asserted off a capacity projection while ten workers were at 96% CPU) +plus the shapes the rule names. Each fixture carries `verified`: whether the +turn ran a verification tool. The diagnosis fixtures set it true on purpose +-- an unhedged root-cause claim is not cleared by having run a tool, only by +instrument-level proof in the same message. """ from __future__ import annotations @@ -113,5 +116,107 @@ def test_fails_open_on_garbage_stdin(self): self.assertEqual(err.getvalue(), "") +class TestQuotedHedgeSpans(unittest.TestCase): + """A hedge cited inside a quote run is exempt wherever it sits in the run, + not only when it is the run's first token.""" + + def test_blocks_unquoted_mid_sentence_hedge(self): + self.assertEqual( + detect.code_hedges( + "I might send you the matcher change near a PR; it should work " + "for the heredoc case in claude.hook.json too." + ), + ["should work"], + ) + + def test_no_hit_hedge_quoted_past_the_first_token(self): + for quote in ('"that should work"', "`that should work`", "'that should work'"): + with self.subTest(quote=quote): + self.assertEqual( + detect.code_hedges(f"I might send you {quote} near a PR."), [] + ) + + def test_no_hit_when_describing_the_hooks_own_triggers(self): + self.assertEqual( + detect.code_hedges( + 'The gate fires on `I think`, "that should work", or a bare ' + "`UNVERIFIED:` next to a code noun in the same PR reply." + ), + [], + ) + + def test_blocks_hedge_when_apostrophes_are_the_only_single_quotes(self): + self.assertEqual( + detect.code_hedges("It's probably still on the branch and I don't think the build ran."), + ["probably"], + ) + + def test_no_hit_apostrophe_does_not_open_a_span_over_a_possessive(self): + self.assertEqual( + detect.code_hedges("The workers' pool probably still holds the stale commit."), + ["probably"], + ) + + def test_no_hit_after_an_unbalanced_opening_quote(self): + self.assertEqual(detect.code_hedges('He said "should work for the test suite'), []) + + def test_quoted_spans_ignores_apostrophes(self): + self.assertEqual(detect.quoted_spans("it's, don't, the workers' pool"), []) + + +class TestBlocksUnhedgedDiagnosis(unittest.TestCase): + def test_blocks_each_diagnosis_fixture(self): + for case in load("diagnosis_fires.json"): + with self.subTest(label=case["label"]): + self.assertIsNotNone( + detect.decide_from_lines(case["reply"], turn_lines(case["verified"])) + ) + + def test_blocks_zombie_claim_even_though_the_turn_ran_a_tool(self): + case = load("diagnosis_fires.json")[0] + self.assertTrue(detect.diagnosis_claims(case["reply"])) + self.assertEqual(detect.code_hedges(case["reply"]), []) + self.assertIsNotNone(detect.decide_from_lines(case["reply"], turn_lines(True))) + + def test_hook_blocks_zombie_claim_with_exit_2(self): + case = load("diagnosis_fires.json")[0] + path = transcript_file(turn_lines(True)) + try: + code, err = run_hook({"last_assistant_message": case["reply"], "transcript_path": path}) + finally: + os.unlink(path) + self.assertEqual(code, 2) + self.assertIn("instrument-level proof", err) + self.assertIn("zombie", err) + + def test_blocks_diagnosis_with_no_transcript_at_all(self): + case = load("diagnosis_fires.json")[2] + self.assertIsNotNone(detect.decide({"last_assistant_message": case["reply"]})) + + +class TestAllowsProvenOrQuotedDiagnosis(unittest.TestCase): + def test_allows_each_silent_diagnosis_fixture(self): + for case in load("diagnosis_silent.json"): + with self.subTest(label=case["label"]): + self.assertIsNone( + detect.decide_from_lines(case["reply"], turn_lines(case["verified"])) + ) + + def test_no_hit_when_the_process_table_ships_in_the_same_message(self): + self.assertEqual(detect.diagnosis_claims(load("diagnosis_silent.json")[0]["reply"]), []) + + def test_allows_diagnosis_when_stop_hook_active(self): + self.assertIsNone(detect.decide({ + "last_assistant_message": load("diagnosis_fires.json")[0]["reply"], + "stop_hook_active": True, + })) + + def test_diagnosis_gate_ignores_an_unreadable_transcript(self): + self.assertIsNotNone(detect.decide({ + "last_assistant_message": load("diagnosis_fires.json")[0]["reply"], + "transcript_path": "/nonexistent/x.jsonl", + })) + + if __name__ == "__main__": unittest.main() diff --git a/engine/hooks/wrong-check-reflect/README.md b/engine/hooks/wrong-check-reflect/README.md index bf877c47..7a71bf15 100644 --- a/engine/hooks/wrong-check-reflect/README.md +++ b/engine/hooks/wrong-check-reflect/README.md @@ -4,7 +4,15 @@ When the assistant admits a prior check/claim was wrong ("Good catch — my earlier check was wrong", "You're right, I misread the file", "I incorrectly assumed…", "the file I cited was a duplicate", "My mistake — I misread it", "I misread the front matter on that skill"), inject a -`/reflect` follow-up. Finish the live correction first. Fail-open. +`/reflect` follow-up. + +A bare "I was wrong" counts, with no named check after it. The retraction +that follows a false claim is often the shortest sentence in the turn, and +requiring it to name the check it retracts let the plainest concession +through. The hypothetical ("if I was wrong about this…"), reported-speech +("the reviewer said I was wrong"), product-blame ("the test was wrong"), +third-person, quote, backtick and fence guards all still hold, so only an +admission asserted in the agent's own voice fires. Finish the live correction first. Fail-open. Once per transcript. Skip if the user already said `/reflect`. Not word-count (`diu-stop`). Not token_audit thrash (`reflect-on-thrash`). diff --git a/engine/hooks/wrong-check-reflect/detect.py b/engine/hooks/wrong-check-reflect/detect.py index c41aa795..465d1d98 100644 --- a/engine/hooks/wrong-check-reflect/detect.py +++ b/engine/hooks/wrong-check-reflect/detect.py @@ -1,5 +1,12 @@ """Detect first-person “my earlier check was wrong” admissions. +A bare “I was wrong” counts. The retraction that follows a false claim is +often the shortest sentence in the turn, and the earlier requirement that it +name the check it retracts let the plainest concession through. The +hypothetical, product-blame, reported-speech, quote and fence guards below +still hold, so only a first-person admission asserted in the agent's own +voice fires. + Assistant text only. Fail-open: parse/IO errors mean no hit. Once per transcript. Skip if the user already asked /reflect. """ @@ -48,6 +55,9 @@ r"(?i)\bi\s+(was\s+wrong|got\s+it\s+wrong)\s+(about|on)\s+" r"(the\s+)?(file|path|source|check|assumption)\b" ), + re.compile( + r"(?i)\bi\s+(?:was\s+wrong|got\s+(?:it|that|this)\s+wrong)\b" + ), re.compile( r"(?i)\bi\s+(read|got|took|marked|logged|noted)\s+(that|this|it)\s+wrong\s+" r"in\s+my\s+(earlier|previous|prior)\s+\w+" @@ -87,6 +97,14 @@ re.compile(r"(?i)\bthe\s+(test|ui|build|product|code)\s+was\s+wrong\b"), re.compile(r"(?i)\bif\b.{0,40}\bmy\s+mistake\b"), re.compile(r"(?i)\bif\s+i\s+misread\s+(it|that|this|the)\b"), + re.compile( + r"(?i)\b(?:if|unless|whether|in\s+case|suppose|assuming)\s+i\s+" + r"(?:was|were)\s+wrong\b" + ), + re.compile( + r"(?i)\b(?:says?|said|thinks?|thought|claims?|claimed|argued|insisted|" + r"told\s+me|telling\s+me)\s+(?:that\s+)?i\s+(?:was|were)\s+wrong\b" + ), ] FOLLOWUP = ( diff --git a/engine/hooks/wrong-check-reflect/tests/test_hooks.py b/engine/hooks/wrong-check-reflect/tests/test_hooks.py index b27d876a..d6bf2dbd 100644 --- a/engine/hooks/wrong-check-reflect/tests/test_hooks.py +++ b/engine/hooks/wrong-check-reflect/tests/test_hooks.py @@ -191,6 +191,57 @@ def test_no_hit_normal_correction_language(self): ) ) + def test_hit_bare_i_was_wrong(self): + match = detect.find_admission("I was wrong. The pool never tracked that slot.") + self.assertIsNotNone(match) + self.assertIn("i was wrong", match.lower()) + + def test_hit_bare_i_was_wrong_conceding_a_live_diagnosis(self): + self.assertIsNotNone(detect.find_admission( + "I was wrong - it **is** genuinely computing. 10 workers in R state " + "at ~96% CPU." + )) + + def test_hit_i_got_that_wrong(self): + self.assertIsNotNone( + detect.find_admission("I got that wrong -- the worker was live the whole time.") + ) + + def test_no_hit_hypothetical_bare_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "If I was wrong about this, then the pool would show a free slot." + )) + + def test_no_hit_unless_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "Unless I was wrong about the ordering, the queue drains first." + )) + + def test_no_hit_reported_speech_someone_said_i_was_wrong(self): + self.assertIsNone(detect.find_admission( + "The reviewer said I was wrong, but the diff shows the guard is present." + )) + + def test_no_hit_third_person_was_wrong(self): + self.assertIsNone(detect.find_admission("He was wrong about the pool, not me.")) + + def test_no_hit_bare_i_was_wrong_inside_code_fence(self): + self.assertIsNone(detect.find_admission( + "Here is the shape:\n```\nI was wrong - it is genuinely computing.\n```\n" + "That is what fires." + )) + + def test_no_hit_bare_i_was_wrong_quoted(self): + self.assertIsNone(detect.find_admission( + 'The hook catches replies like "I was wrong" when they are asserted, ' + "not cited." + )) + + def test_no_hit_bare_i_was_wrong_backticked(self): + self.assertIsNone(detect.find_admission( + "The regex looks for `I was wrong` in assistant text." + )) + def test_no_hit_hypothetical_reversed_word_order(self): self.assertIsNone( detect.find_admission(