hooks: gate the confident wrong diagnosis, and the bare retraction

engine/hooks/hedge-runs-prove-it/README.md

@@ -14,6 +14,58 @@ sentence carries a cannot-verify reason ("cannot verify: no network",
 "would need the live token"), when the hedge is quoted, or when the hedge
 is about something that is not code or state (a company's motive).
 
+## What counts as quoted
+
+A hedge anywhere inside a double-quoted, backticked or single-quoted run is
+someone citing the word, not claiming it. The exemption covers the whole
+run, not just its first token, so a reply explaining this hook to the user
+-- `The gate fires on "that should work"` -- stays silent.
+
+Spans are tested by containment and never stripped out. `CODE_NOUN_RE`
+counts a backticked name as a code noun, so deleting the span would take
+the proximity signal with it and silence real hedges.
+
+A single quote opens a span only when it is not an apostrophe: `it's`,
+`don't` and `the workers' pool` keep their quote as a letter, and a hedge
+beside one still fires.
+
+## The unhedged half
+
+A hedge announces its own missing check. A confident diagnosis does not, so
+it is the more dangerous shape and it used to pass this gate untouched. The
+hook also blocks a reply that asserts live system state -- "it's a zombie",
+"that's the bug", "the root cause is X", "the worker is hung", "this is why
+it's slow" -- within 200 characters of a runtime noun (process, pid, worker,
+task, pool, queue, slot, workflow, socket, lock, CPU, disk) and carries no
+instrument-level proof in the same message.
+
+Running a tool in the turn does not clear this one; a projection that omits
+a field is not proof the state is absent. Only same-message proof clears it:
+a fenced block of real output, a `file:line`, a pid, an exit code, a
+`/proc/<pid>` path, or an explicit `UNVERIFIED:` prefix.
+
+Stays silent on a diagnosis inside a fence, a double-quoted span, a backtick
+span, a markdown blockquote, a hypothetical ("if it's a zombie, ..."), and
+past-tense narration of an old incident ("the task was a zombie").
+
+Mechanical half of `corpus/skills/cat-mode/SKILL.md`'s Verify rule:
+"Unhedged root-cause or fix claims about live system behavior need
+instrument-level proof in the same message, or `UNVERIFIED:`." Four
+independent conditions must hold before it blocks, because a Stop hook's
+effective false-positive rate is what decides whether anyone keeps it on
+(Sadowski et al., "Lessons from Building Static Analysis Tools at Google,"
+CACM 61(4), 2018).
+
+`diu-stop` has a narrower causal closer of its own ("the cause is",
+"because"). It stays silent on the copula shape, and its paragraph check
+skips any paragraph containing an inline backtick, so it is not the lever
+for this class.
+
+No `agent_id` guard: `auto-pr` and `frustration-watchdog` carry one because
+they act on the whole session's behalf, which a subagent must not do. This
+hook only blocks the offending reply and tells that same agent to go get
+evidence -- as correct inside a subagent as outside it.
+
 Mechanical half of the evidence rules in `engine/CLAUDE.core.md` ("never
 claim ... without evidence in the SAME message"; `UNVERIFIED:` is the
 escape hatch, not a free pass). Fail-open on parse or read errors;
@@ -25,7 +77,9 @@ escape hatch, not a free pass). Fail-open on parse or read errors;
 - `claude_stop_check.py` -- Claude Stop entrypoint.
 - `claude.hook.json` / `install_claude_hook.py` -- settings.json merge (idempotent).
 - `tests/fixtures/hedges_{fires,silent}.json` -- sanitized real replies and
-  the rule's own shapes.
+  the rule's own shapes, including a hedge quoted mid-span.
+- `tests/fixtures/diagnosis_{fires,silent}.json` -- unhedged diagnosis
+  claims, including the same claim shipped with its process table.
 - `tests/test_hooks.py`
 
 Tests: `python3 -m unittest discover -s engine/hooks/hedge-runs-prove-it/tests -v`

engine/hooks/hedge-runs-prove-it/detect.py

@@ -1,14 +1,29 @@
-"""hedge-runs-prove-it: a hedge about code or repo state is a prompt to verify.
-
-"I think", "I believe", "probably", "should work", "presumably", or an
-`UNVERIFIED:` prefix next to a code noun (a path, a backticked name, test,
-CI, build, bug, fix, script, hook, PR, merge, branch, commit) means the
-agent has a check it has not run. The reply passes only when the turn ran
-a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:` clause
-says why it cannot be verified ("cannot verify: no network"). Hedges about
-things that are not code or state (a company's motive, a filing date)
-are out of scope. Judgment stays with the model; this file matches
-shapes and fails open.
+"""hedge-runs-prove-it: an unrun check about code or repo state is a prompt to verify.
+
+Two shapes, two bars.
+
+A hedge -- "I think", "I believe", "probably", "should work", "presumably",
+or an `UNVERIFIED:` prefix next to a code noun (a path, a backticked name,
+test, CI, build, bug, fix, script, hook, PR, merge, branch, commit) --
+means the agent has a check it has not run. The reply passes only when the
+turn ran a verification tool (Bash, Read, Grep, Glob) or the `UNVERIFIED:`
+clause says why it cannot be verified ("cannot verify: no network").
+
+An unhedged diagnosis -- "it's a zombie", "that's the bug", "the root cause
+is X", "the worker is hung", "this is why it's slow" -- asserts live system
+state with no hedge word at all, so the hedge bar never sees it. Confidence
+is the more dangerous shape, not the safer one: the claim carries no signal
+that a check is outstanding. Running a tool in the turn does not clear it,
+because a projection that omits a field is not proof the state is absent.
+Only instrument-level proof in the same message clears it: pasted output, a
+`file:line`, a pid, an exit code, or an explicit `UNVERIFIED:` prefix.
+
+Hedges about things that are not code or state (a company's motive, a
+filing date) are out of scope, and so is either shape quoted rather than
+claimed -- anywhere inside a double-quoted, backticked or single-quoted run,
+not merely as its first token -- along with diagnoses inside a fence, a
+blockquote, or a hypothetical. Judgment stays with the model; this file
+matches shapes and fails open.
 """
 from __future__ import annotations
 
@@ -37,12 +52,65 @@
 VERIFY_TOOLS = {"Bash", "Read", "Grep", "Glob", "Monitor", "WebFetch"}
 QUOTED_BEFORE = ('"', "'", "`")
 
+DIAGNOSIS_RE = re.compile(
+    r"(?:\b(?:it'?s|that'?s|this\s+is|they'?re|the\s+[\w-]+(?:\s+[\w-]+)?\s+is)\s+"
+    r"(?:just\s+|simply\s+|actually\s+|basically\s+|clearly\s+|a\s+|an\s+|the\s+)*"
+    r"(?:zombie|hung|hanging|stuck|wedged|deadlocked|dead(?!\s+code)|crashed|"
+    r"leaking|thrashing|starved|orphaned|frozen|spinning|silently\s+failing|"
+    r"corrupt(?:ed)?|misconfigured)\b)|"
+    r"(?:\b(?:that'?s|this\s+is|here'?s)\s+(?:the|your|our|my)\s+"
+    r"(?:bug|root\s+cause|cause|culprit|problem|issue|failure)\b)|"
+    r"(?:\bthe\s+(?:root\s+)?cause\s+is\b)|"
+    r"(?:\bthe\s+reason\s+is\b)|"
+    r"(?:\bwhat'?s\s+(?:happening|going\s+on)\s+is\b)|"
+    r"(?:\b(?:that|this)(?:'?s|\s+is)\s+why\s+(?:it|its|it'?s|the|they|that|this)\b)",
+    re.IGNORECASE,
+)
+RUNTIME_NOUN_RE = re.compile(
+    r"\b(?:process(?:es)?|pids?|threads?|workers?|tasks?|jobs?|runs?|runners?|"
+    r"daemons?|services?|servers?|hosts?|containers?|pods?|nodes?|pools?|"
+    r"queues?|slots?|workflows?|sessions?|agents?|sockets?|ports?|"
+    r"connections?|locks?|loops?|requests?|cpu|memory|disk|cache|database|db|"
+    r"quer(?:y|ies)|logs?|streams?|builds?|tests?|suites?|CI|pipelines?|"
+    r"hooks?|scripts?|commands?|replay)\b",
+    re.IGNORECASE,
+)
+INSTRUMENT_EVIDENCE_RE = re.compile(
+    r"```|\bUNVERIFIED:|/proc/\d+|"
+    r"\b[\w./-]+\.[A-Za-z]{1,6}:\d+\b|"
+    r"\bpids?\b\s*[:=#]?\s*\d+|\bMainPID\b|"
+    r"\bexit\s+(?:code|status)\b|\bexit[_-]?code\b",
+    re.IGNORECASE,
+)
+DIAGNOSIS_LOOKBACK_RE = re.compile(
+    r"\b(?:if|unless|whether|suppose|assuming|in case|maybe|perhaps|"
+    r"might\s+be|could\s+be|may\s+be|not\s+sure|unclear)\b[^.!?\n]*$",
+    re.IGNORECASE,
+)
+LOOKBACK = 120
+FENCE_RE = re.compile(r"```.*?```", re.DOTALL)
+BLOCKQUOTE_RE = re.compile(r"(?m)^\s*>.*$")
+DOUBLE_QUOTE_RE = re.compile(r'"[^"]*"', re.DOTALL)
+BACKTICK_RE = re.compile(r"`[^`]*`", re.DOTALL)
+SINGLE_QUOTE_RE = re.compile(r"(?<![A-Za-z0-9])'(?![\s'])[^'\n]*(?<!\s)'(?![A-Za-z0-9])")
+QUOTE_SPAN_RES = (DOUBLE_QUOTE_RE, BACKTICK_RE, SINGLE_QUOTE_RE)
+
 MESSAGE = (
     "hedge-runs-prove-it: this reply hedges about code or repo state ({hedge}) and "
     "the turn ran no verification (no Bash / Read / Grep). Run prove-it now: verify in "
     "this turn, or write `UNVERIFIED: <claim> -- cannot verify: <reason>`."
 )
 
+DIAGNOSIS_MESSAGE = (
+    "hedge-runs-prove-it: this reply asserts a diagnosis about live system state "
+    "({claim}) with no instrument-level proof in the same message. An unhedged "
+    "root-cause claim is the dangerous shape, not the hedged one -- nothing in it "
+    "signals an outstanding check -- and a projection that omits a field is not "
+    "proof the state is absent. Attach the instrument output here: pasted ps / "
+    "strace / /proc output, a live query's real result, a pid, an exit code, or a "
+    "file:line. Otherwise prefix the claim with `UNVERIFIED:`."
+)
+
 
 def _sentence_after(text: str, start: int) -> str:
     end = len(text)
@@ -53,10 +121,37 @@ def _sentence_after(text: str, start: int) -> str:
     return text[start:end]
 
 
+def quoted_spans(text: str) -> list[tuple[int, int]]:
+    """Character ranges of double-quoted, backticked and single-quoted runs.
+
+    A single quote only opens a span when it is not an apostrophe: "it's",
+    "don't" and "the workers' pool" all keep their quote as a letter, so a
+    hedge beside one is not exempt on that basis.
+    """
+    spans: list[tuple[int, int]] = []
+    for pattern in QUOTE_SPAN_RES:
+        spans.extend((m.start(), m.end()) for m in pattern.finditer(text or ""))
+    return spans
+
+
+def _inside_quoted_span(position: int, spans: list[tuple[int, int]]) -> bool:
+    return any(start <= position < end for start, end in spans)
+
+
 def code_hedges(text: str) -> list[str]:
-    """Hedge phrases near a code noun that lack a cannot-verify reason."""
+    """Hedge phrases near a code noun that lack a cannot-verify reason.
+
+    A hedge anywhere inside a quoted run is someone citing the word, not
+    claiming it, so the whole span is exempt rather than only its first
+    token. The span is tested by containment and never stripped: a
+    backticked name is itself a code noun, so removing the span would take
+    the proximity signal with it.
+    """
     hits: list[str] = []
+    spans = quoted_spans(text or "")
     for match in HEDGE_RE.finditer(text or ""):
+        if _inside_quoted_span(match.start(), spans):
+            continue
         if match.start() and text[match.start() - 1] in QUOTED_BEFORE:
             continue
         window = text[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY]
@@ -68,6 +163,30 @@ def code_hedges(text: str) -> list[str]:
     return hits
 
 
+def _diagnosis_text(text: str) -> str:
+    cleaned = FENCE_RE.sub(" ", text or "")
+    cleaned = BLOCKQUOTE_RE.sub(" ", cleaned)
+    for pattern in QUOTE_SPAN_RES:
+        cleaned = pattern.sub(" ", cleaned)
+    return cleaned
+
+
+def diagnosis_claims(text: str) -> list[str]:
+    """Unhedged diagnoses of live system state carrying no same-message proof."""
+    if not text or INSTRUMENT_EVIDENCE_RE.search(text):
+        return []
+    cleaned = _diagnosis_text(text)
+    hits: list[str] = []
+    for match in DIAGNOSIS_RE.finditer(cleaned):
+        window = cleaned[max(0, match.start() - PROXIMITY): match.end() + PROXIMITY]
+        if not RUNTIME_NOUN_RE.search(window):
+            continue
+        if DIAGNOSIS_LOOKBACK_RE.search(cleaned[max(0, match.start() - LOOKBACK): match.start()]):
+            continue
+        hits.append(match.group(0))
+    return hits
+
+
 def _text_content(data: dict) -> str:
     message = data.get("message")
     content = message.get("content") if isinstance(message, dict) else data.get("content")
@@ -117,7 +236,17 @@ def verified_this_turn(lines: list[dict]) -> bool:
     return False
 
 
+def _diagnosis_feedback(message: str) -> str | None:
+    claims = diagnosis_claims(message)
+    if not claims:
+        return None
+    return DIAGNOSIS_MESSAGE.format(claim=", ".join(f'"{c}"' for c in claims[:3]))
+
+
 def decide_from_lines(message: str, lines: list[dict]) -> str | None:
+    diagnosis = _diagnosis_feedback(message)
+    if diagnosis:
+        return diagnosis
     hedges = code_hedges(message)
     if not hedges:
         return None
@@ -131,6 +260,9 @@ def decide(payload: dict) -> str | None:
     if payload.get("stop_hook_active"):
         return None
     message = payload.get("last_assistant_message") or ""
+    diagnosis = _diagnosis_feedback(message)
+    if diagnosis:
+        return diagnosis
     if not code_hedges(message):
         return None
     transcript_path = payload.get("transcript_path") or payload.get("transcriptPath") or ""

engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_fires.json

@@ -0,0 +1,32 @@
+[
+  {
+    "label": "zombie-vs-slow root cause read off a capacity projection, no process table (real session)",
+    "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the\ntask claims `running`. Checking whether the replay process actually exists:",
+    "verified": true
+  },
+  {
+    "label": "same claim with the status word unbackticked",
+    "reply": "**It's a zombie, not slow.** The pool shows zero slots in use while the task claims running. Checking whether the replay process actually exists:",
+    "verified": true
+  },
+  {
+    "label": "that's the bug, asserted about a worker",
+    "reply": "That's the bug: the worker never re-registers its slot after a restart.",
+    "verified": true
+  },
+  {
+    "label": "root cause named for a queue that has not been instrumented",
+    "reply": "The root cause is the poll loop holding the lock while the queue drains.",
+    "verified": false
+  },
+  {
+    "label": "process called hung with no second sample",
+    "reply": "The replay process is hung on the socket read.",
+    "verified": true
+  },
+  {
+    "label": "this is why it's slow, asserted about a pool",
+    "reply": "This is why it's slow -- the pool serializes every task through one worker.",
+    "verified": false
+  }
+]

engine/hooks/hedge-runs-prove-it/tests/fixtures/diagnosis_silent.json

@@ -0,0 +1,52 @@
+[
+  {
+    "label": "same zombie claim, but the process table ships in the same message",
+    "reply": "**It's a zombie, not slow.** The pool shows zero slots in use.\n\n```\nUID   PID  %CPU STAT ELAPSED CMD\nme   4412  96.2 R    11:19:03 node replay-worker.js\n```",
+    "verified": true
+  },
+  {
+    "label": "hypothetical diagnosis, still on the way to the check",
+    "reply": "If it's a zombie, the pool would still show the slot free, so I am reading the process table next.",
+    "verified": false
+  },
+  {
+    "label": "quoting the user's own diagnosis back",
+    "reply": "You said \"it's a zombie, not slow\" -- reading the process table now.",
+    "verified": false
+  },
+  {
+    "label": "the claim shape shown inside a fence, not asserted",
+    "reply": "The detector fires on this shape:\n\n```\nIt's a zombie, not slow. The pool shows zero slots.\n```\n",
+    "verified": false
+  },
+  {
+    "label": "diagnosis explicitly prefixed UNVERIFIED",
+    "reply": "UNVERIFIED: it's a zombie, not slow -- the pool shows zero slots but I have not looked at the process table.",
+    "verified": true
+  },
+  {
+    "label": "past incident narrated, not a live claim",
+    "reply": "Yesterday the pool showed zero slots and the task was a zombie; that run was killed by hand.",
+    "verified": false
+  },
+  {
+    "label": "describing the rule itself in backticks",
+    "reply": "The gate fires when a reply says `it's a zombie` with no process-table output in the same message.",
+    "verified": false
+  },
+  {
+    "label": "user's line pasted as a blockquote",
+    "reply": "> It's a zombie, not slow. The pool shows zero slots.\n\nThat is the line to check, so I am pulling the process table.",
+    "verified": false
+  },
+  {
+    "label": "ordinary because-clause about the agent's own sequencing",
+    "reply": "That's why I ran the test suite before touching the worker pool.",
+    "verified": false
+  },
+  {
+    "label": "diagnosis carrying a file:line instead of pasted output",
+    "reply": "That's the bug: the worker slot is released before the task row is updated, at packages/pool/slots.py:212.",
+    "verified": false
+  }
+]

engine/hooks/hedge-runs-prove-it/tests/fixtures/hedges_fires.json

@@ -18,5 +18,15 @@
     "label": "probably about a merge state, nothing run",
     "reply": "#229 is probably merged by now since #228 landed and Mergify chains them.",
     "verified": false
+  },
+  {
+    "label": "unquoted mid-sentence hedge, no quote run anywhere near it",
+    "reply": "I might send you the matcher change near a PR; it should work for the heredoc case in claude.hook.json too.",
+    "verified": false
+  },
+  {
+    "label": "apostrophes are letters, not quote delimiters, so the hedge is bare",
+    "reply": "It's probably still on the branch and I don't think the build ran.",
+    "verified": false
   }
 ]