hook: backstop layer catches a stale verdict without needing the model to notice

[EdbertChan]

Summary

The two layers below this one both depend on the model. wrong-check-reflect matches the shape of a retraction in the outgoing text, so it only helps once the model has decided to admit something. principle-flag-your-own-corrections carries the judgment no regex can enumerate, but still needs the model to realise a claim went stale.

Neither catches the silent switch to the corrected value — which is the failure the principle itself names, because the user cannot tell a silent correction from consistency.

This hook needs neither. It reads the transcript's own commands and their results. If a verifier printed ok earlier and failed later, the earlier status is stale as a matter of record.

Review Claim

A stale verdict should be caught from transcript evidence, independently of whether the model admits it or phrases the admission in a recognised way.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Advisory only: stderr plus exit 0. It never blocks a turn, edits a file, or runs a command. Suppressed once per transcript per target, and silent whenever the outgoing message already names the flip. Fail-open on any parse or IO error, and silent with no transcript path. The only new state is a cache dir for that suppression, overridable by env var.

Slice Rationale

Last in the stack because it depends on both layers below being in place — its decisive scenario asserts that it fires while wrong-check-reflect stays silent on the same transcript, which needs that hook's current behavior to compare against.

Kept separate from the detector widening (#309) because that slice changes an existing hook's matching and this adds a new hook with a different mechanism. A reviewer judging "is this regex too broad" and one judging "is transcript-evidence the right backstop" are answering different questions.

Non-goals

• Does not block. A gate can legitimately start failing because the turn broke it deliberately, and the hook cannot know intent.
• Does not fire on fail → pass. That is ordinary repair work; firing there would punish fixing things.
• Does not track non-verifier commands. An ls that succeeds then fails is not a stale verdict.
• Does not change wrong-check-reflect or the corrections principle.

Test Plan

Test Plan

• python3 -m unittest discover -s engine/hooks/verdict-flip-watch/tests → Ran 20 tests ... OK
• python3 scripts/check_hook_test_coverage.py engine/hooks/verdict-flip-watch → OK (1 hook(s) checked)
• python3 scripts/run_skill_scenarios.py → ok all 17 scenario(s) behaved as declared
• python3 engine/skills/make-pr/scripts/preflight.py --base admission-scenarios → ok preflight passed
• Gates: ecosystem boundaries, skill file refs, three-harness, trigger policy, subagent scope contract → all ok

The decisive scenario is stale-green-caught-without-any-admission: on one transcript, verdict-flip-watch fires while wrong-check-reflect stays silent. That is the layering this slice adds, asserted rather than described.

Four transcript fixtures cover the boundaries: the real pass→fail case, a verifier that stays green, fail→pass (a fix), and a flip on a non-verifier command.

• python3 -m unittest tests.test_install → Ran 58 tests in 280.443s ... OK (exit 0)

That last one exceeds the 120s foreground command timeout, so it finished in the background. This PR body and the commit message first recorded it as not run, because the output file was read before the run completed. Corrected: it ran and passed. The wiring lines are at install.sh:239 and install.sh:335, following the hedge-runs-prove-it pattern.

Revert Plan

Revert Plan

• Safe to revert? Yes
• Revert command: git revert <sha>
• Post-revert steps: rerun ./install.sh from the main checkout to drop the hook entry from ~/.claude/settings.json
• Data migration? No. The cache dir under ~/.cache/catstack-verdict-flip-watch can be deleted at will.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JQMWSLRArEfEm1psa7RKdD

---

Note

Low Risk
Advisory-only Stop hook (exit 0, stderr); no blocking or file edits, fail-open on errors, with tests and install wiring matching existing hook patterns.

Overview
Adds verdict-flip-watch, a third advisory Claude Stop hook that flags when the same verifier passed earlier in the transcript and failed later while the outgoing reply does not acknowledge the stale earlier green.

Detection is transcript-driven: it pairs Bash tool_use commands with tool_result output, normalizes verifier targets (check_*, test runners, etc.), classifies pass/fail (fail wins in mixed output), and emits a one-time stderr nudge per transcript/target unless correction vocabulary is already in the message. It is wired through install.sh (symlink + install_claude_hook.py), with unit fixtures/tests and hook coverage checks.

run_skill_scenarios.py now allows ran entries as {cmd, output} so scenarios can simulate real tool results; tests/scenarios/verdict-flip.json asserts the hook fires on silent stale-green (while wrong-check-reflect stays silent), stays quiet when the reply owns the flip, and ignores fail→pass repair runs.

^{Reviewed by Cursor Bugbot for commit 3b1399f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_70ba225e-6073-40f3-952d-e4d363e88fce)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_767c2c99-8656-4358-a259-baec1ea52d17)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_96efb93e-0d91-470a-a6ef-a38bb3dbf84d)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bfe3801e-06d2-44ae-af81-1c23143242b9)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_ba363493-2449-4a5c-ad2a-118915434c30)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_4dd98df1-ac58-4cd9-ad92-39899d9cfb8c)