reflect: make the mechanical gates detect the shapes this session used - #299
Open
EdbertChan wants to merge 1 commit into
Conversation
Four accepted reflect findings from one 10-hour session that shipped four
unproven fixes. Each gate that should have caught it was blind to the exact
shape in front of it.
prove-it-ship-gate (A1): CLAIM_RE had no pattern for "Proof: yes", "proven",
"confirmed working", or "all three asks are done", so six real ship claims
read as silence. has_evidence() also accepted any `#\d{3,6}`, which means the
PR number of the change under review counted as proof that the pipeline ran.
A live-output receipt (workflow id, Actions run, release tag, live-owner
dispatch) is now a distinct predicate from a bare PR reference.
token_audit.py (A2): intervention-must-automate read NO on a session with
four same-class proof demands. told-you needs "i told you"; this user
challenges the evidence instead of blaming the agent, in a politer register
with no pattern. Adds restated-ask and proof-challenge.
pr-schema-gate (A3): `mergify stack push --dry-run` armed the pending
follow-up state, so the real push that followed was blocked. Cost five of the
session's fifteen tool errors, with the workaround already in memory. A
rehearsal must not have the side effects of the real operation.
CLAUDE.learned.md (A5): replaces the interim "prove the real path" prose with
the enumerate-then-probe procedure it was missing -- eight steps from
IEC 61882 HAZOP and Google SRE ch. 12, each attributed, each with a gate.
Repro-then-fix and one-variable isolation both terminate on the first cause;
neither asks whether a second stage also failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVGJAQuwdDArq4oGK2wrSE
Change-Id: Ie25ea152fdc94b3896cc93768c71b49afb77b617Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_29ff57c0-0b3a-4cd7-b0b1-1868050530ad) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four mechanical gates were blind to the exact failure shape they exist to catch. This restores each detector's coverage.
The audited session shipped four fixes in one night, each correct, none making the feature work. Every gate stayed quiet.
The ship gate missed six real "it works" claims and treated the PR number of the change as proof the pipeline ran.
The intervention detector read NO on a session with four proof demands, because the user challenged the evidence rather than blaming the agent.
The schema gate armed its pending state on a
--dry-runrehearsal, then blocked the real push that followed it.The learned rule said prove the real path. It did not say how to find the defects, so this adds the enumerate-then-probe procedure.
Review Claim
Each of the four detectors now fires on the real inputs it previously missed, and stays silent on the near-miss inputs beside them.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Every change is additive to a detector, and every existing test in the repo still passes unchanged:
scripts/run_all_tests.shexits 0 over 32 suites. The two hooks fail open exactly as before -- an unreadable transcript, malformed stdin, or a missing state file still lets the turn through.pr-schema-gateonly ever loosens: a--dry-runrehearsal now arms nothing, so the worst case is one un-armed follow-up requirement, never a newly blocked publication.prove-it-ship-gateonly ever tightens on claims it already understood, and the widened claim patterns are pinned against four near-miss messages from the same session that must stay silent.token_audit.pyadds two frustration kinds; the first-time proof request that must not flag is a test.Assumptions: proposed unconfirmed -- this PR was drafted from a delegated task specification with no interactive turn available to confirm the invariant text.
Slice Rationale
These four are one review unit and one claim: every mechanical surface that observed this session and reported nothing. Splitting them separates the detector from the evidence that it was blind, and each fix is a handful of lines. The remaining accepted findings (A4, a headless
config check; A6, the memory and artifact corrections) are separate units and are not here.Non-goals
diu-stop(finding B6, routing still undecided).validate-pr-body.mjs(backlog B1) -- the receipt predicate lands here, its use in the PR-body gate does not.invoker-cli config check(A4) or correct the memory files and published artifact (A6).Test Plan
Test Plan
Each item is the same suite run twice: once with only the detector reverted (
git stash push <detector>), once whole. Real output pasted.python3 -m unittest discover -s engine/hooks/prove-it-ship-gate/tests(A1)python3 -m unittest discover -s engine/skills/reflect/scripts/tests -p test_token_audit.py(A2)python3 -m unittest discover -s engine/hooks/pr-schema-gate/tests(A3)python3 -m unittest discover -s tests -p test_hazop_procedure.py(A5)bash scripts/run_all_tests.sh-- whole repo,exit=0, 32 suites, zeroFAILEDlines.python3 engine/skills/make-pr/scripts/preflight.py --base origin/mainpython3 scripts/check_no_new_comments.py --base origin/main->ok no new commentspython3 scripts/check_no_dated_provenance.py --base origin/main->ok no dated provenancepython3 scripts/check_codify_has_code.py --base origin/main->ok codify-has-codeDiff atomicity: one warning,
unrelated-areas (corpus, engine, tests). Confirmed intentional -- the detector, its tests, and the rule prose are one claim.Two classifier notes, disclosed rather than worked around:
node engine/skills/draft-pr/scripts/validate-pr-body.mjsrejects this body withReview lane behavior cannot ship with docs files in the same PR. It reaches that by classifyingcorpus/CLAUDE.learned.mdunderdocs-by-extension(**/*.{md,mdx}) indrafter.config.json. The make-pr overlay's own preflight classifies the same file as neutral (preflight.py'sreview_unit_forreturnsNonefor it). Two classifiers over the same input disagree, which is the sibling-classifier-drift shape. Nothing here was relabelled and no rule was loosened to pass -- reporting the disagreement instead.validate-pr-body.mjsis not a CI gate in this repo (.github/workflows/ci.ymlinstalls the toolchain but never invokes it), so this is advisory.check_codify_has_code.pyexists to prevent.Revert Plan
Revert Plan
git revert <sha>$TMPDIR, not the repo, so no stale state survives a revert. Reverting restores the previous detectors exactly.Note
Medium Risk
Changes when Stop/PreToolUse hooks block agent output and shell commands; mis-tuned regex could false-positive ship claims or false-negative proof challenges, but scope is detection logic plus tests, not production data paths.
Overview
Tightens reflect-derived mechanical gates so they catch phrasing and workflows that slipped through in a session that shipped multiple unproven fixes.
prove-it-ship-gate now treats claims like “Proof: yes”, “proven”, “confirmed working”, and “all … are done” as live ship assertions, expands live-context nouns (nightly, pipeline, merge queue, etc.), and stops counting bare
#12345PR refs as evidence. Live-output receipts (workflow ids, Actions run URLs,daily-*tags, live-owner dispatch) satisfy the gate; citing only the PR under review does not.pr-schema-gate no longer treats
mergify stack push --dry-runas a publication that arms the create-pr follow-up lock, so a rehearsal cannot block the real push.token_audit adds restated-ask and proof-challenge frustration patterns and folds them into intervention-must-automate, matching polite repeated “prove it” demands instead of only blunt “you messed up” signals.
CLAUDE.learned.md replaces vague “prove the real path” guidance with an eight-step, attributed enumerate-then-probe procedure (HAZOP / SRE);
tests/test_hazop_procedure.pypins step order, citations, and guide words in that doc.Reviewed by Cursor Bugbot for commit 3481f0e. Bugbot is set up for automated code reviews on this repo. Configure here.