From 3481f0ed7b2f9dc44371c4adbeb7bb27ed642084 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 00:21:09 -0700 Subject: [PATCH] reflect: make the mechanical gates detect the shapes this session used Four accepted reflect findings from one 10-hour session that shipped four unproven fixes. Each gate that should have caught it was blind to the exact shape in front of it. prove-it-ship-gate (A1): CLAIM_RE had no pattern for "Proof: yes", "proven", "confirmed working", or "all three asks are done", so six real ship claims read as silence. has_evidence() also accepted any `#\d{3,6}`, which means the PR number of the change under review counted as proof that the pipeline ran. A live-output receipt (workflow id, Actions run, release tag, live-owner dispatch) is now a distinct predicate from a bare PR reference. token_audit.py (A2): intervention-must-automate read NO on a session with four same-class proof demands. told-you needs "i told you"; this user challenges the evidence instead of blaming the agent, in a politer register with no pattern. Adds restated-ask and proof-challenge. pr-schema-gate (A3): `mergify stack push --dry-run` armed the pending follow-up state, so the real push that followed was blocked. Cost five of the session's fifteen tool errors, with the workaround already in memory. A rehearsal must not have the side effects of the real operation. CLAUDE.learned.md (A5): replaces the interim "prove the real path" prose with the enumerate-then-probe procedure it was missing -- eight steps from IEC 61882 HAZOP and Google SRE ch. 12, each attributed, each with a gate. Repro-then-fix and one-variable isolation both terminate on the first cause; neither asks whether a second stage also failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HVGJAQuwdDArq4oGK2wrSE Change-Id: Ie25ea152fdc94b3896cc93768c71b49afb77b617 --- corpus/CLAUDE.learned.md | 11 +++ engine/hooks/pr-schema-gate/detect.py | 6 +- .../hooks/pr-schema-gate/tests/test_hooks.py | 24 +++++ engine/hooks/prove-it-ship-gate/detect.py | 37 ++++++-- .../prove-it-ship-gate/tests/test_hooks.py | 63 +++++++++++++ .../reflect/scripts/tests/test_token_audit.py | 58 ++++++++++++ engine/skills/reflect/scripts/token_audit.py | 15 ++- tests/test_hazop_procedure.py | 94 +++++++++++++++++++ 8 files changed, 295 insertions(+), 13 deletions(-) create mode 100644 tests/test_hazop_procedure.py diff --git a/corpus/CLAUDE.learned.md b/corpus/CLAUDE.learned.md index 936f2ca..82404ca 100644 --- a/corpus/CLAUDE.learned.md +++ b/corpus/CLAUDE.learned.md @@ -3,6 +3,17 @@ Engine-only install drops this file; reflect Accepted global rules land here. # Evidence rules (apply everywhere, every project) +- **A pipeline is diagnosed by enumerating its parts and probing each one, before any of them is fixed.** Repro-then-fix, class-search, 5 Whys and one-variable isolation each terminate on the first cause they find; none of them asks whether a second, unrelated stage also failed. A staged pipeline whose stages fail independently therefore gets fixed one layer per attempt, each fix correct and none of them making the feature work. Nancy G. Leveson, *CAST Handbook: How to Learn More from Incidents and Accidents*, 2019, https://psas.scripts.mit.edu/home/get_file4.php?name=CAST_Handbook.pdf, names the outcome: "The result of searching for a root cause and claiming success is that the problem is not fixed and further accidents occur. We end up in continual fire-fighting mode: fixing the symptoms of problems but not tack[l]ing the systemic causes… Too often we play a sophisticated 'whack-a-mole' game and do not understand why the losses continue to occur." Run the eight steps below in order. Steps 4, 5 and 7 are load-bearing: 1–3 enumerate, 4–7 turn "think broadly" into something that fails a build. + 1. **Enumerate the parts, before touching anything.** [IEC 61882:2016 cl. 4.2] Write `rca/parts.tsv`, one row per pipeline stage from source to sink: `part_id · design_intent · owning_file_or_config`. Design intent is mandatory — a part with no stated intent cannot be checked for deviation from it. *Gate:* a script derives the stage list from the pipeline definition itself (config, DAG, workflow YAML) and `diff`s it against column 1; non-zero exit if any executable stage is unlisted. This is what catches a stage nobody wrote down. + 2. **Build the guide-word matrix.** [IEC 61882:2016 cl. 4.2, Tables 1–2] `rca/matrix.tsv` = parts × {NO/NOT, MORE, LESS, AS WELL AS, PART OF, REVERSE, OTHER THAN, EARLY, LATE, BEFORE, AFTER}. Every cell gets a named plausible deviation, or the literal `N/A` plus a one-line reason. *Gate:* `awk` asserts zero empty cells and `rows == parts × guidewords`. A blank cell fails. This is the step that surfaces scope-of-allowlist (PART OF) and fallback reachability (OTHER THAN) as separate items rather than as the next thing tripped over. + 3. **Attach an independent probe command to every non-N/A deviation.** [Google SRE ch. 12] Add a `probe_cmd` column. Each probe exits 0/1 on its own and touches only its own part — no probe may depend on another probe having passed. *Gate:* lint fails the file if any non-`N/A` row has an empty `probe_cmd`. + 4. **Run the whole probe set with fixing frozen.** [IEC 61882:2016 cl. 4.1] Execute every probe, record exit codes to `rca/results.tsv`. No edits during this phase. *Gate:* `git diff --quiet && git status --porcelain | wc -l` identical before and after. A dirty tree means something was fixed mid-enumeration and the enumeration is void — restart it. + 5. **Count the defects and publish the number before fixing any of them.** [Cook #3; Leveson CAST] `awk '$exit!=0' rca/results.tsv | wc -l` → `rca/defect_count`. If it is greater than 1, a fix to any one of them may not be reported as resolving the incident. *Gate:* the file exists and is non-empty before the first fix commit. This is the step whose absence produces four fixes in one night. + 6. **Check the causal chart for straight lines.** [Gano — ARMS Reliability 2016] For each failing probe, record at least one **action** cause and one **condition** cause. *Gate:* a script rejects any entry with only one cause type. A straight line of causes means causes are still missing. + 7. **Fix, then re-run the entire probe set — never just the one you fixed.** [Liblit et al. 2005] *Gate:* one run where every probe exits 0; assert `fixed_count == defect_count` from step 5. A green single probe is not evidence; only a green full sweep is. + 8. **Confirm end-to-end and keep the matrix.** [IEC 61882 — HAZOP records are reusable] Run the real pipeline source-to-sink once, capture the output, commit `rca/matrix.tsv` alongside the fix. *Gate:* the end-to-end command exits 0 **and** `rca/matrix.tsv` is in the diff. The next incident starts from the existing matrix, which is how a rediscovered class stops being rediscovered. + + Sources for the procedure: IEC 61882 Ed. 2.0 (2016), *Hazard and operability studies (HAZOP studies) – Application guide*, https://webstore.iec.ch/en/publication/24321 — cl. 3.1.11 admits a logical part ("step in an operational sequence"), cl. 4.2 mandates the per-part guide-word sweep, and cl. 4.1 states that developing risk-treatment actions "is not a primary objective of the HAZOP examination", which is the enumerate-before-you-fix rule. Chris Jones, "Effective Troubleshooting", ch. 12 in Beyer, Jones, Petoff & Murphy (eds.), *Site Reliability Engineering: How Google Runs Production Systems*, O'Reilly, 2016, https://sre.google/sre-book/effective-troubleshooting/ — "examining each component in turn. This strategy is also well-suited for use with data processing pipelines", and "we iteratively hypothesize potential causes for the failure and try to test those hypotheses". Richard I. Cook, *How Complex Systems Fail*, 1998 rev. 2000, https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf — #3, "Catastrophe requires multiple failures – single point failures are not enough." Jack Jager, "Actions or Conditions: What is the Difference and Why Does it Matter?", ARMS Reliability, 2016, https://blog.armsreliability.com/blog/actions-or-conditions-what-is-the-difference-and-why-does-it-matter — "there should never be a straight line, or even a partial straight line of causes within a cause and effect chart." Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken & Michael I. Jordan, "Scalable Statistical Bug Isolation", PLDI 2005, DOI 10.1145/1064978.1065014, https://theory.stanford.edu/~aiken/publications/papers/pldi05.pdf — the only surveyed software method purpose-built for a program containing multiple undiagnosed bugs, with a guarantee of one predictor per distinct defect. - Absence of a field in a projection (CLI, MCP, or API output) is not proof of absent state. Say "not projected" and find the emitter before retrying, resubmitting, or switching agents. Found via `/reflect` on a 2026-09-01 session: a task sat in `needs_input` with its reason stored by the owner but omitted by every headless query; the session guessed three causes (usage limit, agent switch, dirty branch) and resubmitted twice before reading the emitter, which then took four minutes. - When delegating a file-finding task to a subagent and two files could plausibly hold the same answer (a duplicate, a moved file, a same-named symbol in two packages), tell the subagent to state whether each file:line claim is "read-confirmed" (it opened the actual reference/import and traced it) or "name-matched" (it assumes the file is the one in use because the name/path looked right). A subagent that reasons by name-proximity instead of tracing the real reference can hand back a confident wrong file — a judgment call about how the subagent qualifies its own confidence, not something a mechanical check can catch. Found via `/reflect` on a 2026-08-17 session: a subagent named a dead, unused duplicate file as the source of a bug; the citation was relayed as fact for three turns before a direct Read caught it. - A `file:line` citation, mine or a subagent's, also names the ref it was read at: working tree, `HEAD`, `origin/`, or the installed bundle. A working-tree read in a checkout with untracked or modified files under the cited path is name-matched, not read-confirmed, until the same line is shown at the ref the change will actually run on. Subagent prompts that ask for read-confirmed vs name-matched must also ask for the ref. Found via `/reflect` on a 2026-09-01 session: a fix was planned against `scripts/e2e-regression-watch.mjs:330` read on a dirty feature branch after `origin/master` had already removed it (#11593); a launch-time freshness gate was misdiagnosed from an untracked copy of `task-specification-preflight.ts`; three Explore subagents reported branch-local lines as read-confirmed. Third instance after memory `verify-bug-findings-against-master` (2026-07-20). diff --git a/engine/hooks/pr-schema-gate/detect.py b/engine/hooks/pr-schema-gate/detect.py index 85f827e..49eb466 100644 --- a/engine/hooks/pr-schema-gate/detect.py +++ b/engine/hooks/pr-schema-gate/detect.py @@ -189,7 +189,7 @@ def block_message_for(cmd: str) -> str: return BLOCK_MESSAGE.format(cmd=cmd) -MERGIFY_STACK_PUSH = re.compile(r"\bmergify\s+stack\s+push\b") +MERGIFY_STACK_PUSH = re.compile(r"\bmergify\s+stack\s+push\b(?![^;&|\n]*--dry-run\b)") CREATE_PR_TOOL = re.compile(r"\bcreate-pr\.mjs\b") PENDING_TTL_SECONDS = 2 * 60 * 60 @@ -213,7 +213,9 @@ def find_publication_command(raw_text: str) -> str | None: """Return the branch-publishing command in the payload text, or None. Publication actions are allowed to run -- they are only the events that - arm and re-check the follow-up requirement. + arm and re-check the follow-up requirement. A `--dry-run` push publishes + nothing, so it is not a publication action and must not arm the pending + state; arming on a rehearsal blocks the real push that follows it. """ if MERGIFY_STACK_PUSH.search(raw_text): return "mergify stack push" diff --git a/engine/hooks/pr-schema-gate/tests/test_hooks.py b/engine/hooks/pr-schema-gate/tests/test_hooks.py index 1d29524..5437bc6 100644 --- a/engine/hooks/pr-schema-gate/tests/test_hooks.py +++ b/engine/hooks/pr-schema-gate/tests/test_hooks.py @@ -386,6 +386,21 @@ def test_publication_command_recognized(self): def test_unrelated_command_is_not_a_publication(self): self.assertIsNone(detect.find_publication_command("git status && npm test")) + def test_dry_run_push_is_not_a_publication(self): + for cmd in ( + "mergify stack push --dry-run", + "npx mergify stack push --dry-run --branch-prefix stack/EdbertChan", + "cd /repo && mergify stack push --dry-run", + ): + with self.subTest(cmd=cmd): + self.assertIsNone(detect.find_publication_command(cmd)) + + def test_real_push_after_a_dry_run_in_one_command_is_still_a_publication(self): + self.assertEqual( + detect.find_publication_command("mergify stack push --dry-run && mergify stack push"), + "mergify stack push", + ) + def test_sanctioned_followup_recognized(self): self.assertTrue(detect.is_sanctioned_followup(FOLLOWUP_CMD)) @@ -464,6 +479,15 @@ def test_first_stack_push_is_allowed_and_arms_pending(self): self.assertFalse(blocked) self.assertIsNotNone(detect.read_pending(repo)) + def test_dry_run_arms_nothing_and_never_blocks_the_real_push(self): + with _repo_with_tool() as repo: + blocked, _ = _run("mergify stack push --dry-run", repo) + self.assertFalse(blocked) + self.assertIsNone(detect.read_pending(repo)) + blocked, err = _run(STACK_PUSH_CMD, repo) + self.assertFalse(blocked) + self.assertEqual(err, "") + def test_second_stack_push_is_blocked_while_follow_up_is_owed(self): with _repo_with_tool() as repo: self.assertFalse(_run(STACK_PUSH_CMD, repo)[0]) diff --git a/engine/hooks/prove-it-ship-gate/detect.py b/engine/hooks/prove-it-ship-gate/detect.py index d64134e..8e7fc31 100644 --- a/engine/hooks/prove-it-ship-gate/detect.py +++ b/engine/hooks/prove-it-ship-gate/detect.py @@ -27,17 +27,22 @@ r"\b(?:is|are|now|been|got|successfully|and)\s+(?:already\s+|now\s+|fully\s+)?(?:shipped|deployed|live|landed)\b|" r"\b(?:now works|fully fixed|working end[- ]to[- ]end|done and shipped|" r"tested,? and shipped|everything'?s? (?:landed|deployed|live)|" - r"confirmed live|is up and running|running in production)\b", + r"confirmed live|is up and running|running in production)\b|" + r"\bproof:\s*(?:yes|it works|confirmed)\b|" + r"\b(?:proven|proved)\b|" + r"\bconfirmed\s+(?:working|running|fixed|green|end[- ]to[- ]end)\b|" + r"\b(?:all|both|everything)\b[^.!?\n]{0,60}\b(?:is|are)\s+done\b", re.IGNORECASE, ) NEGATED_CLAIM_RE = re.compile( r"\b(?:not|never|n't|isn't|aren't|wasn't|without being|not yet|hasn't|haven't)\s+" - r"(?:been\s+|yet\s+|actually\s+|fully\s+)?(?:shipped|deployed|live|landed|done|merged)\b", + r"(?:been\s+|yet\s+|actually\s+|fully\s+)?(?:shipped|deployed|live|landed|done|merged|proven|proved)\b", re.IGNORECASE, ) LIVE_NOUN_RE = re.compile( r"\b(?:linear|deploy(?:ed|ment|s)?|production|prod|do-?1|droplet|digital\s*ocean|" r"webhook|slack|external api|live mine|posthog|stripe|sentry|live path|" + r"nightly|pipeline|merge queue|(?:real|scheduled|next)\s+tick|" r"live (?:worker|owner|host|server|tick))\b", re.IGNORECASE, ) @@ -47,11 +52,17 @@ # commit sha, a ticket/PR id, a fenced block, an exit code, a PID, or a # timestamp. Narrative like "ran it against production" is not evidence. EVIDENCE_RE = re.compile( - r"https?://\S+|```|\b[0-9a-f]{7,40}\b|\b[A-Z]{2,6}-\d{1,6}\b|#\d{3,6}\b|" + r"https?://\S+|```|\b[0-9a-f]{7,40}\b|\b[A-Z]{2,6}-\d{1,6}\b|" r"\bexit[_ ]code\b|\bEXIT_CODE\b|\bPID\b|\bMainPID\b|" r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}", re.IGNORECASE, ) + +LIVE_RECEIPT_RE = re.compile( + r"\bwf-\d{10,}-\d+\b|actions/runs/\d+|\bdaily-\d{8}\b|" + r"\bDelegated to live owner\b", + re.IGNORECASE, +) UNVERIFIED_RE = re.compile(r"\bUNVERIFIED:\s*live path\b", re.IGNORECASE) # Bash commands this turn that touch a live surface. A pytest run does not. @@ -75,8 +86,12 @@ def claims_live_ship(message: str) -> bool: return False +def has_live_receipt(message: str) -> bool: + return bool(LIVE_RECEIPT_RE.search(message or "")) + + def has_evidence(message: str) -> bool: - return bool(EVIDENCE_RE.search(message or "")) + return bool(EVIDENCE_RE.search(message or "")) or has_live_receipt(message) def _is_user_line(data: dict) -> bool: @@ -162,10 +177,12 @@ def decide(payload: dict) -> str | None: if any(LIVE_COMMAND_RE.search(c) for c in commands): return None return ( - "prove-it-ship-gate: this message claims done/shipped/live for work with a " - "live side effect (Linear, deploy, production host, webhook, external API) " - "but shows no live evidence -- no URL, sha, ticket/PR id, fenced output, exit " - "code, or live command this turn. Fixture tests and UI registration do not " - "prove the live path ran. Paste the live evidence in this message, or prefix " - "the claim with `UNVERIFIED: live path`." + "prove-it-ship-gate: this message claims done/shipped/live/proven for work " + "with a live side effect (Linear, deploy, production host, webhook, external " + "API) but shows no live evidence -- no URL, sha, ticket id, fenced output, " + "exit code, live-output receipt, or live command this turn. Fixture tests, UI " + "registration, a dry run, and the PR number of this change do not prove the " + "live path ran; only an id the pipeline itself emitted does (a workflow id, " + "an Actions run URL, a release tag, a live-owner dispatch). Paste that " + "evidence in this message, or prefix the claim with `UNVERIFIED: live path`." ) diff --git a/engine/hooks/prove-it-ship-gate/tests/test_hooks.py b/engine/hooks/prove-it-ship-gate/tests/test_hooks.py index 13a851e..0d29023 100644 --- a/engine/hooks/prove-it-ship-gate/tests/test_hooks.py +++ b/engine/hooks/prove-it-ship-gate/tests/test_hooks.py @@ -161,6 +161,69 @@ def test_fails_open_on_garbage_stdin(self): self.assertEqual(err.getvalue(), "") +SESSION_CLAIM_SHAPES = [ + "Proof: yes, and it caught a real bug. The scorer, copied standalone to DO1, " + "scores both heavy sessions from the rollout log.", + "Real tick proven end to end. Duplicate probe cancelled.", + "Proven, on DO1, against the real sessions. All commands run this turn.", + "All three asks are done. PR 12006 merged at 21:15:22 UTC. Workers on DO1 " + "are picking the new code up.", + "The nightly pipeline is working end to end again.", + "Three things confirmed working that were broken an hour ago: nightly cut, " + "the fleet model pin, and the session miner on DO1.", +] + +SESSION_NON_CLAIMS = [ + "I'll check the deployed timeout value in the worker config.", + "Actively working on the nightly bump script now.", + "This is not deployed to production yet.", + "Reading the repair plan builder locally while, on DO1, enabling the e2e toggle.", +] + +LIVE_RECEIPTS = [ + "Real tick proven end to end on DO1: filed wf-1788934611938-7.", + "The nightly pipeline is working end to end again -- " + "https://github.com/EdbertChan/catstack/actions/runs/34259072426 is green.", + "Proven on DO1: the cut published daily-20260909.", + "All three asks are done. Delegated to live owner; the follow-up landed.", +] + +PR_REFERENCE_ONLY = [ + "All three asks are done. PR #12027 fixes all three, with a six-check " + "self-test and this live proof on DO1.", + "The nightly pipeline is working end to end again -- see #12023.", +] + + +class TestSessionClaimShapes(unittest.TestCase): + def test_fires_on_each_claim_shape_the_session_used(self): + for text in SESSION_CLAIM_SHAPES: + with self.subTest(text=text[:60]): + self.assertTrue(detect.claims_live_ship(text)) + self.assertIsNotNone(detect.decide({"last_assistant_message": text})) + + def test_silent_on_near_miss_progress_talk(self): + for text in SESSION_NON_CLAIMS: + with self.subTest(text=text[:60]): + self.assertFalse(detect.claims_live_ship(text)) + self.assertIsNone(detect.decide({"last_assistant_message": text})) + + +class TestLiveReceiptVsPrReference(unittest.TestCase): + def test_silent_when_a_live_output_receipt_is_present(self): + for text in LIVE_RECEIPTS: + with self.subTest(text=text[:60]): + self.assertTrue(detect.has_live_receipt(text)) + self.assertIsNone(detect.decide({"last_assistant_message": text})) + + def test_fires_when_the_only_id_is_the_changes_own_pr_number(self): + for text in PR_REFERENCE_ONLY: + with self.subTest(text=text[:60]): + self.assertFalse(detect.has_live_receipt(text)) + self.assertFalse(detect.has_evidence(text)) + self.assertIsNotNone(detect.decide({"last_assistant_message": text})) + + if __name__ == "__main__": unittest.main() diff --git a/engine/skills/reflect/scripts/tests/test_token_audit.py b/engine/skills/reflect/scripts/tests/test_token_audit.py index 698398d..d428134 100644 --- a/engine/skills/reflect/scripts/tests/test_token_audit.py +++ b/engine/skills/reflect/scripts/tests/test_token_audit.py @@ -1206,6 +1206,64 @@ def test_agent_blame_and_same_type_must_automate(self): os.unlink(path) +class TestProofChallengeAndRestatedAsk(unittest.TestCase): + """A user who challenges the evidence instead of blaming the agent is the + same intervention class in a politer register. Four real turns from one + session, plus the first-time proof request that must stay unflagged.""" + + REAL_TURNS = [ + ("and can you prove that this all works? you sohuld be able to generate " + "the same report with the script standalone run from digital ocean 1", + {"proof-challenge"}), + ("wait you said it caught a real bug. If there is a defect, can you " + "/prove-it that its fixed?", {"proof-challenge"}), + ("all i asked was to prove that the miner works on its own? does it work?", + {"restated-ask", "proof-challenge"}), + ("did you prove that 12027 will make itw ork? is the proof atached? /prove-it", + {"proof-challenge"}), + ] + FIRST_TIME_ASK = ( + "that means that you missed root cause. We should investigate any " + "root-cause finding principles like 5 whys or other litterature" + ) + + def _kinds(self, texts): + usage = {"input_tokens": 1, "output_tokens": 1} + lines = [ + claude_user_text_line(t, ts=f"2026-09-09T0{i}:00:00Z") + for i, t in enumerate(texts) + ] + lines.append(claude_assistant_line("m1", "u1", [{"type": "text", "text": "ok"}], usage)) + path = write_jsonl(lines) + try: + with redirect_stdout(io.StringIO()): + result = token_audit.audit_claude(path) + finally: + os.unlink(path) + return result + + def test_detects_each_real_proof_challenge_turn(self): + for text, expected in self.REAL_TURNS: + with self.subTest(text=text[:50]): + result = self._kinds([text]) + kinds = {k for f in result["frustration"]["flagged"] for k in f["kinds"]} + self.assertTrue(expected <= kinds, f"{expected} not in {kinds}") + + def test_four_proof_challenges_flag_intervention_must_automate(self): + result = self._kinds([t for t, _ in self.REAL_TURNS]) + flag = next(f for f in result["flags"] if f["name"] == "intervention-must-automate") + self.assertEqual(flag["value"], "yes") + self.assertIn("proof-challenge", flag["rationale"]) + + def test_silent_on_a_first_time_root_cause_request(self): + result = self._kinds([self.FIRST_TIME_ASK]) + kinds = {k for f in result["frustration"]["flagged"] for k in f["kinds"]} + self.assertNotIn("proof-challenge", kinds) + self.assertNotIn("restated-ask", kinds) + flag = next(f for f in result["flags"] if f["name"] == "intervention-must-automate") + self.assertEqual(flag["value"], "no") + + class TestOmpFrustrationAndOut(unittest.TestCase): """OMP mode gained --out and the frustration detector together; the interruption shapes (customType=interrupted-thinking, 'Skipped due to diff --git a/engine/skills/reflect/scripts/token_audit.py b/engine/skills/reflect/scripts/token_audit.py index 8ee6eac..c053084 100644 --- a/engine/skills/reflect/scripts/token_audit.py +++ b/engine/skills/reflect/scripts/token_audit.py @@ -130,13 +130,26 @@ def _direct_run_targets(command): ("accusation", re.compile(r"\byou('?re| are) (thrashing|not listening|ignoring)\b|\bignoring me\b", re.I)), ("agent-blame", re.compile(r"\byou (fucked up|messed up|broke)\b|\byou('?ve| have) (fucked|messed) up\b", re.I)), ("multi-question-marks", re.compile(r"\?\?\?+")), + ("restated-ask", re.compile( + r"\ball i (asked|wanted|said)\b" + r"|\bthat'?s not what i (asked|said)\b" + r"|\bi only asked\b" + r"|\bmy (original|actual) (ask|question) was\b", re.I)), + ("proof-challenge", re.compile( + r"\b(can|could|did|will) you /?prove[- ]?it\b" + r"|/prove-it\b" + r"|\bis the proof att?ach?ed\b" + r"|\bwhere'?s the proof\b" + r"|\bprove (to me )?that (it|this|that|the|#?\d)", re.I)), ] # Same-type user intervention. One correction can be cheap. Repeating the # class (told-you / accusation / agent-blame twice, two of those kinds in # one session, or a verbatim re-send) is the automate-me trigger. Product # blame ("the ui is messed up") does not match agent-blame. -INTERVENTION_KINDS = frozenset({"told-you", "accusation", "agent-blame"}) +INTERVENTION_KINDS = frozenset({ + "told-you", "accusation", "agent-blame", "restated-ask", "proof-challenge", +}) # function_call_output / custom_tool_call_output payloads carry their exit # status as prose ("Process exited with code 1" for exec_command, diff --git a/tests/test_hazop_procedure.py b/tests/test_hazop_procedure.py new file mode 100644 index 0000000..44f9ffb --- /dev/null +++ b/tests/test_hazop_procedure.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""The learned evidence rules must carry the enumerate-before-you-fix +procedure as an ordered, attributed, citable eight steps -- not as a slogan. + +A prose rule that names no method and no source is the shape that already +lost: three always-loaded surfaces told this session to prove the real path +and it shipped four unproven fixes anyway. These assertions pin the parts +that make the rule checkable by a reader: every step present, every step +attributed to the method it came from, and every method resolvable to an +author/title/year/URL that was actually fetched. +""" +import os +import re +import unittest + + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +LEARNED = os.path.join(REPO_ROOT, "corpus", "CLAUDE.learned.md") + +STEP_ATTRIBUTIONS = [ + (1, "[IEC 61882:2016 cl. 4.2]"), + (2, "[IEC 61882:2016 cl. 4.2, Tables 1–2]"), + (3, "[Google SRE ch. 12]"), + (4, "[IEC 61882:2016 cl. 4.1]"), + (5, "[Cook #3; Leveson CAST]"), + (6, "[Gano — ARMS Reliability 2016]"), + (7, "[Liblit et al. 2005]"), + (8, "[IEC 61882 — HAZOP records are reusable]"), +] + +CITATION_URLS = [ + "https://webstore.iec.ch/en/publication/24321", + "https://sre.google/sre-book/effective-troubleshooting/", + "https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf", + "https://blog.armsreliability.com/blog/actions-or-conditions-what-is-the-difference-and-why-does-it-matter", + "https://theory.stanford.edu/~aiken/publications/papers/pldi05.pdf", + "https://psas.scripts.mit.edu/home/get_file4.php?name=CAST_Handbook.pdf", +] + +GUIDE_WORDS = [ + "NO/NOT", "MORE", "LESS", "AS WELL AS", "PART OF", "REVERSE", + "OTHER THAN", "EARLY", "LATE", "BEFORE", "AFTER", +] + + +def read_learned(): + with open(LEARNED, encoding="utf-8") as handle: + return handle.read() + + +class TestHazopProcedure(unittest.TestCase): + def setUp(self): + self.text = read_learned() + + def test_eight_numbered_steps_each_carry_their_method_attribution(self): + for number, attribution in STEP_ATTRIBUTIONS: + with self.subTest(step=number): + pattern = re.compile( + rf"^\s*{number}\.\s+\*\*.+?\*\*\s+{re.escape(attribution)}", + re.MULTILINE, + ) + self.assertRegex(self.text, pattern) + + def test_every_cited_method_resolves_to_a_fetched_source_url(self): + for url in CITATION_URLS: + with self.subTest(url=url): + self.assertIn(url, self.text) + + def test_guide_word_matrix_lists_every_iec_61882_guide_word(self): + for word in GUIDE_WORDS: + with self.subTest(word=word): + self.assertIn(word, self.text) + + def test_defect_count_is_published_before_the_first_fix_commit(self): + self.assertIn("rca/defect_count", self.text) + self.assertIn("before the first fix commit", self.text) + + def test_leveson_whack_a_mole_passage_is_quoted_verbatim(self): + self.assertIn( + "we play a sophisticated 'whack-a-mole' game and do not understand " + "why the losses continue to occur.", + self.text, + ) + + def test_enumeration_is_separated_from_fixing(self): + self.assertIn( + 'is not a primary objective of the HAZOP examination', + self.text, + ) + self.assertIn("never just the one you fixed", self.text) + + +if __name__ == "__main__": + unittest.main()