reflect: make the mechanical gates detect the shapes this session used

corpus/CLAUDE.learned.md

@@ -3,6 +3,17 @@ Engine-only install drops this file; reflect Accepted global rules land here.
 
 # Evidence rules (apply everywhere, every project)
 
+- **A pipeline is diagnosed by enumerating its parts and probing each one, before any of them is fixed.** Repro-then-fix, class-search, 5 Whys and one-variable isolation each terminate on the first cause they find; none of them asks whether a second, unrelated stage also failed. A staged pipeline whose stages fail independently therefore gets fixed one layer per attempt, each fix correct and none of them making the feature work. Nancy G. Leveson, *CAST Handbook: How to Learn More from Incidents and Accidents*, 2019, https://psas.scripts.mit.edu/home/get_file4.php?name=CAST_Handbook.pdf, names the outcome: "The result of searching for a root cause and claiming success is that the problem is not fixed and further accidents occur. We end up in continual fire-fighting mode: fixing the symptoms of problems but not tack[l]ing the systemic causes… Too often we play a sophisticated 'whack-a-mole' game and do not understand why the losses continue to occur." Run the eight steps below in order. Steps 4, 5 and 7 are load-bearing: 1–3 enumerate, 4–7 turn "think broadly" into something that fails a build.
+  1. **Enumerate the parts, before touching anything.** [IEC 61882:2016 cl. 4.2] Write `rca/parts.tsv`, one row per pipeline stage from source to sink: `part_id · design_intent · owning_file_or_config`. Design intent is mandatory — a part with no stated intent cannot be checked for deviation from it. *Gate:* a script derives the stage list from the pipeline definition itself (config, DAG, workflow YAML) and `diff`s it against column 1; non-zero exit if any executable stage is unlisted. This is what catches a stage nobody wrote down.
+  2. **Build the guide-word matrix.** [IEC 61882:2016 cl. 4.2, Tables 1–2] `rca/matrix.tsv` = parts × {NO/NOT, MORE, LESS, AS WELL AS, PART OF, REVERSE, OTHER THAN, EARLY, LATE, BEFORE, AFTER}. Every cell gets a named plausible deviation, or the literal `N/A` plus a one-line reason. *Gate:* `awk` asserts zero empty cells and `rows == parts × guidewords`. A blank cell fails. This is the step that surfaces scope-of-allowlist (PART OF) and fallback reachability (OTHER THAN) as separate items rather than as the next thing tripped over.
+  3. **Attach an independent probe command to every non-N/A deviation.** [Google SRE ch. 12] Add a `probe_cmd` column. Each probe exits 0/1 on its own and touches only its own part — no probe may depend on another probe having passed. *Gate:* lint fails the file if any non-`N/A` row has an empty `probe_cmd`.
+  4. **Run the whole probe set with fixing frozen.** [IEC 61882:2016 cl. 4.1] Execute every probe, record exit codes to `rca/results.tsv`. No edits during this phase. *Gate:* `git diff --quiet && git status --porcelain | wc -l` identical before and after. A dirty tree means something was fixed mid-enumeration and the enumeration is void — restart it.
+  5. **Count the defects and publish the number before fixing any of them.** [Cook #3; Leveson CAST] `awk '$exit!=0' rca/results.tsv | wc -l` → `rca/defect_count`. If it is greater than 1, a fix to any one of them may not be reported as resolving the incident. *Gate:* the file exists and is non-empty before the first fix commit. This is the step whose absence produces four fixes in one night.
+  6. **Check the causal chart for straight lines.** [Gano — ARMS Reliability 2016] For each failing probe, record at least one **action** cause and one **condition** cause. *Gate:* a script rejects any entry with only one cause type. A straight line of causes means causes are still missing.
+  7. **Fix, then re-run the entire probe set — never just the one you fixed.** [Liblit et al. 2005] *Gate:* one run where every probe exits 0; assert `fixed_count == defect_count` from step 5. A green single probe is not evidence; only a green full sweep is.
+  8. **Confirm end-to-end and keep the matrix.** [IEC 61882 — HAZOP records are reusable] Run the real pipeline source-to-sink once, capture the output, commit `rca/matrix.tsv` alongside the fix. *Gate:* the end-to-end command exits 0 **and** `rca/matrix.tsv` is in the diff. The next incident starts from the existing matrix, which is how a rediscovered class stops being rediscovered.
+
+  Sources for the procedure: IEC 61882 Ed. 2.0 (2016), *Hazard and operability studies (HAZOP studies) – Application guide*, https://webstore.iec.ch/en/publication/24321 — cl. 3.1.11 admits a logical part ("step in an operational sequence"), cl. 4.2 mandates the per-part guide-word sweep, and cl. 4.1 states that developing risk-treatment actions "is not a primary objective of the HAZOP examination", which is the enumerate-before-you-fix rule. Chris Jones, "Effective Troubleshooting", ch. 12 in Beyer, Jones, Petoff & Murphy (eds.), *Site Reliability Engineering: How Google Runs Production Systems*, O'Reilly, 2016, https://sre.google/sre-book/effective-troubleshooting/ — "examining each component in turn. This strategy is also well-suited for use with data processing pipelines", and "we iteratively hypothesize potential causes for the failure and try to test those hypotheses". Richard I. Cook, *How Complex Systems Fail*, 1998 rev. 2000, https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf — #3, "Catastrophe requires multiple failures – single point failures are not enough." Jack Jager, "Actions or Conditions: What is the Difference and Why Does it Matter?", ARMS Reliability, 2016, https://blog.armsreliability.com/blog/actions-or-conditions-what-is-the-difference-and-why-does-it-matter — "there should never be a straight line, or even a partial straight line of causes within a cause and effect chart." Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken & Michael I. Jordan, "Scalable Statistical Bug Isolation", PLDI 2005, DOI 10.1145/1064978.1065014, https://theory.stanford.edu/~aiken/publications/papers/pldi05.pdf — the only surveyed software method purpose-built for a program containing multiple undiagnosed bugs, with a guarantee of one predictor per distinct defect.
 - Absence of a field in a projection (CLI, MCP, or API output) is not proof of absent state. Say "not projected" and find the emitter before retrying, resubmitting, or switching agents. Found via `/reflect` on a 2026-09-01 session: a task sat in `needs_input` with its reason stored by the owner but omitted by every headless query; the session guessed three causes (usage limit, agent switch, dirty branch) and resubmitted twice before reading the emitter, which then took four minutes.
 - When delegating a file-finding task to a subagent and two files could plausibly hold the same answer (a duplicate, a moved file, a same-named symbol in two packages), tell the subagent to state whether each file:line claim is "read-confirmed" (it opened the actual reference/import and traced it) or "name-matched" (it assumes the file is the one in use because the name/path looked right). A subagent that reasons by name-proximity instead of tracing the real reference can hand back a confident wrong file — a judgment call about how the subagent qualifies its own confidence, not something a mechanical check can catch. Found via `/reflect` on a 2026-08-17 session: a subagent named a dead, unused duplicate file as the source of a bug; the citation was relayed as fact for three turns before a direct Read caught it.
 - A `file:line` citation, mine or a subagent's, also names the ref it was read at: working tree, `HEAD`, `origin/<base>`, or the installed bundle. A working-tree read in a checkout with untracked or modified files under the cited path is name-matched, not read-confirmed, until the same line is shown at the ref the change will actually run on. Subagent prompts that ask for read-confirmed vs name-matched must also ask for the ref. Found via `/reflect` on a 2026-09-01 session: a fix was planned against `scripts/e2e-regression-watch.mjs:330` read on a dirty feature branch after `origin/master` had already removed it (#11593); a launch-time freshness gate was misdiagnosed from an untracked copy of `task-specification-preflight.ts`; three Explore subagents reported branch-local lines as read-confirmed. Third instance after memory `verify-bug-findings-against-master` (2026-07-20).

engine/hooks/pr-schema-gate/detect.py

@@ -189,7 +189,7 @@ def block_message_for(cmd: str) -> str:
     return BLOCK_MESSAGE.format(cmd=cmd)
 
 
-MERGIFY_STACK_PUSH = re.compile(r"\bmergify\s+stack\s+push\b")
+MERGIFY_STACK_PUSH = re.compile(r"\bmergify\s+stack\s+push\b(?![^;&|\n]*--dry-run\b)")
 CREATE_PR_TOOL = re.compile(r"\bcreate-pr\.mjs\b")
 
 PENDING_TTL_SECONDS = 2 * 60 * 60
@@ -213,7 +213,9 @@ def find_publication_command(raw_text: str) -> str | None:
     """Return the branch-publishing command in the payload text, or None.
 
     Publication actions are allowed to run -- they are only the events that
-    arm and re-check the follow-up requirement.
+    arm and re-check the follow-up requirement. A `--dry-run` push publishes
+    nothing, so it is not a publication action and must not arm the pending
+    state; arming on a rehearsal blocks the real push that follows it.
     """
     if MERGIFY_STACK_PUSH.search(raw_text):
         return "mergify stack push"

engine/hooks/pr-schema-gate/tests/test_hooks.py

@@ -386,6 +386,21 @@ def test_publication_command_recognized(self):
     def test_unrelated_command_is_not_a_publication(self):
         self.assertIsNone(detect.find_publication_command("git status && npm test"))
 
+    def test_dry_run_push_is_not_a_publication(self):
+        for cmd in (
+            "mergify stack push --dry-run",
+            "npx mergify stack push --dry-run --branch-prefix stack/EdbertChan",
+            "cd /repo && mergify stack push --dry-run",
+        ):
+            with self.subTest(cmd=cmd):
+                self.assertIsNone(detect.find_publication_command(cmd))
+
+    def test_real_push_after_a_dry_run_in_one_command_is_still_a_publication(self):
+        self.assertEqual(
+            detect.find_publication_command("mergify stack push --dry-run && mergify stack push"),
+            "mergify stack push",
+        )
+
     def test_sanctioned_followup_recognized(self):
         self.assertTrue(detect.is_sanctioned_followup(FOLLOWUP_CMD))
 
@@ -464,6 +479,15 @@ def test_first_stack_push_is_allowed_and_arms_pending(self):
             self.assertFalse(blocked)
             self.assertIsNotNone(detect.read_pending(repo))
 
+    def test_dry_run_arms_nothing_and_never_blocks_the_real_push(self):
+        with _repo_with_tool() as repo:
+            blocked, _ = _run("mergify stack push --dry-run", repo)
+            self.assertFalse(blocked)
+            self.assertIsNone(detect.read_pending(repo))
+            blocked, err = _run(STACK_PUSH_CMD, repo)
+            self.assertFalse(blocked)
+            self.assertEqual(err, "")
+
     def test_second_stack_push_is_blocked_while_follow_up_is_owed(self):
         with _repo_with_tool() as repo:
             self.assertFalse(_run(STACK_PUSH_CMD, repo)[0])

engine/hooks/prove-it-ship-gate/detect.py

@@ -27,17 +27,22 @@
     r"\b(?:is|are|now|been|got|successfully|and)\s+(?:already\s+|now\s+|fully\s+)?(?:shipped|deployed|live|landed)\b|"
     r"\b(?:now works|fully fixed|working end[- ]to[- ]end|done and shipped|"
     r"tested,? and shipped|everything'?s? (?:landed|deployed|live)|"
-    r"confirmed live|is up and running|running in production)\b",
+    r"confirmed live|is up and running|running in production)\b|"
+    r"\bproof:\s*(?:yes|it works|confirmed)\b|"
+    r"\b(?:proven|proved)\b|"
+    r"\bconfirmed\s+(?:working|running|fixed|green|end[- ]to[- ]end)\b|"
+    r"\b(?:all|both|everything)\b[^.!?\n]{0,60}\b(?:is|are)\s+done\b",
     re.IGNORECASE,
 )
 NEGATED_CLAIM_RE = re.compile(
     r"\b(?:not|never|n't|isn't|aren't|wasn't|without being|not yet|hasn't|haven't)\s+"
-    r"(?:been\s+|yet\s+|actually\s+|fully\s+)?(?:shipped|deployed|live|landed|done|merged)\b",
+    r"(?:been\s+|yet\s+|actually\s+|fully\s+)?(?:shipped|deployed|live|landed|done|merged|proven|proved)\b",
     re.IGNORECASE,
 )
 LIVE_NOUN_RE = re.compile(
     r"\b(?:linear|deploy(?:ed|ment|s)?|production|prod|do-?1|droplet|digital\s*ocean|"
     r"webhook|slack|external api|live mine|posthog|stripe|sentry|live path|"
+    r"nightly|pipeline|merge queue|(?:real|scheduled|next)\s+tick|"
     r"live (?:worker|owner|host|server|tick))\b",
     re.IGNORECASE,
 )
@@ -47,11 +52,17 @@
 # commit sha, a ticket/PR id, a fenced block, an exit code, a PID, or a
 # timestamp. Narrative like "ran it against production" is not evidence.
 EVIDENCE_RE = re.compile(
-    r"https?://\S+|```|\b[0-9a-f]{7,40}\b|\b[A-Z]{2,6}-\d{1,6}\b|#\d{3,6}\b|"
+    r"https?://\S+|```|\b[0-9a-f]{7,40}\b|\b[A-Z]{2,6}-\d{1,6}\b|"
     r"\bexit[_ ]code\b|\bEXIT_CODE\b|\bPID\b|\bMainPID\b|"
     r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}",
     re.IGNORECASE,
 )
+
+LIVE_RECEIPT_RE = re.compile(
+    r"\bwf-\d{10,}-\d+\b|actions/runs/\d+|\bdaily-\d{8}\b|"
+    r"\bDelegated to live owner\b",
+    re.IGNORECASE,
+)
 UNVERIFIED_RE = re.compile(r"\bUNVERIFIED:\s*live path\b", re.IGNORECASE)
 
 # Bash commands this turn that touch a live surface. A pytest run does not.
@@ -75,8 +86,12 @@ def claims_live_ship(message: str) -> bool:
     return False
 
 
+def has_live_receipt(message: str) -> bool:
+    return bool(LIVE_RECEIPT_RE.search(message or ""))
+
+
 def has_evidence(message: str) -> bool:
-    return bool(EVIDENCE_RE.search(message or ""))
+    return bool(EVIDENCE_RE.search(message or "")) or has_live_receipt(message)
 
 
 def _is_user_line(data: dict) -> bool:
@@ -162,10 +177,12 @@ def decide(payload: dict) -> str | None:
         if any(LIVE_COMMAND_RE.search(c) for c in commands):
             return None
     return (
-        "prove-it-ship-gate: this message claims done/shipped/live for work with a "
-        "live side effect (Linear, deploy, production host, webhook, external API) "
-        "but shows no live evidence -- no URL, sha, ticket/PR id, fenced output, exit "
-        "code, or live command this turn. Fixture tests and UI registration do not "
-        "prove the live path ran. Paste the live evidence in this message, or prefix "
-        "the claim with `UNVERIFIED: live path`."
+        "prove-it-ship-gate: this message claims done/shipped/live/proven for work "
+        "with a live side effect (Linear, deploy, production host, webhook, external "
+        "API) but shows no live evidence -- no URL, sha, ticket id, fenced output, "
+        "exit code, live-output receipt, or live command this turn. Fixture tests, UI "
+        "registration, a dry run, and the PR number of this change do not prove the "
+        "live path ran; only an id the pipeline itself emitted does (a workflow id, "
+        "an Actions run URL, a release tag, a live-owner dispatch). Paste that "
+        "evidence in this message, or prefix the claim with `UNVERIFIED: live path`."
     )

engine/hooks/prove-it-ship-gate/tests/test_hooks.py

@@ -161,6 +161,69 @@ def test_fails_open_on_garbage_stdin(self):
         self.assertEqual(err.getvalue(), "")
 
 
+SESSION_CLAIM_SHAPES = [
+    "Proof: yes, and it caught a real bug. The scorer, copied standalone to DO1, "
+    "scores both heavy sessions from the rollout log.",
+    "Real tick proven end to end. Duplicate probe cancelled.",
+    "Proven, on DO1, against the real sessions. All commands run this turn.",
+    "All three asks are done. PR 12006 merged at 21:15:22 UTC. Workers on DO1 "
+    "are picking the new code up.",
+    "The nightly pipeline is working end to end again.",
+    "Three things confirmed working that were broken an hour ago: nightly cut, "
+    "the fleet model pin, and the session miner on DO1.",
+]
+
+SESSION_NON_CLAIMS = [
+    "I'll check the deployed timeout value in the worker config.",
+    "Actively working on the nightly bump script now.",
+    "This is not deployed to production yet.",
+    "Reading the repair plan builder locally while, on DO1, enabling the e2e toggle.",
+]
+
+LIVE_RECEIPTS = [
+    "Real tick proven end to end on DO1: filed wf-1788934611938-7.",
+    "The nightly pipeline is working end to end again -- "
+    "https://github.com/EdbertChan/catstack/actions/runs/34259072426 is green.",
+    "Proven on DO1: the cut published daily-20260909.",
+    "All three asks are done. Delegated to live owner; the follow-up landed.",
+]
+
+PR_REFERENCE_ONLY = [
+    "All three asks are done. PR #12027 fixes all three, with a six-check "
+    "self-test and this live proof on DO1.",
+    "The nightly pipeline is working end to end again -- see #12023.",
+]
+
+
+class TestSessionClaimShapes(unittest.TestCase):
+    def test_fires_on_each_claim_shape_the_session_used(self):
+        for text in SESSION_CLAIM_SHAPES:
+            with self.subTest(text=text[:60]):
+                self.assertTrue(detect.claims_live_ship(text))
+                self.assertIsNotNone(detect.decide({"last_assistant_message": text}))
+
+    def test_silent_on_near_miss_progress_talk(self):
+        for text in SESSION_NON_CLAIMS:
+            with self.subTest(text=text[:60]):
+                self.assertFalse(detect.claims_live_ship(text))
+                self.assertIsNone(detect.decide({"last_assistant_message": text}))
+
+
+class TestLiveReceiptVsPrReference(unittest.TestCase):
+    def test_silent_when_a_live_output_receipt_is_present(self):
+        for text in LIVE_RECEIPTS:
+            with self.subTest(text=text[:60]):
+                self.assertTrue(detect.has_live_receipt(text))
+                self.assertIsNone(detect.decide({"last_assistant_message": text}))
+
+    def test_fires_when_the_only_id_is_the_changes_own_pr_number(self):
+        for text in PR_REFERENCE_ONLY:
+            with self.subTest(text=text[:60]):
+                self.assertFalse(detect.has_live_receipt(text))
+                self.assertFalse(detect.has_evidence(text))
+                self.assertIsNotNone(detect.decide({"last_assistant_message": text}))
+
+
 if __name__ == "__main__":
     unittest.main()