From 8db36b86f532e1122402d2d55c5c184776a0e481 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Tue, 8 Sep 2026 23:18:08 -0700 Subject: [PATCH 1/2] reflect: prove pipelines by their real output, not a substitute Four fixes in one night each passed a cheap proof and then failed one layer down on the real path: a unit test, a config read-back, four static workflow assertions, and a dry run. Every fix was correct; none of them made the feature work. Adds three evidence rules: a pipeline change is proven only by a real run that emits its real output, a dry run never proves a write path, and an identity/binary/database change must be verified as the service resolves it rather than as the operator's shell does. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HVGJAQuwdDArq4oGK2wrSE Change-Id: Ib661b78e918424539f46bf8d421cedabfba92c86 --- corpus/CLAUDE.learned.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/corpus/CLAUDE.learned.md b/corpus/CLAUDE.learned.md index 82404ca..adb05b4 100644 --- a/corpus/CLAUDE.learned.md +++ b/corpus/CLAUDE.learned.md @@ -14,6 +14,9 @@ Engine-only install drops this file; reflect Accepted global rules land here. 8. **Confirm end-to-end and keep the matrix.** [IEC 61882 — HAZOP records are reusable] Run the real pipeline source-to-sink once, capture the output, commit `rca/matrix.tsv` alongside the fix. *Gate:* the end-to-end command exits 0 **and** `rca/matrix.tsv` is in the diff. The next incident starts from the existing matrix, which is how a rediscovered class stops being rediscovered. Sources for the procedure: IEC 61882 Ed. 2.0 (2016), *Hazard and operability studies (HAZOP studies) – Application guide*, https://webstore.iec.ch/en/publication/24321 — cl. 3.1.11 admits a logical part ("step in an operational sequence"), cl. 4.2 mandates the per-part guide-word sweep, and cl. 4.1 states that developing risk-treatment actions "is not a primary objective of the HAZOP examination", which is the enumerate-before-you-fix rule. Chris Jones, "Effective Troubleshooting", ch. 12 in Beyer, Jones, Petoff & Murphy (eds.), *Site Reliability Engineering: How Google Runs Production Systems*, O'Reilly, 2016, https://sre.google/sre-book/effective-troubleshooting/ — "examining each component in turn. This strategy is also well-suited for use with data processing pipelines", and "we iteratively hypothesize potential causes for the failure and try to test those hypotheses". Richard I. Cook, *How Complex Systems Fail*, 1998 rev. 2000, https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf — #3, "Catastrophe requires multiple failures – single point failures are not enough." Jack Jager, "Actions or Conditions: What is the Difference and Why Does it Matter?", ARMS Reliability, 2016, https://blog.armsreliability.com/blog/actions-or-conditions-what-is-the-difference-and-why-does-it-matter — "there should never be a straight line, or even a partial straight line of causes within a cause and effect chart." Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken & Michael I. Jordan, "Scalable Statistical Bug Isolation", PLDI 2005, DOI 10.1145/1064978.1065014, https://theory.stanford.edu/~aiken/publications/papers/pldi05.pdf — the only surveyed software method purpose-built for a program containing multiple undiagnosed bugs, with a guarantee of one predictor per distinct defect. +- A pipeline or worker change is proven only by one uninterrupted run of the real path, at production settings, that produces the pipeline's real external output: a workflow id, a PR number, a tag, a merged commit. A unit test, a static workflow assertion, a config read-back, and a dry run are pre-checks, not proof — each one is a substitute for the real path, and the defects live exactly at the seam the substitute skips. When the real run fails, expect the next layer to fail too: keep re-running the real path until it emits the output, rather than shipping after the first green. Found via `/reflect` on a 2026-09-08 session, four times in a row on the same night: a repair-plan header fix passed its unit test then died at dispatch on an agent/model mismatch; the config fix read back correctly then killed the owner at boot on a third key (`defaultExecutionHarness`) that overrides the one edited; a nightly-release fix passed four static workflow assertions then failed on dispatch with `GitHub Actions is not permitted to create or approve pull requests`; and a session-miner fix passed a dry run that filed two sessions, then filed zero for real because the submit path shelled a script that cannot reach the running owner and the default pool name was a remote-target id. Each fix was correct; none of them made the feature work. +- Never accept a dry run as proof of a write path. Dry run proves the decision and never the effect. Same 2026-09-08 session, twice: `mergify stack push --dry-run` listed the PRs it would create while the real push was blocked by a guard, and the session miner's dry run reported `filed 2` while the real tick reported `submit failed`. +- When a fix changes which identity, binary, or database a process uses, verify the one the *service* resolves, not the one your shell resolves. Same session: `which invoker-cli` from an ssh login returned a 0.1.0 system install under `/usr/lib/node_modules`, while the owner process's own `PATH` resolved a different 0.1.1 binary under `~/.local/npm-global/bin`; and a worker's inventory query shelled a checkout script that opened a database holding a week-old snapshot (60 workflows, newest 8 days old) instead of the live owner's (18 workflows, newest that day). - Absence of a field in a projection (CLI, MCP, or API output) is not proof of absent state. Say "not projected" and find the emitter before retrying, resubmitting, or switching agents. Found via `/reflect` on a 2026-09-01 session: a task sat in `needs_input` with its reason stored by the owner but omitted by every headless query; the session guessed three causes (usage limit, agent switch, dirty branch) and resubmitted twice before reading the emitter, which then took four minutes. - When delegating a file-finding task to a subagent and two files could plausibly hold the same answer (a duplicate, a moved file, a same-named symbol in two packages), tell the subagent to state whether each file:line claim is "read-confirmed" (it opened the actual reference/import and traced it) or "name-matched" (it assumes the file is the one in use because the name/path looked right). A subagent that reasons by name-proximity instead of tracing the real reference can hand back a confident wrong file — a judgment call about how the subagent qualifies its own confidence, not something a mechanical check can catch. Found via `/reflect` on a 2026-08-17 session: a subagent named a dead, unused duplicate file as the source of a bug; the citation was relayed as fact for three turns before a direct Read caught it. - A `file:line` citation, mine or a subagent's, also names the ref it was read at: working tree, `HEAD`, `origin/`, or the installed bundle. A working-tree read in a checkout with untracked or modified files under the cited path is name-matched, not read-confirmed, until the same line is shown at the ref the change will actually run on. Subagent prompts that ask for read-confirmed vs name-matched must also ask for the ref. Found via `/reflect` on a 2026-09-01 session: a fix was planned against `scripts/e2e-regression-watch.mjs:330` read on a dirty feature branch after `origin/master` had already removed it (#11593); a launch-time freshness gate was misdiagnosed from an untracked copy of `task-specification-preflight.ts`; three Explore subagents reported branch-local lines as read-confirmed. Third instance after memory `verify-bug-findings-against-master` (2026-07-20). From 58138ad4fa74fe9b36adf40ac1a92cc6f5b93b5a Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 13:06:44 -0700 Subject: [PATCH 2/2] reflect: state the pipeline-proof rules without dated provenance The two new evidence bullets carried "Found via /reflect on a 2026-09-08 session" framing, which the no-dated-provenance gate bans in rule prose: a standing rule states the principle, not the incident date it came from. Rewrote both to keep every concrete failure mechanism -- repair-plan header fix dying at dispatch on an agent/model mismatch, the config fix killing the owner at boot on defaultExecutionHarness, the nightly-release fix failing on "GitHub Actions is not permitted to create or approve pull requests", the session-miner dry run filing 2 then 0 for real, and mergify stack push --dry-run listing PRs while the real push was blocked -- while dropping the date and the "Found via" citation. Voice now matches the neighbouring "Same session:" bullet. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- corpus/CLAUDE.learned.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/corpus/CLAUDE.learned.md b/corpus/CLAUDE.learned.md index adb05b4..b21fcb2 100644 --- a/corpus/CLAUDE.learned.md +++ b/corpus/CLAUDE.learned.md @@ -14,8 +14,8 @@ Engine-only install drops this file; reflect Accepted global rules land here. 8. **Confirm end-to-end and keep the matrix.** [IEC 61882 — HAZOP records are reusable] Run the real pipeline source-to-sink once, capture the output, commit `rca/matrix.tsv` alongside the fix. *Gate:* the end-to-end command exits 0 **and** `rca/matrix.tsv` is in the diff. The next incident starts from the existing matrix, which is how a rediscovered class stops being rediscovered. Sources for the procedure: IEC 61882 Ed. 2.0 (2016), *Hazard and operability studies (HAZOP studies) – Application guide*, https://webstore.iec.ch/en/publication/24321 — cl. 3.1.11 admits a logical part ("step in an operational sequence"), cl. 4.2 mandates the per-part guide-word sweep, and cl. 4.1 states that developing risk-treatment actions "is not a primary objective of the HAZOP examination", which is the enumerate-before-you-fix rule. Chris Jones, "Effective Troubleshooting", ch. 12 in Beyer, Jones, Petoff & Murphy (eds.), *Site Reliability Engineering: How Google Runs Production Systems*, O'Reilly, 2016, https://sre.google/sre-book/effective-troubleshooting/ — "examining each component in turn. This strategy is also well-suited for use with data processing pipelines", and "we iteratively hypothesize potential causes for the failure and try to test those hypotheses". Richard I. Cook, *How Complex Systems Fail*, 1998 rev. 2000, https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf — #3, "Catastrophe requires multiple failures – single point failures are not enough." Jack Jager, "Actions or Conditions: What is the Difference and Why Does it Matter?", ARMS Reliability, 2016, https://blog.armsreliability.com/blog/actions-or-conditions-what-is-the-difference-and-why-does-it-matter — "there should never be a straight line, or even a partial straight line of causes within a cause and effect chart." Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken & Michael I. Jordan, "Scalable Statistical Bug Isolation", PLDI 2005, DOI 10.1145/1064978.1065014, https://theory.stanford.edu/~aiken/publications/papers/pldi05.pdf — the only surveyed software method purpose-built for a program containing multiple undiagnosed bugs, with a guarantee of one predictor per distinct defect. -- A pipeline or worker change is proven only by one uninterrupted run of the real path, at production settings, that produces the pipeline's real external output: a workflow id, a PR number, a tag, a merged commit. A unit test, a static workflow assertion, a config read-back, and a dry run are pre-checks, not proof — each one is a substitute for the real path, and the defects live exactly at the seam the substitute skips. When the real run fails, expect the next layer to fail too: keep re-running the real path until it emits the output, rather than shipping after the first green. Found via `/reflect` on a 2026-09-08 session, four times in a row on the same night: a repair-plan header fix passed its unit test then died at dispatch on an agent/model mismatch; the config fix read back correctly then killed the owner at boot on a third key (`defaultExecutionHarness`) that overrides the one edited; a nightly-release fix passed four static workflow assertions then failed on dispatch with `GitHub Actions is not permitted to create or approve pull requests`; and a session-miner fix passed a dry run that filed two sessions, then filed zero for real because the submit path shelled a script that cannot reach the running owner and the default pool name was a remote-target id. Each fix was correct; none of them made the feature work. -- Never accept a dry run as proof of a write path. Dry run proves the decision and never the effect. Same 2026-09-08 session, twice: `mergify stack push --dry-run` listed the PRs it would create while the real push was blocked by a guard, and the session miner's dry run reported `filed 2` while the real tick reported `submit failed`. +- A pipeline or worker change is proven only by one uninterrupted run of the real path, at production settings, that produces the pipeline's real external output: a workflow id, a PR number, a tag, a merged commit. A unit test, a static workflow assertion, a config read-back, and a dry run are pre-checks, not proof — each one is a substitute for the real path, and the defects live exactly at the seam the substitute skips. When the real run fails, expect the next layer to fail too: keep re-running the real path until it emits the output, rather than shipping after the first green. Four times in a row in one session, each green on its pre-check and dead on the real path: a repair-plan header fix passed its unit test then died at dispatch on an agent/model mismatch; a config fix read back correctly then killed the owner at boot on a third key (`defaultExecutionHarness`) that overrides the one edited; a nightly-release fix passed four static workflow assertions then failed on dispatch with `GitHub Actions is not permitted to create or approve pull requests`; and a session-miner fix passed a dry run that filed two sessions, then filed zero for real because the submit path shelled a script that cannot reach the running owner and the default pool name was a remote-target id. Each fix was correct; none of them made the feature work. +- Never accept a dry run as proof of a write path. Dry run proves the decision and never the effect. Same session, twice: `mergify stack push --dry-run` listed the PRs it would create while the real push was blocked by a guard, and the session miner's dry run reported `filed 2` while the real tick reported `submit failed`. - When a fix changes which identity, binary, or database a process uses, verify the one the *service* resolves, not the one your shell resolves. Same session: `which invoker-cli` from an ssh login returned a 0.1.0 system install under `/usr/lib/node_modules`, while the owner process's own `PATH` resolved a different 0.1.1 binary under `~/.local/npm-global/bin`; and a worker's inventory query shelled a checkout script that opened a database holding a week-old snapshot (60 workflows, newest 8 days old) instead of the live owner's (18 workflows, newest that day). - Absence of a field in a projection (CLI, MCP, or API output) is not proof of absent state. Say "not projected" and find the emitter before retrying, resubmitting, or switching agents. Found via `/reflect` on a 2026-09-01 session: a task sat in `needs_input` with its reason stored by the owner but omitted by every headless query; the session guessed three causes (usage limit, agent switch, dirty branch) and resubmitted twice before reading the emitter, which then took four minutes. - When delegating a file-finding task to a subagent and two files could plausibly hold the same answer (a duplicate, a moved file, a same-named symbol in two packages), tell the subagent to state whether each file:line claim is "read-confirmed" (it opened the actual reference/import and traced it) or "name-matched" (it assumes the file is the one in use because the name/path looked right). A subagent that reasons by name-proximity instead of tracing the real reference can hand back a confident wrong file — a judgment call about how the subagent qualifies its own confidence, not something a mechanical check can catch. Found via `/reflect` on a 2026-08-17 session: a subagent named a dead, unused duplicate file as the source of a bug; the citation was relayed as fact for three turns before a direct Read caught it.