testing docs: authoritative tier inventory, standing order, and cheap-tier drift guard (#89 phase 1) - #91
Merged
Conversation
…-tier drift guard (#89 phase 1) Documents every live eval tier in docs/testing.md — proves / cannot-prove / trigger / cost / local run per tier, the shared statistical spine (repeat:, k-of-N floor via evals/paid/pass-rate.sh, FAULT-vs-verdict separation, fail-closed starvation, PROMPTFOO_RETRY_5XX), and the demonstration discipline — linked from evals/README.md and the root AGENTS.md eval-discipline section. Planned L1 probes and the L2/L3 composition layers are linked to #89/#88 rather than specified (per the scope-split comment). Standing order (root AGENTS.md + the doc itself): any PR that adds, removes, renames, or re-scopes an eval tier, workflow job, or per-plugin eval pack MUST update docs/testing.md in the same PR. The order bites: evals/cheap/check-testing-doc.sh derives the live inventory dynamically (job display names from .github/workflows/*.yml, eval dirs from evals/*/, pack kinds from plugins/*/evals/*/) and compares it in both directions against the doc's machine-readable LIVE-INVENTORY block; wired into evals/cheap/run.sh as a repo-level gate (inert in the counterfeit tier's synthetic root, which lacks evals/counterfeits/). Proven red first on a deliberate mismatch, then green. cheap: 1177 passed, 0 failed; counterfeit corpus: 22 passed, 0 failed; branch protection in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7481ba3da9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…nularity (review findings) Three defects Codex review found in the testing-doc drift guard, all verified real and each proven red-then-green: 1. Inertness marker was evals/counterfeits/ — itself a tracked eval-dir, so a PR removing that tier made the guard silently skip the exact drift it exists to catch. Marker is now `.git` (dir at the repo root, file in a worktree; never created in the counterfeit synthetic temp root, never tracked inventory). Proven: temp copy with .git and evals/counterfeits/ removed now fails run.sh with the stale 'eval-dir: evals/counterfeits' entry instead of skipping. 2. Workflow discovery globbed only *.yml; GitHub Actions also picks up *.yaml, so a .yaml workflow could join or leave without an inventory change. Both extensions now scanned. Proven: a probe .yaml workflow goes red until the doc names it. 3. Pack inventory collapsed to distinct pack KINDS, so one plugin gaining or losing a pack of an existing kind left the set unchanged while the doc's per-plugin claims went stale. Entries are now plugin-qualified (pack: graveyard/pier, ...), LIVE-INVENTORY block regenerated via --print (34 -> 70 entries), doc prose updated. Proven both directions: removing plugins/graveyard/evals/pier and adding plugins/grill-me/evals/promptfoo each go red where the kinds-only set stayed green. Gates: evals/cheap/run.sh 1177 passed 0 failed (worktree — .git-as-file covered); evals/counterfeits/run.sh 22 passed 0 failed (guard inert in the synthetic root); ci/check_branch_protection.py in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
Resolved the expected evals/cheap/run.sh conflict by keeping both appended sections: the testing-doc drift guard (this PR) followed by the RQ-001 behavior-surface trigger map (merged via #90). Gates on the merged tree: cheap 1510 passed / 0 failed, counterfeits 22/0, branch protection in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
JRichlen
pushed a commit
that referenced
this pull request
Aug 30, 2026
…line guard Main's new testing-doc drift guard (#91) and this branch's timeline guard both claimed section 20 at the same append point. The resolution keeps both — main's testing-doc drift stays §20, the design-trajectory timeline guard becomes §21 — followed by the RQ-001 section as before. cheap tier green after resolution (1183 passed, 0 failed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
This was referenced Aug 30, 2026
JRichlen
pushed a commit
that referenced
this pull request
Aug 30, 2026
…ing.md per the standing order The run.sh sections auto-merged cleanly. Per the standing order that #91 landed, this PR re-scopes the routing tier (typed composition line + the trajectory leg), so docs/testing.md's routing entry now documents both legs, the two contract validators, the offline tests, the second pass-rate gate, and the updated trigger paths and call counts. Gates on the merged tree: cheap 1512/0, counterfeits 22/0, branch protection and behavior-surfaces lockstep in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #89 (phase 1) — deterministic deliverables only: the testing documentation, the standing order, and the cheap-tier doc-drift guard. L1 probes are the next increment of #89; the L2/L3 composition and trajectory layers are owned by #88 per the scope-split comment on #89 and are only linked, not specified.
What's in this PR
docs/testing.md— authoritative inventory of every live tier: what each proves, what it structurally cannot prove, when it fires, rough cost, and the exact local run command. Also covers the shared statistical spine (repeat:, k-of-N pass-rate floor viaevals/paid/pass-rate.sh, FAULT-vs-verdict separation, fail-closed starvation,PROMPTFOO_RETRY_5XX), the demonstration discipline (human review gate and why it cannot be machine-enforced), and a "planned" section that links Routing evals: model specialist + Redgate composition and trajectories #88/Testing handoff: document the existing eval architecture, plan the agentic/composition test strategy, and add standing orders that keep both current #89 rather than re-specifying them. Linked fromevals/README.mdand the rootAGENTS.md/CLAUDE.mdeval-discipline section.AGENTS.md, mirrored in the doc): any PR that adds, removes, renames, or re-scopes an eval tier, workflow job, or per-plugin eval pack MUST updatedocs/testing.mdin the same PR.evals/cheap/check-testing-doc.sh, wired intoevals/cheap/run.shas a clearly-delimited section at the end (surgical, before the summary block). It derives the live inventory dynamically (workflow job display names parsed from.github/workflows/*.ymlwith${{ matrix.* }}suffixes normalized away, eval dirs fromevals/*/, distinct pack kinds fromplugins/*/evals/*/) and compares it both directions against a machine-readableLIVE-INVENTORYblock in the doc. A missingdocs/testing.mdfails closed in the real repo; the gate is inert in the counterfeit tier's synthetic root (marker:evals/counterfeits/, which the synthetic root never copies — it does copy.github/workflows/evals.yml, which is why that couldn't be the marker).--printemits the current live list for updating the doc.Tier inventory documented (verified against reality, not the issue's list)
evals/cheap/run.sh)evals/counterfeits/run.sh)ci/install-smoke.sh+evals/cheap/run-one.sh)evals/routing/)scale.yml: redgate + agent-compiler stress)refresh-examples.ymlscheduled biweekly /pages.ymlon main)docs/**pushTiers the issue's inventory list missed: the grader-model check (
confirm grader model resolves— a frozen required check), the paid multi-plugin gate (advisory), and the example-gallery workflows (refresh-examples.yml+pages.yml) as a verification surface. All are now documented and machine-inventoried.Red-then-green proof of the drift guard (verbatim)
Deliberate mismatch (replaced the live routing-tier entry in the doc's block with a fabricated one), exercising both directions:
Restored:
Gates run before commit
evals/cheap/run.sh— 1177 passed, 0 failed (includes the new gate)evals/counterfeits/run.sh— 22 passed, 0 failed (confirms the new run.sh section stays inert in the synthetic root and the baseline stays calibrated)python3 ci/check_branch_protection.py --repo .— 6 checks + 2 safety paths in sync; no CI job names touchedNo skill prose changed (only repo-level docs/AGENTS.md governance + a deterministic check), so no behavioral-tier run or demonstration comment applies.
🤖 Generated with Claude Code
Generated by Claude Code