Eval CI: trigger behavioral packs on behavior-bearing surfaces - #90
Merged
Merged
Conversation
…(RQ-001) The behavioral tier's per-plugin filter only watched the promptfoo pack and the shared paid harness, and the routing filter only watched SKILL.md and the marketplace — so a command, skill reference, plugin AGENTS.md, hook, or manifest edit could steer the model differently and merge green without any behavioral evaluation. - Freeze the behavior-bearing path definition in ci/behavior-surfaces.json (behavior-surfaces-as-code, same pattern as ci/required-checks.json). - Expand the behavioral-run and routing paths-filter blocks in evals.yml to the full behavior surface: skills/**/SKILL.md, skills/**/references/**, commands/**, plugin AGENTS.md, hooks/**, plugin.json, plus the existing pack/harness paths; routing adds plugin.json and nested-skill SKILL.md. Guard scripts deliberately stay the deep tier's surface. - Add ci/check_behavior_surfaces.py: offline, deterministic drift guard that keeps spec and workflow in verbatim lockstep both directions, and proves a counterfeit table of changed-file cases — including behavior edits OUTSIDE SKILL.md — selects exactly the right packs (and nothing spurious). Proven RED against the pre-fix filters, GREEN after. - Wire it into evals/cheap/run.sh as a delimited section before the summary; inert in roots without the workflow/spec, green in the counterfeit tier's synthetic root. - Every behavioral/routing leg now announces EVALUATED vs SKIPPED via ::notice:: and a step-summary line, so green can never be confused with not-exercised; the checker asserts both markers stay present. No job names changed (branch-protection lock verified in sync); semantic thresholds and paid spend gating untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 142511327b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4 tasks
Review on PR #90 found two shipped model-instruction shapes the trigger map missed: declared agent files (plugin.json 'agents' component — present in fleet-playbook-curator and tailscale-wif) and companion prompts beside a SKILL.md (fleet-playbook-curator's PROMPT.md). Add plugins/<p>/agents/** and plugins/<p>/skills/**/PROMPT.md to ci/behavior-surfaces.json and the workflow filter, with counterfeit table cases proven RED against the previous filter set and GREEN after. The root AGENTS.md stays deliberately absent: it is repo governance at the marketplace root, outside every plugin source dir, and never shipped by an install (ci/install-smoke.sh walks marketplace source -> plugin dir -> declared components) — an all-legs trigger for it would be exactly the spurious paid spend RQ-001's acceptance criteria forbid. A counterfeit negative case now pins that decision. Also state the honest limit in the spec's _comment: selection guarantees the matching pack RUNS; it does not put the changed file into the pack's prompt — packs inject SKILL.md today, and injecting the other triggering surfaces is behavioral-pack design tracked under the testing-strategy work (issue #89). cheap tier: 1178 passed, 0 failed. counterfeits: 22 passed. branch protection in sync (6 checks + 2 safety paths). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
JRichlen
pushed a commit
that referenced
this pull request
Aug 30, 2026
Resolved the expected evals/cheap/run.sh conflict by keeping both appended sections: the testing-doc drift guard (this PR) followed by the RQ-001 behavior-surface trigger map (merged via #90). Gates on the merged tree: cheap 1510 passed / 0 failed, counterfeits 22/0, branch protection in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #87
What changed
The behavioral tier's per-plugin filter only watched
plugins/<p>/evals/promptfoo/**+evals/paid/**— not even that plugin's ownSKILL.md— and the routing filter watched onlyevals/routing/**,plugins/*/skills/*/SKILL.md, and the marketplace. A command, skill reference, pluginAGENTS.md, hook, or manifest edit could steer the model and merge green without any behavioral evaluation.ci/behavior-surfaces.json(new) — behavior-surfaces-as-code, the single source of truth for behavior-bearing paths, same pattern asci/required-checks.json. Per-plugin:skills/**/SKILL.md,skills/**/references/**,commands/**,AGENTS.md,hooks/**,.claude-plugin/plugin.json, the promptfoo pack, and the shared paid harness. Routing: the routing pack, everySKILL.md, everyplugin.json, andmarketplace.json. Guard scripts (skills/**/scripts/**) deliberately stay the deep tier's frozen surface..github/workflows/evals.yml— thebehavioral-runandrouting-evalpaths-filter blocks now carry those patterns verbatim. No job renamed;python3 ci/check_branch_protection.py --repo .reportsbranch protection in sync: 6 checks + 2 safety paths.ci/check_behavior_surfaces.py(new) — deterministic, offline, no model call. Asserts (1) spec ↔ workflow filter lockstep, both directions; (2) a counterfeit table of changed-file cases, evaluated against the globs extracted from the workflow itself, proving behavior edits outsideSKILL.mdselect the right pack and unrelated edits select nothing (no spurious paid spend); (3) both legs still announce EVALUATED vs SKIPPED.evals/cheap/run.sh— one clearly delimited section appended before the summary wires the checker in (self-test +--repo .). Inert in roots without the workflow/spec; green in the counterfeit tier's synthetic root.::notice::and a$GITHUB_STEP_SUMMARYline sayingEVALUATEDorSKIPPED, extending the existing "announce that this leg did NOT run" pattern, so green can never be confused with not-exercised. The checker fails if either marker disappears.Semantic thresholds (
pass-rate.shfloors,repeat:), spend gating (fork-PR guards, per-plugin billing), the routing result schema, and composition are untouched (#88 owns the schema).Counterfeit proof — RED on the pre-fix filters
python3 ci/check_behavior_surfaces.py --repo .run against the unmodified workflow frommain(verbatim):Counterfeit proof — GREEN after the fix
Same command against this branch (verbatim):
Test evidence
evals/cheap/run.sh— exit 0, 1178 passed, 0 failed (includes the newbehavior-surface trigger mapsection and the checker's 10-case--self-test, which itself re-proves RED-on-pre-fix/GREEN-on-fix against synthetic filter sets)evals/counterfeits/run.sh— exit 0, 22 passed, 0 failed (baseline calibration stays green with the new section firing in the synthetic root)python3 ci/check_branch_protection.py --repo .— in sync, 6 checks + 2 safety paths (no frozen check name touched)yaml.safe_load)🤖 Generated with Claude Code
Generated by Claude Code