Conversation
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… feat/agentic-test-framework # Conflicts: # docs/testing.md
The committed .redgate/INDEX.md was generated under a C-locale sort; under en_US.UTF-8 the same corpus sorts slice2-reconcile before slice2-reconcile-r2 and the cheap-tier drift gate reported a phantom drift. Pin the sort so the index is byte-identical regardless of the invoking shell's locale. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gx6juLZRTy4fVdS666GJAN
…ted runners, verified outcome, CI-gated verdict) Manual-dispatch workflow jori-benchmark.yml plus evals/paid/jori-benchmark/: one fresh runner per attempt, one headless 'claude -p' session per attempt, two arms with exposure parity (baseline = plain prompt; jori = '/jori' prefix + --plugin-dir plugins/jori; tools, model, budget cap and workspace identical), a deterministic two-sided outcome verifier with a reward-hack guard (tests' digests), and an aggregator that counts every attempt, keeps unreported cost UNKNOWN (never 0), refuses a verdict across differing realized models or n<2, and calls 'jori cheaper' only when the 95% bootstrap CI of the mean-cost difference is below zero. Free self-test (13 checks) runs first and locally. No scheduled or automatic trigger; docs/testing.md tier map, section and inventory updated in the same change (87 live entries). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gx6juLZRTy4fVdS666GJAN
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The new evals/routing/subject-provider-config.test.py hard-fails when PyYAML is unavailable, which can break the offline cheap tier on environments that don’t preinstall PyYAML.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces the new jori coordination plugin and adds a manual, paid benchmark workflow to measure whether using /jori + the plugin changes the harness-reported cost to complete a verified activity versus a baseline run. It also updates the routing/trajectory subject provider configuration to use GLM with mandatory reasoning passthrough and adds offline guards around that contract and OpenRouter price-monitoring controls.
Changes:
- Add the
plugins/jori/plugin (skill, references, cheap + promptfoo evals, command, context fragment) and wire it into the marketplace + routing roster. - Add a manual-dispatch GitHub Actions workflow and supporting harness (
evals/paid/jori-benchmark/) to run matched baseline vs jori attempts and aggregate a bootstrap-CI verdict on mean cost difference. - Add OpenRouter subject-provider config contract test + model-pricing offline controls and default-off monitoring workflow/docs.
File summaries
| File | Description |
|---|---|
| README.md | Adds jori to the plugin list. |
| plugins/redgate/skills/criteria-contract/scripts/criteria-index.sh | Makes run directory sorting locale-stable. |
| plugins/jori/skills/jori/SKILL.md | Adds the new Jori coordination skill/invariant. |
| plugins/jori/skills/jori/references/orchestration.md | Adds routing guardrails and calibration guidance reference. |
| plugins/jori/skills/jori/references/model-equivalence.md | Adds sourced cross-provider “task-fit hypothesis” guidance and boundaries. |
| plugins/jori/skills/jori/assets/dashboard-template.md | Adds a portable dashboard template for complex work. |
| plugins/jori/skills/jori/agents/openai.yaml | Adds implicit-invocation interface metadata for OpenAI surface. |
| plugins/jori/README.md | Adds install/use documentation for the new plugin. |
| plugins/jori/evals/promptfoo/promptfooconfig.yaml | Adds behavioral promptfoo pack for Jori invariant/authority boundaries + calibration stubs. |
| plugins/jori/evals/promptfoo/prompt.txt | Adds prompt template for tool-less behavioral evaluation. |
| plugins/jori/evals/promptfoo/calibration-stub.md | Adds invariant-free stub used as negative control. |
| plugins/jori/evals/promptfoo/calibration-reference.md | Adds invariant-free routing reference stub used as negative control. |
| plugins/jori/evals/cheap/checks.sh | Adds offline cheap checks for Jori wiring and load-bearing phrases. |
| plugins/jori/context/AGENTS.fragment.md | Adds optional activation fragment for persistent/global instruction inclusion. |
| plugins/jori/commands/jori.md | Adds /jori command entrypoint. |
| plugins/jori/AGENTS.md | Adds cross-harness entrypoint describing workflow and invariant. |
| plugins/jori/.codex-plugin/plugin.json | Adds Codex plugin manifest metadata. |
| plugins/jori/.claude-plugin/plugin.json | Adds Claude plugin manifest metadata. |
| evals/routing/trajectory/promptfooconfig.yaml | Switches routing trajectory subject provider to GLM + passthrough reasoning. |
| evals/routing/subject-provider-config.test.py | Adds offline regression test for the GLM passthrough contract. |
| evals/routing/roster.txt | Registers jori in the routing roster. |
| evals/routing/promptfooconfig.yaml | Switches routing subject provider to GLM + passthrough reasoning. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/setup.sh | Adds fixture staging for the benchmark activity. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/verify.sh | Adds deterministic verifier (tests pass + tests unmodified). |
| evals/paid/jori-benchmark/tasks/multi-module-fix/prompt.md | Adds activity prompt defining the task + constraints. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/oracle.sh | Adds oracle fix used only by self-test to prove verifier can pass. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/tests/init.py | Adds benchmark fixture test package initializer. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/tests/test_textutil.py | Adds fixture test for text utilities bug. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/tests/test_stats.py | Adds fixture test for stats bug. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/tests/test_dates.py | Adds fixture test for dates bug. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/calc/textutil.py | Adds buggy fixture module to be fixed by the agent. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/calc/stats.py | Adds buggy fixture module to be fixed by the agent. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/calc/dates.py | Adds buggy fixture module to be fixed by the agent. |
| evals/paid/jori-benchmark/tasks/multi-module-fix/fixture/calc/init.py | Adds fixture package initializer. |
| evals/paid/jori-benchmark/self-test.sh | Adds offline self-test proving discriminating plumbing before spending. |
| evals/paid/jori-benchmark/run-arm.sh | Adds per-attempt runner (baseline vs jori) that records harness usage/cost + outcome. |
| evals/paid/jori-benchmark/README.md | Documents benchmark design, parity assumptions, and limits. |
| evals/paid/jori-benchmark/aggregate.py | Adds aggregation + bootstrap CI verdict logic and report rendering. |
| evals/counterfeits/fixtures/18-jori-invariant/mutate.sh | Adds counterfeit mutation to ensure cheap checks reject invariant drift. |
| evals/counterfeits/fixtures/18-jori-invariant/DEFECT.md | Documents counterfeit fixture expectation for invariant removal detection. |
| evals/cheap/run.sh | Adds gates for subject-provider config test and model-pricing offline controls. |
| docs/testing.md | Updates eval inventory and documents new benchmark + model-pricing monitor. |
| docs/model-pricing.md | Adds documentation for default-off OpenRouter price monitoring and staged strategy. |
| docs/examples/PLAN.md | Updates example coverage counts and includes jori in the graded-pack list. |
| ci/model-pricing/test_monitor.py | Adds unit tests for model-pricing monitor invariants. |
| ci/model-pricing/STRATEGY.md | Adds bounded agent prompt/spec for staged price strategy proposals. |
| ci/model-pricing/policy.json | Adds shipped (disabled) policy for monitored models and spending ceilings. |
| ci/model-pricing/monitor.py | Adds scan/review implementation with durable reservation ledger and strict bounds. |
| ci/model-pricing/initial-state.json | Adds initial durable state template for the state branch. |
| .github/workflows/model-pricing.yml | Adds default-off scheduled/manual monitoring workflow with offline tests. |
| .github/workflows/jori-benchmark.yml | Adds manual-dispatch benchmark workflow (self-test → matrix attempts → report). |
| .claude-plugin/marketplace.json | Registers the new jori plugin in the marketplace. |
Review details
- Files reviewed: 53/54 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| try: | ||
| import yaml | ||
| except ImportError as error: | ||
| raise SystemExit("PyYAML is required for subject provider config test") from error |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d291bc4b7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| bad "route/step contract: routing pack present without trajectory/step-contract.test.js (fail-closed)" | ||
| fi | ||
| if [ -f evals/routing/subject-provider-config.test.py ]; then | ||
| if out="$(python3 evals/routing/subject-provider-config.test.py 2>&1)"; then |
There was a problem hiding this comment.
The required cheap job in .github/workflows/evals.yml only checks out the repository and runs this script; it never installs PyYAML, while subject-provider-config.test.py exits immediately when import yaml fails. On a clean runner or developer environment without that undeclared package, this unconditional invocation turns the required cheap gate red (confirmed in the provided environment). Make the test standard-library-only or explicitly provision the dependency before running it.
AGENTS.md reference: AGENTS.md:L62-L71
Useful? React with 👍 / 👎.
| ws="${1:?workspace path}" | ||
| cd "$ws" || { echo '{"outcome":"fault","reason":"workspace missing"}'; exit 2; } | ||
| tests_ok=false; untouched=false | ||
| if ( cd "$ws" && sha256sum --status -c .tests.sha256 ) 2>/dev/null; then untouched=true; fi |
There was a problem hiding this comment.
Because .tests.sha256 is stored in the same $ws where the benchmarked Claude process has Bash, Write, and Edit access, an attempt can modify the tests and regenerate this checksum; the verifier then reports both tests_pass and tests_untouched as true. The self-test only edits tests without updating the digest, so it misses this reward-hack path. Compare against hashes kept outside the editable workspace or against the immutable fixture instead.
Useful? React with 👍 / 👎.
| merge-multiple: true | ||
| - name: aggregate every attempt and print the verdict | ||
| run: | | ||
| python3 evals/paid/jori-benchmark/aggregate.py --results results --out report |
There was a problem hiding this comment.
When repeats is at least 3 and a matrix leg fails before producing or uploading its JSON, this always() report job downloads the surviving artifacts and can still satisfy the aggregator's two-known-cost threshold. Because the requested repeat count and expected leg IDs are never passed or checked here, the summary can emit a cost verdict from an incomplete, potentially biased sample while claiming every attempt was aggregated. Pass the expected matrix to the aggregator and make missing or duplicate legs inconclusive.
Useful? React with 👍 / 👎.
| "known_cost_attempts": len(costs), "unknown_cost_attempts": unknown_cost, | ||
| "total_cost_usd": total if costs else "unavailable", | ||
| "mean_cost_usd": mean_or_unavail(costs), | ||
| "cost_per_success_usd": (total / succ) if (costs and succ) else "unavailable", |
There was a problem hiding this comment.
If a successful attempt has total_cost_usd: "UNKNOWN", succ still includes that attempt while total excludes its cost, so this produces a falsely low numeric cost per success—for example, one $1 success plus one unknown-cost success is reported as $0.50. Use only successes with known costs in both numerator and denominator, or mark the ratio unavailable whenever a successful attempt has unknown cost.
Useful? React with 👍 / 👎.
|
@claude fix the failing checks |
|
Deferred as an optional cost benchmark while #117 focuses on demonstrating one useful behavior improvement. This PR contains unique benchmark code; its branch is preserved for reopening or reuse. No benchmark execution, merge or quality claim is implied. |
What
A manual-dispatch-only workflow (
.github/workflows/jori-benchmark.yml) plusevals/paid/jori-benchmark/that runs identical headless Claude Code attempts of one activity with and without the/joriprefix and plugin, records the harness's own billed cost and usage per attempt, and reports whether Jori changed the cost to complete the activity. Verdict only when the bootstrap interval of the mean-cost difference excludes zero; unknown costs stay unknown; all attempts count; a cheaper arm with lower success is flagged.Why
Jori's routing is uncalibrated: no counterfactual has ever been run. This makes the measurement possible without any scheduled or automatic spend.
Evidence
evals/paid/jori-benchmark/self-test.sh: 13 passed, 0 failed (dry-run never calls a model; verifier fails the untouched fixture and a reward hack, passes the oracle)evals/cheap/run.sh: green; testing-doc guard in sync;git diff --checkcleanCost
Nothing runs on merge. A dispatch with the defaults (3 repeats, $2.00 cap) is at most 6 attempts × $2.00 on the repo's
ANTHROPIC_API_KEY.🤖 Generated with Claude Code
https://claude.ai/code/session_01Gx6juLZRTy4fVdS666GJAN