feat: add jori coordination plugin

[JRichlen]

Add Jori coordination and optional model-price monitoring

Adds Jori to the marketplace for coordinating complex work through bounded agents and an adaptable project dashboard. The package includes a command, activation context, templates, evidence-based handoffs, and sourced cross-provider routing guidance. GitHub guidance distinguishes Copilot Auto's HyDRA router from the HydraFusion research preview and preserves uncertainty about undocumented controls.

The three active subject evaluation suites now use z-ai/glm-5.3-flash with explicit max reasoning passed through to OpenRouter. Claude Sonnet remains the independent Jori judge. These configuration changes do not establish cross-model quality parity or change installed Codex models.

The optional price monitor checks public OpenRouter metadata daily after activation. It watches five subject candidates plus separate Sonnet OpenRouter pricing, retaining provider, tier, quantization, context, cache, and pricing conditions. Material changes can trigger a bounded GLM strategy proposal. Results are artifacts containing evidence and an allowlisted apply:false configuration-change plan; the workflow cannot create PRs or apply model changes.

Monitoring ships disabled, with strategy disabled and both spending allowances at $0. Proposed activation ceilings are $0.05 per review and $1 per month, backed by a dedicated limited key and durable reservations committed before requests. Unchanged prices require no model call. Activation, paid calibration of proposals, and model promotion require review.

Validation

Reviewed implementation commit: 2567cf0598848f1f4fd52beaaf594d069b4765d4. Current PR head b12d8942a3a307b172e12db225b5acdc5e8c6c48 merges main and retains that commit and all three GLM/max configurations.

• Local full cheap suite: 1,239 passed, 0 failed; counterfeit corpus: 23/0; monitor tests: 8/0; focused install smoke: 10/0; isolated Jori: 16/0.
• All three subject configurations passed the static native-reasoning passthrough contract. Pinned Promptfoo source was inspected; its renderer was unavailable locally. CI subsequently produced real GLM responses with the configured native reasoning.effort: max setting.
• Public metadata normalized 55 routes across six models. A deliberate removal of the monthly reservation guard caused the offline suite to fail. The tests do not establish live scheduling, provider billing enforcement, or strategy quality.
• The initial authorized run was canceled when main was merged into the PR. Model processes started but produced no results artifacts; request count, tokens and cost are unknown. The automatically triggered replacement run at b12d894 completed with failures. No manual rerun was initiated.
• Current CI results: Jori 21/24, with all 12/12 skill-present rows passing and three calibration-control failures; routing 56/70, with four of fourteen scenarios below the 0.80 pass-rate floor; trajectory 24/25, meeting every per-scenario gate. Twelve routing rows ended at the 4,096-token output limit without a route line; two completed with incorrect tuples. The trajectory pack used an 8,192-token limit and retains its one truncated row as a failed result. These results do not establish cross-model or plugin-effect superiority.
• Current CI deterministic, counterfeit, install, and grader-resolution checks passed. Actual deep Pier execution was inapplicable and skipped. No active evaluation remains.
• Monitor workflow validation passed its offline controls; the detect/strategy job was skipped. Recurring monitoring remains inactive.
• The current-head real-input demonstration includes exact input, native worker output, demonstrated rules, and an observed gap. It demonstrates Jori planning rather than GLM execution or measured savings.

The previous run at 1c93242 returned 119 subject HTTP 403 errors from the configured monthly key limit, with no valid model outputs or grader verdicts. Its failures are transport evidence, not a quality comparison. The historical token-cost comment covers only its two named runs; it is not a monthly bill or current-run estimate.

Recurring monitoring is not activated. No merge or automatic model promotion is included.

[JRichlen]

Jori demonstration — right-sizing an orchestration plan

This task predates Jori. A planning-only worker read the committed Jori skill and reference, then answered the exact pre-skill input below. The wrapper allowed no child agents, external API calls, or edits; the runtime did not independently expose a model or effort setting.

Before (verbatim user input):

Remember you can work on multiple things at once. Model and breadth/width should probably have an association so we have a cost control guardrail too. Like we don't want Astra fanning out and burning token budget when a more effective strategy could be using Terra or Luna fan out wide. Create guidance around good general best practices and criteria to right size orchestration, teaming strategies, model fit, and other factors that go into how we control orchestration and multi-agent behavior and strategy around different problem types, objectives and needs. Sometime we need a very specific question answered, sometimes that question is very unknown and we need to explore multiple paths for confidence. Balance using some lighter agent to coordinate the test in the tasks where you believe the to be a significant chance of high input and output - but don't let the criteria sacrifice quality where lack of would delay us - i'm worried my bias will inherently make it into these criteria so please use logic to set baselines and note what should be observed and calibrated over time

After (verbatim planning output):

Proposed orchestration plan:

• Use one Sol worker at medium effort to draft guidance from task shape, uncertainty, failure consequences, and expected input/output volume. Give a separate reviewer the acceptance criteria and draft to challenge both unnecessary premium usage and false economy. Keep delegation single-level; bound rounds and output before dispatch.
• Recommend Luna at low effort for precise extraction, triage, and mechanical aggregation; Terra for bounded coding; Sol for analysis and review. Propose an authorized Astra lead for demanding ambiguity or consequential synthesis. These are starting hypotheses to validate, not demonstrated rankings.
• For a known question, start with one suitable worker. For an unknown problem, begin with finite decomposition, then fan out distinct evidence-gathering paths. Lightweight coordination fits clear test protocols; semantic reconciliation needs a capable lead. Model names alone cannot establish affordable worker-count ratios.
• Accept the guidance only if representative task examples produce justified routes, explicit evidence requirements, and whole-cycle allowances covering coordination, review, retries, and duplicated context. Dollar cost remains unknown until applicable rates and usage estimates are available.
• Stop when acceptance checks pass. Escalate unresolved consequential uncertainty, repeated failed checks, or proposed expansion beyond authority. Premium fanout requires approval.
• Calibrate using matched task classes: first-pass acceptance, cost to accepted outcome, rework, latency, review escapes, and evidence coverage. Preserve raw failures; distinguish transport problems from model quality.

Rules shown: the output scales route to uncertainty, quality consequence, I/O burden, cost, and latency; keeps delegation single-level; bounds work before dispatch; includes whole-cycle allowance; and treats model roles as calibration hypotheses rather than proven rankings.

Observed miss: The plan names a separate reviewer but leaves its model and effort, and the round and output allowances, unspecified. Those details still need a dispatch brief.

Limits: A real planning worker ran; the proposed worker team was not dispatched or executed. This is not a no-skill baseline and does not establish cost savings, causal lift, or validated Sol/Luna/Terra/Astra settings.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T22:49:01.966134Z	06772d8	PR opened
🔒 Security Review	✅ Completed 2026-09-05T23:02:49.113559Z	06772d8	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

The new plugins/jori/evals/promptfoo/promptfooconfig.yaml diverges from established repo conventions (schema header + subject token budget), which can reduce editor validation and increase rubric flakiness.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new jori plugin to the agent-plugins marketplace, providing a coordination layer for complex work (bounded delegation, authority control, and evidence-backed status), plus supporting eval coverage and marketplace/routing registration.

Changes:

• Adds the plugins/jori/ plugin surface (skill, command, activation fragment, and dashboard template).
• Registers jori across marketplace + routing roster, and documents the new eval packs in docs/testing.md.
• Introduces jori evals: a cheap invariant pack, a promptfoo behavioral pack, and a counterfeit fixture to ensure invariant drift is caught.

File summaries

File	Description
README.md	Adds jori to the top-level plugin list.
plugins/jori/skills/jori/SKILL.md	Defines the Jori coordination invariant and operating rules.
plugins/jori/skills/jori/references/orchestration.md	Adds routing/calibration reference material for coordination decisions.
plugins/jori/skills/jori/assets/dashboard-template.md	Provides a portable dashboard template for complex work.
plugins/jori/skills/jori/agents/openai.yaml	Adds OpenAI agent metadata for implicit invocation + default prompt.
plugins/jori/README.md	Documents install/use guidance for the jori plugin.
plugins/jori/evals/promptfoo/promptfooconfig.yaml	Adds behavioral promptfoo pack (including calibration negative controls).
plugins/jori/evals/promptfoo/prompt.txt	Defines the promptfoo single-turn/tool-less harness prompt.
plugins/jori/evals/promptfoo/calibration-stub.md	Adds invariant-free stub for discriminating-scenario calibration.
plugins/jori/evals/cheap/checks.sh	Adds cheap-tier checks for manifest wiring and load-bearing invariant phrases.
plugins/jori/context/AGENTS.fragment.md	Adds optional activation fragment for persistent coordination behavior.
plugins/jori/commands/jori.md	Adds /jori command entrypoint content.
plugins/jori/AGENTS.md	Adds plugin entrypoint documentation and restates the invariant.
plugins/jori/.codex-plugin/plugin.json	Registers the plugin for Codex with UI metadata and default prompt.
plugins/jori/.claude-plugin/plugin.json	Registers the plugin for Claude with marketplace-facing metadata.
evals/routing/roster.txt	Adds jori to the routing roster list.
evals/counterfeits/fixtures/18-jori-invariant/mutate.sh	Adds a counterfeit mutation to ensure invariant drift is detected.
evals/counterfeits/fixtures/18-jori-invariant/DEFECT.md	Documents the expected counterfeit failure substring.
docs/testing.md	Documents Jori’s behavioral pack scope/limits and adds jori/* packs to the inventory list.
.claude-plugin/marketplace.json	Registers jori in the marketplace catalog.

Review details

• Files reviewed: 22/22 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JRichlen]

CI cost estimate from recorded tokens

$0.8919 across both completed runs, calculated before rounding. This is a published-rate estimate for recorded evaluation tokens, not an invoice or the full workflow bill.

Run	Model / role	Input tokens	Output tokens	Estimated USD
33996506512, 06772d8	Nemotron 3 Ultra / subjects	228,539	81,824	$0.2942823
Same run	Claude Sonnet 5 / Jori grader	13,930	8,148	$0.1093400
33997937122, 80b9096	Nemotron 3 Ultra / subjects	267,546	82,973	$0.3163136
Same run	Claude Sonnet 5 / Jori grader	29,838	11,224	$0.1719160

Rates verified September 5, 2026: OpenRouter nvidia/nemotron-3-ultra-550b-a55b $0.50/M input, $2.20/M output; Anthropic Claude Sonnet 5 $2/M input, $10/M output. Formula: (input × input rate + output × output rate) / 1,000,000.

Individual result usage reconciles with aggregate totals: 724,022 tokens, covering Jori, routing, trajectory, and recorded Jori grading. Cached tokens are zero; reasoning is included in output and counted once. The grader identity comes from the evaluation configuration; exports independently identify the subject model. Failed rows are included: the second run's Jori pack was 21/24, including two GitHub-case failures and one authority-control failure.

Excluded: calls/retries absent from the artifacts, model-resolution pings, native Codex coordination/reviews, runner charges, and account/provider billing adjustments. Transport request counts and final billed cost remain unknown. Exported cost: 0 fields are not evidence of free usage.

[JRichlen]

Current Jori demonstration — bounded orchestration planning

A real native planning worker (requested Luna, medium effort) read the Jori skill and both references at 2567cf0598848f1f4fd52beaaf594d069b4765d4, then answered this preserved user input from before the original skill was written. The proposed team was not dispatched. No external model API call was made. This demonstrates planning behavior; it is not a GLM evaluation.

Before — exact user input:

Remember you can work on multiple things at once. Model and breadth/width should probably have an association so we have a cost control guardrail too. Like we don't want Astra fanning out and burning token budget when a more effective strategy could be using Terra or Luna fan out wide. Create guidance around good general best practices and criteria to right size orchestration, teaming strategies, model fit, and other factors that go into how we control orchestration and multi-agent behavior and strategy around different problem types, objectives and needs. Sometime we need a very specific question answered, sometimes that question is very unknown and we need to explore multiple paths for confidence. Balance using some lighter agent to coordinate the test in the tasks where you believe the to be a significant chance of high input and output - but don't let the criteria sacrifice quality where lack of would delay us - i'm worried my bias will inherently make it into these criteria so please use logic to set baselines and note what should be observed and calibrated over time

After — exact worker response:

Use a proportional, evidence-driven routing policy rather than a fixed rule that equates model size with worker count. Choose a route from task uncertainty, acceptance risk, input/output burden, latency, coordination overhead, and the available operational allowance. Treat model, breadth, depth, reviews, retries, and duplicated context as one whole-cycle cost.

For a bounded known question, start with one smallest sufficient worker and a concrete acceptance check. Luna is a provisional fit for extraction, triage, summaries, bookkeeping, and small edits. Terra is a provisional fit for balanced bounded coding. Sol is appropriate when ordinary implementation or review has meaningful rework risk. Astra is reserved for authorized ambiguity, synthesis, or high-consequence judgment. These are role baselines, not quality rankings or guarantees; verify the active runtime, effort setting, tools, and model actually used.

For an exploratory unknown, begin with a finite discovery or planning round. Fan out only independent paths that can produce distinct evidence. Use a capable lead for decomposition when the unknowns are semantic, and reconcile results explicitly. Shared prompts, sources, or dependencies make agreement correlated, so extra width should not be treated as confidence. A lighter coordinator may aggregate clear high-I/O protocol checks, but semantic judgment and conflict resolution need a capable model.

Associate stronger models with narrower, higher-value assignments when quality risk is high; allow wider Terra or Luna work only when each branch has a separate decision purpose and its expected evidence justifies coordination cost. Do not derive universal worker-count ratios from model names. Keep the current subject or provider unchanged until a reviewed artifact supports a switch.

For each route, record task class, acceptance rule, requested and actual model/effort, breadth, depth, context duplication, review and retry work, wall time, usage, failures, rework, source coverage, and cost when available. Separate provider or transport failures from model quality. Calibrate with matched acceptance tests and an authorized stronger reference sample. Escalate or stop on evidence, not worker confidence. Ask before materially widening, deepening, increasing cost, changing trajectory, or adopting core instructions. Simple work gets no dashboard or orchestration ceremony.

Rules demonstrated: separate known questions from discovery; fan out only distinct evidence paths; consider duplicated context, review and retries; keep model roles provisional; retain approval boundaries.

Observed gap: calibration calls for a stronger reference but does not explicitly require an independent model-family judge for our CI comparisons. That requirement still needs to be specified in the evaluation brief.

Limits: no no-skill baseline, causal improvement, or savings were measured. This output does not validate GLM dispatch, monitor strategy quality, scheduling, or unattended behavior.

[JRichlen]

@claude fix failing checks and add baseline no skill tests or benchmarking to evaluate skills

[JRichlen]

@copilot resolve the merge conflicts in this pull request and fix failing checks

[Copilot]

@copilot resolve the merge conflicts in this pull request and fix failing checks

The branch is already clean: the latest merge commit has no unresolved index entries or conflict markers. I ran the repository cheap suite and isolated Jori checks (all passed: 1,239/0 overall and 16/0 Jori). The latest CI failure is the unrelated routing tier: 4 scenarios fell below its 0.80 floor, with missing ROUTE lines in model output; Jori, install, trajectory, and deterministic checks passed. I did not change unrelated routing behavior.

[JRichlen]

Superseded as a review surface by PR #115. This exact head is an ancestor of both #115 and #116; Jori code is preserved there and on this branch. Closing without merge or branch deletion. Current initiative: #117.