The hang-up
Every behavioral pack runs one subject model (openrouter:nvidia/nemotron-3-ultra-550b-a55b) and one grader (anthropic:messages:claude-sonnet-5), three repeats per scenario at a 0.6 floor. So the tier's claim, "the prose steers a model", is a claim about one cheap model, graded by one model, with a 2-of-3 majority. A hostile reader states it as: AI grades AI, on a sample of three, for a model nobody runs these skills on. All four objections are answerable with measurements, none of which the repo takes today.
Four measurements, in leverage order
1. Subject-model matrix (breaks "one model")
promptfoo already fans a config out over every entry in providers:. Add two subjects to each pack alongside the baseline: one from a different model family at similar cost, and one of the models that actually runs these skills (a Claude model via anthropic:messages:, since the marketplace targets Claude Code first). Keep the baseline provider as the required leg; the new legs report advisory per the plan's rule that no layer is promoted to a required check on the PR that creates it.
evals/paid/pass-rate.sh gains --by-provider: group rows by (provider.id, scenario) instead of scenario alone, print one floor verdict per provider, exit on the baseline provider's verdict only. Fixture-tested red-first in the cheap tier (§ pass-rate self-tests), same shape as the fail-open fixture.- Promotion rule for a new subject, stated up front: the calibration stub (the
without-skill negative control) must still fail under that subject on every scenario. A stub that passes under a new model means the rubric is not discriminating for that model, and the new leg is a finding about the rubric, not a green.
- Cost: roughly 3× the current per-pack spend, still cents. Manual dispatch first (
workflow_dispatch input subjects=), scheduled once the promotion rule has held for two refreshes.
2. Human calibration set (breaks "AI grades AI")
The only measurement that leaves the model-grades-model circle. Sample ~20 stored subject outputs across packs (with-skill, without-skill, stub), strip the labels, have the maintainer grade them blind against the same rubric text the grader sees, and compute agreement (percent and Cohen's κ) with the grader's verdicts.
- Store at
evals/paid/calibration/human-labels.json with the output hash each label refers to, so the set is reproducible and cannot drift from the outputs.
- A cheap-tier check asserts the agreement report exists and names its κ before any grader-dependent tier is promoted to required; the threshold is set after the first measurement, not before (per rule 7, prove the gate red first, then choose the bar).
- Cost: about an hour of human time, once per grader change.
3. Grader cross-check and self-consistency (bounds "two of three")
- Self-consistency: re-grade each stored output three times with the same grader at temperature 0. Agreement rate is the label-noise floor; if it is below the pass-rate floor, 2-of-3 cannot distinguish a skill effect from grader noise and the repeat count must rise. promptfoo caches subject responses, so this is grader calls only.
- Cross-grader: re-grade the same outputs with a second grader from a different family (an OpenRouter model). Per-scenario disagreement is a rubric-ambiguity signal; the scenarios that disagree are the first candidates for a deterministic contract (the routing tier's
route-contract.js pattern) instead of a rubric.
- Report both as
evals/paid/calibration/grader-report.json, rendered into docs/testing.md's statistical-spine section and the timeline's spine text, which currently states the numbers by hand.
4. Real-agent runs for the protocol (breaks "two plugins")
The deep tier has pier packs for graveyard and fleet-playbook-curator only. The L3 layer in docs/testing-plan.md (trajectory runs with artifact audit) puts redgate under a real agent in the sandbox across harnesses. That is already planned in #89; this issue only notes that it is the answer to "the strong tier covers two of twenty-four" and should not be reordered behind the three items above.
What can be built without API keys
Items 1 and 3 have offline halves: the --by-provider grouping, the κ computation, the fixtures, and the cheap-tier self-tests can all land red-first with no model calls. Item 2's harness (sampling, hashing, label file, agreement script) is offline too; only the labeling hour and the grader runs need keys and a human.
Standing orders this touches
docs/testing.md inventory must change in the same PR that adds any job or pack (cheap tier enforces it).- No new required check on the PR that creates it.
- Findings go here, with run numbers, not only into a PR body that will be squashed.
Related: #89 (testing handoff), #84 (trigger taxonomy), the timeline's disclosed limit ("one subject model, one grader").
The hang-up
Every behavioral pack runs one subject model (
openrouter:nvidia/nemotron-3-ultra-550b-a55b) and one grader (anthropic:messages:claude-sonnet-5), three repeats per scenario at a 0.6 floor. So the tier's claim, "the prose steers a model", is a claim about one cheap model, graded by one model, with a 2-of-3 majority. A hostile reader states it as: AI grades AI, on a sample of three, for a model nobody runs these skills on. All four objections are answerable with measurements, none of which the repo takes today.Four measurements, in leverage order
1. Subject-model matrix (breaks "one model")
promptfoo already fans a config out over every entry in
providers:. Add two subjects to each pack alongside the baseline: one from a different model family at similar cost, and one of the models that actually runs these skills (a Claude model viaanthropic:messages:, since the marketplace targets Claude Code first). Keep the baseline provider as the required leg; the new legs report advisory per the plan's rule that no layer is promoted to a required check on the PR that creates it.evals/paid/pass-rate.shgains--by-provider: group rows by(provider.id, scenario)instead of scenario alone, print one floor verdict per provider, exit on the baseline provider's verdict only. Fixture-tested red-first in the cheap tier (§ pass-rate self-tests), same shape as the fail-open fixture.without-skillnegative control) must still fail under that subject on every scenario. A stub that passes under a new model means the rubric is not discriminating for that model, and the new leg is a finding about the rubric, not a green.workflow_dispatchinputsubjects=), scheduled once the promotion rule has held for two refreshes.2. Human calibration set (breaks "AI grades AI")
The only measurement that leaves the model-grades-model circle. Sample ~20 stored subject outputs across packs (with-skill, without-skill, stub), strip the labels, have the maintainer grade them blind against the same rubric text the grader sees, and compute agreement (percent and Cohen's κ) with the grader's verdicts.
evals/paid/calibration/human-labels.jsonwith the output hash each label refers to, so the set is reproducible and cannot drift from the outputs.3. Grader cross-check and self-consistency (bounds "two of three")
route-contract.jspattern) instead of a rubric.evals/paid/calibration/grader-report.json, rendered intodocs/testing.md's statistical-spine section and the timeline's spine text, which currently states the numbers by hand.4. Real-agent runs for the protocol (breaks "two plugins")
The deep tier has pier packs for graveyard and fleet-playbook-curator only. The L3 layer in
docs/testing-plan.md(trajectory runs with artifact audit) puts redgate under a real agent in the sandbox across harnesses. That is already planned in #89; this issue only notes that it is the answer to "the strong tier covers two of twenty-four" and should not be reordered behind the three items above.What can be built without API keys
Items 1 and 3 have offline halves: the
--by-providergrouping, the κ computation, the fixtures, and the cheap-tier self-tests can all land red-first with no model calls. Item 2's harness (sampling, hashing, label file, agreement script) is offline too; only the labeling hour and the grader runs need keys and a human.Standing orders this touches
docs/testing.mdinventory must change in the same PR that adds any job or pack (cheap tier enforces it).Related: #89 (testing handoff), #84 (trigger taxonomy), the timeline's disclosed limit ("one subject model, one grader").