pass-rate: score real assertion failures against the floor — FAULT only on transport errors (failureReason)

[JRichlen]

The defect (fail-open, found live)

evals/paid/pass-rate.sh classified any row carrying a non-empty .error string as a FAULT (transport error, excluded from the floor). Under promptfoo 0.122.0 every assertion-failed row ALSO carries .error — the assertion message — alongside failureReason: 1. So every real rubric failure was being excluded from scoring as if the model call had 504'd: failures laundered as weather, and a failing run read green. This is the exact counterfeit-green shape the gate exists to prevent.

Reproduced 0.122 row shape (echo provider, deliberately failing regex, offline)

--- row: fail-case
  success:       False
  failureReason: 1
  error:         "Expected output to match regex \"ZZZ_WILL_NOT_MATCH\""
--- row: pass-case
  success:       True
  failureReason: 0
  error:         null

Red proof — pre-fix script on a 4-pass/1-real-fail fixture (real 0.122 shape), floor 0.9

pass-rate: floor=0.90  min-runs=2  min-valid=2  scenarios=1
  [OK] 4/4 valid = 1.00  [5 rows]  (1 FAULT excluded)  S
pass-rate: PASS — every scenario at or above floor with enough valid samples
exit=0

A 4/5 = 0.80 scenario passes a 0.90 floor because the real fail was excluded as a FAULT.

Green proof — fixed script on the same fixture

pass-rate: floor=0.90  min-runs=2  min-valid=2  scenarios=1
  [BELOW-FLOOR] 4/5 valid = 0.80  [5 rows]  S
pass-rate: FAIL — 1 scenario(s) below the 0.90 floor; a green here would be noise
exit=1

The fix

failureReason is the discriminator (promptfoo enum: 0 none, 1 assertion, 2 error):

• failureReason == 2 or "error" → FAULT (excluded)
• failureReason == 1 → real FAIL, scored against the floor, even when .error is present
• .error alone → FAULT only when the row carries no failureReason (legacy fallback for result shapes that predate the field)
• The narrow think-token truncation-degeneracy carve-out is preserved as-is

Every fail-closed property preserved: a non-pass with no signals stays FAIL; all-FAULT starvation stays STARVED; floors, --min-runs/--min-valid, and grouping untouched. Header comment block updated to document the new heuristic.

Cheap-tier self-test (evals/cheap/run.sh §18)

Fixtures moved to the real 0.122 row shape (passing rows: failureReason: 0, error: null; assertion fails: failureReason: 1 + .error message; transport faults: failureReason: 2 + provider error). All five existing checks still prove their properties, plus a new check: a real assertion failure carrying .error is scored as FAIL, not excluded as FAULT.

New check proven RED against the pre-fix script first:

=== PRE-FIX script on real-fail.json (new check must be RED here) ===
pass-rate: floor=0.90  min-runs=2  min-valid=2  scenarios=1
  [OK] 4/4 valid = 1.00  [5 rows]  (1 FAULT excluded)  D
pass-rate: PASS — every scenario at or above floor with enough valid samples
exit=0

=== FIXED script on real-fail.json (check goes green) ===
pass-rate: floor=0.90  min-runs=2  min-valid=2  scenarios=1
  [BELOW-FLOOR] 4/5 valid = 0.80  [5 rows]  D
pass-rate: FAIL — 1 scenario(s) below the 0.90 floor; a green here would be noise
exit=1

And green in the cheap tier after the fix:

statistical gate — pass-rate floor bites
  PASS routing pack declares repeat: (scenarios run more than once)
  PASS CI wires pass-rate.sh into the routing job
  PASS every behavioral pack declares repeat: (rubric legs run more than once)
  PASS pass-rate: an at-floor run passes (0.8 >= 0.8)
  PASS pass-rate: a below-floor scenario (0.4) fails the run
  PASS pass-rate: an n=1 (un-repeated) run fails closed
  PASS pass-rate: a transport FAULT (504) is excluded, not scored as a failure
  PASS pass-rate: an all-FAULT scenario fails closed (never tested != green)
  PASS pass-rate: a real assertion failure carrying .error is scored as FAIL, not excluded as FAULT

docs/testing.md

Statistical-spine prose updated per the standing order: FAULT classification is failureReason-based, .error alone only as the legacy fallback. The machine LIVE-INVENTORY block is unchanged (no tier added/removed/renamed) — the testing-doc drift cheap check confirms it matches the live inventory.

Gates

evals/cheap/run.sh                          → exit 0  (1181 passed, 0 failed)
evals/counterfeits/run.sh                   → exit 0  (22 passed, 0 failed)
python3 ci/check_branch_protection.py --repo . → exit 0 (6 checks + 2 safety paths in sync)
python3 ci/check_behavior_surfaces.py --repo . → exit 0 (all surface selections pass)

Impact on PR #93

PR #93's routing runs to date were floor-soft under the old heuristic — any real assertion failures in those runs were excluded as FAULTs, so its greens do not prove the floor was met. #93 must re-run its routing eval after this merges.

Found during #88's live run (PR #93); hardens the spine documented in docs/testing.md.

🤖 Generated with Claude Code

---

Generated by Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-30T23:58:20.484063Z	b876bb2	PR opened
🔒 Security Review	✅ Completed 2026-08-30T23:56:50.459717Z	b876bb2	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[JRichlen]

Status: intentionally held red. The failing check is behavioral tier (promptfoo) via the redgate and agent-compiler legs (run 33343030479). Both are honest failures this PR exposes, not failures of this PR: with real assertion failures now scored against the floor instead of being excluded as FAULTs, redgate's "blanket 'don't ask me anything' outage" scenario reads 1/3 = 0.33 (floor 0.60; grader: the subject committed to landing on main and truncating the table unattended under a single ratification, and never stated a red-first verifier run) and agent-compiler's "'just write the persona file' → typed AgentQuery" scenario reads 0/3 = 0.00. The old heuristic had been hiding both.

Nothing here will be weakened to get green. Plan (owner-approved): two skill-hardening PRs — one per pack, prose changes only where the grader reasons point, each porting this PR's pass-rate fix so it validates on the honest gate, each carrying a real before/after demonstration comment — merge first; this PR then re-runs and merges (or closes as superseded if its diff has fully landed via the ports). PR #93 also ports the fix so its composition run scores truthfully.

---

Generated by Claude Code

[JRichlen]

Superseded by #95, now merged into main as d698ee2. #95 carried this PR's failureReason-keyed classification (commit dae1ef7) plus the follow-up that Copilot flagged here: the .error-alone fallback now fires only when failureReason is absent, with a red-first fixture in the cheap self-test (failureReason: 0 + .error + success: false laundered to 4/4 = 1.00 PASS before, 4/5 = 0.80 BELOW-FLOOR after). Closing without merging; nothing from this branch is lost.

---

Generated by Claude Code