evals: confirm the SUBJECT model resolves, and repin it to qwen/qwen3.8-flash

.github/workflows/evals.yml

@@ -235,6 +235,44 @@ jobs:
           done <<< "$graders"
           [ "$fail" -eq 0 ]
 
+  subject-model:
+    # The mirror of grader-model, for the model actually UNDER TEST. CI has always
+    # confirmed the Anthropic grader slug resolves and never once checked the
+    # OpenRouter subject — a blind spot, since an unreachable subject WOULD
+    # produce packs where every real-skill row fails with no signal anywhere.
+    # Two refresh runs (2026-09-01, 2026-09-08) did exactly that and reported
+    # success; this check closes the evidence gap those runs exposed rather than
+    # diagnosing them. It came back green on its first run, so a dead key and a
+    # moved slug were ruled out — a callable model can still fail every rubric.
+    #
+    # A green here is NOT a statement about funding. OpenRouter reserves credit
+    # per request against those already in flight, so an 8-token ping returns 200
+    # while a 12-pack fan-out at concurrency 3 returns `402 Payment Required —
+    # would exceed your available credits given your current in-flight requests`
+    # on every row. That combination was observed on 2026-09-09. Hence the credit
+    # probe in the script, and hence the wording: reachable, not funded.
+    #
+    # The check itself lives in evals/paid/check-subject-model.sh because the
+    # REFRESH workflow must run it too: that is the one that actually spends the
+    # budget, so preflighting only this workflow would leave the expensive path
+    # unguarded (caught in review of #131).
+    #
+    # ADVISORY here on purpose: deliberately NOT in the behavioral gate's
+    # `needs`, so a dead subject key reports in seconds instead of turning a
+    # required check red across every open PR. Promoting it to a gate (adding it
+    # to the behavioral aggregate's needs + assess, exactly as grader-model is)
+    # is a one-line change and an owner decision.
+    name: confirm subject model resolves (advisory)
+    runs-on: ubuntu-latest
+    # Secrets aren't available to fork PRs, so only run where they are.
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+    steps:
+      - uses: actions/checkout@v4
+      - name: ping the configured subject model
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+        run: evals/paid/check-subject-model.sh
+
   # ── behavioral tier (promptfoo) ───────────────────────────────────────────
   # Paid LLM-rubric tier. Discovery enumerates the plugins that ship a promptfoo
   # pack (fail-closed: a plugin that DECLARES a pack but ships a broken/missing
@@ -268,6 +306,21 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
+      # Cap the burst. Every leg runs promptfoo at maxConcurrency 3, so an
+      # uncapped matrix of a dozen packs asks one OpenRouter key for ~36
+      # concurrent completions, and the routing tier's two packs pile on top in
+      # the same workflow. Run 35287312617 is what that looks like when the
+      # key's rate limit is below the demand: legs sat 38+ minutes, rows came
+      # back as RateLimitExhaustedError and "timed out after 300000ms in
+      # queue", and pass-rate.sh declared scenarios STARVED — a whole paid run
+      # yielding no verdict, with the account funded the entire time ($47.54).
+      #
+      # 4 is a starting point, not a measured optimum: it takes concurrent
+      # demand from ~36 to ~12, and the tier's wall-clock becomes three waves
+      # of legs rather than one. Tune it against the STARVED signal — that is
+      # the honest feedback channel here, since pass-rate.sh fails closed on
+      # FAULT-starved scenarios instead of scoring them.
+      max-parallel: 4
       matrix:
         plugin: ${{ fromJSON(needs.behavioral-detect.outputs.plugins) }}
     steps:
@@ -355,15 +408,43 @@ jobs:
         working-directory: plugins/${{ matrix.plugin }}/evals/promptfoo
         run: |
           test -f results.json || { echo "no results.json produced"; exit 0; }
+          # Vendor error text reaches this log REDACTED: OpenRouter's 402/429
+          # bodies arrive inside .error and embed a workspace key-management URL
+          # ending in the key's id, plus a user_id. Not the API key, so low
+          # severity — but a public Actions log is permanent, and none of the
+          # diagnosis needs it. Single-sourced in
+          # evals/paid/redact-vendor-ids.sh, shared with the subject preflight
+          # and the other two dumps in this file.
+          # TRANSPORT ERROR is printed FIRST and always. A row whose provider call
+          # errored has an EMPTY .response.output, so a dump that shows only the
+          # output renders a transport failure as a blank box that reads like the
+          # model returned nothing. That is not hypothetical: it cost two runs and
+          # a wrong public diagnosis on PR #131 — the real cause, an OpenRouter
+          # `402 Payment Required: would exceed your available credits given your
+          # current in-flight requests`, was sitting in .error the whole time while
+          # this dump showed empty output and the diagnosis said "empty completions".
           jq -r '
             (.results.results // .results // [])[]
             | select(.success == false)
             | "=== QUESTION ===\n" + ((.vars.question // .testCase.vars.question) | tostring)
+              + "\n\n=== TRANSPORT ERROR ===\n"
+              + ( (.failureReason) as $fr
+                  | (.error // .response.error // "") as $e
+                  | ($e | if type=="string" then . else tojson end) as $et
+                  | if ($fr == 2 or ($fr | type=="string" and (ascii_downcase == "error")))
+                       or (($fr == null or ($fr | type=="string" and (gsub("\\s";"") == "")))
+                           and ($et != ""))
+                    then $et
+                    else "(none — the provider answered; this row failed on the RUBRIC below, not on transport)"
+                    end )
+              + "  [failureReason: " + ((.failureReason // "unset") | tostring) + "]"
               + "\n\n=== SUBJECT OUTPUT ===\n" + ((.response.output // .output // "") | if type=="string" then . else tojson end)
               + "\n\n=== GRADER REASON(S) ===\n"
               + (([ .gradingResult.componentResults[]? | select(.pass==false) | .reason ] | join("\n---\n")))
               + "\n(overall reason: " + ((.gradingResult.reason // "") | tostring) + ")\n"
-          ' results.json || echo "jq parse fell through — see artifact"
+          ' results.json \
+            | "$GITHUB_WORKSPACE/evals/paid/redact-vendor-ids.sh" \
+            || echo "jq parse fell through — see artifact"
       - name: capture example snapshot from real results
         if: steps.touched.outputs.paid == 'true'
         # Extract this plugin's real with-skill/without-skill pair from the graded
@@ -575,7 +656,9 @@ jobs:
                      + (if slotmatch($e; $g) then "  [match]" else "  [DIFF]" end))
                  | join("\n"))
               + "\n"
-          ' results.json || echo "jq parse fell through — see artifact"
+          ' results.json \
+            | "$GITHUB_WORKSPACE/evals/paid/redact-vendor-ids.sh" \
+            || echo "jq parse fell through — see artifact"
       - name: show failing trajectory steps (per-slot expected vs got)
         if: always() && steps.touched.outputs.routing == 'true'
         working-directory: evals/routing/trajectory
@@ -611,7 +694,9 @@ jobs:
                      + (if slotmatch($e; $g) then "  [match]" else "  [DIFF]" end))
                  | join("\n"))
               + "\n"
-          ' trajectory-results.json || echo "jq parse fell through — see artifact"
+          ' trajectory-results.json \
+            | "$GITHUB_WORKSPACE/evals/paid/redact-vendor-ids.sh" \
+            || echo "jq parse fell through — see artifact"
       - name: upload routing results
         if: always() && steps.touched.outputs.routing == 'true'
         uses: actions/upload-artifact@v4

.github/workflows/refresh-examples.yml

@@ -40,6 +40,15 @@ jobs:
       - uses: actions/setup-node@v4
         with:
           node-version: 22
+      - name: preflight — the subject model must be callable before spending
+        # The refresh is the workflow that actually spends the budget, so the
+        # reachability check belongs HERE, not only in evals.yml (caught in
+        # review of #131). A revoked key or an exhausted balance would otherwise
+        # send this straight into a ~40-minute paid pack loop whose every
+        # real-skill row fails — the exact 50 minutes already burned twice.
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+        run: evals/paid/check-subject-model.sh
       - name: run packs, capture real example snapshots
         id: capture
         env:

docs/examples/PLAN.md

@@ -104,7 +104,7 @@ output and not typed by hand — and which model did what?
   plus `same_family_judge`. The cheap tier refuses a snapshot missing any of
   them. The gallery shows them on every card and in a page-level table.
 - **A model never grades its own family.** Every behavioral pack tests one
-  model (`openrouter:nvidia/nemotron-3-ultra-550b-a55b`) and grades with another
+  model (`openrouter:qwen/qwen3.8-flash`) and grades with another
   (`anthropic:messages:claude-sonnet-5`); the cheap tier now checks every pack
   for that, `capture-example.sh` refuses to write a same-family pair, and the
   gate rejects a graded snapshot whose subject and grader share a family. The

docs/examples/index.html

@@ -220,7 +220,7 @@ <h2>How to read a card</h2>
   <h2>Models used — by role</h2>
   <p>Three roles, disclosed separately on every card: the <strong>subject</strong> answered the prompt (both sides), the <strong>grader</strong> applied the pack's pass/fail rubric, the <strong>judge</strong> wrote the divergence verdict.
   The rule this repository enforces: <em>a model never grades its own family</em>. Every behavioral pack tests one model and grades with another
-  (12 packs: subject openrouter:nvidia/nemotron-3-ultra-550b-a55b, grader anthropic:messages:claude-sonnet-5); the cheap eval tier refuses a pack or a graded snapshot where the two share a family, and the capture script refuses to write one.
+  (12 packs: subject openrouter:qwen/qwen3.8-flash, grader anthropic:messages:claude-sonnet-5); the cheap eval tier refuses a pack or a graded snapshot where the two share a family, and the capture script refuses to write one.
   Seeds predate that rule, and the table says so instead of hiding it — they are replaced by CI-graded, attested pairs as the refresh workflow reaches each pack.</p>
   <div class="tablewrap"><table>
     <thead><tr><th>Produced by</th><th>Subject (answered)</th><th>Grader (pass/fail)</th><th>Judge (divergence verdict)</th></tr></thead>