Add a frozen concise-review adaptation experiment

.gitignore

@@ -39,3 +39,5 @@ plugins/*/evals/promptfoo/results.matrix.json
 plugins/*/evals/promptfoo/promptfooconfig.regrade.*.yaml
 plugins/*/evals/promptfoo/replay.*.json
 plugins/*/evals/promptfoo/results.regrade.*.json
+# Adaptive-review raw outputs, human labels, and judge evidence stay private.
+/evals/paid/adaptive-review/runs/

docs/testing.md

@@ -33,6 +33,7 @@ structurally cannot.
 | [subject-model matrix](#subject-model-matrix-manual-advisory) | `subject-matrix.yml` + `evals/paid/subject-matrix.sh` | (1 + subjects) × the pack's usual cents | manual dispatch only | no — advisory; the baseline subject decides the job, extra subjects never do |
 | [calibration sheet](#calibration-sheet-manual-no-model-calls) | `calibration-sheet.yml` + `evals/paid/calibration/` | free (no model calls) | manual dispatch only | no — writes a blind sheet to a `calibration/<run-id>` branch for a human to label |
 | [grader agreement](#grader-agreement-manual-grading-only) | `grader-agreement.yml` + `evals/paid/calibration/regrade.sh` | grading spend only (no subject calls) | manual dispatch only | no — reports agreement and kappa between graders on a finished run's outputs |
+| [adaptive-review experiment](#adaptive-review-experiment-offline-contract) | `evals/paid/adaptive-review/` | offline contract is free; actor/judge runs are separately authorized | contract self-test on every cheap run; inference is manual only | no — reports improvement, regression, or inconclusive; never installs |
 | [scale](#scale-tier) | `plugins/{redgate,agent-compiler}/evals/scale/` | free, offline, minutes | path-gated (`plugins/redgate/**`, `plugins/agent-compiler/**`) | no — evidence, not a merge gate |
 | [deep](#deep-tier-pier) | `plugins/<p>/evals/pier/` | dollars + minutes (sandboxed agents) | path-gated to the safety surface (`plugins/*/skills/**/scripts/**`, `plugins/*/evals/pier/**`) | yes — `deep tier (pier)` (aggregate) |
 | [example gallery](#example-gallery-refresh--pages) | `refresh-examples.yml` / `pages.yml` | real API budget per refresh | scheduled (1st + 15th, 06:00 UTC) / on `docs/**` push to main | no — review-gated PR / publish |
@@ -317,6 +318,27 @@ that it is green because it did not run, never silently.
   `evals/paid/calibration/regrade.sh --self-test`; the cheap tier §18b
   fixture-tests the overlay and the replay provider.
 
+## adaptive-review experiment (offline contract)
+
+- **What it proves.** `evals/paid/adaptive-review/self-test.sh` proves that the
+frozen public corpus and both `agent-compiler` images have not drifted, and
+that the experiment-specific scorer distinguishes improvement, regression,
+and inconclusive fixtures. It also fails closed when external judge inputs
+lack a separate authorization bound to their exact digest.
+- **What it cannot prove.** The offline fixtures are scorer tests, not evidence
+that concise review helps a person. `status.json` therefore remains
+inconclusive until the pinned local actor, blind human labels, and independent
+frontier judge all produce retained evidence. Detailed run artifacts stay in
+the git-ignored private `runs/` directory.
+- **Fires.** The contract self-test runs inside every cheap-tier invocation.
+Actor and judge inference are manual only; there is no workflow, schedule,
+fallback route, automatic installation, or new required check.
+- **Cost.** Free for the committed self-test. Any external grading needs its own
+input-bound authorization and positive spending limit.
+- **Local run.** `evals/paid/adaptive-review/self-test.sh`; the experiment
+procedure and evidence schemas are in
+`evals/paid/adaptive-review/README.md`.
+
 ## scale tier
 
 - **What it proves.** The same invariants the cheap tier proves once, held

evals/cheap/run.sh

@@ -454,6 +454,20 @@ if [ -f "evals/paid/discover-paid-packs.sh" ]; then
   fi
 fi
 
+# --- 11c. Adaptive-review experiment contract -------------------------------
+# The experiment never calls a model here. This recompiles both treatment
+# images, verifies the frozen corpus, and exercises the evidence scorer's
+# improvement/regression/inconclusive and authorization branches.
+if [ -f "evals/paid/adaptive-review/self-test.sh" ] \
+   && [ -f "plugins/agent-compiler/scripts/compile.py" ]; then
+  group "adaptive-review experiment self-test"
+  if bash evals/paid/adaptive-review/self-test.sh >/dev/null 2>&1; then
+    ok "adaptive-review frozen experiment and scorer self-test"
+  else
+    bad "adaptive-review experiment drifted or its scorer no longer fails closed"
+  fi
+fi
+
 # --- 12. Install-smoke coverage (every registered plugin) -------------------
 # The install smoke test (ci/install-smoke.sh) proves ONE plugin installs
 # structurally. In CI it is fanned out over a matrix enumerated from

evals/paid/adaptive-review/README.md

@@ -0,0 +1,125 @@
+# Adaptive review v1 experiment
+
+This is the smallest benefit experiment for
+[#117](https://github.com/JRichlen/agent-plugins/issues/117). It is **not an
+installed plugin, scheduled job, model driver, or new general eval harness**.
+Runtime expansion remains paused. `status.json` stays `inconclusive` until a
+qualified run supplies complete evidence.
+
+## What is frozen
+
+- `corpus.json`: two development and six held-out excerpts from public project
+  issues and pull requests. `corpus.sha256` detects any post-registration edit.
+- `queries/`: baseline and one candidate AgentQuery.
+- `registry/`: one candidate behavior plus the two views. The committed
+  `compiled/` images and agents are generated by the existing
+  `agent-compiler`; `self-test.sh` recompiles and compares them byte for byte.
+- `experiment.json`: the feedback source, scoped inspectable preference,
+  qualified actor route, thresholds, approval boundary, monitoring rule, and
+  rollback. `experiment.sha256` makes threshold changes explicit.
+
+The recurrence record is intentionally honest: #117 explicitly selected this
+experiment, but no three cited sightings exist, so it makes no
+`recurrence-detector` threshold claim.
+
+## Preregistered decision
+
+Only the six held-out pairs count. **Improvement** requires all of:
+
+- at least 4 candidate wins and at most 1 baseline win under blind human review;
+- mean comprehension lift at least `+0.75` on a 1–5 scale;
+- median reduction of at least 1 substantive edit before posting;
+- every candidate output judged acceptable by the human;
+- zero required facts or actions missing from candidate outputs according to
+  either human or judge;
+- no truncated/non-stop actor response;
+- independent frontier-judge agreement with human acceptability labels of at
+  least `0.80`, Cohen's kappa of at least `0.60` when label variance makes it
+  defined, at least `0.80` winner agreement, and at least 4 judge candidate
+  wins.
+
+A lost candidate requirement, more baseline than candidate wins, or negative
+comprehension/edit delta is a **regression** and may reject the candidate
+without paying for a judge. Missing judge evidence blocks an improvement but
+does not erase a human-observed regression. Everything else is **inconclusive**.
+Even improvement produces only
+`eligible-for-scoped-human-approval`; the scorer never installs anything.
+
+## Run contract
+
+1. Run both committed agents against the identical rendered prompt for every
+   corpus row. The actor must be a pinned local Qwen 27B, direct 32K context,
+   one sequence, with no external fallback. Use the approved
+   `agent-request/v1` capability when it becomes available; do not duplicate
+   its transport or authority semantics here.
+2. Retain a private `run.json` with schema `adaptive-review-run/v1`: exact git,
+   experiment, corpus, prompt, and image hashes; actor model digest; a private
+   random blinding salt; every raw output; finish reason; and numeric input
+   tokens, output tokens, and cost for every call. Generated run directories
+   are git-ignored because detailed lab evidence stays private. Keep `run.json`
+   hidden from the human labeler because it contains the arm mapping.
+   Run from a clean committed checkout: the evaluator resolves the recorded
+   commit and verifies its experiment, corpus, prompt, and compiled images.
+3. Create the human sheet:
+
+   ```sh
+   python3 evaluate.py blind --run runs/<id>/run.json \
+     --out runs/<id>/human-labels.json
+   ```
+
+   The sheet contains the source task, required facts/actions, frozen rubric,
+   and output hashes/texts, but no arm names. Output order is randomized with
+   system entropy and IDs are salted, so the mapping cannot be reconstructed
+   from the public experiment. For each output, label comprehension (1–5),
+   minimum substantive edits before posting, acceptability, and missing
+   required IDs; then choose a winner or tie.
+4. Give the exact same output set and task requirements to an independent
+   frontier judge from another model family. Its record must include pinned
+   model identity and a separate authorization receipt bound to the exact
+   canonical review-input digest and a finite positive spending limit. Preserve
+   the receipt, labels, and total usage as
+   `adaptive-review-judge-labels/v1`.
+5. Evaluate without rerunning either model:
+
+   ```sh
+   python3 evaluate.py evaluate \
+     --run runs/<id>/run.json \
+     --human runs/<id>/human-labels.json \
+     --judge runs/<id>/judge-labels.json \
+     --out runs/<id>/report.json
+   ```
+
+The evaluator rejects contaminated hashes, missing pairs, changed review
+inputs, unknown requirement IDs, unqualified actors, missing or same-family
+judge identity, unbound external inputs, non-finite/unknown usage, and
+overspend. Missing judge evidence cannot advance the candidate.
+
+The authorization receipt is inspectable provenance, not cryptographic
+attestation. “Held-out” means excluded from development runs; these public
+repository excerpts were not hidden from the experiment author.
+
+## Lifecycle and rollback
+
+The corpus captures the goal (**understand**); the compiled view is the sole
+treatment (**select**); the qualified actor produces retained outputs
+(**act**); blind labels record edits and comprehension (**collect feedback**);
+the scorer compares paired arms (**compare**); only an explicit human may
+approve a review-only install (**approve or reject**). Any approved install
+must record the baseline image hash, repeat this held-out check after material
+feedback or model changes (**monitor**), and restore that baseline/remove the
+candidate on regression (**roll back**).
+
+The preference is visible in `experiment.json`, limited to issue review,
+corrected by versioning this experiment, and deleted by removing the candidate.
+There is no hidden profile, schedule, remote memory, learning/coding transfer,
+or automatic curation.
+
+## Offline verification
+
+```sh
+evals/paid/adaptive-review/self-test.sh
+```
+
+This recompiles both arms, verifies the corpus checksum, exercises improvement,
+regression, inconclusive, and missing-authorization fixtures, and confirms the
+checked-in status cannot imply an unrun benefit.

evals/paid/adaptive-review/compiled/baseline.image.json

@@ -0,0 +1 @@
+{"behavior":[],"capabilities":[],"compilerVersion":"0.1.0","diagnostics":[],"effectCeiling":["filesystem:read"],"effects":[],"hash":"sha256:ef278a1088b6f09cb60582b8fb3b9b565c4900ed9e17fb55b84e572a80bfe0ec","query":{"effectCeiling":["filesystem:read"],"name":"issue-review-baseline","role":"reviewer","stance":["goal-aligned"],"task":"issue-review","views":["view.issue-review-baseline"]},"registryRevision":"sha256:7079b3cc6b6ec21b2e1dc92a339dd500efb422ff250b7aff634be3e7c732bc8b","schemaVersion":"0.1","views":["view.issue-review-baseline"]}

evals/paid/adaptive-review/compiled/baseline.md

@@ -0,0 +1,11 @@
+---
+name: issue-review-baseline
+description: Compiled reviewer agent for issue-review. Effects capped at: filesystem:read.
+---
+
+<!-- compiled by agent-compiler; imageHash: sha256:ef278a1088b6f09cb60582b8fb3b9b565c4900ed9e17fb55b84e572a80bfe0ec; rendererVersion: 0.1.0; DO NOT EDIT BY HAND — edit the registry modules and recompile -->
+
+# issue-review-baseline
+
+Stance: goal-aligned.
+

evals/paid/adaptive-review/compiled/candidate.image.json

@@ -0,0 +1 @@
+{"behavior":[{"content":"Replace chronology and repeated rationale with the shortest causal context needed to understand why each retained fact or action matters.","id":"behavior.issue-review-concise#compress-context","kind":"rule","provenance":{"lines":[21,23],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"},"strength":"must"},{"content":"Would deleting this sentence change a decision, constraint, risk, or action? If not, remove it.","id":"behavior.issue-review-concise#deletion-test","kind":"probe","provenance":{"lines":[29,31],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"}},{"content":"Lead with the decision, requested outcome, or next action the issue exists to support.","id":"behavior.issue-review-concise#lead-with-goal","kind":"rule","provenance":{"lines":[9,11],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"},"strength":"must"},{"content":"Separate decisions, actions, gates, and deferred work so a maintainer can scan what happens next without rereading the background.","id":"behavior.issue-review-concise#make-actions-scannable","kind":"rule","provenance":{"lines":[25,27],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"},"strength":"must"},{"content":"Can every required fact and action in the source be located in the revision without inference?","id":"behavior.issue-review-concise#preservation-test","kind":"probe","provenance":{"lines":[33,35],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"}},{"content":"Preserve every requirement, constraint, unresolved risk, and explicit action that can change the decision or its execution.","id":"behavior.issue-review-concise#preserve-requirements","kind":"rule","provenance":{"lines":[13,15],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"},"strength":"must"},{"content":"Use only domain lenses that affect the stated goal; omit generic review categories that produce no finding.","id":"behavior.issue-review-concise#select-lenses","kind":"rule","provenance":{"lines":[17,19],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"},"strength":"must"},{"content":"Never call an issue concise by dropping a required fact, action, limitation, or approval boundary.","id":"behavior.issue-review-concise#shorter-by-omission","kind":"antipattern","provenance":{"lines":[37,39],"module":"behavior.issue-review-concise","source":"behaviors/concise-review.md","version":"1.0.0"}}],"capabilities":[],"compilerVersion":"0.1.0","diagnostics":[],"effectCeiling":["filesystem:read"],"effects":[],"hash":"sha256:50140878d2d6d271604ae745ff607c8e9bc710fe2ba376d7cef034016e2680f6","query":{"effectCeiling":["filesystem:read"],"name":"issue-review-concise","role":"reviewer","stance":["goal-aligned"],"task":"issue-review","views":["view.issue-review-concise"]},"registryRevision":"sha256:7079b3cc6b6ec21b2e1dc92a339dd500efb422ff250b7aff634be3e7c732bc8b","schemaVersion":"0.1","views":["view.issue-review-concise"]}

evals/paid/adaptive-review/compiled/candidate.md

@@ -0,0 +1,28 @@
+---
+name: issue-review-concise
+description: Compiled reviewer agent for issue-review. Effects capped at: filesystem:read.
+---
+
+<!-- compiled by agent-compiler; imageHash: sha256:50140878d2d6d271604ae745ff607c8e9bc710fe2ba376d7cef034016e2680f6; rendererVersion: 0.1.0; DO NOT EDIT BY HAND — edit the registry modules and recompile -->
+
+# issue-review-concise
+
+Stance: goal-aligned.
+
+## Rules
+
+- **MUST** Replace chronology and repeated rationale with the shortest causal context needed to understand why each retained fact or action matters.  <!-- behavior.issue-review-concise#compress-context -->
+- **MUST** Lead with the decision, requested outcome, or next action the issue exists to support.  <!-- behavior.issue-review-concise#lead-with-goal -->
+- **MUST** Separate decisions, actions, gates, and deferred work so a maintainer can scan what happens next without rereading the background.  <!-- behavior.issue-review-concise#make-actions-scannable -->
+- **MUST** Preserve every requirement, constraint, unresolved risk, and explicit action that can change the decision or its execution.  <!-- behavior.issue-review-concise#preserve-requirements -->
+- **MUST** Use only domain lenses that affect the stated goal; omit generic review categories that produce no finding.  <!-- behavior.issue-review-concise#select-lenses -->
+
+## Probes to run
+
+- Would deleting this sentence change a decision, constraint, risk, or action? If not, remove it.  <!-- behavior.issue-review-concise#deletion-test -->
+- Can every required fact and action in the source be located in the revision without inference?  <!-- behavior.issue-review-concise#preservation-test -->
+
+## Never
+
+- Never call an issue concise by dropping a required fact, action, limitation, or approval boundary.  <!-- behavior.issue-review-concise#shorter-by-omission -->
+