feat(verifier): 入职考试 CI+校准回流+rubric shadow（W5-C3 .github#226，ADR-0072） - #64
Merged
Conversation
pipeline/verifier-exam/：run_exam.py（冻结校验 fail-closed+四分项门+成绩按 judge_id@版本@prompt_hash 存档+replay/api 判官适配）+calibrate.py（owner merge/reject 事件→校准样本静默回流+敏感度/特异度 Wilson CI+下界不及格自动 升人类信号）+rubric_shadow.py（五维 AI 可读性 rubric，仅记录不阻断+负债申报） +钉版考试集（freeze_hash 锚 exam-pin.yaml）+31 例自测；workflow verifier-exam （selftest/考试/拒上岗门语义演示/每日 drift 对账/校准回流/rubric shadow）。
|
Warning Review limit reached
Next review available in: 55 seconds Limit details: You’ve used all 10 included reviews currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (22)
Comment |
PR Summary by QodoAdd verifier entrance exam CI with calibration feedback and rubric shadow
AI Description
Diagram
High-Level Assessment
Files changed (22)
|
There was a problem hiding this comment.
Pull request overview
该 PR 为 verifier 入职考试体系引入完整的 CI 关卡实现：包含冻结考试集校验的考试 runner、基于 GitHub PR owner 行为的校准回流与 Wilson CI 计算、以及 AI 可读性 rubric 的 shadow 记录管道，并配套单元测试与 GitHub Actions 工作流来演示/执行“拒上岗门”语义。
Changes:
- 新增
run_exam.py：支持考试集冻结校验（fail-closed）、回放/真实判官两种模式、四分项门阈值判定与成绩存档键。 - 新增
calibrate.py：从 PR merge/reject 事件静默回流校准样本，并输出校正分与 Wilson CI（needs_human 仅信号不阻断）。 - 新增
rubric_shadow.py+ rubric/prompt/fixtures/workflow：五维 rubric shadow 出分与负债申报（仅记录不阻断），并通过 workflow 定时产出 artifacts。
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| pipeline/verifier-exam/run_exam.py | 考试 runner：冻结校验、分项评测、门禁判定、成绩落档与 replay/api 判官适配 |
| pipeline/verifier-exam/calibrate.py | 校准回流与出分校准：owner 行为采样、Wilson CI、needs_human 信号输出 |
| pipeline/verifier-exam/rubric_shadow.py | AI 可读性 rubric shadow：上下文提取、回放/（预留）api 打分、负债申报与记录落盘 |
| pipeline/verifier-exam/rubrics/ai-readability-v1.yaml | 五维 rubric 契约与锚档描述（含 annotation_debt_policy） |
| pipeline/verifier-exam/exam-policy.yaml | 考试门阈值真源（四分项指标与校准参数默认值） |
| pipeline/verifier-exam/exam-pin.yaml | 供应链钉版：archive 正本 freeze_hash 锚与 drift 检查依据 |
| pipeline/verifier-exam/examset/v1/manifest.json | 考试集冻结 manifest（逐文件 sha256 + freeze_hash） |
| pipeline/verifier-exam/examset/v1/rewardbench2-generative.jsonl | RewardBench2 形态的成对偏好题集（冻结副本） |
| pipeline/verifier-exam/examset/v1/llmbar-adversarial.jsonl | LLMBar 形态的对抗成对偏好题集（冻结副本） |
| pipeline/verifier-exam/examset/v1/null-canaries.jsonl | null-model 金丝雀集合（冻结副本） |
| pipeline/verifier-exam/prompts/v1/pairwise-judge.md | 成对偏好判题 prompt（参与 prompt_hash） |
| pipeline/verifier-exam/prompts/v1/canary-judge.md | 金丝雀判负 prompt（参与 prompt_hash） |
| pipeline/verifier-exam/judge-configs/glm-4.5-air.json | 候选判官示例配置（锁定采样参数并记录） |
| pipeline/verifier-exam/fixtures/judge-gold.json | 回放判官金标准 fixture（用于自测/CI） |
| pipeline/verifier-exam/fixtures/judge-canary-miss.json | 回放 fixture：单点金丝雀翻转用于拒上岗测试 |
| pipeline/verifier-exam/fixtures/judge-positional.json | 回放 fixture：位置偏差靶子（双序一致率应为 0） |
| pipeline/verifier-exam/fixtures/rubric-scores-ci.json | rubric shadow 的 CI 回放打分 fixture（含缺维触发负债申报） |
| pipeline/verifier-exam/tests/examutil.py | 测试工具：合成考试集/fixture/运行入口与记录读取 |
| pipeline/verifier-exam/tests/test_exam.py | run_exam 关键语义测试：冻结、金丝雀、双序边界、存档键、输入隔离 |
| pipeline/verifier-exam/tests/test_calibrate.py | calibrate 关键语义测试：Wilson、去重回流、needs_human 边界与非阻断 |
| pipeline/verifier-exam/tests/test_rubric_shadow.py | rubric shadow 关键语义测试：契约完整性、负债申报、不阻断 |
| .github/workflows/verifier-exam.yml | CI 工作流：selftest、考试（replay/api）、drift 对账、校准回流、rubric shadow artifacts |
Suppressed comments (1)
.github/workflows/verifier-exam.yml:97
- 同上，门语义演示步骤也固定使用
judge-configs/glm-4.5-air.json；如果 workflow_dispatch 选择了其他model_alias，演示与主考试会出现不一致。建议这里同样使用inputs.model_alias选择 judge-config。
python3 pipeline/verifier-exam/run_exam.py \
--judge-config pipeline/verifier-exam/judge-configs/glm-4.5-air.json \
--replay-fixture pipeline/verifier-exam/fixtures/judge-positional.json \
--out "$RUNNER_TEMP/rejected" --run-id gate-demo > /dev/null 2>&1
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+254
to
+263
| def gate(sections: dict, policy: dict) -> bool: | ||
| """任一分项 value < min → False（拒上岗）。阈值真源=policy，不硬编码。""" | ||
| for name, spec in policy.get("sections", {}).items(): | ||
| if name not in sections: | ||
| raise ExamError(f"policy 声明的分项缺结果: {name}") | ||
| metric = METRIC_KEY[spec["metric"]] | ||
| sections[name]["metric"] = metric | ||
| sections[name]["threshold"] = spec["min"] | ||
| sections[name]["pass"] = sections[name][metric] >= spec["min"] | ||
| return all(sections[n]["pass"] for n in policy.get("sections", {})) |
Comment on lines
+187
to
+195
| def _call(self, user_text: str) -> str: | ||
| pf = self.tmp / "verifier-exam-prompt.txt" | ||
| pf.write_text(user_text, encoding="utf-8") | ||
| r = subprocess.run(self.base_args + ["--prompt-file", str(pf), "--tag", self.tag], | ||
| capture_output=True, text=True, timeout=180) | ||
| if r.returncode != 0: | ||
| return "" # 调用失败=不可解析=计错（不中断整场考试，逐条记录） | ||
| return r.stdout | ||
|
|
Comment on lines
+123
to
+133
| pf = Path("rubric-prompt.txt") | ||
| pf.write_text(prompt, encoding="utf-8") | ||
| s = judge_cfg.get("sampling", {}) | ||
| args = ["bash", str(Path(repo_root) / "scripts" / "llm-call.sh"), | ||
| "--model", judge_cfg["model_alias"], "--prompt-file", str(pf), | ||
| "--tag", f"rubric-shadow@{judge_cfg.get('judge_id', 'unknown')}"] | ||
| if s.get("temperature") is not None: | ||
| args += ["--temperature", str(s["temperature"])] | ||
| r = subprocess.run(args, capture_output=True, text=True, timeout=180) | ||
| if r.returncode != 0: | ||
| raise RubricError(f"llm-call.sh 失败 rc={r.returncode}: {r.stderr[:200]}") |
Comment on lines
+105
to
+107
| args = ["gh", "api", f"repos/{repo}/pulls", | ||
| "-f", "state=closed", "-f", "sort=updated", "-f", "direction=desc", | ||
| "-f", "per_page=30"] |
Comment on lines
+77
to
+80
| python3 pipeline/verifier-exam/run_exam.py \ | ||
| --judge-config pipeline/verifier-exam/judge-configs/glm-4.5-air.json \ | ||
| --judge-mode "$JUDGE_MODE" \ | ||
| --out verifier-exam/results --run-id "$RUN_ID" > "$RUNNER_TEMP/exam.json" |
Code Review by Qodo
1. Freeze anchor never enforced
|
Comment on lines
+69
to
+72
| manifest = json.loads(mpath.read_text(encoding="utf-8")) | ||
| files, freeze = compute_freeze(exam_dir) | ||
| for name, info in manifest.get("files", {}).items(): | ||
| if name not in files: |
There was a problem hiding this comment.
1. Freeze anchor never enforced 🐞 Bug ⛨ Security
load_and_verify() validates the vendored files only against their adjacent, PR-editable manifest and never checks exam-pin.yaml, so changing the exam files and recomputing the manifest is accepted without creating a new version. The daily drift job also compares only the upstream manifest metadata to the pin, not the vendored or upstream file bytes, so it cannot detect this replacement.
Agent Prompt
## Issue description
The runner trusts a manifest that can be changed together with the frozen exam, so the pinned freeze hash is not enforced.
## Issue Context
Load `exam-pin.yaml`, require its version and freeze hash to match the locally recomputed exam, and make drift verification recompute both upstream and vendored file hashes rather than trusting upstream manifest metadata.
## Fix Focus Areas
- pipeline/verifier-exam/run_exam.py[64-83]
- pipeline/verifier-exam/exam-pin.yaml[8-14]
- .github/workflows/verifier-exam.yml[146-160]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+105
to
+107
| args = ["gh", "api", f"repos/{repo}/pulls", | ||
| "-f", "state=closed", "-f", "sort=updated", "-f", "direction=desc", | ||
| "-f", "per_page=30"] |
There was a problem hiding this comment.
2. Collection sends post requests 🐞 Bug ≡ Correctness
fetch_events_online() adds -f parameters without --method GET, which makes gh api send POST requests to read-only pulls and reviews endpoints. Scheduled collection therefore fails before producing calibration samples.
Agent Prompt
## Issue description
`gh api -f` changes these collection calls to POST, but both endpoints must be read with GET.
## Issue Context
Add `--method GET` to both pulls and reviews calls while preserving query parameters.
## Fix Focus Areas
- pipeline/verifier-exam/calibrate.py[103-123]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+80
to
+82
| return {"schema": SCHEMA_SAMPLE, "sample_id": "cal-" + hashlib.sha256(dk.encode()).hexdigest()[:10], | ||
| "dedup_key": dk, "ts": ts, "source": "owner-action", "repo": repo, | ||
| "pr": pr["number"], "owner_action": action, "judge_verdict": None} |
There was a problem hiding this comment.
3. Verdicts never join samples 🐞 Bug ≡ Correctness
Every collected calibration sample is created with judge_verdict: null, and neither the CLI nor workflow contains a step that fills it. Since score() accepts only positive/negative verdicts, calibration permanently remains insufficient-calibration and never computes sensitivity or specificity.
Agent Prompt
## Issue description
Collected owner actions never receive corresponding judge verdicts, so no calibration pair can be scored.
## Issue Context
Add a deterministic join key and an ingestion/join command or collect the verifier verdict with each event before invoking `score`.
## Fix Focus Areas
- pipeline/verifier-exam/calibrate.py[76-82]
- pipeline/verifier-exam/calibrate.py[135-151]
- .github/workflows/verifier-exam.yml[183-195]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+57
to
+58
| env: | ||
| JUDGE_MODE: ${{ inputs.judge_mode || 'replay' }} # zizmor：模板值经 env 间接引用 |
There was a problem hiding this comment.
4. Calibration uses replay accuracy 🐞 Bug ≡ Correctness
Scheduled runs have no workflow-dispatch inputs, so JUDGE_MODE defaults to replay and the calibration job receives the gold fixture's exam accuracy rather than a real verifier's score. Even after joins are implemented, reports would calibrate the replay fixture instead of the deployed judge.
Agent Prompt
## Issue description
The scheduled calibration pipeline always feeds replay accuracy into calibration.
## Issue Context
Source raw score and verdicts from the actual licensed verifier being calibrated, or run a credentialed API exam in a separately protected scheduled job.
## Fix Focus Areas
- .github/workflows/verifier-exam.yml[55-58]
- .github/workflows/verifier-exam.yml[162-195]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+70
to
+74
| - name: 入职考试（任一分项不过=exit 1 拒上岗） | ||
| id: exam | ||
| env: | ||
| LLM_API_KEY: ${{ secrets.LLM_API_KEY }} | ||
| RUN_ID: ${{ github.run_id }} # zizmor：模板值经 env 间接引用 |
There was a problem hiding this comment.
5. Secret reaches pr code 🐞 Bug ⛨ Security
The exam step unconditionally injects LLM_API_KEY while executing run_exam.py from the pull request checkout, including in default replay mode. A same-repository PR that modifies the runner can read or exfiltrate the organization secret despite the comment claiming PR-modifiable code cannot access it.
Agent Prompt
## Issue description
A secret is exposed to code controlled by pull-request changes.
## Issue Context
Do not inject the key in replay/PR jobs. Put credentialed API execution in a protected workflow/job that runs trusted code and is not triggered from an untrusted PR checkout.
## Fix Focus Areas
- .github/workflows/verifier-exam.yml[17-25]
- .github/workflows/verifier-exam.yml[60-80]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+188
to
+195
| python3 pipeline/verifier-exam/calibrate.py collect \ | ||
| --repo "$REPO_FULL" --out calibration/samples.jsonl | ||
| - name: 出分即校准（CI 下界<及格线→needs_human 信号，不阻断） | ||
| run: | | ||
| set -euo pipefail | ||
| python3 pipeline/verifier-exam/calibrate.py score \ | ||
| --samples calibration/samples.jsonl --raw-score "${RAW_ACC:-0.0}" \ | ||
| --out calibration/report.json |
There was a problem hiding this comment.
10. Calibration history never accumulates 🐞 Bug ≡ Correctness
Each scheduled calibration run starts from a fresh checkout and creates a new local calibration/samples.jsonl, because the workflow only uploads the file after scoring and never restores a prior artifact or other durable store. As a result, append_samples can only deduplicate/accumulate within a single run, and the seven-day collector window causes older calibration evidence to be permanently lost over time.
Agent Prompt
## Issue description
Scheduled calibration runs do not restore prior `calibration/samples.jsonl` data, so each run starts with an empty/new local samples file and `append_samples` only deduplicates/accumulates within that single run. This prevents calibration history from building over time and causes events older than the collector window (e.g., seven days) to be permanently lost.
## Issue Context
The workflow performs a fresh checkout on each scheduled job, writes and scores `calibration/samples.jsonl`, and uploads it as an artifact, but does not download/restore that artifact (or otherwise persist the JSONL) at the start of the next run. Implement a durable storage approach with an explicit restore/update strategy (e.g., repository-backed versioned store or artifact download before collection and upload after scoring) so JSONL history and deduplication persist across runs, and ensure concurrent scheduled executions cannot overwrite each other.
## Fix Focus Areas
- .github/workflows/verifier-exam.yml[176-203]
- pipeline/verifier-exam/calibrate.py[85-100]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+119
to
+122
| rv = subprocess.run(["gh", "api", f"repos/{repo}/pulls/{pr['number']}/reviews", | ||
| "-f", "per_page=30"], capture_output=True, text=True, timeout=60) | ||
| if rv.returncode == 0: | ||
| pr["_reviews"] = json.loads(rv.stdout) |
There was a problem hiding this comment.
11. Review evidence is truncated 🐞 Bug ≡ Correctness
Reject classification examines only the first 30 reviews of an unmerged closed PR. If its CHANGES_REQUESTED review appears on a later page, the PR is treated as ambiguous and skipped despite having the required rejection evidence.
Agent Prompt
## Issue description
Review collection requests only one page of 30 reviews. This can omit a `CHANGES_REQUESTED` review and prevent a closed PR from becoming a reject calibration sample.
## Issue Context
`classify_pr` depends on `_reviews` containing a `CHANGES_REQUESTED` state. Fetch all relevant review pages before classification.
## Fix Focus Areas
- pipeline/verifier-exam/calibrate.py[66-73]
- pipeline/verifier-exam/calibrate.py[117-123]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+77
to
+80
| python3 pipeline/verifier-exam/run_exam.py \ | ||
| --judge-config pipeline/verifier-exam/judge-configs/glm-4.5-air.json \ | ||
| --judge-mode "$JUDGE_MODE" \ | ||
| --out verifier-exam/results --run-id "$RUN_ID" > "$RUNNER_TEMP/exam.json" |
There was a problem hiding this comment.
12. Selected model is ignored 🐞 Bug ≡ Correctness
The workflow dispatch exposes a model_alias selector, but the exam command always loads the checked-in fixed GLM config and never uses the input, so dispatching a different candidate still evaluates glm-4.5-air. This makes API model selection ineffective and can mislead results by recording the wrong model identity, preventing the intended candidate-model entrance exam workflow.
Agent Prompt
## Issue description
The workflow-dispatch `model_alias` input is declared but never consumed by the exam execution. As a result, the exam always runs with the static checked-in GLM configuration (and records `glm-4.5-air`), so selecting another candidate model via API has no effect and produces misleading model identity/results.
## Issue Context
Update the workflow and runner so the requested alias meaningfully affects which judge/model configuration is used during the exam. Either (a) resolve `model_alias` to a trusted, allow-listed judge config path/identifier, or (b) generate an ephemeral configuration from the input while keeping fixed prompts/sampling/settings for comparability; alternatively, if dynamic selection is not intended, remove `model_alias` and require explicit config selection.
## Fix Focus Areas
- .github/workflows/verifier-exam.yml[23-27]
- .github/workflows/verifier-exam.yml[70-80]
- pipeline/verifier-exam/run_exam.py[169-184]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+171
to
+181
| s = judge_config.get("sampling") or {} | ||
| self.model = judge_config["model_alias"] | ||
| self.base_args = ["bash", str(repo_root / "scripts" / "llm-call.sh"), "--model", self.model] | ||
| if s.get("temperature") is not None: | ||
| self.base_args += ["--temperature", str(s["temperature"])] | ||
| if s.get("max_tokens") is not None: | ||
| self.base_args += ["--max-tokens", str(s["max_tokens"])] | ||
| if s.get("thinking"): | ||
| self.base_args += ["--thinking", str(s["thinking"])] | ||
| if s.get("seed") is not None: | ||
| self.base_args += ["--seed", str(s["seed"])] |
There was a problem hiding this comment.
13. Top-p setting is not applied 🐞 Bug ≡ Correctness
The judge records top_p as a locked sampling parameter in the config and archives it in results as if applied, but ApiJudge never forwards it and llm-call.sh has no --top-p option. This can cause actual API behavior to differ from the sampling configuration claimed in the result, undermining reproducibility when provider defaults differ from the config.
Agent Prompt
## Issue description
`top_p` is accepted in judge configurations and is archived in results as if it were applied, but `ApiJudge` never forwards it to provider calls and `llm-call.sh` does not currently accept or transmit a top-p value, so the recorded sampling configuration may not match actual API behavior.
## Issue Context
Implement consistent `top_p` support end-to-end (validate/accept via the wrapper, include in provider requests, and pass through from `ApiJudge`), and ensure any usage/recording schema reflects what was truly applied; alternatively, if `top_p` is not supported for a given pathway/provider, reject configs containing unsupported sampling fields or avoid recording them as “locked/applied” to prevent false reproducibility claims.
## Fix Focus Areas
- pipeline/verifier-exam/run_exam.py[169-181]
- pipeline/verifier-exam/run_exam.py[266-281]
- pipeline/verifier-exam/judge-configs/glm-4.5-air.json[5-10]
- scripts/llm-call.sh[24-57]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+135
to
+138
| for d in DIMENSIONS: | ||
| m = re.search(rf"^DIM\s+{d}\s+([01](?:\.\d+)?)\s*$", r.stdout, re.M | re.I) | ||
| dims[d] = {"score": float(m.group(1)) if m else None, | ||
| "basis": "api" if m else "unparseable"} |
There was a problem hiding this comment.
14. Api scores exceed rubric range 🐞 Bug ☼ Reliability
The rubric API parser accepts values such as 1.5 even though the rubric scale is [0,1], then includes them directly in the recorded dimensional mean. A malformed judge response can therefore produce an impossible shadow score instead of an explicit no-data debt.
Agent Prompt
## Issue description
The API-mode score regex accepts decimals above one (for example `1.5`) and stores them without validation, violating the rubric's declared `[0,1]` scale.
## Issue Context
Treat any missing, duplicate, malformed, non-finite, or out-of-range dimension score as unparseable/no-data so annotation debt is emitted rather than calculating an invalid mean.
## Fix Focus Areas
- pipeline/verifier-exam/rubric_shadow.py[119-139]
- pipeline/verifier-exam/rubric_shadow.py[166-182]
- pipeline/verifier-exam/rubrics/ai-readability-v1.yaml[5-7]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
动机
宪法 §4C 持证上岗制 / ADR-0072：verifier 入职考试（CI 关卡）+ 校准集循环 +
AI 可读性 rubric shadow 的操作面。范式署名：LLM-as-a-Verifier（arXiv:2607.05391）、
RewardBench2/LLMBar（考试集形态）、MT-Bench 位置交换（arXiv:2306.05685）、
null-model 攻击实证（arXiv:2410.07137）、敏感度/特异度校准（arXiv:2511.21140，ICML'26）。
考试集冻结正本在 archive（配套 PR：archive
w5c3-evalsets）；执照面在 agent-registry（配套 PR：agent-registry
w5c3-registry）。变更清单
pipeline/verifier-exam/run_exam.py：考试 runner——冻结校验 fail-closed（哈希不符=exit 2不许开考）；四分项门（rb2 ≥0.70 / llmbar ≥0.60 / 金丝雀 negative_rate=1.00 / 双序 ≥0.90，
阈值真源 exam-policy.yaml）；任一不过=exit 1 拒上岗；成绩按
judge_id@exam_version@prompt_hash12存档（JSONL，同键追加留史）；replay（零真实 LLM）/api（经 scripts/llm-call.sh 计量唯一入口）判官适配；输入隔离（黄金标签永不进判官通道）。
pipeline/verifier-exam/calibrate.py：校准回流（AC-3）——owner merge/reject GitHub 事件→校准样本（dedup 幂等，零额外操作）；出分带校正值±Wilson CI；CI 下界<及格线→
needs_human 信号（不阻断，exit 0）；owner 确认率 ≥20% 指标位（从 0 起累积）。
pipeline/verifier-exam/rubric_shadow.py+rubrics/ai-readability-v1.yaml（AC-4）：五维 rubric（可定位性/入口清晰度/模块深度/命名词表/示例新鲜度）判据分解+每档锚定描述；
仅记录不阻断；标注负债显式申报（annotation_debt 字段）。
pipeline/verifier-exam/examset/v1/（钉版副本，与 archive 正本字节一致）+exam-pin.yaml（freeze_hash 供应链锚）+ prompts/v1 + judge-configs + fixtures。pipeline/verifier-exam/tests/：31 例（零真实 LLM）。.github/workflows/verifier-exam.yml：selftest / 考试（无凭据=回放模式）+ 拒上岗门语义演示（位置偏差判官必须被拒）/ 成绩存档断言 / 每日 freeze drift 对账（archive 正本 vs pin）/
每日校准回流 / rubric shadow（全 job 顶层最小权限+timeout+concurrency，actions SHA 钉版）。
AC 映射（.github#226）
全锁定）When
verifier-examworkflow 运行 Then 考试集全过才可注册（执照注册校验在agent-registry 配套 PR，replay 成绩被拒）；任一分项不过 exit 1；成绩按
judge_id@版本@prompt_hash 存档（workflow
verifier-exam-resultsartifact，键断言步骤）。test_every_real_canary_judged_negative逐条断言 24/24；单点翻转（judge-canary-miss fixture）→ negative_rate=23/24 → exit 1。
calibrate.py collect消费GitHub PR 事件）Then 自动成为校准样本（dedup_key 幂等，零额外操作）；出分=校正值±Wilson CI；
test_boundary_ci_lower_equals_pass_line_not_escalated钉死"CI 下界严格低于及格线才升人类"。（
test_all_zero_scores_still_exit_zero：全零极端分仍 exit 0 + blocking=false）；标注负债有申报记录（annotation_debt：数据不足维度显式 insufficient-data+reason）。
测试方法（本地已跑，零网络零真实推理）
python3 -m unittest discover -s pipeline/verifier-exam/tests -v→ 31 例全绿：金丝雀 24/24 判负、双序边界（gate 函数 0.89 拒/0.90 过 + 100 条集成 0.90 过/0.89 拒）、
位置偏差判官 agreement=0.0 被拒、冻结篡改 exit 2、存档键完整性（采样参数变同键/
prompt 变新键）、Wilson 已知值 (0.7,10)→(0.3965,0.8920)、rubric 不阻断+负债申报。
run_exam.py（gold 轨道 exit 0：四分项 1.0/1.0/1.0/1.0；positional 轨道 exit 1：dual_order=0.0）；workflow 全 run 块
bash -n通过、YAML 可解析。风险与回滚
llm-call.sh 失败=分项计错（fail-closed）；校准样本暂存 artifact（retention 90 天）→
容量上来后转仓库存档（follow-up）。
不受影响（ADR-0072 回滚条款）；rubric shadow 本身零阻断。
Card: Cloudbird-Software/.github#226
ADR: ADR-0072（archive
adr/ADR-0072-verifier-entrance-exam-calibration.md）