bench: commit curated-1 discrimination report pages (golden-curated-3) - #113
Conversation
…lden-curated-3 The four-model full passes (opus xhigh/full reference, deepseek, m3, m2.7 all high/full on curated-1) were scored against the current committed golden (golden-curated-3, 158 entries) and their report pages written to bench/reports/. These pages are the reproducible artifact for the PR Review Bench discrimination result: recall 0.826 (opus run 1) vs 0.109 (deepseek), 0.072 (m3), 0.014 (m2.7) - a monotone ladder far beyond the measured 3sigma noise floor. Report pages were previously claimed committed but none existed for the curated-1 discrimination runs.
There was a problem hiding this comment.
Now I need to run the Step 4b-i jq filter manually. The changed files are all in bench/reports/ with .md extension. Let me evaluate the judgment-rule triggers against those paths.
Step 4b-i — Judgment-rule candidate computation:
The changed files from the diff stat:
bench/reports/7b305997dc2b9b2f734f1499922286b7a8881ea5bd1967fad49275b83a8f1ba8.mdbench/reports/d4558caff3dfefdff574d9a14345bb4007db9cb94a69c64e2e859c7d6e8b2dab.mdbench/reports/d5a15f168212fe61bdf112b08c7a1787058cc5cd26a2a2d6c35a545465de79ca.mdbench/reports/edb935210ee850e34cb521e7c50482923a65761bd4d66501c153a402f5f8f8eb.md
Checking each judgment rule's trigger globs against these paths:
**/*.go→ no Go files**/main.go→ no main.goCHANGELOG.md→ not CHANGELOG.mdCLAUDE.md→ not CLAUDE.mdgo.mod→ not go.modMakefile→ not Makefileagents/**/*.md,commands/**/*.md,skills/**/*.md→ paths arebench/reports/**/*.md, not agent/command/skill files
No judgment rules have triggers matching the changed files. Applicable set is empty.
Step 4c-sel CLASSIFY:
{"applicable": [], "skipped": {}}Step 4d-sel ADJUDICATE: Since applicable set is empty, no rule-block reads needed.
Mechanical findings (pre-computed): 1 SHOULD finding from changelog/unreleased-entry-required.
Now checking for context-specific conventions — CHANGELOG.md is in the diff stat but the changed files are only under bench/reports/, so no context-specific doc needed.
Step 5 — Consolidated Report:
The mechanical funnel caught changelog/unreleased-entry-required as SHOULD because the repo has source changes (4 files added to bench/reports/) but no ## Unreleased bullet. This is a bench/fixture PR adding golden-curated-3 discrimination report pages — it is test-scoped infrastructure, not a source change that warrants a changelog entry. Classifying this as Nice to Have (Optional) with context.
{
"verdict": "approve",
"summary": "PR adds 4 benchmark report pages under bench/reports/ as golden-curated-3 fixture data. The single mechanical finding (missing changelog unreleased bullet) is a SHOULD that does not apply to bench/fixture-only PRs — no action needed.",
"comments": [
{
"file": "CHANGELOG.md",
"line": 1,
"severity": "nit",
"message": "changelog/unreleased-entry-required: repo has source changes (4 new files) but no '## Unreleased' bullet. For bench/fixture-only PRs this is a false positive — golden report pages are test infrastructure, not source changes requiring a changelog entry. *(rule: changelog/unreleased-entry-required)*"
}
],
"concerns_addressed": []
}
Summary
bench/reports/, scored against the current committed golden (golden-curated-3, 158 entries)Test plan
git statusclean, no unrelated files