Skip to content

bench: commit curated-1 discrimination report pages (golden-curated-3) - #113

Merged
bborbe merged 1 commit into
masterfrom
chore/bench-report-pages
Aug 18, 2026
Merged

bench: commit curated-1 discrimination report pages (golden-curated-3)#113
bborbe merged 1 commit into
masterfrom
chore/bench-report-pages

Conversation

@bborbe

Copy link
Copy Markdown
Owner

Summary

  • Commit the four curated-1 discrimination report pages to bench/reports/, scored against the current committed golden (golden-curated-3, 158 entries)
  • Models: opus (xhigh/full reference), deepseek-v4-flash-max, m3, m2.7 (all high/full) on the full 20-PR curated set
  • Recall ladder: opus 0.826 (run 1) > deepseek 0.109 > m3 0.072 > m2.7 0.014 — a monotone model-strength ladder far beyond the measured noise floor (Opus 3σ ≈ 15.6% of mean)
  • These pages are the reproducible artifact for the PR Review Bench discrimination result; they were previously claimed committed but none existed

Test plan

  • Report pages render with correct Configuration blocks (model is the only differing field among the three high/full configs)
  • Each page shows the full curated fixture scored (20/20 PRs, rows skipped 0)
  • git status clean, no unrelated files

…lden-curated-3
The four-model full passes (opus xhigh/full reference, deepseek, m3, m2.7
all high/full on curated-1) were scored against the current committed
golden (golden-curated-3, 158 entries) and their report pages written to
bench/reports/. These pages are the reproducible artifact for the PR
Review Bench discrimination result: recall 0.826 (opus run 1) vs 0.109
(deepseek), 0.072 (m3), 0.014 (m2.7) - a monotone ladder far beyond the
measured 3sigma noise floor. Report pages were previously claimed
committed but none existed for the curated-1 discrimination runs.

@ben-s-pull-request-reviewerben-s-pull-request-reviewerBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I need to run the Step 4b-i jq filter manually. The changed files are all in bench/reports/ with .md extension. Let me evaluate the judgment-rule triggers against those paths.

Step 4b-i — Judgment-rule candidate computation:

The changed files from the diff stat:

  • bench/reports/7b305997dc2b9b2f734f1499922286b7a8881ea5bd1967fad49275b83a8f1ba8.md
  • bench/reports/d4558caff3dfefdff574d9a14345bb4007db9cb94a69c64e2e859c7d6e8b2dab.md
  • bench/reports/d5a15f168212fe61bdf112b08c7a1787058cc5cd26a2a2d6c35a545465de79ca.md
  • bench/reports/edb935210ee850e34cb521e7c50482923a65761bd4d66501c153a402f5f8f8eb.md

Checking each judgment rule's trigger globs against these paths:

  • **/*.go → no Go files
  • **/main.go → no main.go
  • CHANGELOG.md → not CHANGELOG.md
  • CLAUDE.md → not CLAUDE.md
  • go.mod → not go.mod
  • Makefile → not Makefile
  • agents/**/*.md, commands/**/*.md, skills/**/*.md → paths are bench/reports/**/*.md, not agent/command/skill files

No judgment rules have triggers matching the changed files. Applicable set is empty.

Step 4c-sel CLASSIFY:

{"applicable": [], "skipped": {}}

Step 4d-sel ADJUDICATE: Since applicable set is empty, no rule-block reads needed.

Mechanical findings (pre-computed): 1 SHOULD finding from changelog/unreleased-entry-required.

Now checking for context-specific conventions — CHANGELOG.md is in the diff stat but the changed files are only under bench/reports/, so no context-specific doc needed.

Step 5 — Consolidated Report:

The mechanical funnel caught changelog/unreleased-entry-required as SHOULD because the repo has source changes (4 files added to bench/reports/) but no ## Unreleased bullet. This is a bench/fixture PR adding golden-curated-3 discrimination report pages — it is test-scoped infrastructure, not a source change that warrants a changelog entry. Classifying this as Nice to Have (Optional) with context.


{
"verdict": "approve",
"summary": "PR adds 4 benchmark report pages under bench/reports/ as golden-curated-3 fixture data. The single mechanical finding (missing changelog unreleased bullet) is a SHOULD that does not apply to bench/fixture-only PRs — no action needed.",
"comments": [
{
"file": "CHANGELOG.md",
"line": 1,
"severity": "nit",
"message": "changelog/unreleased-entry-required: repo has source changes (4 new files) but no '## Unreleased' bullet. For bench/fixture-only PRs this is a false positive — golden report pages are test infrastructure, not source changes requiring a changelog entry. *(rule: changelog/unreleased-entry-required)*"
}
],
"concerns_addressed": []
}

@bborbe
bborbe merged commit 0cc0a4e into masterAug 18, 2026
1 check passed
@bborbe
bborbe deleted the chore/bench-report-pages branch August 18, 2026 17:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@bborbe