Skip to content

Promote 1 three-model-agreed m3 finding into the golden set - #111

Merged
bborbe merged 1 commit into
masterfrom
feature/golden-promote-triangulation
Aug 18, 2026
Merged

Promote 1 three-model-agreed m3 finding into the golden set#111
bborbe merged 1 commit into
masterfrom
feature/golden-promote-triangulation

Conversation

@bborbe

Copy link
Copy Markdown
Owner

What

golden-curated-2golden-curated-3 (157 → 158 entries): one finding, agreed on by three independent models (m3, m2.7, deepseek — three different families) and deepseek-verified against the actual PR with runnable evidence, is promoted to accepted.

entryevidence chain
tts-mcp#13CHANGELOG.md (9 bullets violate conventional-prefix)1) m3's full pass flagged it, 2) m2.7's full pass flagged the same path, 3) deepseek's full pass flagged the same path, 4) deepseek re-verified against tts-mcp#13 head, confirmed the 9 bullets added in this PR all lack feat:/fix:/etc. prefixes (a MUST-level rule)

Why this is the cleanest entry yet

This is the first entry backed by three models' independent agreement rather than two. Three different model families (one cheap MiniMax, one cheap-from-a-different-family m2.7, one reference-tier deepseek) all flagged the same issue. Then a fourth confirmation pass against the actual PR with a strong model and runnable evidence (the bullets added by this PR are all prefix-less; the rule is MUST-level).

The triangulation principle's working

The [[Triangulate the Golden Set Across Models]] pass surfaced 16 three-way agreement locations. Of those:

  • 3 were re-statements of existing entries (aliasing guard caught them) → dropped
  • 12 were rejected on defensible grounds (DRY, taste, pre-existing, hypothetical testability) → not promoted
  • 1 was genuinely new → this PR

Three-way agreement is necessary but not sufficient. The aliasing guard remains essential — promoting a re-statement would have silently double-counted recall. This PR shows the full mechanism working correctly: cross-model agreement → re-statement check → strong-model verification → promotion.

Score deltas (vs golden-curated-2 → golden-curated-3)

modelrecall beforerecall afterhits
opus (self-match)0.8320.826114
m30.0660.07210
deepseek0.1090.10915

m3 gained 1 hit (its own finding now counts). deepseek unchanged. opus self-match dropped slightly — the new entry is a finding opus missed, so the self-match tautology loosens, as expected when the set grows toward truth.

@ben-s-pull-request-reviewerben-s-pull-request-reviewerBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I have everything I need. Let me perform the in-session classification and adjudication.

Step 4b-i candidate rules (judgment rules with trigger matching changed files):

Changed files: CHANGELOG.md, bench/golden.json

From rules/index.json, the only judgment rule with trigger CHANGELOG.md is:

  • changelog/conventional-prefix-required (owner: agent-auditor)

Step 4c-sel CLASSIFY:

The rule's applies_when: "a bullet under ## Unreleased in CHANGELOG.md does not start with one of the recognised conventional prefixes"

The new bullet in the diff reads:

- feat: promote 1 m3 finding, three-model-agreed ...

It starts with feat: — a recognized prefix. The applies_when condition does not hold → skipped.

All other judgment rules in the index have triggers like **/*.go, go.mod, agents/**/*.md, etc. — none match CHANGELOG.md or bench/golden.json.

Step 4d-sel ADJUDICATE: No applicable rules → no adjudication.

Step 4a mechanical funnel: 0 findings (already confirmed above).

Result:selector clean — no adjudication needed


Step 5: Consolidated Report

Must Fix (Critical): None.

Should Fix (Important): None.

Nice to Have (Optional): None.

Traceability:

  • Candidates: 1 rule matched by Step 4b-i glob filter
  • Applicable: 0 rules selected by Step 4c-sel
  • Skipped: changelog/conventional-prefix-required → bullet has proper feat: prefix

{
"verdict": "approve",
"summary": "PR adds a single well-formed CHANGELOG entry and a golden-set entry for a three-model-agreed finding. The changelog bullet correctly uses the `feat:` prefix. No violations of any applicable judgment or mechanical rule.",
"comments": [],
"concerns_addressed": [
"tests: bench/golden.json new entry tts-mcp#13 has valid JSON structure, correct signature array, and proper provenance string — verified by in-session review",
"tests: adjudications array append is genuinely new (aliasing-checked per PR description) — no duplicate"
]
}

@bborbe
bborbe merged commit eb94415 into masterAug 18, 2026
1 check passed
@bborbe
bborbe deleted the feature/golden-promote-triangulation branch August 18, 2026 12:01
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@bborbe