Promote 1 three-model-agreed m3 finding into the golden set - #111
Conversation
There was a problem hiding this comment.
Now I have everything I need. Let me perform the in-session classification and adjudication.
Step 4b-i candidate rules (judgment rules with trigger matching changed files):
Changed files: CHANGELOG.md, bench/golden.json
From rules/index.json, the only judgment rule with trigger CHANGELOG.md is:
changelog/conventional-prefix-required(owner:agent-auditor)
Step 4c-sel CLASSIFY:
The rule's applies_when: "a bullet under ## Unreleased in CHANGELOG.md does not start with one of the recognised conventional prefixes"
The new bullet in the diff reads:
- feat: promote 1 m3 finding, three-model-agreed ...
It starts with feat: — a recognized prefix. The applies_when condition does not hold → skipped.
All other judgment rules in the index have triggers like **/*.go, go.mod, agents/**/*.md, etc. — none match CHANGELOG.md or bench/golden.json.
Step 4d-sel ADJUDICATE: No applicable rules → no adjudication.
Step 4a mechanical funnel: 0 findings (already confirmed above).
Result:selector clean — no adjudication needed
Step 5: Consolidated Report
Must Fix (Critical): None.
Should Fix (Important): None.
Nice to Have (Optional): None.
Traceability:
- Candidates: 1 rule matched by Step 4b-i glob filter
- Applicable: 0 rules selected by Step 4c-sel
- Skipped:
changelog/conventional-prefix-required→ bullet has properfeat:prefix
{
"verdict": "approve",
"summary": "PR adds a single well-formed CHANGELOG entry and a golden-set entry for a three-model-agreed finding. The changelog bullet correctly uses the `feat:` prefix. No violations of any applicable judgment or mechanical rule.",
"comments": [],
"concerns_addressed": [
"tests: bench/golden.json new entry tts-mcp#13 has valid JSON structure, correct signature array, and proper provenance string — verified by in-session review",
"tests: adjudications array append is genuinely new (aliasing-checked per PR description) — no duplicate"
]
}Uh oh!
There was an error while loading. Please reload this page.
What
golden-curated-2→golden-curated-3(157 → 158 entries): one finding, agreed on by three independent models (m3, m2.7, deepseek — three different families) and deepseek-verified against the actual PR with runnable evidence, is promoted toaccepted.tts-mcp#13CHANGELOG.md(9 bullets violate conventional-prefix)tts-mcp#13head, confirmed the 9 bullets added in this PR all lackfeat:/fix:/etc. prefixes (a MUST-level rule)Why this is the cleanest entry yet
This is the first entry backed by three models' independent agreement rather than two. Three different model families (one cheap MiniMax, one cheap-from-a-different-family m2.7, one reference-tier deepseek) all flagged the same issue. Then a fourth confirmation pass against the actual PR with a strong model and runnable evidence (the bullets added by this PR are all prefix-less; the rule is MUST-level).
The triangulation principle's working
The [[Triangulate the Golden Set Across Models]] pass surfaced 16 three-way agreement locations. Of those:
Three-way agreement is necessary but not sufficient. The aliasing guard remains essential — promoting a re-statement would have silently double-counted recall. This PR shows the full mechanism working correctly: cross-model agreement → re-statement check → strong-model verification → promotion.
Score deltas (vs golden-curated-2 → golden-curated-3)
m3 gained 1 hit (its own finding now counts). deepseek unchanged. opus self-match dropped slightly — the new entry is a finding opus missed, so the self-match tautology loosens, as expected when the set grows toward truth.