diff --git a/CHANGELOG.md b/CHANGELOG.md index 0018c92..1f4401f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/). * MINOR version when you add functionality in a backwards-compatible manner, and * PATCH version when you make backwards-compatible bug fixes. +## Unreleased + +- feat: promote 1 m3 finding, three-model-agreed (m3+m2.7+deepseek all flagged `tts-mcp#13` `CHANGELOG.md` conventional-prefix violation) and deepseek-verified with runnable evidence, into the golden set (`golden-curated-2` → `golden-curated-3`, 157 → 158 entries). Of the 16 three-way agreement locations surfaced by [[Triangulate the Golden Set Across Models]], only 1 was genuinely new and verified — 3 others were re-statements of existing entries (caught by the aliasing guard, correctly dropped), 12 were rejected on defensible grounds (DRY, documented contract, pre-existing). Net effect on scores vs `golden-curated-2` (re-scored against the enlarged 158-entry set): m3 recall 0.066 → **0.072** (+1 hit, m3's own finding now counts), deepseek 0.109 → 0.109 (no change), opus 0.832 → **0.826** (−1 hit, the new entry is a finding opus missed — the self-match tautology loosening, as expected when the set grows). The triangulation principle's doing its job: agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting + ## v0.44.0 - feat: promote 2 m3 findings deepseek-verified as real into the golden set (`golden-curated-1` → `golden-curated-2`, 155 → 157 entries). `recurring-task-creator#30` — `make precommit` genuinely red (govulncheck reproduced GO-2026-6179/6180 against `golang.org/x/mod@v0.37.0`, neither in `VULNCHECK_IGNORE`); `discord-assistant#5` — `strip_wake_phrase` lstrip omits the apostrophe, so `"hey bot's weather"` becomes `"'s weather"` (reproduced exactly). These are the first entries backed by **two models' independent agreement** (m3 found, deepseek verified with runnable evidence) rather than a single model's opinion — the evidence class the golden-set design has always specified for promotion. Two other deepseek-verified candidates (unbounded `uncaughtException` swallow, supervise orphan) were dropped: the aliasing check showed they were re-statements of entries the set already held, so promoting them would have double-counted. Net effect on scores vs `golden-curated-1`: m3 recall 0.052 → 0.066, deepseek 0.096 → 0.109, opus self-match 0.844 → 0.832 (new entries are findings opus missed — the tautology loosening, as intended) diff --git a/bench/golden.json b/bench/golden.json index 62e297f..ef31971 100644 --- a/bench/golden.json +++ b/bench/golden.json @@ -1,5 +1,5 @@ { - "version": "golden-curated-2", + "version": "golden-curated-3", "created": "2026-08-09", "prs_version": "curated-1", "baseline": { @@ -2035,6 +2035,19 @@ "line_when_seen": 864, "excerpt": "**`shim/claude_openai_shim.py:864`** \u2014 `strip_wake_phrase()` uses `text[m.end():].lstrip(\" ,.:;-\u2014\")` and omits the apostrophe; `\"hey bot's weather\"` strips to `\"'s weather\"` \u2014 the model receives a pro", "provenance": "m3 finding, deepseek-verified 2026-08-18 (runnable evidence); genuinely new \u2014 NOT a re-statement of an existing entry (aliasing-checked)" + }, + { + "pr_id": "tts-mcp#13", + "path": "CHANGELOG.md", + "signature": [ + "### added", + "conventional-prefix" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": null, + "excerpt": "**`CHANGELOG.md:12`** \u2014 All 9 bullets under `## Unreleased` violate the conventional-prefix requirement: none starts with `feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, or `perf:` (e.g. lin", + "provenance": "three-model agreement (m3+m2.7+deepseek, 2026-08-18 triangulation); deepseek-verified with runnable evidence; aliasing-checked genuinely new" } ], "adjudications": [ @@ -2048,6 +2061,7 @@ "2026-08-10 ADJUDICATE curated-1: 115 findings across 20 PRs. 113 new entries (94 accepted / 19 rejected); 2 findings were the SAME issue an existing entry already held, re-reported in different words, so those two keys were RELAXED rather than duplicated (python-skeleton#3 --ignore-vuln and sudo tee -a). One issue, one entry - a duplicate is what inflated recall to a spurious 1.000 in v0.37.0.", "2026-08-10 REJECTED grew 1 -> 20. The rule applied: a finding is rejected when it is a correct observation that names no defect - polish, taste, or a consequence the finding itself concludes is intended. The clearest cluster is github-releaser-agent#8, where four findings disqualify themselves in their own text ('worth a follow-up rather than in isolation', 'not a correctness fix', 'so not a deviation', 'raise as a dark-factory issue rather than here'). Before this, precision could not be lost by any configuration.", "2026-08-10 Judgment line held consistently, and it is a line: 'harmless today' hardening on correct code is rejected (unquoted shell expansion), while a latent defect where the code does something other than its shape implies is accepted (check() closing over the outer const). Reachability is not correctness. Comment-requests on correct code are rejected even when they argue regression risk.", - "2026-08-18 PROMOTE 2 m3 findings deepseek-verified as real: recurring-task-creator#30 vulncheck (govulncheck reproduced GO-2026-6179/6180, CI genuinely red) and discord-assistant#5 apostrophe strip (reproduced exactly). Two more deepseek-verified candidates (unbounded uncaughtException swallow, supervise orphan) turned out to be RE-STATEMENTS of existing entries \u2014 the aliasing check caught them as duplicates and they were dropped rather than promoted twice. First entries backed by two models' independent agreement. glm-5.2 retired; glm-5.3[1m] only." + "2026-08-18 PROMOTE 2 m3 findings deepseek-verified as real: recurring-task-creator#30 vulncheck (govulncheck reproduced GO-2026-6179/6180, CI genuinely red) and discord-assistant#5 apostrophe strip (reproduced exactly). Two more deepseek-verified candidates (unbounded uncaughtException swallow, supervise orphan) turned out to be RE-STATEMENTS of existing entries \u2014 the aliasing check caught them as duplicates and they were dropped rather than promoted twice. First entries backed by two models' independent agreement. glm-5.2 retired; glm-5.3[1m] only.", + "2026-08-18 TRIANGULATE: 1 m3 finding deepseek-verified as real and three-model-agreed (m3+m2.7+deepseek all flagged tts-mcp#13 CHANGELOG.md conventional-prefix violation; deepseek-verified with runnable evidence); promoted to golden-curated-3. Of 16 three-way agreement locations, 15 were rejected on defensible grounds or were re-statements of existing entries; only 1 was genuinely new. Agreement across models is necessary but not sufficient; aliasing checks remain essential to avoid silent double-counting." ] -} +} \ No newline at end of file