Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/).
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.

## Unreleased

- feat: promote 2 m3 findings deepseek-verified as real into the golden set (`golden-curated-1` → `golden-curated-2`, 155 → 157 entries). `recurring-task-creator#30` — `make precommit` genuinely red (govulncheck reproduced GO-2026-6179/6180 against `golang.org/x/mod@v0.37.0`, neither in `VULNCHECK_IGNORE`); `discord-assistant#5` — `strip_wake_phrase` lstrip omits the apostrophe, so `"hey bot's weather"` becomes `"'s weather"` (reproduced exactly). These are the first entries backed by **two models' independent agreement** (m3 found, deepseek verified with runnable evidence) rather than a single model's opinion — the evidence class the golden-set design has always specified for promotion. Two other deepseek-verified candidates (unbounded `uncaughtException` swallow, supervise orphan) were dropped: the aliasing check showed they were re-statements of entries the set already held, so promoting them would have double-counted. Net effect on scores vs `golden-curated-1`: m3 recall 0.052 → 0.066, deepseek 0.096 → 0.109, opus self-match 0.844 → 0.832 (new entries are findings opus missed — the tautology loosening, as intended)

## v0.43.3

- fix: regenerate `rules/index.json` — `git-workflow/never-direct-commit-to-master` was missing its `@commits` trigger, failing `make precommit` (`check-index`) on every push.
Expand Down
31 changes: 29 additions & 2 deletions bench/golden.json
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
{
"version": "golden-curated-1",
"version": "golden-curated-2",
"created": "2026-08-09",
"prs_version": "curated-1",
"baseline": {
Expand DownExpand Up@@ -2009,6 +2009,32 @@
"state": "rejected",
"line_when_seen": null,
"excerpt": "**Commits `2312772`, `75cde50`** \u2014 subjects are dark-factory prompt slugs (`004-spec-002-tag-commit-seam`), not imperative verbs. Repo-wide tooling convention across all prompt-driven commits; raise a"
},
{
"pr_id": "recurring-task-creator#30",
"path": "go.mod",
"signature": [
"GO-2026-6179",
"vulncheck"
],
"rule_id": null,
"state": "accepted",
"line_when_seen": 95,
"excerpt": "**`go.mod:95`** \u2014 `make precommit` fails on this branch with `vulncheck` exit 1: `GO-2026-6179` and `GO-2026-6180` against `golang.org/x/mod@v0.37.0` (sumdb transparency-log verification bypass + unau",
"provenance": "m3 finding, deepseek-verified 2026-08-18 (runnable evidence); genuinely new \u2014 NOT a re-statement of an existing entry (aliasing-checked)"
},
{
"pr_id": "discord-assistant#5",
"path": "shim/claude_openai_shim.py",
"signature": [
"strip_wake_phrase",
"lstrip"
],
"rule_id": null,
"state": "accepted",
"line_when_seen": 864,
"excerpt": "**`shim/claude_openai_shim.py:864`** \u2014 `strip_wake_phrase()` uses `text[m.end():].lstrip(\" ,.:;-\u2014\")` and omits the apostrophe; `\"hey bot's weather\"` strips to `\"'s weather\"` \u2014 the model receives a pro",
"provenance": "m3 finding, deepseek-verified 2026-08-18 (runnable evidence); genuinely new \u2014 NOT a re-statement of an existing entry (aliasing-checked)"
}
],
"adjudications": [
Expand All@@ -2021,6 +2047,7 @@
"2026-08-10 MATCHER path removed from identity. Demonstrated: the 'write-scoped token still minted under --skip-post' issue is entry-keyed to pkg/factory/runner.go but was re-reported anchored at cmd/run-task/main.go - the body named runner.go explicitly. Scored as a miss purely on the anchor choice.",
"2026-08-10 ADJUDICATE curated-1: 115 findings across 20 PRs. 113 new entries (94 accepted / 19 rejected); 2 findings were the SAME issue an existing entry already held, re-reported in different words, so those two keys were RELAXED rather than duplicated (python-skeleton#3 --ignore-vuln and sudo tee -a). One issue, one entry - a duplicate is what inflated recall to a spurious 1.000 in v0.37.0.",
"2026-08-10 REJECTED grew 1 -> 20. The rule applied: a finding is rejected when it is a correct observation that names no defect - polish, taste, or a consequence the finding itself concludes is intended. The clearest cluster is github-releaser-agent#8, where four findings disqualify themselves in their own text ('worth a follow-up rather than in isolation', 'not a correctness fix', 'so not a deviation', 'raise as a dark-factory issue rather than here'). Before this, precision could not be lost by any configuration.",
"2026-08-10 Judgment line held consistently, and it is a line: 'harmless today' hardening on correct code is rejected (unquoted shell expansion), while a latent defect where the code does something other than its shape implies is accepted (check() closing over the outer const). Reachability is not correctness. Comment-requests on correct code are rejected even when they argue regression risk."
"2026-08-10 Judgment line held consistently, and it is a line: 'harmless today' hardening on correct code is rejected (unquoted shell expansion), while a latent defect where the code does something other than its shape implies is accepted (check() closing over the outer const). Reachability is not correctness. Comment-requests on correct code are rejected even when they argue regression risk.",
"2026-08-18 PROMOTE 2 m3 findings deepseek-verified as real: recurring-task-creator#30 vulncheck (govulncheck reproduced GO-2026-6179/6180, CI genuinely red) and discord-assistant#5 apostrophe strip (reproduced exactly). Two more deepseek-verified candidates (unbounded uncaughtException swallow, supervise orphan) turned out to be RE-STATEMENTS of existing entries \u2014 the aliasing check caught them as duplicates and they were dropped rather than promoted twice. First entries backed by two models' independent agreement. glm-5.2 retired; glm-5.3[1m] only."
]
}
Loading