feat(rag): eval-gate ON #213's ranking wins (diversity demotion + linear freshness) - #242
Merged
Conversation
…on + linear freshness) #213 shipped documentDiversityPenalty and the linear freshness curve config-gated and default-OFF, to be turned on only after eval-gating. Live evals show a strict improvement with no regression on the safety bar, so both are now ON by default: - documentDiversityPenalty 0 -> 0.02 (gentle same-document crowding demotion, capped 0.12) - freshness.mode step -> linear (ramped decay instead of the harsh cliff) Also fixes resolveRankingConfig, which hardcoded the freshness `mode` fallback to "step" instead of deriving it from defaultRankingConfig, so changing the default alone would not have taken effect at runtime without an explicit env override. Eval evidence (golden retrieval + eval:quality --rag-only, live): - retrieval mrr@10 0.7283 -> 0.7569, document/content recall@5 stays 1.0, 0 failures - grounded_supported 0.967 -> 1.0, unsupported_correct unchanged, citation/numeric failure 0 - verify:cheap green (980/980) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
4 tasks
14 tasks
BigSimmo added a commit
that referenced
this pull request
Aug 8, 2026
… not only the first adoption (#1742) `validateCandidateSourceBinding` required `candidateSourceHead` to name a commit whose `AWAITING_BASELINE` still declared the canonical six, and the current tree to declare none. That models the FIRST adoption and nothing else. After #1729 landed, no commit on main declares the six again, so no future capture head could ever satisfy it -- the binding was satisfiable exactly once, and that once has passed. The consequence is only latent today, and this commit does not pretend otherwise: `validateLinuxVisualBaselineSet` short-circuits on `declaredPaths.length === 0` and no surface yet declares `baseline.status: "committed"`, so the whole contract is dormant and refreshing a golden is currently ungated. The defect bites the moment those surfaces are declared (#242) -- at which point the first intentional design change would leave the goldens red with no supported way to re-adopt them, which is precisely backwards for a gate meant to make design changes provable. So the binding now accepts either shape: FIRST ADOPTION candidate head declares the six, current declares none. REFRESH both ends empty, suite byte-identical. A refresh is the ordinary case once baselines exist: a surface is deliberately re-shot and its goldens replaced while the suite itself does not change at all. Normalisation already blanks the AWAITING values on both sides, so the "suite changed beyond the AWAITING declaration" check stays exact for both. Nothing is loosened beyond that. A candidate head with five ids, seven ids, a dynamic set, a spread or a duplicate still fails, as do a non-empty current list, a non-ancestor head, a hash mismatch, a wrong platform and an unreviewed set. The one previously-untested shape -- a refresh -- now has its own case, because the gap was invisible precisely because nothing exercised it. Verified: tests/design-system-adoption.test.ts 51 passed (51), including the new refresh case; check:design-system-adoption and check:design-system-contract exit 0; format:check clean. Refs #242, #118
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#213 shipped two retrieval-time ranking improvements config-gated and default-OFF, to be flipped on only after eval-gating. This PR does the eval-gating and turns them on by default:
documentDiversityPenalty0 → 0.02 — gentle same-document crowding demotion (capped 0.12; RC7/CI-16).freshness.modestep → linear — ramped freshness decay instead of the harsh cliff (CI-17).It also fixes a latent bug in
resolveRankingConfig: the freshnessmodefallback was hardcoded to"step"instead of derived fromdefaultRankingConfig, so changing the default alone would not have taken effect at runtime without an explicitRAG_RANKING_CONFIGoverride. The override path is unchanged; settingdocumentDiversityPenalty:0+freshness.mode:"step"restores exact pre-flip behavior.Eval evidence (live)
mrr@10grounded_supported_rateunsupported_correct_rate*The one failing unsupported case (
unsupported-close-title-noise) fails identically on baseline — it's a pre-existing flaky false-positive control, not caused by this change. The flip also cleared the twoagitationgrounding cases in the after-run.verify:cheap: green, 980/980. Confirmed the flip takes effect by default (clean env): goldenmrr@10=0.7569.Clinical governance preflight
grounded_supportedimproves to 1.0;unsupported_correctunchanged (safety bar held); citation/numeric-grounding failure stay 0; no source-governance change.RAG_RANKING_CONFIGwithout a code change.🤖 Generated with Claude Code