fix(rag): refresh the ranking snapshot before its 19 Aug expiry (#304), at zero cost - #1876
Conversation
…ry (#304) The snapshot was generatedAt 2026-07-20 and tests/ranking-tuning.test.ts:79-91 fails it outside a 30-day window, so it would have gone red on 2026-08-19 and taken the whole unit suite with it — not just a RAG test. Refreshed WITHOUT a paid dispatch. #304 assumed a new eval-canary repository dispatch was required. It is not: run 31329507691 (2026-08-09) is green and its eval-canary-output artifact had not expired, and build-ranking-snapshot.ts stamps generatedAt at build time, so regenerating from that artifact resets the full window at zero cost. That source run was verified contract-conformant before building from it: 36 cases, document_recall@5=1, content_recall@5=1, mrr@10=0.8921, content_mrr@10=0.9406, ndcg@10=0.9308, irrelevant_source_rate@10=0.0917, failed_cases=0 — an exact match for the pinned baseline. generatedAt 2026-07-20T18:22:02.934Z -> 2026-08-12T19:21:20.567Z sourceRunId 29763761133 -> 31329507691 Window now runs to 2026-09-11. tests/ranking-tuning.test.ts: 12 passed. The diff is large (2954+/353-) because the candidate set tracks the live corpus between the two runs, which is what the freshness gate exists to force. It is NOT a behaviour change: the snapshot has zero runtime consumers — only scripts/ and tests/ read it — so it is an offline ranking-tuning fixture, not a retrieval input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017paT42ZVMf8jaLtkjFxdy5
The snapshot is written by scripts/build-ranking-snapshot.ts via JSON.stringify(…, 2), which always expands arrays, while Prettier collapses short ones. The two can never agree, so every regeneration would otherwise land an unrelated ~200-line reflow on a RAG-protected fixture. This is the same reasoning already recorded in this file for the Supabase generated types, the drift manifest and the Therapy Compass export: keep the generator's formatting so regeneration stays churn-free. Note the previous committed snapshot was ALSO Prettier-unclean — this is pre-existing, and only surfaces when the snapshot is itself in a push's changed set. A whole-tree `prettier --check .` now passes; before this change the snapshot was the only offender in the repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This pull request has been ignored for the connected project Preview Branches by Supabase. |
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in:59 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
Comment |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:4d833d8b79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
BigSimmo
commented
Aug 12, 2026
@codex Address the currently unresolved inline review comment on this PR against the current GitHub head. Make ranking-snapshot freshness derive from the source canary evidence timestamp rather than local rebuild time, update the generator/snapshot/schema/tests as needed, preserve offline reproducibility, run focused ranking-tuning/format checks, and push the minimal fix. Do not trigger a paid/provider-backed canary, merge, close, rebase, force-push, deploy, or alter auto-merge. Please report the exact commit SHA and test results. |
Preserve the eval-canary run_started_at timestamp in a sidecar and make the offline freshness gate use that immutable source evidence rather than the local rebuild time.
Resolve the shared review ledger in favour of current main, as required for append-only ledgers. The branch-local row claimed that rebuild time reset evidence freshness, so it is not propagated after the source-canary guard supersedes that premise.
Summary
Testing
|
CI triageCI failed on this PR. Automated classification of the 2 failed job(s):
Compared with main CI run #10109 (success). Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger. |
Summary
Refreshes the ranking snapshot ahead of a hard deadline.
scripts/fixtures/rag-ranking-candidate-snapshot.v1.jsonwasgeneratedAt 2026-07-20T18:22:02.934Z, andtests/ranking-tuning.test.ts:79-91fails it outside a 30-day window. It would have gone red on 2026-08-19, taking the entire unit suite with it rather than failing as something recognisably RAG-shaped. NewgeneratedAtis2026-08-12T19:21:20.567Z, so the window now runs to 11 September.No paid dispatch was needed.
#304assumed the refresh required a neweval-canaryrepository dispatch (~$1–2 of live OpenAI + Supabase). It does not. Canary run 31329507691 (2026-08-09) is green, itseval-canary-outputartifact had not expired, andbuild-ranking-snapshot.tsstampsgeneratedAtat build time — so regenerating from the existing artifact resets the full window at zero cost.sourceRunIdmoves29763761133 → 31329507691.The source run was verified contract-conformant before building from it, not assumed: 36 cases,
document_recall@5=1,content_recall@5=1,mrr@10=0.8921,content_mrr@10=0.9406,ndcg@10=0.9308,irrelevant_source_rate@10=0.0917,failed_cases=0. An exact match for the pinned baseline.Keeps the generated snapshot out of Prettier's reach. The generator writes
JSON.stringify(…, 2), which always expands arrays, while Prettier collapses short ones — the two can never agree, so every future regeneration would land an unrelated ~200-line reflow on a RAG-protected fixture..prettierignorealready records exactly this reasoning for the Supabase generated types, the drift manifest and the Therapy Compass export. Worth noting: the previous committed snapshot was also Prettier-unclean — this is pre-existing and only surfaces when the snapshot is itself in a push's changed set. A whole-treeprettier --check .now passes; before this change the snapshot was the repository's only offender.RAG impact: no retrieval behaviour change — the snapshot has zero runtime consumers. Only
scripts/build-ranking-snapshot.ts,scripts/tune-search-weights.ts,scripts/lib/ranking-tuning.ts,scripts/pr-policy.mjsandtests/ranking-tuning.test.tsread it (grepoversrc/returns nothing), so it is an offline ranking-tuning fixture rather than a retrieval input. The diff is large (2954+/353-) because the candidate set tracks the live corpus between the two canary runs, which is precisely what the freshness gate exists to force; no ranking weight, comparator, selection rule or RPC is touched, and no tuner recommendation is adopted.Verification
npm run verify:cheapnpm run testis the last link in theverify:cheap:internalchain (… && npm run lint && npm run typecheck && npm run test), so reaching a green unit suite proves the ~30 static gates,lintandtypecheckall passed ahead of it.npx vitest run tests/ranking-tuning.test.ts— 12 passed, including the freshness-window pin this PR exists to satisfy.npx prettier --check .— "All matched files use Prettier code style!"UI verification not run: no
src/change, no route, component or style touched.npm run eval:retrieval:qualitynot run — it needs live Supabase and OpenAI keys, which this container does not have. That is not a gap here: the numbers above are a live retrieval eval, taken from the canary run this snapshot is built from.Risk and rollout
validateRankingSnapshotplus its own 12-test suite. The.prettierignoreentry is additive and matches three existing precedents in the same file.git reverteither commit independently. Reverting the snapshot restores the 2026-07-20 file and, with it, the 19 August expiry.Notes
#304should be archived when this merges, with a correction recorded: its premise that a fresh operator-identity dispatch was required is only conditionally true. While a recent green canary artifact is unexpired, regeneration is free — a cheaper path worth checking before spending on a dispatch next time.The row's other claim does hold: the GitHub MCP toolset has no repository-dispatch call. That is a toolset gap rather than a capability one — plain HTTPS to the REST API with an operator token works — but it was not needed here.
Generated by Claude Code