refactor(rag): extract second-stage ranking

[BigSimmo]

Motivation

• Reduce the rag.ts monolith and complete the X3 maturity work by moving a cohesive second-stage rerank unit into its own module while preserving existing behaviour and public exports.
• Make the src/lib/rag codebase easier to reason about, enforce the maintainability budget, and unblock further domain-directory extractions.
• Respect the repository RAG safeguards by declaring RAG impact: no retrieval behaviour change because this is a verbatim extraction with the same public API and comparator precedence.

Description

• Added src/lib/rag/rag-second-stage.ts containing the second-stage engagement predicate, scoring adjustments, document-diversity penalties, subject-match protection, and retrieval-layer telemetry extracted unchanged from rag.ts.
• Updated src/lib/rag/rag.ts to import and re-export applySecondStageRerankIfNeeded and to call the extracted unit at the same pipeline points so the public orchestration and behaviour remain identical.
• Ratcheted the rag.ts maintainability budget in scripts/check-maintainability-budgets.mjs (budget reduced to reflect the extraction) and updated documentation in docs/codebase-index.md and docs/maturity-backlog-workorders.md to record the change.
• Kept the change isolated to a single X3 structural unit; ACL, host hardening, and provider-gated items remain out of scope per the maturity work orders.

Testing

• Ran focused unit tests: npm run test -- tests/rag-second-stage-ranking.test.ts tests/rag-fast-path-ordering.test.ts tests/rag-document-lookup-escalation-rescue.test.ts and they all passed.
• Ran the repository preflight and offline retrieval checks: npm run verify:pr-local -- --files src/lib/rag/rag.ts,src/lib/rag/rag-second-stage.ts,scripts/check-maintainability-budgets.mjs,docs/codebase-index.md,docs/maturity-backlog-workorders.md, npm run typecheck, npm run lint, npm run test (full Vitest), npm run build, npm run check:rag:fixtures, and npm run eval:rag:offline; these completed and passed the offline/CI-local checks.
• Ran formatting: npm run format / npm run format:check and the tree is formatted.
• Not run (provider-backed / approval-required): npm run eval:retrieval:quality, npm run check:production-readiness, and npm run verify:ui because they are live/provider-backed or UI gates and require separate approval.

---

Codex Task

RAG impact: no retrieval behaviour change — verbatim second-stage ranking extraction; orchestration, comparator precedence, and call sites are preserved.

Verification after current-main reconciliation

• 4 focused RAG test files passed (28 tests).
• [x]
  pm run typecheck passed.
• [x]
  pm run check:maintainability-budgets passed with
  ag.ts at 4351/4351 lines.
• [x]
  pm run eval:rag:offline passed: 36 golden cases, 23 suites, 574 tests.
• [x]
  pm run check:production-readiness:ci returned READY with expected missing-local-secret warnings.

Risk and rollout

This is a structural extraction only. Revert the squash commit if module resolution, public re-exports, or retrieval telemetry differs; no provider configuration or production data is changed.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use.
• No patient-identifiable document workflow was introduced or expanded.
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy).
• Service-role keys and private document access remain server-only.
• Demo/synthetic content remains clearly separated from real clinical sources.
• Source metadata, review status, and outdated/unknown-source behavior remain conservative.
• Deployment classification/TGA SaMD impact is unchanged because retrieval behavior is unchanged.

Summary by CodeRabbit

• Improvements

  • Improved search result ranking through a second-stage relevance and evidence review.
  • Better balances results across visual, table-based, clinical, and other evidence types.
  • Added safeguards for clinical content and improved result diversity.
  • Enhanced ranking explanations and retrieval performance telemetry.

• Documentation

  • Updated technical documentation to reflect second-stage ranking capabilities and maintainability progress.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:51 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8f1b5dfd-3e62-4959-98f5-c81545695fe1

📥 Commits

Reviewing files that changed from the base of the PR and between f7d099d and 226355f.

📒 Files selected for processing (3)

• docs/branch-review-ledger.md
• docs/maturity-backlog-workorders.md
• src/lib/rag/rag.ts

📝 Walkthrough

Walkthrough

Second-stage RAG reranking and telemetry helpers move from rag.ts into rag-second-stage.ts. rag.ts re-exports the extracted functions, while documentation and maintainability budgets reflect the new module boundary.

Changes

RAG reranking extraction

Layer / File(s)	Summary
Reranking helpers and scoring src/lib/rag/rag-second-stage.ts	Adds retrieval-layer telemetry, rerank gating, feature-based scoring, evidence boosts, and quality penalties.
Reranking application flow src/lib/rag/rag-second-stage.ts	Applies diversity and clinical-subject adjustments, recalculates scores and ranks, and records second-stage telemetry.
RAG integration and tracking src/lib/rag/rag.ts, docs/..., scripts/check-maintainability-budgets.mjs	Moves the implementation out of rag.ts, preserves its re-exports, and updates module documentation and the line budget from 4,543 to 4,351.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• BigSimmo/Database#649: Modifies the same second-stage reranking and ranking configuration behavior.
• BigSimmo/Database#901: Refactors the same second-stage reranking function.
• BigSimmo/Database#915: Updates second-stage ranking order and deterministic result handling.

Suggested reviewers:cursoragent

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: extracting second-stage ranking from rag.ts.
Description check	✅ Passed	The description covers motivation, changes, testing, risk, and governance, but it diverges from the exact template headings and omits a plain Notes section.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[BigSimmo]

@CodeRabbit review PR

[BigSimmo]

@codex review PR

[coderabbitai]

@BigSimmo I’ll review PR #1472, with particular attention to preserving reranking behavior, exports, comparator precedence, and extraction boundaries.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit:ef2a4fd4e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

This PR refactors the RAG pipeline by extracting the second-stage reranking logic out of src/lib/rag/rag.ts into a dedicated module, aiming to reduce the rag.ts monolith while preserving existing ranking behavior and the public @/lib/rag/rag export surface.

Changes:

• Extracted second-stage rerank logic (engagement predicate, scoring adjustments, document diversity penalty, subject-match protection, and related telemetry helpers) into src/lib/rag/rag-second-stage.ts.
• Updated rag.ts to import and re-export applySecondStageRerankIfNeeded from the new module and to use the extracted telemetry helpers.
• Tightened the maintainability budget for rag.ts and updated RAG module documentation/workorder notes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/lib/rag/rag.ts	Removes in-file second-stage rerank implementation and switches to importing/re-exporting the extracted module while keeping orchestration in rag.ts.
src/lib/rag/rag-second-stage.ts	New module containing the extracted second-stage rerank implementation and retrieval-layer telemetry helpers.
scripts/check-maintainability-budgets.mjs	Ratchets the rag.ts line-count budget downward to reflect the extraction.
docs/maturity-backlog-workorders.md	Records the extraction as X3 maturity progress (but currently contains stale post-extraction budget numbers).
docs/codebase-index.md	Adds rag-second-stage.ts to the documented RAG module cluster.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[BigSimmo]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

[Copilot]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply t...

Fixed in the latest commit — updated docs/maturity-backlog-workorders.md line 99 to say (4,780 → 4,351; budget ratcheted to 4,351), matching the enforced budget in scripts/check-maintainability-budgets.mjs and the actual rag.ts line count.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/maturity-backlog-workorders.md`:
- Around line 95-99: Correct the `#086` second-stage extraction entry in the
ratchet ledger to use the established 4,543-line baseline from the hydration
entry, update the resulting post-extraction count and ratcheted budget
consistently, and verify or correct the duplicated `#086` PR reference so it does
not repeat the coverage-gate bullet.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 30c8ce83-8689-4dac-9ee4-9e0d51a5e9a0

📥 Commits

Reviewing files that changed from the base of the PR and between db50738 and f7d099d.

📒 Files selected for processing (5)

• docs/codebase-index.md
• docs/maturity-backlog-workorders.md
• scripts/check-maintainability-budgets.mjs
• src/lib/rag/rag-second-stage.ts
• src/lib/rag/rag.ts

[BigSimmo]

Temporarily paused during the owner-authorized ordered merge sweep so main cannot move between exact-head review and merge. The branch and all commits are preserved; this PR will be reopened at its verified place in the queue.

[BigSimmo]

@cursoragent this PR will be re-opened soon. Please resolve any failing CI, run bugbot and resolve any bugbot comments against current repo and fix any merge conflicts and also fix any errors and address and fix all comments or suggestions that are likely to improve the PR and Codebase. Please implement smallest required fix according to best code practices. Check your work. Leave this PR closed for now but still make all these changes and run the review. I want it ready to go for when I re-open it.

[cursor]

PR #1472 is ready to reopen and left CLOSED.

Done

• Synced origin/main (merge-clean; 0 behind, 12 ahead)
• Fixed CodeRabbit: baseline 4,543 → 4,351, labeled PR #1472 (no #086 collision); thread resolved
• Bugbot: no actionable findings; second-stage extraction looks behavior-preserving
• Fixed Cursor Cloud verify noise: inspectOriginRemote now reads stored remote.origin.url (not insteadOf-rewritten get-url)
• Local proof: verify:pr-local green — 444 files / 4645 tests, build + RAG fixtures; budgets 4336/4351

Branch tip

febf00b7 on codex/complete-repository-maturity-programme
#1472

GitHub still shows the frozen closed head f7d099d6; reopen should attach the current tip. Hosted ci.yml will run again after reopen (not available while closed).

[BigSimmo]

@coderabbitai please run a new review of this closed PR for me

[coderabbitai]

@BigSimmo I’m running a fresh full review of closed PR #1472, including the extracted second-stage RAG reranking behavior and its API-preservation boundaries.

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 37 minutes.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[BigSimmo]

@cursoragent resolve merge conflict

[cursor]

Resolved. GitHub’s DIRTY was behind-but-clean, not a content fight.

• Fetched origin/main (d766d53a)
• git merge-tree clean; merge completed with no conflict markers (auto-merged ledger + mockup files from #1437)
• Pushed tip 226355fe — 0 behind / 15 ahead, merge-tree clean
• GitHub now: mergeable: true, mergeable_state: blocked (checks pending, not conflicts)

Complicated conflicts: none.