eval(retrieval): add content_mrr@10 passage-rank metric by BigSimmo · Pull Request #248 · BigSimmo/Database

BigSimmo · 2026-07-03T12:36:41Z

What & why

The golden retrieval eval measured document-level rank (mrr@10, over expectedDocumentSubstrings) and content-level recall (content_recall@5, binary "is the answer passage in the top 5") — but nothing measured how high the answer-bearing passage ranks. That left rerank/chunking changes (plan items RC8/RC10/RC13) unprovable: a change that lifts the right passage from #5→#1 leaves content recall at 1.0 and doc-level mrr@10 is blind to passage order within a document.

This adds contentReciprocalRankAt10 — the mean over a case's expected content terms of the reciprocal rank of the earliest top-10 result carrying that term (1.0 iff every term rides the rank-1 passage). Surfaced as content_mrr_at_10, averaged only over cases that declare content terms (content_mrr_case_count) so structural zeros don't dilute the signal, plus a per-case contentRR@10 column and a human-summary line.

This is the Phase-2 enabler from the RAG plan ("build the missing chunk-level metric first so ranking changes are provable"). Gate future ranking/chunking changes on it.

Live baseline (golden set, project `sjrfecxgysukkwxsowpy`)

cases=24 document_recall@5=1 content_recall@5=1 top_k_hit_rate=1
mrr@10=0.7569 content_mrr@10=0.8979 (over 24 content-term cases)
median_latency=1009ms p90=6231ms failed_cases=0

The new metric already discriminates where the existing ones are blind:

lithium-therapy-monitoring: rr@10=0.00 (doc-name unmatched in top-10) but contentRR@10=0.63 — answer content is ranking; the doc miss is a naming artifact, not a content miss.
opioid-withdrawal-doses: rr@10=0.25 but contentRR@10=1.00 — answer passages at rank 1.
Weakest passage-rank cases (insomnia-assessment-management, schizophrenia-overview at 0.50) become visible rerank-tuning targets.

Safety / scope

Measurement-only. No change to retrieval, ranking, synthesis, source rendering, or any clinical output. Purely an addition to the offline eval harness.
Purely additive: the summary type is inferred (ReturnType<…>), so reindex-eval-gate.ts and the other summary consumers are unaffected.
Reuses existing content-matching helpers (contentExpectationAlternatives / textContainsClinicalTerm / resultContentText).

Tests / checks

New focused guard: demoting the answer passage below a distractor drops the metric 1.0→0.5 while content recall holds 1.0.
✅ tests/eval-retrieval.test.ts + tests/eval-quality.test.ts (21/21) · typecheck · eslint · prettier
✅ Live eval:retrieval:quality — 24/24 pass, baseline above

Note: distinct from #243's "answer-targeting metric", which is a separate answer-quality metric in eval-answer-quality.ts — no overlap.

The golden retrieval eval measured document-level rank (mrr@10, over expectedDocumentSubstrings) and content-level *recall* (contentRecall@5, binary "is the answer passage in the top 5"), but nothing measured how HIGH the answer-bearing passage ranks. That left rerank/chunking changes that lift the right passage from #5 to #1 unprovable: content recall stays 1.0 across that move and doc-level mrr is blind to passage order within a document. Add contentReciprocalRankAt10 — the mean over a case's expected content terms of the reciprocal rank of the earliest top-10 result carrying that term (1.0 iff every term rides the rank-1 passage). Surface it in the summary as content_mrr_at_10, averaged only over cases that declare content terms (content_mrr_case_count) so structural zeros don't dilute the signal, plus a per-case contentRR@10 column and the human summary line. Purely additive: the summary type is inferred, so reindex-eval-gate and the other summary consumers are unaffected. Guarded by a focused test proving the metric drops to 0.5 when the answer passage is demoted below a distractor while content recall stays 1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:fcc931ad11

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The retrievalLimitForGoldenCase fix landed without a prettier pass, failing the required verify check (format:check). Formatting-only; no logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:b4f273427b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@10

Two remaining P2 review findings on the passage-rank metric: - Quality-eval path truncation: runRetrievalQualityCases (eval-quality.ts) still fetched topK=testCase.topK, so content_mrr@10 silently degraded to content_mrr@topK for the many fixtures with topK<10. Now uses retrievalLimitForGoldenCase(=max(topK,10)), matching the standalone runner. hit@K semantics are unchanged (evaluateGoldenRetrievalCase still slices to the case's topK), so only positions 9-10 are exposed for the @10 metric. - Missing first-class table/visual evidence: resultContentEvidenceText matched table_facts and a few image fields but not typed index-unit content or the accessible table markdown / table rows, so a hit carrying the answer only in a medication-chart row / risk-matrix cell / flowchart step was invisible to contentReciprocalRankAt10. Now includes index_unit title+content and image accessibleTableMarkdown + flattened tableRows. The retrieval_synopsis stays in the evidence text: it is the stored first-class passage representation used in ranking/display, and the metadata-distractor test already guards the title/file/section vector that the earlier fix excluded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

BigSimmo · 2026-07-03T14:56:37Z

Addressed the Codex review threads (commits 1f56972c0 + 927a6ee9c), verified on a fresh live golden run: 24/24 pass, content_recall@5=1.0, content_mrr@10=0.9048 (up from 0.8979).

Restrict content rank to passage text — done: contentReciprocalRankAt10 matches against resultContentEvidenceText (synopsis + content + table-facts + images + index-unit), which excludes document title/file/section metadata. Guarded by the new metadata-distractor test.
Fetch enough results for an @10 metric (standalone runner) — done: retrievalLimitForGoldenCase = max(topK, 10).
Fetch ten results in the quality eval path — fixed in 927a6ee9c: runRetrievalQualityCases (eval-quality.ts) now fetches retrievalLimitForGoldenCase(testCase); hit@K semantics unchanged (still sliced to the case's topK).
Include visual/index evidence in content rank text — fixed in 927a6ee9c: evidence text now includes typed index_unit title+content and image accessibleTableMarkdown + flattened tableRows, so table/visual answers are matched (lifted content_mrr@10 0.8979→0.9048).
Strip synopsis metadata — keeping retrieval_synopsis: it's the stored first-class passage representation used in ranking/display, so excluding it would under-count real answer content. A false-match would require a golden content term to exactly equal a section label (none do), and the metadata-distractor test already guards the title/file/section vector.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:a1a6b7f345

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-03T15:13:27Z

 query: testCase.query,
 ownerId,
- topK: testCase.topK,
+ topK: retrievalLimitForGoldenCase(testCase),


Keep latency evals at the fixture topK
When npm run eval:retrieval:latency runs, this path now upgrades every fixture with topK < 10 to topK: 10. searchChunksWithTelemetry uses topK throughout selection/reranking and visual-evidence attachment, so this changes the workload being timed rather than only fetching extra rows for the new @10 quality metric; latency reports for the existing topK=8 cases can regress or stop being comparable even when the production topK=8 path is unchanged. Gate the larger fetch to quality/combined scoring, or explicitly separate latency runs for topK=10.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-03T15:13:27Z

 return normalized(
- [
- result.title,
- result.file_name,
- result.section_heading,
- result.section_path?.join(" "),
- result.retrieval_synopsis,
- result.content,
- tableFactText,
- imageText,
- ]
- .filter(Boolean)
- .join(" "),
+ [result.retrieval_synopsis, result.content, tableFactText, imageText, indexUnitText].filter(Boolean).join(" "),


Include memory-card evidence in contentRR
For memory-boosted results, downstream answer/source paths use result.memory_cards[*].content as structured evidence, but the new content-rank evidence text omits it. In broad-summary or comparison cases where the top chunk carries an expected term only through an attached memory card, contentReciprocalRankAt10 records a miss even though the answer path has usable evidence, creating false passage-rank regressions. Include memory-card content in this evidence text alongside table, visual, and index-unit content.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-03T15:13:27Z

+export function retrievalLimitForGoldenCase(testCase: GoldenRetrievalCase) {
+ return Math.max(testCase.topK, 10);


Only widen retrieval when content rank needs it
Captured RAG eval cases are converted with expectedContentTerms: [] and topK: 8, but this helper still upgrades them to a 10-result retrieval even though they are excluded from content_mrr@10. Because searchChunksWithTelemetry uses topK to size candidate budgets and selection before the evaluator later slices back to hit@8, these cases can pass or fail based on candidates/ranking that the actual topK=8 path would not have produced; return the original topK when there are no content terms to score.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-03T15:13:27Z

- ]
- .filter(Boolean)
- .join(" "),
+ [result.retrieval_synopsis, result.content, tableFactText, imageText, indexUnitText].filter(Boolean).join(" "),


Include table-fact metadata snippets in contentRR
When a table hit carries the answer only in table_facts[*].metadata fields such as accessible_table_markdown, table_text_snippet, or cells, this evidence text still reduces table facts to the parsed scalar columns before scoring contentReciprocalRankAt10. The answer and verification paths render/check those metadata snippets, so table-heavy cases can be scored as contentRR misses even though the ranked result contains usable table evidence; add the table-fact metadata text to the evidence string.
Useful? React with 👍 / 👎.

#248 — settled without touching live. check:drift does cover the missing-index class: all four 20260705180000 indexes are in schema.sql and in the drift manifest, indexes compare by name on table + def_hash, and the drift allowlist is empty, so a run would have failed with four missing_live findings. The gap is cadence, not coverage — live-drift.yml is weekly plus manual and gates nothing. Also records a genuinely uncovered sub-class: schema_drift_snapshot() reads pg_get_indexdef but never indisvalid/indisready, so an index left invalid by a failed CREATE INDEX CONCURRENTLY compares byte-identical while the planner cannot use it. The live forensic (partial apply vs manual drop vs history repair) remains operator work behind the approved window. #210 — half already fixed, and its prescribed fix refuted. npm run typecheck has used tsconfig.typecheck.json since 450690f, which excludes .next entirely; verified green with .next/dev/types/validator.ts present. Dropping the dev-types glob from tsconfig.json does not hold: Next 16 emits it deliberately and writeConfigurationDefaults pushes it back into an existing include on every next dev/build. What remains is narrower — the Playwright isolated tsconfig inherits the repo-root globs, confirmed by probe, and Next's own dev-types filter does not apply because useTypeScriptCli defaults true. Recorded as not yet proven end-to-end. Also records the session-start marker defect fixed in the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STG6AU5J4gxrFJagP4pRti

chatgpt-codex-connectorBot reviewed Jul 3, 2026
View reviewed changes

Comment threadscripts/eval-retrieval.ts Outdated
Comment threadscripts/eval-retrieval.ts

BigSimmoand others added 2 commits July 3, 2026 20:57

Fix retrieval content MRR scoring
1f56972

style: prettier-format eval-retrieval content_mrr scoring
b4f2734
The retrievalLimitForGoldenCase fix landed without a prettier pass, failing the required verify check (format:check). Formatting-only; no logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

BigSimmo enabled auto-merge (squash) July 3, 2026 13:01

chatgpt-codex-connectorBot reviewed Jul 3, 2026
View reviewed changes

Comment threadscripts/eval-retrieval.ts Outdated
Comment threadscripts/eval-retrieval.ts
Comment threadscripts/eval-retrieval.ts Outdated

Merge branch 'main' into claude/rag-passage-metric
a1a6b7f

BigSimmo merged commit a70e3b2 into mainJul 3, 2026
4 checks passed

chatgpt-codex-connectorBot reviewed Jul 3, 2026
View reviewed changes

BigSimmo deleted the claude/rag-passage-metric branch July 5, 2026 11:43

This was referenced Aug 13, 2026
docs(issues): keep #248 open and track residual live drift #1906
Merged
docs(issues): reconcile 17 queued ledger requests #1987
Merged

Navigation Menu

eval(retrieval): add content_mrr@10 passage-rank metric - #248

eval(retrieval): add content_mrr@10 passage-rank metric#248
BigSimmo merged 5 commits into
mainfrom
claude/rag-passage-metric

BigSimmo commented Jul 3, 2026

Uh oh!

chatgpt-codex-connectorBot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BigSimmo commented Jul 3, 2026

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		export function retrievalLimitForGoldenCase(testCase: GoldenRetrievalCase) {
		return Math.max(testCase.topK, 10);

Conversation

BigSimmo commented Jul 3, 2026

What & why

Live baseline (golden set, project sjrfecxgysukkwxsowpy)

Safety / scope

Tests / checks

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BigSimmo commented Jul 3, 2026

Uh oh!

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotJul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Live baseline (golden set, project `sjrfecxgysukkwxsowpy`)