fix(eval): cost readout counts zero-usage cases as $0 + follow-up ledger capture

[BigSimmo]

Summary

Two small post-program items:

Cost readout fix (issue #14). Every 44-case canary printed Estimated cost USD: n/a despite the pricing rates being set since PR #1017. Root cause: extractive/unsupported cases make no provider call, their per-case estimate was null, and the all-or-nothing total nulled on any null case — so one extractive answer hid the whole run's spend. A no-usage case costs exactly $0 when rates are configured; null now stays reserved for rates-unconfigured, so the total keeps meaning "cannot estimate". One-line change in scripts/eval-quality.ts plus an aggregation pin test (priced + zero-cost cases sum; a genuinely unpriced case still yields n/a).

Outstanding-issues capture (#12–#17). The post-program follow-up queue is now durable ledger memory: the monitoring retrieval-depth trio, comparison-class coverage, this cost fix, the parked E-3d residual, the operator metadata-refresh task, and the Sunday 2026-07-26 scheduled-artifact read.

RAG impact: no retrieval behaviour change — eval reporting + docs only; no src/lib/rag/** edit.

Verification

• tests/eval-quality.test.ts — 28/28 including the new aggregation pin
• typecheck, prettier clean

Risk and rollout

• Risk: low — eval report arithmetic only; gates and thresholds untouched.
• Rollback: plain revert.
• Provider or production effects: none (the fix changes how already-collected usage is summed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved evaluation cost reporting: cases with configured cost rates now show $0 when no provider activity is recorded, while other missing-usage scenarios remain unavailable to avoid incorrect totals.
  • Updated total cost aggregation to stay unavailable when any case has an unpriceable (null) estimate.

• Tests

  • Added coverage for summing priced, zero-cost, and unpriceable evaluation cases in quality reports.

• Documentation

  • Refreshed the outstanding-issues tracker with new open items covering monitoring, comparison, evaluation readouts, governance metadata, and scheduled-run artifact handling.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b5614d68-2c72-4017-875d-b13365c81d44

📥 Commits

Reviewing files that changed from the base of the PR and between 2ed9ce1 and eb1046b.

📒 Files selected for processing (2)

• docs/outstanding-issues.md
• scripts/eval-quality.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/eval-quality.ts

---

📝 Walkthrough

Walkthrough

The RAG evaluation report now distinguishes zero-cost cases that never contacted the provider from attempted cases with missing usage data. The outstanding-issues tracker advances its identifier and adds six open items.

Changes

RAG evaluation cost handling

Layer / File(s)	Summary
Configured-rate cost aggregation and validation scripts/eval-quality.ts, tests/eval-quality.test.ts	estimatedCostUsd returns 0 for cases that never used the provider when rates are configured, while attempted but unmeasured cases remain null; tests verify priced, zero-cost, and unpriceable aggregation.

Outstanding-issues ledger

Layer / File(s)	Summary
Issue tracker metadata and entries docs/outstanding-issues.md	The issues:next-id pointer changes from 018 to 024, and open items #018–#023 are added.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• BigSimmo/Database#1017: Adds related RAG pricing-rate configuration.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately reflects the two main changes: eval cost readout fix and outstanding-issues ledger capture.
Description check	✅ Passed	The description covers Summary, Verification, and Risk/Rollback, and includes the key implementation details; only the Notes/clinical preflight sections are light or omitted.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:2ed9ce125d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/outstanding-issues.md`:
- Line 48: Fix the unbalanced Markdown strikethrough markers in issue `#015` by
removing or correctly balancing the ~~ delimiters around “6 generation attempts”
and “$2-4”. Ensure only the intended text is struck through and the remaining
description renders normally.
- Line 47: Update issue `#014` in the outstanding-issues document to reflect that
the PR resolves configured-rate and zero-cost eval cost reporting. Move it to
the resolved archive or rewrite the row to describe only any remaining
validation work, ensuring the outstanding-issues report no longer presents the
completed estimator fix as open.
In `@scripts/eval-quality.ts`:
- Around line 1141-1145: Update the estimatedCostUsd calculation in the
evaluation result to distinguish provider calls from cases with no provider
call. Use openAIRequestIds or modelUsed together with an explicit usage-present
signal, rather than hasOpenAIUsage alone, so generated answers with missing
token metadata return null while confirmed no-call cases return $0 and
usage-bearing calls still use estimateCostUsd.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 209d5a86-8960-477e-9b76-80c8a462760f

📥 Commits

Reviewing files that changed from the base of the PR and between ce32fe1 and 2ed9ce1.

📒 Files selected for processing (3)

• docs/outstanding-issues.md
• scripts/eval-quality.ts
• tests/eval-quality.test.ts