Uh oh!
There was an error while loading. Please reload this page.
revert(S13b): Risk rubric reverted + samuel-wright seed enriched - #21
Merged
Conversation
Follow-up to PR #19 (the original S13 Risk-prompt calibration). Live re-eval on PR #19 produced a regression the original design didn't anticipate: | Metric | Pre-S13 | PR #19 (rubric) | Pre-S13 retry (post-revert) | | Specificity | 30.8% | 0% | 0% | | FPs (out of 13 TNs) | 9 | 13 | 13 | | Sensitivity | 100% | 100% | 100% | The pre-S13 retry column is the key finding: reverting the rubric to the original 1-paragraph form reproduces the same specificity 0% the rubric did. The LLM API has changed baseline behavior between 2026-07-07 (date of the committed pre-S13 report) and 2026-07-08 (today) — it now calls every patient with active Conditions `riskLevel: 'critical'` regardless of bundle evidence. This is *not* caused by the S13 PR; the trigger is upstream of this code. Actions in this PR: 1. Revert the S13 rubric prompt to the prior 1-paragraph form. The `export` of `buildPrompt` is kept (TDD surface) and a JSDoc paragraph documents the reversion + points at design-risk-calibration.md for the audit trail. 2. Trim the 4 rubric-pinning tests to 2: kept the citation requirement regression guard (GD11) and the bundle grounding guard. Removed the rubric-anchor and threshold-text tests (the rubric they pinned is gone). 7/7 riskAgent.test.ts green; 43/43 across src/eval/ + src/agents/ green. 3. Rewrite the S13 disclosures in `apps/api/src/scripts/eval.ts`'s `renderMarkdown()` from "rubric-mirrors-seed" to "S13b = reversion + seed enrichment" — both at the Methodology banner and above the Risk-FP list. The next `npm run eval` will print the rewritten disclosures. 4. Enrich `apps/api/src/fhir-data/seed-patients.ts`'s `samuel-wright` with an Encounter (CHF inpatient, discharged 36h ago) + 2 Observations (BNP 380 pg/mL — well above the 200 threshold — and K+ 3.5). His `riskScore: 79` plus the post-discharge tasks ("Daily weight monitoring", "Sodium-restricted diet education") implied a CHF admit with BNP evidence; the bundle previously didn't carry the evidence. Re-import FHIR (idempotent PUT) propagated the change. 5. Refresh the 4 plan/verification/review docs: - design-risk-calibration.md = historical (rubric reverted) - implementation-plan-risk-calibration.md = historical + S13b additions - verification-s13.md = active post-mortem (failure-mode trigger, live-eval data, LLM-variance diagnosis) - review-s13.md = S13b two-axis review NOT in this commit: - docs/eval-report.{md,json} NOT regenerated. The fresh-cache eval reports a specificity regression (0%) that's not caused by this PR and would mislead any downstream reader. The pre-S13 committed reports (2026-07-07 timestamps) stay as the canonical artifact. The follow-up live-eval data point is recorded in verification-s13.md §4. Cross-slice follow-up (verification-s13.md §6): 1. LLM-side variance investigation — determine why the API state changed between 2026-07-07 and 2026-07-08 (model version, system prompt, defaults). 2. v2 Risk rubric with few-shot examples instead of abstract anchors. 3. Clinician validation of labels via `npm run review:render`. 4. Re-run eval after (1) to confirm the calibration gains are reproducible. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This follow-up to PR #19 (the original S13 Risk-prompt calibration) reverts the rubric after live re-eval revealed an unexpected LLM-side regression, and ships a seed-data fix for the patient whose label-evidence gap was the original motivation for the calibration.
What happened
PR #19 was authored to fix the rubric-analyzer's biggest gap (9 FPs / 13 TNs → specificity 30.8%). Live re-eval on it showed a regression that the original design didn't anticipate:
The post-revert column is the key finding: re-running the original pre-S13 code (no rubric) reproduces the same specificity 0%. So the rubric isn't the cause — the LLM API has changed baseline behavior between 2026-07-07 (date of the committed pre-S13 report) and 2026-07-08 (today). Today's LLM calls every patient with active Conditions
riskLevel: 'critical'regardless of bundle evidence.This PR:
exportretained for TDD + a JSDoc noting the reversion).samuel-wright's seed with Encounter + Observations — the single data fix the rubric-mirrors-seed analysis identified as the actual gap.What this PR does NOT do
docs/eval-report.{md,json}. Today's fresh-cache eval reports specificity 0% — a regression that is not caused by this PR (verified above) and would mislead any downstream reader. The 2026-07-07 pre-S13 committed reports stay as the canonical artifact. The live-eval data point + the LLM-variance diagnosis are recorded inverification-s13.md§4.Verification
npx jest src/agents/riskAgent.test.tsnpx jest src/eval/ src/agents/npx tsc --noEmitnpm run import(idempotent PUT)Cross-slice follow-up (verification-s13.md §6)
npm run review:render(long-term path to a real-clinical rubric regardless of LLM variance).Audit trail
docs/plans/caresync-ai/design-risk-calibration.md— full S13 design rationale + why the rubric failed (retained as historical)docs/plans/caresync-ai/implementation-plan-risk-calibration.md— original phases A-E + S13b additionsdocs/plans/caresync-ai/verification-s13.md— active post-mortem with the failure-mode trigger, live-eval data, and the full LLM-variance diagnosisdocs/plans/caresync-ai/review-s13.md— S13b two-axis review🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com