test(eval): recognize real Discharge Planning docs in golden alias - #206
Merged
Conversation
… alias The golden fixtures reference synthetic MHSP.* filenames that were superseded when the corpus was repopulated with real named documents. Most alias-match, but the Discharge alias listed only "…to Discharge" mental-health variants, not the "Discharge Planning for Community Patients / Inpatients" documents that actually rank for the admission-vs-discharge comparison. That made admission-discharge-comparison report a false top-5 recall miss on the non-existent MHSP.Discharge.pdf even though a real, relevant discharge document was retrieved. Add "Discharge Planning" to the Discharge alias. Any "Discharge Planning …" document is unambiguously a discharge document, so this recognizes a genuine hit rather than masking a miss. admission-discharge-comparison now passes (grounded, 5 citations); eval-quality unit tests 11/11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
15 tasks
BigSimmo added a commit
that referenced
this pull request
Aug 2, 2026
…+8) (#1576) * DS V2 PR-E: answer safety, form foundation, announcements (slices 6+7+8) Build the five specified safety components so a degraded answer, an absent clinical value, an invalid field and a route change all become states the type system and assistive technology can see. Slice 6 - answer safety. VerificationNotice owns eight approved strings (four states x clinician/plain); AnswerCard requires both `state: AnswerState` and `verification`, and a degraded state cannot be constructed without the `onOpenSource` route that turns the caution into an affordance. DoseLine takes a structured dose model, composes Quantity, and marks overdue in three channels (amber rule + "Source review overdue" + StatusMark shape). MissingValue replaces the bare dash with four phrases. DateDisplay takes ISO only and renders <time>. AnswerFooter now names an absent field instead of dropping it - on a provenance strip the absence is the governance signal. answerClipboardText carries the degraded caveat out of the app and routes provenance through the one existing clipboardProvenanceLine() implementation. Slice 7 - FormField/FieldHint/FieldError/ErrorSummary. Hint and error are both in the DOM and both in describedBy when invalid; caller ids merge ahead of them rather than being overwritten; required/optional is label text; ErrorSummary takes focus rather than announcing. Slice 8 - LiveAnnouncer/RouteAnnouncer/announce. One singleton owns every announcement, with a dedupe window and a queue gap; route change moves focus to the new <h1> unless focus sits inside a dialog or a preserve-focus workflow. Step 0 contract pre-check passed with no change to src/lib/rag/** or source-review: ready, stale_evidence and source_only all project from fields the retrieval layer already decides, so no component infers staleness from a date. partial_retrieval has no producer and is recorded as such in SPEC, COMPONENTS, /issues #206 and tests/answer-state-contract.test.ts. All components are built, not registered - registration and product adoption stay in PR 13. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ds-v2): tighten answer-safety states after clinical governance review Eight findings from the PR 6 clinical governance review, all in code this branch introduced, all safety-positive. No product surface imports any of these modules yet, so nothing rendered changes. Counting and identity - answerStateFromRetrieval() counts and keys by document, not chunk. RagAnswer.sources is chunk-level, so the previous projection produced duplicate React keys, repeated a document as several overdue rows, and reported fractions that were wrong in both directions — including the direction that under-warns ("1 of 6" when half the documents are stale). - Where chunks of one document disagree on governance status, the more severe reading wins. Governance status carried, not collapsed - OverdueSource gains `status`, and DoseRow replaces the optional `overdue` boolean with a required `status: DocumentStatus`. An overdue status additionally requires `source`, so a warned row with no route back to the document is unrepresentable. `outdated` (superseded) now renders the slashed mark and the word "superseded" rather than the half-ring "review overdue" vocabulary of a document still in force. Cautions that cannot argue against themselves - stale_evidence with an empty overdue list now throws in development (mirroring the partial_retrieval guard) and, in production, states the caution without a count instead of "0 of 3 sources are past their review date". Same rule in answerClipboardText(). - VerificationNotice falls back to the most cautionary wording for an unrecognised state, and logs once, rather than failing open to `ready`. - The stale clinician wording says "some of which are past their review date" rather than asserting every cited source is overdue. - SourceOnlyBody no longer claims "nothing has been paraphrased": the tier is inferred from the routing mode and the extractive builder composes sections, so the surface cannot stand behind verbatim fidelity. Provenance precision - DateDisplay renders a date-only ISO as a date, even for kind="generated". It previously printed "14/03/2026, 08:00" for "2026-03-14" — a precision that was never recorded, on a provenance strip. Clipboard (exceeds the original SPEC slice-8 scope, deliberately) - answerClipboardText() now carries attribution and "Verify against the linked source documents before clinical use." on every state including `ready`, enumerates the cited documents, and suppresses the single-document provenance line where it would contradict a multi-source stale caveat. A copied answer loses the banner, the notice and the links; unattributed clinical prose in a record reads as clinician-endorsed. It remains narrower than formatAnswerRenderCopyText and is not a replacement for it. Recorded rather than fixed, so PR 13 cannot inherit them as clearance: AnswerState has no channel for an ungrounded answer (#208), the clipboard constraint above (#209), and the Gate 1 contrast pair for --warning at text tier plus the per-process logged-once Sets (#210). All three are also written into SPEC.md and COMPONENTS.md. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ledger): record PR-E answer-safety review at 8ad91e3 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ds-v2): resolve PR-E review findings for answer safety Filter staleness to cited supporting sources, reject rolled calendar dates, make duplicate production announcers a no-op, re-focus ErrorSummary on repeated submits, clear-then-set identical live announcements, and realign AnswerCard/DoseLine dtsPropsFor declarations. * fix(ds-v2): suppress duplicate LiveAnnouncer without lint cascade Keep production duplicate ownership in an effect with a targeted set-state-in-effect exemption, matching the repo pattern, so the duplicate never registers a listener and unmounts its live regions. * fix(ds-v2): address Copilot findings on banner and form tests Import ComponentProps from react, require exact overdue/sourceCount equality for the totality headline, and derive the partial-retrieval numerator from the rendered missing list. * fix(ds-v2): address CodeRabbit findings on answer-safety surfaces Merge the broken OverdueSource doc comment, omit clipboard provenance when metadata is absent, keep unidentified overdue sources as stale, give ErrorSummary an accessible heading name, fail over LiveAnnouncer ownership when the active instance unmounts, harden MissingValue phrase lookup against prototype keys, and treat sourceCount underflow as totality. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Smallest safe fix for the
admission-discharge-comparisongolden failure (item #2 from triage).Root cause
The golden fixtures reference synthetic
MHSP.*filenames that no longer exist — the corpus was repopulated with real named documents. TheclinicalDocumentAliaseslayer maps the old names to real ones, but theDischargealias listed only "…to Discharge" mental-health variants, not the "Discharge Planning for Community Patients / Inpatients" documents that actually rank for this query. So the case reported a false top-5 recall miss on the non-existentMHSP.Discharge.pdf, even though a real, relevant discharge document ("Discharge Planning for Community Patients (NMHS).pdf") was retrieved and the answer was grounded.Fix
Add
"Discharge Planning"to theDischargealias. Any "Discharge Planning …" document is unambiguously a discharge document, so this recognizes a genuine hit rather than masking a miss (the concern I flagged during triage).Verification
admission-discharge-comparisonnow passes: grounded, 5 citations, no failures.tests/eval-quality.test.ts11/11; typecheck/lint/prettier clean.Not included (deliberately): the NOCC and agitation-table failures are a separate, deeper issue — the deterministic document-list / table answers are mangled by the clinical-prose sanitizer (
sanitizeStructuredText) they shouldn't run through. That has no small safe fix (the sanitizer is shared and load-bearing for real LLM prose), so it's left as a scoped follow-up rather than risking regressions.🤖 Generated with Claude Code