Uh oh!
There was an error while loading. Please reload this page.
feat(S2): single-agent analysis with citation enforcement - #3
Merged
Conversation
Iteration 2 for S2 (single-agent analysis with citation enforcement): Risk agent (Claude Sonnet 5, structured output) over Maria's $everything bundle, streamed over SSE to one feed box, with the Seam-2 citation validator dropping any fhirResourceId absent from the retrieved bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the Risk agent (OpenAI gpt-5.5, GD13-revised from Claude Sonnet 5), its SSE streaming route, and the citation validator (Seam 2) that drops any fhirResourceId absent from the retrieved bundle before it reaches the UI, plus the frontend Run Analysis control and streaming feed box. Includes fixes from this session's verification-before-completion and code-review pass: GD11 citation enforcement now also covers the narrated token stream (not just structured flags), the SSE route degrades gracefully instead of hanging/crashing on an agent failure, the OpenAI client is constructed lazily so a missing key can't crash the process at boot, and the duplicated PatientBundle/AgentFlag types are each now a single exported source. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 5, 2026
Merged
manjula25 added a commit
that referenced
this pull request
Jul 9, 2026
…hors-rule + few-shot S16 commit 3 replaces the post-S13b 1-paragraph buildPrompt with the v2 rubric from design-risk-calibration-v2.md §"The v2 rubric": - 3 calibration anchors (multi-condition comorbidity, recent inpatient discharge ≤30d, abnormal labs) addressing S13 failure mode #2 (loose abstract anchors). - Explicit "0 anchors met is ALWAYS riskLevel='low'" hard rule addressing S13 failure mode #3 (any-condition → critical over-call). - 3 worked examples using actual seed-text bundle shapes (james-okafor for 0 anchors, maria-chen for 1 anchor, synthetic `bob` for 2 anchors) addressing S13 failure mode #1 (negative-instruction vs clinical-judgment). 3 new TDD tests pin the v2 structure so future agents cannot silently regress it to either the S13b 1-paragraph form or a partial v2 (same audit-trail discipline as S13b's rubric-pins removal). All 10 riskAgent tests pass (7 existing S13b regression-guard tests + 3 new v2 structure pins). **2x2 acceptance gate (per prd-s16.md D6 + implementation-plan-s16.md §"Phase C"):** | Metric | Target | Actual | Pass | |-------------------------------|--------|-------------|------| | Dev-labeled 16 specificity | ≥30% | 69.2% | ✓ | | Dev-labeled 16 sensitivity | ≥67% | 100.0% | ✓ | | Held-out 10 specificity | ≥30% | 50.0% | ✓ | | Held-out 10 sensitivity | ≥50% | n/a (denom 0) |⚠️ | The held-out sensitivity denominator-0 is a property of the held-out labels (none of pop-0011..pop-0020 meet labelFromBundle's riskScoreFor ≥ 75 threshold), not a v2 rubric failure — see verification-s16.md §1 + rubric-eval-result.md §"Why held-out sensitivity is undefined" for the full structural explanation. **Verdict:** gate passes on the meaningful, measurable signals. Dev-labeled specificity recovered from 0% (post-S13b over-call) to 69.2%; FPs dropped from 9 → 4 on the dev-labeled 16. Pillar P2 lifts 4 → 5; total HL7 evaluation moves 89.2 → ~91.0 per prd-s16.md D10. **OpenAI quota incident (audit trail):** the first live eval run (96 LLM calls) produced the 2x2 numbers above and wrote them into docs/eval-report.{md,json}. A subsequent cleanup re-run hit 429 quota exceeded on all 24 cache misses; the partial eval-report was reverted with `git checkout HEAD -- docs/eval-report.{md,json}` so the working tree stays consistent. The v2 numbers are preserved in verification-s16.md §5 (verbatim reproduction of the dev-labeled + held-out Risk sections + confusion matrices) and rubric-eval-result.md §"Gate result". docs/eval-report.{md,json} regeneration is a 1-command follow-up (`cd apps/api && npx tsx src/scripts/eval.ts`) once quota refreshes — not a code change. Co-Authored-By: Claude <noreply@anthropic.com>
manjula25 added a commit
that referenced
this pull request
Jul 9, 2026
…+ G5) Completes the Phase G post-merge verification deliverables that don't require live LLM calls (the eval-report.regen sidecar + variance probe re-run at items G1 + G2 remain blocked on OpenAI quota refresh — see rubric-eval-result.md §"Recovery steps deferred to post-merge"). **docs/plans/caresync-ai/review-s16.md** (181 lines, S14/S15 two-axis pattern: Standards + Spec + Self-review + Aggregated verdict): - Standards axis: 6 baseline smells + 1 cross-cutting tradeoff (eval.ts Status line add/revert during quota incident), each left as-is with the reasoning + the specific 3 TDD test names anchoring the binding contract. - Spec axis: 2 real defects from commit 3 sub-agent mid-flight (commit 2's OpenAI Responses API rejects seed+temperature per Option A pivot; commit 3's interrupted Phase C eval fixed manually this session), 3 documented design tradeoffs (held-out sensitivity N/A from label-set limitation; `--risk-only` flag absent per S15's CLI flag set; eval.ts Status line add/revert decision rationale). - Self-review: 8 per-concern verdicts — 3-commit independent revertabil- ity, TDD discipline preserved (3 v2 pins written before buildPrompt rewrite; RED → GREEN transcript in verification-s16.md §3), S13b regression-guard tests (citation requirement + bundle grounding) preserved, runRiskAgent not modified, no other agent touched, variance probe survives any future commit-3 reversion, 2x2 numbers preserved with S13b-style audit discipline per the quota incident, all 6 implementation-plan constraints honored, pillar P2 advances per prd-s16.md D10. - Aggregated verdict: PASS with one deferred follow-up (eval-report sidecar regen — 1 command, blocked on OpenAI quota). **reports/HL7-Challenge-Evaluation.2026-07-09-post-s16.md** (HL7 evaluation re-run capturing P2 lift + new total): - Pillar P2 4 → 5 (dev-labeled specificity recovered 0% → 69.2%, +39.2 pp above 30% target; sensitivity preserved at 100%; held-out specificity 50%, above 30% target by 20 pp). Contribution delta +3.6 at 18% weight. - All other pillars unchanged (P1=5, P3=5, P4=4, P5=5, P6=5, P7=4, P8=4, P9=3). Weighted total: 89.2 → 92.8 (+3.6). - Audit-trail note on the +1.2 delta discrepancy in the post-S15 file (reports 89.2 but pillar contributions sum to 90.4) preserved in §C + sources — the +3.6 P2-lift contribution delta is consistent with the `score × weight / 5` formula used in pillar rows. - 8 open questions (4 carried, 4 new surfaced by S16): held-out sensitivity structural N/A (label-set, not rubric), 4 residual FPs at moderate-vs- high boundary (v3 rubric follow-up), model card authoring now unblocked per implementation-plan §"Open follow-ups" #3, latent SDOH regex bug in confidenceScorer.ts:172 (5-minute follow-up PR). - AI-leverage multiplier M=1.15 unchanged (v2 rubric is LLM-only ingredient — hand-crafted rule-based scorer implementing the same calibration would lose the model's cross-domain clinical reasoning). - Band: Finalist (85-100) unchanged. - Strongest dimension: P1 + P3 + P2 (first time three pillars share 5 since pre-S13 baseline); biggest gap: P4 (governance, model card, 0/26 clinician validation, parallel-track not gated by S16). **Eval-report.{md,json} regen status:** unchanged on disk (deferred to `cd apps/api && npx tsx src/scripts/eval.ts` once OpenAI quota refreshes). v2 numbers preserved canonically in verification-s16.md §5. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gpt-5.5, GD13-revised from Claude Sonnet 5) with aPOST /api/patients/:id/analysisSSE route, streaming clinical narration and structured findings to the client.fhirResourceId— in structured flags and in the free-text narration stream — is validated against the patient's retrieved bundle before it reaches the UI; fabricated citations are dropped/redacted.PatientDetail, with FHIR citation chips on validated findings.verification-before-completion+code-reviewpass, run before merge: the SSE route now degrades gracefully instead of hanging/crashing on an agent failure; narration citations are now enforced (previously only structured flags were checked); the OpenAI client is now built lazily so a missing key can't crash the process at boot;PatientBundle/AgentFlagtypes are deduplicated to single sources.Test plan
FHIR_BASE_URL=http://localhost:8080/fhir npm run test:api— 15 suites / 61 tests passingnpm run test:web— 5 files / 14 tests passingnpm run test:e2e— 3 Playwright specs passing (login/panel, analysis streaming, social-worker denial)npm run build/npm run lint— both apps clean (pre-existing warn-level warnings only)gpt-5.5+ HAPI (D3): ~70 streamed tokens, 9 findings, all 9fhirResourceIds independently verified against a fresh bundle fetch; fabrication-drop proven by running the productionvalidateCitationsagainst the real flags plus one synthetic fabricated oneFull detail in
docs/plans/caresync-ai/implementation-plan.md(Iteration 2),verification.md,review.md, and the changelog underdocs/superpowers/specs/feature-caresync-s2-single-agent-analysis/.🤖 Generated with Claude Code