revert(S13b): Risk rubric reverted + samuel-wright seed enriched

apps/api/src/agents/riskAgent.test.ts

@@ -143,85 +143,58 @@ describe('runRiskAgent (B1 revised — mocked OpenAI client, no live call)', ()
 
 // S13 A2 — TDD pins on `buildPrompt`'s structural properties. The agent's
 // classification comes from the LLM (non-deterministic), so TDD can't pin
-// "patient X gets riskLevel=high" — but it CAN pin that the prompt carries
-// the calibration rubric (D3), the citation requirement (GD11, regression
-// guard), and the bundle's resource grounding (regression guard). These
-// are the load-bearing properties — if any silently regress, the
-// calibration breaks without a test failure.
+// "patient X gets riskLevel=high" — but it CAN pin that the prompt preserves
+// the citation requirement (GD11 regression guard) and the bundle's
+// resource grounding (regression guard). These are the load-bearing
+// properties — if any silently regress, the agent falls back to training-
+// data priors or hallucinates citations, breaking G3 / G4.
 //
-// Fixture bundle mirrors `riskScoreFor()`'s evidence: 2 chronic conditions,
-// a recent inpatient encounter (enc-<id> class would be inpatient in real
-// data; not modeled here at the schema level, but the Encounter resource
-// line is included so the prompt's recency anchor has grounded text to
-// find), and 3 Observations.
-describe('buildPrompt (S13 — Risk rubric calibration)', () => {
- const rubricFixtureBundle = {
+// History note (S13b): this `describe` block previously pinned a 3-anchor
+// rubric that mirrored `riskScoreFor()` ≥ 75. Live re-eval showed the
+// rubric caused the model to over-call (every patient including no-evidence
+// ones got `riskLevel: 'critical'` — specificity 30.8% → 0%). The rubric
+// was reverted. The two survival tests below are the load-bearing
+// properties the prompt MUST keep, regardless of future calibration work.
+describe('buildPrompt (S13 — structural surface)', () => {
+ const fixtureBundle = {
  resources: [
  { resourceType: 'Condition', id: 'fixture-cond-1' },
- { resourceType: 'Condition', id: 'fixture-cond-2' },
  { resourceType: 'Encounter', id: 'fixture-enc-1' },
  { resourceType: 'Observation', id: 'fixture-obs-1' },
- { resourceType: 'Observation', id: 'fixture-obs-2' },
- { resourceType: 'Observation', id: 'fixture-obs-3' },
  ],
  validIds: new Set([
  'Condition/fixture-cond-1',
- 'Condition/fixture-cond-2',
  'Encounter/fixture-enc-1',
  'Observation/fixture-obs-1',
- 'Observation/fixture-obs-2',
- 'Observation/fixture-obs-3',
  ]),
  };
 
- // A2.1 — D3 rubric's three evidence anchors must all be present.
- it('buildPrompt includes the rubric anchors (multi-condition comorbidity, recent inpatient discharge, abnormal labs)', () => {
- const prompt = buildPrompt(rubricFixtureBundle);
- expect(prompt.toLowerCase()).toContain('multi-condition comorbidity');
- expect(prompt.toLowerCase()).toContain('recent inpatient discharge');
- expect(prompt.toLowerCase()).toContain('abnormal labs');
- // Lab thresholds: BNP >200, HbA1c >9.0, eGFR <30 — the calibration
- // target that mirrors `riskScoreFor()` ≥ 75.
- expect(prompt).toContain('BNP');
- expect(prompt).toContain('200');
- expect(prompt).toContain('HbA1c');
- expect(prompt).toContain('9.0');
- expect(prompt).toContain('eGFR');
- expect(prompt).toContain('30');
- });
+ // A2.1 (formerly rubric-anchors) — REMOVED. The rubric itself was reverted
+ // after live re-eval showed it caused the model to over-call. See the
+ // JSDoc on `buildPrompt` and verification-s13.md §3.
 
- // A2.2 — the four risk-level tiers must be named explicitly in the rubric
- // and a count threshold must appear (so the model can't be ambiguous about
- // which bucket a patient falls into).
- it('buildPrompt includes the threshold text and all four risk-level tiers', () => {
- const prompt = buildPrompt(rubricFixtureBundle);
- expect(prompt).toContain('low');
- expect(prompt).toContain('moderate');
- expect(prompt).toContain('high');
- expect(prompt).toContain('critical');
- expect(prompt.toLowerCase()).toMatch(/at least 2|two or more|≥2/);
- expect(prompt).toContain('30 days');
- });
+ // A2.2 (formerly threshold text) — REMOVED. Same reason as A2.1.
 
- // A2.3 — GD11 regression guard. The calibration must NOT displace the
- // citation requirement (Risk agent's core architectural innovation — see
- // `P4 Trust/Safety` evidence in `HL7-Challenge-Evaluation.md`).
+ // A2.3 — GD11 regression guard. The prompt must keep the citation
+ // requirement intact (Risk agent's core architectural innovation — see
+ // `P4 Trust/Safety` evidence in `HL7-Challenge-Evaluation.md`). If this
+ // test fails, a future edit to `buildPrompt` has dropped the citation
+ // contract and the eval's confusion matrix can no longer be trusted.
  it('buildPrompt preserves the citation requirement (GD11 regression guard)', () => {
- const prompt = buildPrompt(rubricFixtureBundle);
+ const prompt = buildPrompt(fixtureBundle);
  expect(prompt).toContain('fhirResourceId');
  expect(prompt.toLowerCase()).toContain('fabricated citations');
  });
 
- // A2.4 — grounding regression guard. The rubric must NOT displace the
- // bundle's resources; the agent still has to reason from the actual FHIR
- // data, not from training-data priors.
+ // A2.4 — grounding regression guard. The prompt must keep embedding the
+ // bundle's resources. If a future edit drops or hardcodes the resource
+ // list, the agent stops reasoning from the actual FHIR data and falls
+ // back to priors — same over-calling failure mode that the S13 rubric
+ // hit, but at the data layer instead.
  it('buildPrompt embeds the bundle resources (grounding regression guard)', () => {
- const prompt = buildPrompt(rubricFixtureBundle);
+ const prompt = buildPrompt(fixtureBundle);
  expect(prompt).toContain('Condition/fixture-cond-1');
- expect(prompt).toContain('Condition/fixture-cond-2');
  expect(prompt).toContain('Encounter/fixture-enc-1');
- // And one Observation by id to confirm the prompt is iterating
- // through the bundle rather than hardcoding a list.
- expect(prompt).toContain('Observation/fixture-obs-2');
+ expect(prompt).toContain('Observation/fixture-obs-1');
  });
 });

apps/api/src/agents/riskAgent.ts

@@ -66,41 +66,29 @@ const REPORT_RISK_TOOL = {
 };
 
 /**
- * S13 — exported for TDD unit tests (see `riskAgent.test.ts` A2.x). The
- * prompt's structural properties (rubric anchors, citation requirement,
- * bundle embedding) are the load-bearing calibration surface — they
- * are what `riskAgent.test.ts` pins.
+ * S13 — exported for TDD unit tests (`riskAgent.test.ts` pins the prompt's
+ * structural surface — citation requirement + bundle embedding).
  *
- * The embedded rubric below mirrors `fhir-data/population.ts:127-134`'s
- * `riskScoreFor()` ≥ 75 threshold (D3, design-risk-calibration.md §3) so the
- * agent's `riskLevel` output aligns with the seed-heuristic label used in
- * `data/eval/labels.json`. **Clinician validation of labels is the
- * long-term path to a real-clinical rubric** — this is the conservative
- * interim step that fixes the eval's 9-FP false-positive rate without
- * metric gaming.
+ * History note: an S13 rubric mirroring `fhir-data/population.ts:127-134`'s
+ * `riskScoreFor()` ≥ 75 threshold was authored + TDD-pinned (see
+ * docs/plans/caresync-ai/design-risk-calibration.md), but live re-eval
+ * showed it caused the model to over-call (specificity regressed from 30.8%
+ * → 0% on the 16-patient held-out set — every patient including the
+ * no-evidence ones got `riskLevel: 'critical'`). The rubric was reverted to
+ * the prior one-paragraph form, keeping the export + JSDoc update. The
+ * follow-up fix is to enrich the seed data (a single, surgical change in
+ * `apps/api/src/fhir-data/seed-patients.ts`'s `samuel-wright` entry) so the
+ * label-evidence gap is closed for that one patient — the rest of the eval
+ * findings remain honest under the original prompt. Clinician validation of
+ * labels remains the long-term path to a real-clinical rubric.
  */
 export function buildPrompt(bundle: PatientBundle): string {
  const resourceLines = bundle.resources.map((r) => `- ${r.resourceType}/${r.id}: ${JSON.stringify(r)}`).join('\n');
 
  return [
  'You are a clinical risk-assessment agent. Narrate your reasoning briefly in plain text, then report your findings by calling the report_risk tool exactly once.',
  '',
- 'You are the Risk agent on a care-coordination platform, assessing 30-day hospital readmission risk.',
- '',
- '## Risk rubric (S13 calibration)',
- '',
- 'Assign the patient\'s `riskLevel` using these anchors. A patient is high or critical risk when they meet at least 2 of the 3 anchors below. A patient is moderate risk when they meet exactly 1 anchor. A patient is low risk when they meet 0 anchors.',
- '',
- 'Anchor A — Multi-condition comorbidity: the patient has at least 2 active Conditions from this set: diabetes (ICD-10 E11.9), CHF (ICD-10 I50.9), depression (ICD-10 F33.1), CKD (ICD-10 N18.3).',
- '',
- 'Anchor B — Recent inpatient discharge: any Encounter whose end is within the last 30 days and whose class indicates inpatient or acute care (not just any recent encounter).',
- '',
- 'Anchor C — Abnormal labs: any of BNP > 200 pg/mL, HbA1c > 9.0%, or eGFR < 30 mL/min/1.73m² in the Observations.',
- '',
- 'Justify your `riskLevel` choice in the narration by naming which anchors the patient meets. Do not call a patient high or critical when fewer than 2 anchors are met — over-calling risk produces non-actionable alerts. Do not call a patient low when 2 or more anchors are clearly present.',
- '',
- '## Patient record (FHIR)',
- '',
+ "You are the Risk agent on a care-coordination platform, assessing 30-day hospital readmission risk.",
  "Below is the patient's complete retrieved FHIR record (one resource per line, as `ResourceType/id: <resource JSON>`).",
  '',
  resourceLines,

apps/api/src/fhir-data/seed-patients.ts

@@ -123,7 +123,20 @@ export const PANEL_PATIENTS: SeedPatient[] = [
  gender: 'male',
  birthDate: '1955-01-30',
  phone: '+1-555-0194',
+ // S13 follow-up — the seed previously carried only a 1-condition (CHF)
+ // record, but `riskScore: 79` plus the post-discharge tasks ("Daily weight
+ // monitoring", "Sodium-restricted diet education") implied a CHF inpatient
+ // admit with BNP evidence. The S13 Risk rubric (≥2 of {multi-condition,
+ // recent inpatient discharge ≤30d, abnormal labs}) couldn't see that
+ // evidence because the bundle didn't carry it, and the post-S13 eval
+ // flipped him from TP to FN. Enriching the seed with the encounter + obs
+ // the label implied gives the agent real evidence to evaluate against.
  conditions: [{ id: 'samuel-wright-chf', system: 'ICD-10', code: 'I50.9', display: 'Heart failure, unspecified' }],
+ observations: [
+ { id: 'samuel-wright-bnp', loincCode: '30934-4', display: 'Natriuretic peptide B', value: 380, unit: 'pg/mL' },
+ { id: 'samuel-wright-potassium', loincCode: '2823-3', display: 'Potassium', value: 3.5, unit: 'mmol/L' },
+ ],
+ encounter: { id: 'samuel-wright-chf-admit', conditionId: 'samuel-wright-chf', dischargedHoursAgo: 36 },
  riskScore: 79,
  tasks: [
  { id: 'samuel-wright-task-weight', description: 'Daily weight monitoring check-in', priority: 'high', dueInDays: 0 },

apps/api/src/scripts/eval.ts

@@ -196,11 +196,11 @@ function renderMarkdown(labels: LabelRow[], run: EvalRunResult, metrics: Metrics
  );
  lines.push('');
  lines.push(
- '**Status (S13):** Risk-agent prompts now include an explicit clinical rubric (≥2 of {multi-condition comorbidity, recent inpatient ' +
- 'discharge ≤30d, abnormal labs: BNP>200, HbA1c>9.0, eGFR<30}) that mirrors `fhir-data/population.ts:127-134` `riskScoreFor()` ≥ 75. ' +
- 'The Risk-specificity and PPV numbers below reflect that alignment with the synthetic ground truth, not with a real clinical ' +
- 'reference standard. See `docs/plans/caresync-ai/design-risk-calibration.md` §2 D3 / §3 for the calibration rationale. Clinician ' +
- 'validation of labels remains the long-term path to a real-clinical rubric — this calibration is the conservative interim step.'
+ '**Status (S13b):** The S13 calibration attempt (Risk-prompt rubric mirroring `riskScoreFor()` ≥ 75) was reverted after live re-eval ' +
+ 'showed it caused the model to over-call (specificity regressed from 30.8% → 0% on the 16-patient held-out set). The follow-up ' +
+ 'fix in this slice is a single seed-data change — `apps/api/src/fhir-data/seed-patients.ts`\'s `samuel-wright` entry now carries ' +
+ 'the Encounter + Observations his label implied but the seed previously omitted. See `docs/plans/caresync-ai/verification-s13.md` ' +
+ '§3 + §6 for the full reversion log. Clinician validation of labels remains the long-term path to a real-clinical rubric.'
  );
  lines.push('');
  lines.push('## Methodology');
@@ -309,8 +309,8 @@ function renderMarkdown(labels: LabelRow[], run: EvalRunResult, metrics: Metrics
  lines.push('### Risk false positives (agent over-called risk)');
  lines.push('');
  lines.push(
- '**Note (S13):** The Risk agent\'s prompt rubric was authored to mirror the synthetic seed heuristic. The specificity number above reflects ' +
- 'that alignment — see `docs/plans/caresync-ai/design-risk-calibration.md` for the calibration rationale.'
+ '**Note (S13b):** The S13 risk-rubric was reverted after live re-eval showed it over-called. The remaining false positives above reflect ' +
+ 'the pre-S13 baseline (seed-derived labels vs the LLM\'s general clinical priors); see `docs/plans/caresync-ai/verification-s13.md` for the reversion log.'
  );
  lines.push('');
  if (errors.risk.falsePositives.length === 0) {