Uh oh!
There was an error while loading. Please reload this page.
Feature/s15 evaluation gaps - #26
Merged
Merged
Conversation
… log) Slice planning artifacts for S15, which closes sub-gaps 1 (no held-out eval set) and 2 (0/16 clinician-validated labels) of the HL7 evaluation's biggest-risk decomposition. Sub-gap 3 (Risk agent 9-FP rate) is reserved for S16 with its own design-risk-calibration-v2.md. - grill-evaluation-gaps.md: 6-question grill + derived Q7-Q9 (S15/S16 split, held-out set shape, labeling rules, eval-report shape, CLI ergonomics, outreach log architecture, verification matrix, out-of-scope) - prd-s15.md: problem statement + 4-commit solution + 18 user stories + D1-D11 implementation decisions + T1-T5 testing decisions + explicit out-of-scope - implementation-plan-s15.md: 4 atomic commits + Phase E verification + rollback table + D1-D10 definition of done + open follow-ups Engagement is NOT a verification gate; the slice ships the path (review: render + review:apply already built in S14) and the tracking (outreach log) without requiring a clinician to volunteer. Audit-trail-clean: every commit is "wire an existing thing to a real output," same shape as S14. S16 owns the Risk rubric work. Co-Authored-By: Claude <noreply@anthropic.com>
…a.heldOutRows
Declares the 10 held-out patients (pop-0011..pop-0020) — already part
of generatePopulation()'s 500-patient cohort at
apps/api/src/fhir-data/population.ts:13 — as a labeled held-out set
in data/eval/labels.json via _meta.heldOutRows. Adds 10 label rows
with mechanically-derived labels (careGap, risk, sdoh) per the
existing _meta.labelingRules; actionPlanner is qualitative-only.
The held-out labels are a credible-enough apples-to-apples held-out
set for a POC: bundles are independently generated (deterministic
seed) and labels come from the same dev-interpreted labeling rules
as the dev-labeled 16. Held-out SDOH sub-metric will report 0 data
points because generatePopulation()'s buildSdohForIndex(i) only seeds
explicit AHC-HRSN screenings for i in {4, 9} (pop-0005 negative,
pop-0010 positive); rows without an AHC-HRSN Observation are unlabeled
(null) per _meta.labelingRules.sdoh. The Care Gap + Risk held-out
sub-metrics are populated (9/10 careGap-positive, 10/10 risk-low).
No agent or eval-harness changes; commit 3 will render the new rows
as a Held-out evaluation section in docs/eval-report.md.
Spec: prd-s15.md D2 + grill-evaluation-gaps.md §2 + §3New apps/api/src/eval/labelFromBundle.ts exporting labelFromBundle(bundle, dim): boolean | null. Pure function — no I/O, no LLM. Mirrors the dev-interpreted _meta.labelingRules (careGap: Condition → LOINC, risk: riskScoreFor ≥ 75, sdoh: AHC-HRSN positive/negative/absent). Both the dev-labeled 16 and the held-out 10 score against this single function. Also exports the existing riskScoreFor() from population.ts (no behavior change — pure mechanical exposure so this new module can call it). The Risk branch derives (conditionCount, recencyHours) from the bundle and delegates to riskScoreFor; tests stub the import via jest.mock so they're deterministic. No eval-harness wiring yet; commit 3 calls this from scripts/eval.ts. Spec: prd-s15.md D4 + grill-evaluation-gaps.md §3 Co-Authored-By: Claude <noreply@anthropic.com>
…tion Splits apps/api/src/scripts/eval.ts into two parallel patient loops (dev-labeled baseline + held-out evaluation) plus a third Outreach section. Adds --dev-only, --held-out-only, --no-live CLI flags. Reuses eval/computeMetrics.ts (same function called twice). Status disclosure now reports "N clinician-validated / 16 dev-labeled / 10 held-out" with three counts. The held-out section uses labelFromBundle (commit 2) applied to the 10 independently-generated bundles; the dev-labeled section uses the existing _meta.labelingRules path (unchanged). Spec: prd-s15.md D5 + D6 + grill-evaluation-gaps.md §4 + §5
Adds data/eval/clinician-outreach.json (initial empty invitations[]) + apps/api/src/eval/outreachSchema.ts (pure validator) + 5 unit tests + apps/api/src/scripts/outreach-validate.ts (I/O script) + new npm run outreach:validate. Extends apps/api/src/scripts/eval.ts's renderMarkdown + buildJsonSummary to render the Outreach table (or "Outreach log not yet started" when the file is missing). Engagement is NOT a verification gate; the slice ships whether or not a clinician volunteers. The Outreach table makes the gap visible in docs/eval-report.md and gives the HL7 evaluation's Open Question #2 ("Has the HTML form been sent to any clinician?") a real answer from the artifact itself. Spec: prd-s15.md D7 + D8 + grill-evaluation-gaps.md §6 Co-Authored-By: Claude <noreply@anthropic.com>
Phase E artifacts per docs/plans/caresync-ai/implementation-plan-s15.md
§Phase E. Documents the 5-row verification matrix (all pass), the
two mid-flight defects caught by sub-agents (POPULATION_SIZE=500 in
Commit 1, SDOH regex in Commit 2), and the 7 open follow-ups (most
owned by S16 or post-S15).
Also regenerates docs/eval-report.{md,json} with the S15 three-section
layout: Status line "0 of 26 clinician-validated / 16 dev-labeled /
10 held-out", Dev-labeled baseline (16 patients), Held-out evaluation
(10 patients), Outreach, Error analysis (dev-labeled + held-out),
Data-availability gaps combined. Outreach sub-object in the JSON
summary reads { fileExists: true, ok: true, errors: [], invitations: [] }
— the empty-state disclosure that makes the engagement gap visible
without gating the slice.
Verification matrix: 5/5 pass. TDD: 21 tests across 3 eval/ suites
(labelFromBundle, outreachSchema, computeMetrics, errorAnalysis);
3 round-trip tests in scripts/eval.test.ts; 300/300 tests across
46 suites in apps/api (full regression). npx tsc --noEmit clean.
Spec: prd-s15.md §7 (verification matrix) + grill-evaluation-gaps.md
§7 (verification matrix) + implementation-plan-s15.md §Phase ERe-runs the HL7 rubric against the post-S15 state. P6 upgrades from
4 → 5 because the held-out section now renders with per-agent
sensitivity/specificity/PPV. All other pillars unchanged. P2 + P4
remain bounded by the engagement sub-gap (still 0 clinician-validated)
and the Risk agent 9-FP sub-gap (S16's scope, parallel-track).
Sources cite the regenerated docs/eval-report.{md,json} (commit
a56e93a, 2026-07-08T18:08:16Z) for the current eval state.
Co-Authored-By: Claude <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (20)
WalkthroughThis PR adds an S15 slice: a pure ChangesS15 Evaluation Slice
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant runHarness
participant FhirReadService
participant labelFromBundle
participant runEval
participant OutreachValidator
CLI->>runHarness: run with flags (--dev-only/--held-out-only/--no-live)
runHarness->>runHarness: load labels + heldOutRows
runHarness->>FhirReadService: getPatientBundle(heldOutPatientId)
FhirReadService-->>runHarness: bundle
runHarness->>labelFromBundle: labelFromBundle(bundle, dim)
labelFromBundle-->>runHarness: expected label or null
runHarness->>runEval: runEval(labels, fhirService, db, noLive)
runEval-->>runHarness: scored results
runHarness->>OutreachValidator: readAndValidateOutreach()
OutreachValidator-->>runHarness: ok/errors + invitations
runHarness-->>CLI: markdown + JSON report
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Bug Fixes