feat(eval): isolated Docling lab benchmark harness + Gate B decision-record template (packet S6 / B3)

[BigSimmo]

Summary

Implements packet S6 from docs/rag-improvement/HANDOVER.md: the isolated Docling lab benchmark under eval/docling/ per docs/rag-improvement/README.md §B3. The deliverable is the harness plus a Gate B decision-record template — no benchmark verdict is recorded; that is a separate owner-reviewed dispatch run.

• Isolated lab under eval/docling/ with its own fully hashed Python 3.11 lockfile (pip-compile --generate-hashes, 1,720 sha256 hashes, CPU-only torch via the pytorch CPU index) and its own venvs. Nothing under worker/**, Dockerfile.worker, src/lib/extractors/document.ts, or the database is modified — the legacy extractor and the worker's production lock are consumed strictly read-only as the comparator.
• Sandbox: Docker image on the repo's existing digest-pinned node:24-bookworm-slim base; benchmark runs with --network=none (egress block), non-root lab user, --cap-drop=ALL, read-only rootfs and repo mount, 2 CPUs / 6 GB memory / 256 pids / 1 GB tmpfs; per-document wall clocks (120 s, hostile 60 s) enforced by process-group SIGKILL with peak RSS measured via os.wait4; whole-run 3600 s cap; output caps (64 MB per-doc, 512 MB raw, 1 MB report) enforced fail-closed. Docling layout/TableFormer models are prefetched at image build because the run has no network.
• Fixtures: 36 synthetic fixtures across 6 difficulty strata plus a 10-file hostile corpus (truncated, malformed xref, deep nesting, 64 MB compression bomb, encrypted, zero-byte, absurd MediaBox, mislabelled PNG, huge page tree, prompt-injection with planted canary), all generated at run time from the committed ground-truth manifest by a seeded, self-checking PyMuPDF generator (output byte-identical across runs except the AES-salted encrypted fixture). S4 fixture posture throughout: invented drug names only, letters-only CANARY-* leak detectors, real-source denylist enforced by validation.
• Comparison measures (README §B3): parse success, resource bounds, cell-level table precision/recall/F1, exact number/unit/comparator checks, hostile containment — scored per stratum by harness/score.py with definitions documented next to the arithmetic.
• Aggregate-only reports:report/build-report.mjs rebuilds measurements through a numeric allowlist, stamps the six-field S4 report key imported from scripts/rag-adversarial-contract.mjs (never re-declared), scans the serialised report for canary/real-source leaks and fails closed printing counts, never tokens. Extractor identities travel outside the key like S4's promptVersion.
• Gate B decision-record template: human template docs/rag-improvement/gate-b-decision-record.md + machine twin eval/docling/report/gate-b-decision-record.template.json (all gates pending_owner_run), mirroring baseline-record.md's recorded-vs-pending provenance discipline, with pre-agreed thresholds that must be completed before a run is dispatched. Validated by the same module (--validate-record, template and final modes).
• Dispatch-only workflow:.github/workflows/docling-lab.yml is workflow_dispatch only, permissions: contents: read, ubuntu-24.04, allowlisted pinned action SHAs, uploads only the aggregate report artifact on success. It is not referenced by pr-required or any automatic trigger.
• Offline CI coverage:tests/docling-lab-contract.test.ts (20 tests, no docling install needed) pins the manifest contract, imported report-key order, the aggregate-only allowlist (a canary planted in polluted measurements must not survive into the report), gate-status discipline, template validity, and lockfile/env cross-checks. npm run check:docling-lab is the matching CLI gate.
• Repo plumbing: eval/ added to docs/codebase-index.md, scripts-index inventory regenerated (+2 npm scripts: generate:docling-lab-lock, check:docling-lab), .gitignore entry for eval/docling/out/, HANDOVER §2 S6 status row updated in this PR.

RAG impact: no retrieval behaviour change — isolated dispatch-only eval lab under eval/docling/; no runtime, worker, extractor, ranking, or retrieval code is touched, and the benchmark never runs in PR CI.

Verification

• npm run verify:pr-local — heavy fail-closed plan (unknown eval/** scope + workflow file): completed: check:runtime, check:installed-lock-parity, format:changed, check:npm-ci-dry-run, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline, check:github-actions, check:ci-scope, check:gitleaks-pinned, check:ci-triage, check:pr-policy, check:gate-manifest, check:skills, check:pr-mergeability, check:verification-plan, lint, typecheck, test, build, check:rag:fixtures, check:medication-interactions, check:medication-lexicon-report · failed: (none) · not reached: (none)
• npm run check:docling-lab — docling-lab contract passed (36 fixtures, 10 hostile, 6 canaries; Gate B template valid).
• node scripts/run-vitest.mjs run tests/docling-lab-contract.test.ts — Tests 20 passed (20)
• npm run check:github-actions — GitHub Actions pin check passed.
• Docs gates — docs link check passed: 1832 repo path references resolve. · docs script-ref check passed: 494 npm-run reference(s) resolve to real scripts. · docs/codebase-index.md coverage OK · Docs inventory current: 245 script files, 250 npm scripts.
• End-to-end smoke (real legacy engine over the full 46-document generated corpus, local venv): 30/30 text-bearing fixtures parsed, table cell F1 1.0, all 144 non-OCR assertions found, hostile 10/10 contained with 0 crash artifacts and 0 canary echoes; scanned_ocr 0/6 locally only because the smoke venv has no tesseract (the lab image installs it). The report chain (score.py → build-report.mjs) produced a canary-clean aggregate report from that run.
• The docker build / full benchmark run is deliberately not executed in this session: benchmark runs are manual/dispatch-only per the packet, and the verdict is an owner-reviewed run.

UI verification not run: no UI, routing, styling, or browser behaviour is touched — the diff is an isolated eval lab, docs, and a dispatch-only workflow.

Risk and rollout

• Risk: Low. Everything new lives under eval/docling/, tests/, docs, and one workflow_dispatch-only workflow; no production, worker, extractor, or retrieval path changes. The workflow cannot fire automatically and uploads only an aggregate, canary-scanned report. The two npm scripts are additive.
• Rollback: revert this PR's single squash commit; nothing else references the new files. The lab has no persistent state — eval/docling/out/ is gitignored run output.
• Provider or production effects: None. No OpenAI/Supabase/live-data access anywhere in the lab; the sandbox run is egress-blocked by construction. Lock generation used the public PyPI and pytorch CPU package indexes only.

Clinical Governance Preflight

Completed because this PR adds evaluation infrastructure adjacent to clinical extraction (it changes no clinical behaviour):

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

• For the Gate B owner run: copy the decision-record template and agree the thresholds before dispatching .github/workflows/docling-lab.yml; results land as a docling-lab-report-<run id> artifact.
• Known fixture-hardness limitation (affects B4 planning): the v1 table strata are cleanly ruled grids and the legacy extractor already scores cell F1 1.0 on them locally, leaving the table-heavy improvement target little headroom. Consider a docling-lab-fixtures.v2 with unruled/merged-cell/rotated-header tables before treating the table-heavy delta as decisive. Details in eval/docling/README.md.
• The encrypted hostile fixture is the only non-byte-deterministic output (AES-256 salts); everything else is byte-identical across generator runs (no_new_id, pinned dates, fixed seed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01XX3AXYHiGXiFfL2VGFiEMn

---

Generated by Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:39 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 95 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7437e741-ffde-44f7-97b6-af5fd564574f

📥 Commits

Reviewing files that changed from the base of the PR and between 9c660af and 8bc9950.

📒 Files selected for processing (26)

• .github/workflows/docling-lab.yml
• .gitignore
• docs/branch-review-records/6ea4a18560d833a6c86edeeebf1828e2a11801d2e55027be54889272b93e7a96.record.md
• docs/codebase-index.md
• docs/rag-improvement/HANDOVER.md
• docs/rag-improvement/gate-b-decision-record.md
• docs/scripts-index.md
• eval/docling/Dockerfile
• eval/docling/README.md
• eval/docling/fixtures/generate_fixtures.py
• eval/docling/fixtures/manifest.v1.json
• eval/docling/generate-lock.mjs
• eval/docling/harness/entry.sh
• eval/docling/harness/run-legacy.ts
• eval/docling/harness/run_corpus.py
• eval/docling/harness/run_docling.py
• eval/docling/harness/score.py
• eval/docling/report/build-report.mjs
• eval/docling/report/gate-b-decision-record.template.json
• eval/docling/report/lab-config.json
• eval/docling/report/lab-contract.mjs
• eval/docling/requirements.in
• eval/docling/requirements.txt
• eval/docling/run-lab.sh
• package.json
• tests/docling-lab-contract.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[BigSimmo]

Unit coverage failure on 0531220: reproduced locally — both job steps pass on the identical tree. Ran the coverage job's exact commands (npm run test:coverage, then npm run check:coverage-inventory) on a checkout of 0531220:

• test:coverage — exit 0; 646 files / 6945 tests passed; summary Statements 56.36% · Branches 53.57% · Functions 59.11% · Lines 57.72%, all above the 52/49/54/53 floors.
• check:coverage-inventory — PASS — 1128 source files cover src/, scripts/, worker/, supabase/functions/.

So the CI failure looks environmental, not a defect in this diff. The most plausible cause is the coverage job's timeout-minutes: 20: the instrumented full suite has grown by roughly 800 tests this week (S1c #2052, S5 #2056, this PR), and it took ~7 minutes locally on 4 workers — a 2-core hosted runner under v8 instrumentation may now exceed 20. Worth checking whether the job log shows tests still executing at cancellation; if so, the durable fix is raising that job's timeout (or sharding), which would affect every PR, not just this one.

Also worth noting: the earlier PR required failures on 356948a→0a899d2→e06967e were each on heads superseded minutes later by the next update-branch sync — cancel-in-progress: true fails the aggregate on every superseded head. With main moving this fast, the repo's settle-then-sync-once guidance will avoid burning a CI run per sync; the failed 0531220 run is itself already superseded by 8bc9950, whose fresh run is the one to watch. Every gate in this PR's own scope remains green locally at the current tree.

---

Generated by Claude Code