[Epic] Dimensions 5-7 roadmap: peer-group scoring, HITL calibration, lineage packet, real extraction

[stranske]

Why

The 2026-06-28 audit's field/opportunity research (Code/Audits/Inv-Man-Intake/2026-06-28-05-field-and-opportunities.md)
found the contracts and the explainable-scoring stance are at/ahead of the institutional manager-DD field, but the
highest-leverage gaps are adjacent problems the existing machinery almost solves. This epic tracks them so
dimensions 5–7 (approach-vs-field / missed-opportunities / tools) are not lost; each child is a separately-scoped
2-week issue. This is a roadmap epic, not a single mergeable change.

Scope

Track and sequence the four highest-leverage opportunities. Each becomes its own AGENT_ISSUE_FORMAT child issue
with its own test/verification gate before implementation.

Non-Goals

• This epic itself ships no code; do NOT implement everything in one PR.
• Do NOT claim extraction accuracy/throughput on fixtures (the README disclaimer is correct) — any accuracy claim
  must be backed by a real-document eval set.
• Scaffold-only completion does NOT count: closing this epic requires the child issues to be filed (and the
  peer-group child to land with a test), not a single placeholder.

Tasks (each → its own child issue)

• Peer-group / percentile relative scoring (highest leverage, no ML). The asset-class partition
  (scoring/weights.py:LAUNCH_ASSET_CLASSES) and component metrics already exist; add a cohort store
  (DuckDB/SQLite) and emit a percentile rank vs. asset-class cohort alongside the absolute 0–1 score. Child issue
  must include a deterministic test on a seeded cohort.
• Close the HITL calibration loop. CorrectionRecord + escalations are captured but never measured. Build
  a job that joins corrections to extractions → field-level precision/recall + confidence calibration, used to
  tune the 0.85/0.75/0.60 thresholds in config/extraction_thresholds.yaml with evidence.
• Regulatory-defensibility lineage packet. Assemble existing provenance/versioning/trace data into a
  one-shot per-decision lineage export (no new capture needed).
• Make extraction real behind the provider Protocol. Evaluate Docling (MIT, local) as a provider
  implementation + instructor for schema-valid LLM output + DeepEval (PR-gate) / LangSmith datasets (drift).
  Keep the stlite demo fixture-backed.

Acceptance Criteria

• Four child issues are filed (one per task), each AGENT_ISSUE_FORMAT-compliant with its own named
  test/verification gate, and linked back to this epic.
• The peer-group child issue includes a concrete deterministic test
  (tests/scoring/test_peer_group.py::test_percentile_rank_against_seeded_cohort) as its gate.
• This epic links Code/Audits/Inv-Man-Intake/2026-06-28-05-field-and-opportunities.md as the source analysis.

Implementation Notes

• Source analysis with citations: 2026-06-28-05-field-and-opportunities.md (commercial peers: Canoe, Accelex,
  Backstop, Dynamo; tools: Docling, instructor, DeepEval).
• Sequence by leverage: peer-group → HITL calibration → lineage packet → real extraction.

[stranske]

Child issues filed (2026-06-28), sequenced by leverage:

• [Epic #699][P2] Peer-group / percentile relative scoring (cohort store, no ML) #710 — Peer-group / percentile relative scoring (highest leverage, no ML)
• [Epic #699][P2] Close the HITL calibration loop — measure corrections vs extractions to tune thresholds #711 — Close the HITL calibration loop (corrections vs extractions → threshold tuning)
• [Epic #699][P3] Assemble a regulatory-defensibility lineage packet from existing provenance #712 — Regulatory-defensibility lineage packet (assemble existing provenance)
• [Epic #699][P3] Real document extraction via Docling behind the provider Protocol + eval harness #713 — Real document extraction via Docling behind the provider Protocol + eval harness

Each is AGENT_ISSUE_FORMAT with a named test gate + deliberate-break. Source analysis: Code/Audits/Inv-Man-Intake/2026-06-28-05-field-and-opportunities.md.

[stranske]

Additional children filed (2026-06-28) from the deepened dim-5/6/7 study (deep-research verified competitor teardowns + PDF methodology — see Code/Audits/Inv-Man-Intake/2026-06-28-05b-field-deepdive.md, -05c-competitor-teardowns.md, and Code/Audits/2026-06-28-pdf-extraction-methodology.md):

• [Epic #699][P1] Extraction-accuracy regression gate + drift monitor (on existing LangSmith traces) #714 — Extraction-accuracy regression gate + drift monitor (MISSED-OPP-3) — P1
• [Epic #699][P1] Cross-check ensemble extraction — generalize conflict_resolver to all key numeric fields #715 — Cross-check ensemble extraction, generalizing conflict_resolver (MISSED-OPP-4) — P1
• [Epic #699][P2] Weight-table governance — sensitivity report + versioned approval changelog #716 — Weight-table governance: sensitivity + approval changelog (MISSED-OPP-6) — P2
• [Epic #699][P2] Document-type classifier → per-type expected-field/threshold profiles #717 — Document-type classifier → per-type field/threshold profiles (FIELD-5/MISSED-OPP-7) — P2

Full child set: #710 #711 #712 #713 #714 #715 #716 #717.

NOTE: PDF extraction is a cross-repo concern (Counter_Risk, Pension-Data, Manager-Database, Inv-Man-Intake all re-implement it independently). A dedicated shared-tool initiative is being spun out separately — see Code/Audits/2026-06-28-fleet-pdf-extraction-survey.md. #713 (Docling) should consume that shared tool rather than add a 5th private implementation.

[stranske]

Dependency notes (2026-06-29)

• [Epic #699][P3] Real document extraction via Docling behind the provider Protocol + eval harness #713 (real extraction / Docling) — depends on the spun-out shared-PDF-tool session; per the 2026-06-29 Pyodide audit it must provide a Pyodide-viable native-text mode (pdfplumber/pypdf) for the Tier-A browser app PLUS a Docling/OCR mode that runs outside the browser (Docling/torch + Tesseract are not Pyodide-viable). Not 'run Docling in the browser.'
• [Epic #699][P1] Extraction-accuracy regression gate + drift monitor (on existing LangSmith traces) #714 (drift/regression gate) — depends on [Epic #699][P2] Close the HITL calibration loop — measure corrections vs extractions to tune thresholds #711 (the golden set / correction-eval loop supplies the labels).
• [Epic #699][P2] Peer-group / percentile relative scoring (cohort store, no ML) #710 (peer-group), [Epic #699][P3] Assemble a regulatory-defensibility lineage packet from existing provenance #712 (lineage packet), [Epic #699][P2] Weight-table governance — sensitivity report + versioned approval changelog #716 (weight governance), [Epic #699][P2] Document-type classifier → per-type expected-field/threshold profiles #717 (doc-type classifier) are largely independent and parallelizable.

Operator-app children #721-#727 are tracked on epic #718.

[stranske]

Closing: all children verified done in the 2026-07-09 implementation-verification pass (Code/Audits/Inv-Man-Intake/2026-07-09-implementation-verification.md). #710-#717 each merged with their named acceptance test present + real (deliberate-break gates confirmed, e.g. #714's test_extraction_regression_gate_fails_on_perturbed_golden_value); Docling (#713) correctly an optional extra so the core stays Pyodide-viable; CI green on main 708123e.