Packet S6b: Gate B PASS — Docling lab benchmark run, decision record, and harness fixes

docs/branch-review-records/381b41511e145024c0d34d173ed186f1659871b294a41e390fa36fdf6f5bdfb9.record.md

@@ -0,0 +1 @@
+| 2026-08-18 | claude/docling-gate-b-eval-5czgln | 397f40d96c4c08c5da61b8fcf07e166a93cf7aba | packet S6b: Gate B run + decision record (eval/docling harness fixes + docs/rag-improvement records) | Gate B PASS recorded from evidence run 32176604314; four latent harness defects fixed (setuptools pin, libGL, torch.compile toolchain, HTML-entity scoring) | verify:pr-local selected plan green (lint, typecheck, docs/ledger contracts); npm run test 673 files / 7281 passed / 4 skipped; check:rag:fixtures 36 golden / 26 suites; check:docling-lab contract passed; gate-b record valid (final mode) |

docs/outstanding-issues-inbox/abc21f52-0d39-490d-9676-d106b9af4302.json

@@ -0,0 +1,14 @@
+{
+ "version": 2,
+ "id": "abc21f52-0d39-490d-9676-d106b9af4302",
+ "createdOn": "2026-08-18",
+ "action": "add",
+ "payload": {
+ "pri": "P2",
+ "type": "task",
+ "summary": "Build packet B4: Docling worker shadow mode (WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow) — authorised by the Gate B PASS of 2026-08-18",
+ "detail": "Gate B passed on 2026-08-18 (evidence run 32176604314 at 8a92378; record docs/rag-improvement/gate-b-decision-record-2026-08-18.md): design-only authorisation for README section B4 per HANDOVER S7+. Scope: typed WORKER_DOCUMENT_EXTRACTOR_MODE env defaulting to legacy, shadow runs after legacy success on a 1-5 percent cohort selected by src/lib/index-quality.ts signals, aggregate metadata only, no chunk/embedding/index writes, kill switch, one-step rollback to legacy; ingestion-worker-reviewer reviews the PR. Two caveats travel from the decision record: the table-heavy leg passed at parity-on-ceiling (fixtures.v2 hardness corpus precedes any table-quality promotion argument), and docling runs eager at roughly 9-19 s/doc on 2 CPUs vs legacy's 1 s — cohort sizing must budget for it.",
+ "source": "packet S6b Gate B decision record, 2026-08-18",
+ "issueUlid": "01M0B66FJ69DGA6R5W8QN9V7CY"
+ }
+}

docs/outstanding-issues-inbox/fb87f710-b46c-452a-b349-e3ce71c36959.json

@@ -0,0 +1,14 @@
+{
+ "version": 2,
+ "id": "fb87f710-b46c-452a-b349-e3ce71c36959",
+ "createdOn": "2026-08-18",
+ "action": "add",
+ "payload": {
+ "pri": "P3",
+ "type": "task",
+ "summary": "Docling lab fixtures.v2 table-hardness corpus: add unruled, merged-cell, and rotated-header tables so the Gate B table-heavy improvement leg has measurable headroom",
+ "detail": "The v1 corpus's table strata are cleanly ruled grids on which the legacy extractor already scores cell F1 1.0 (S6 smoke run and the S6b Gate B run), so the pre-agreed table-heavy improvement target was set to 0 pp (parity at ceiling) by owner decision on 2026-08-18. Before any table-heavy delta is treated as decisive for a Docling promotion beyond B4 shadow design, add a docling-lab-fixtures.v2 stratum set where the legacy find_tables path is expected to degrade: unruled tables, merged cells, rotated headers. Fixture-hardness change only — eval/docling/ manifest + generator, no worker or extractor edit. See eval/docling/README.md 'Known limitation (v1 corpus)' and docs/rag-improvement/gate-b-decision-record-2026-08-18.md.",
+ "source": "packet S6b (Gate B run), owner threshold decision 2026-08-18",
+ "issueUlid": "01M0AXXCJMBSBE9BYR4QX6QBC8"
+ }
+}

docs/rag-improvement/gate-b-decision-record-2026-08-18.json

@@ -0,0 +1,106 @@
+{
+ "recordVersion": "docling-lab-gate-b.v1",
+ "status": "pass",
+ "note": "Owner-run Gate B decision record for the Docling lab benchmark (packet S6b). Thresholds were agreed and committed BEFORE any benchmark dispatch (branch commit 6d05c07525e9bc30b43edd0eaeda279a57ae607b, 2026-08-18). The recorded evidence run is workflow run 32176604314 (artifact docling-lab-report-32176604314) at commit 8a923787c81e4256d4e22e45eed65ccb26f5fba8 — the first run in the dispatch chain whose docling pass is a valid measurement; see runHistory. Human twin: docs/rag-improvement/gate-b-decision-record-2026-08-18.md. Validate: node eval/docling/report/build-report.mjs --validate-record docs/rag-improvement/gate-b-decision-record-2026-08-18.json --final.",
+ "preAgreedThresholds": {
+ "agreedBeforeRun": true,
+ "parseSuccessNonInferiorityMarginPp": 0,
+ "numericExactnessNonInferiorityMarginPp": 0,
+ "tableHeavyCellF1ImprovementTargetPp": 0,
+ "thresholdRationale": "Zero-pp non-inferiority margins match the programme's zero-tolerance posture (36/36 golden fixture, zero per-case regressions): docling must lose no parse and no dose/unit/comparator assertion legacy preserves, in any stratum. The table-heavy improvement target is 0 pp by explicit owner decision (this session, 2026-08-18): the v1 corpus's cleanly ruled tables put the legacy extractor at cell F1 1.0 in the S6 smoke run, so parity at ceiling satisfies the improvement leg, the headroom caveat is recorded in the decision, and a docling-lab-fixtures.v2 table-hardness follow-up (unruled, merged-cell, rotated-header tables) is queued in the issues inbox.",
+ "resourceCeilings": "eval/docling/report/lab-config.json (sandbox + outputCaps) at the run's commit_sha"
+ },
+ "reportKey": {
+ "commit_sha": "8a923787c81e4256d4e22e45eed65ccb26f5fba8",
+ "dataset_version": "docling-lab-fixtures.v1",
+ "eval_config_version": "docling-lab-config-v1",
+ "model_version": "answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol",
+ "embedding_version": "text-embedding-3-small@1536",
+ "index_version": "20260818113000_forward_codify_hybrid_owner_matches_bodies"
+ },
+ "reportKeyNote": "Copied verbatim from the run artifact's stamped reportKey. model_version, embedding_version, and index_version are programme-wide comparability qualifiers not exercised by this extraction benchmark — no model, embedding, or database call occurs inside the sandbox.",
+ "qualifiers": {
+ "extractorVersions": {
+ "legacy": "src/lib/extractors/document.ts + worker/python/extract_pdf_assets.py (pymupdf==1.28.0, worker/python/requirements.txt)",
+ "docling": "docling==2.120.2 (eval/docling/requirements.txt)"
+ },
+ "workflowRunUrl": "https://github.com/BigSimmo/Database/actions/runs/32176604314",
+ "artifactName": "docling-lab-report-32176604314",
+ "doclingPhaseEnvironment": "TORCHDYNAMO_DISABLE=1 (eager mode; the sandbox image ships no C++ toolchain for torch.compile — eval/docling/harness/entry.sh)"
+ },
+ "runHistory": [
+ {
+ "runId": "32164356999",
+ "commit": "0216f18e97c224ce9bf50799ba8c4837909e781f",
+ "outcome": "infrastructure failure — docker build: pip --require-hashes refused torch's setuptools>=77.0.3 runtime requirement (lock generated without --allow-unsafe); fixed by 779af80"
+ },
+ {
+ "runId": "32165911181",
+ "commit": "779af8094827d9cd8ad32d3d164ca2b0c5fe6e4e",
+ "outcome": "infrastructure failure — docling model prefetch: rapidocr imports cv2, slim base lacked libGL.so.1/glib; fixed by 958a70d"
+ },
+ {
+ "runId": "32166445937",
+ "commit": "958a70d90b373ee45d26645f4cc1cddf492209d0",
+ "outcome": "workflow green but not a valid docling measurement — all 46 docling documents errored uniformly; cause locked in never-uploaded raw output; diagnostic error surface added by c66faf8"
+ },
+ {
+ "runId": "32171549648",
+ "commit": "c66faf89072011dc3307b86bcc2be8425e1cbaff",
+ "outcome": "diagnostic run — named the failure: InvalidCxxCompiler (torch.compile needs a C++ compiler the sandbox image deliberately lacks); fixed by 6c6e80e (TORCHDYNAMO_DISABLE=1)"
+ },
+ {
+ "runId": "32174653778",
+ "commit": "6c6e80ea1cdc31865f01c8c8e8aac06b40cb55b6",
+ "outcome": "first valid docling pass — parse 36/36, table F1 parity, hostile clean, resources in bounds; numeric exactness scored -25 to -62.5pp purely because docling's markdown export HTML-escapes comparators ('>= 1.4 mmol/L' as '>= 1.4 mmol/L') and the scorer did not unescape; values verified intact locally; scorer made escape-neutral by 8a92378"
+ },
+ {
+ "runId": "32176604314",
+ "commit": "8a923787c81e4256d4e22e45eed65ccb26f5fba8",
+ "outcome": "recorded evidence run — all five gates pass under the pre-agreed thresholds"
+ }
+ ],
+ "gates": [
+ {
+ "id": "parse_success",
+ "caseCount": 36,
+ "status": "recorded",
+ "result": "PASS — docling 36/36 and legacy 36/36; comparison.perStratum[*].parseSuccessDeltaPp = 0 in all six strata (rule: >= 0, margin 0 pp)",
+ "evidence": "https://github.com/BigSimmo/Database/actions/runs/32176604314 artifact docling-lab-report-32176604314"
+ },
+ {
+ "id": "resource_bounds",
+ "caseCount": 46,
+ "status": "recorded",
+ "result": "PASS — no timeout, signal kill, or output-cap breach on any of 46 documents for either engine; docling max wallClockMsP95 12,558 ms (cap 120,000 ms; hostile cap 60,000 ms), max peakRssBytesMax 1,504,313,344 bytes (~1.40 GiB, cgroup cap 6 GiB), run completed inside the 3,600 s cap, report 1,285 bytes (cap 1 MiB)",
+ "evidence": "https://github.com/BigSimmo/Database/actions/runs/32176604314 artifact docling-lab-report-32176604314"
+ },
+ {
+ "id": "table_precision_recall",
+ "caseCount": 12,
+ "status": "recorded",
+ "result": "PASS — tableCellPrecision/Recall/F1 = 1.0 for both engines on table_simple and table_heavy; tableCellF1DeltaPp = 0 on both (non-inferiority rule: >= 0); table-heavy improvement leg met at the pre-agreed 0 pp parity-at-ceiling target — the v1 corpus gives legacy F1 1.0, so no improvement was demonstrable by construction (fixtures.v2 follow-up queued)",
+ "evidence": "https://github.com/BigSimmo/Database/actions/runs/32176604314 artifact docling-lab-report-32176604314"
+ },
+ {
+ "id": "numeric_exactness",
+ "caseCount": 36,
+ "status": "recorded",
+ "result": "PASS — docling 162/162 and legacy 162/162 dose/unit/comparator assertions found; comparison.perStratum[*].assertionCoverageDeltaPp = 0 in all six strata (rule: >= 0, margin 0 pp); scored escape-neutrally (html.unescape in score.py normalise) so entity-encoded comparators compare by value",
+ "evidence": "https://github.com/BigSimmo/Database/actions/runs/32176604314 artifact docling-lab-report-32176604314"
+ },
+ {
+ "id": "hostile_containment",
+ "caseCount": 10,
+ "status": "recorded",
+ "result": "PASS — docling containedCount 10/10 and legacy 10/10; containedDeltaCount 0 (rule: >= 0), crashArtifactTotal 0 (rule: = 0), canaryEchoTotal 0 (rule: = 0); every hostile failure was a clean bounded error within its per-document limit",
+ "evidence": "https://github.com/BigSimmo/Database/actions/runs/32176604314 artifact docling-lab-report-32176604314"
+ }
+ ],
+ "decision": {
+ "outcome": "pass",
+ "ownerSignoff": "Thresholds, every dispatch (six approvals), and the parity-at-ceiling table target were explicitly owner-approved in the S6b session on 2026-08-18; final countersign is the owner's review/merge of the packet S6b pull request.",
+ "date": "2026-08-18",
+ "consequence": "Gate B pass authorises DESIGNING packet B4 (WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow worker shadow mode) only, with two explicit caveats carried forward: (1) the table-heavy improvement leg was satisfied at parity-on-ceiling, not by a demonstrated gain — a docling-lab-fixtures.v2 hardness corpus is queued before any promotion beyond shadow design leans on a table-quality argument; (2) docling ran in eager mode (TORCHDYNAMO_DISABLE=1) at ~9-19 s per document on 2 CPUs versus legacy's ~1 s — shadow-cohort sizing must budget for that. The worker remains untouched; the ingestion-worker-reviewer subagent reviews the B4 PR."
+ }
+}