feat(worker): packet B4 — docling shadow extraction mode (WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow, default legacy)

.env.example

@@ -237,6 +237,20 @@ WORKER_INLINE_ENRICHMENT=false
 # `npm run eval:assertions` run before enabling. Fail-open; nothing consumes
 # the annotations yet.
 WORKER_MEDSPACY_ASSERTION=false
+# Packet B4 (Gate B PASS 2026-08-18): "legacy" (default) or "shadow". In shadow mode
+# docling additionally runs AFTER the legacy index generation is committed, on a
+# deterministic cohort of PDFs selected by index-quality signals (tables / OCR /
+# layout), and writes only an aggregate numeric record to
+# documents.metadata.shadow_extraction. No chunk/embedding/index/table-fact writes.
+# Kill switch + one-step rollback: set back to "legacy" (no migration, no reindex).
+WORKER_DOCUMENT_EXTRACTOR_MODE=legacy
+# Percentage of eligible documents that shadow-run (authorised window 1-5; owner-approved 2).
+WORKER_SHADOW_EXTRACTION_COHORT_PERCENT=2
+# Interpreter of the docling venv. Dockerfile.worker sets /opt/docling-venv/bin/python
+# together with DOCLING_ARTIFACTS_PATH=/opt/docling-models, TORCHDYNAMO_DISABLE=1 and
+# HF_HUB_OFFLINE=1 (models are baked at build; no run-time HuggingFace fetch). Leave
+# unset locally: shadow mode then records `runtime_unavailable` without spawning anything.
+# WORKER_DOCLING_PYTHON_BIN=/opt/docling-venv/bin/python
 PYTHON_BIN=python
 # Optional when Tesseract is installed outside PATH on Windows.
 TESSERACT_CMD=C:\Program Files\Tesseract-OCR\tesseract.exe

Dockerfile.worker

@@ -10,6 +10,14 @@
 # - Tesseract OCR (Debian package, bundles English language data).
 # - Python venv with worker/python/requirements.txt (PyMuPDF, Pillow,
 # pytesseract). The venv's `python` matches the PYTHON_BIN default.
+# - Packet B4 (docs/rag-improvement/README.md §B4, Gate B PASS 2026-08-18):
+# a SECOND, isolated Python venv (/opt/docling-venv) built from the Gate B
+# lab's hashed lock (eval/docling/requirements.txt — docling==2.120.2 by
+# construction, CPU-only torch) plus docling's models baked at build time
+# (/opt/docling-models). Consumed only by worker/python/shadow_docling_extract.py
+# when WORKER_DOCUMENT_EXTRACTOR_MODE=shadow; the OCR venv is untouched
+# (numpy 1.26 vs 2.4 forces the split). Not enabling shadow costs nothing at
+# run time — the mode defaults to "legacy" and is set as a Railway variable.
 #
 # Build: docker build -f Dockerfile.worker -t clinical-kb-worker .
 # Run: docker run --env-file <secrets> clinical-kb-worker
@@ -51,14 +59,36 @@ RUN for attempt in 1 2 3; do \
  done
 
 FROM node-base AS runner
+# libgl1 + libglib2.0-0 are OpenCV's runtime shared libraries: docling's rapidocr
+# stage imports cv2 during `docling-tools models download` and again at run time,
+# and the slim base ships neither (Gate B run 32165911181 failed the prefetch with
+# `ImportError: libGL.so.1`). ca-certificates covers the build-time model fetch.
 RUN apt-get update \
- && apt-get install -y --no-install-recommends python3 python3-venv tesseract-ocr \
+ && apt-get install -y --no-install-recommends python3 python3-venv tesseract-ocr ca-certificates libgl1 libglib2.0-0 \
  && rm -rf /var/lib/apt/lists/*
 WORKDIR /app
 COPY worker/python/requirements.txt worker/python/requirements.txt
 RUN python3 -m venv /opt/ocr-venv
 RUN /opt/ocr-venv/bin/pip install --no-cache-dir --upgrade --require-hashes -r worker/python/requirements.txt \
  && /opt/ocr-venv/bin/pip check
+# Packet B4: docling shadow-extraction venv from the Gate B lab's hashed lock (read-only
+# consumption; the lock is regenerated only via `npm run generate:docling-lab-lock`).
+# Same image shape as eval/docling/Dockerfile, so shadow measurements are taken with the
+# extractor identity the decision record names. Kept in its own cacheable layer set.
+COPY eval/docling/requirements.txt /tmp/docling-requirements.txt
+RUN python3 -m venv /opt/docling-venv \
+ && /opt/docling-venv/bin/pip install --no-cache-dir --require-hashes -r /tmp/docling-requirements.txt \
+ && /opt/docling-venv/bin/pip check \
+ && rm /tmp/docling-requirements.txt
+# Prefetch docling's models at build time: HF_HUB_OFFLINE=1 below forbids any run-time
+# fetch, so a missing model fails the shadow run loudly instead of reaching HuggingFace
+# from a production worker.
+RUN /opt/docling-venv/bin/docling-tools models download --output-dir /opt/docling-models \
+ && chmod -R a+rX /opt/docling-models
+ENV WORKER_DOCLING_PYTHON_BIN=/opt/docling-venv/bin/python
+ENV DOCLING_ARTIFACTS_PATH=/opt/docling-models
+ENV TORCHDYNAMO_DISABLE=1
+ENV HF_HUB_OFFLINE=1
 ENV PATH="/opt/ocr-venv/bin:${PATH}"
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1

docs/branch-review-records/b97fde4db6c54bbb8d8febcffd266be8c6a46d0467edf4f9225254d87202b5f0.record.md

@@ -0,0 +1 @@
+| 2026-08-19 | claude/docling-worker-shadow-mode-b6fa17 | 7a30ec3f8b17b97aeb7f17efa003f25c3ea6a61c | Packet B4 docling worker shadow mode (PR #2170): worker/shadow-extraction.ts, worker/python/shadow_docling_extract.py, worker/main.ts post-commit shadow call, worker/prerequisites.ts, worker/validate-runtime.ts, src/lib/env.ts B4 envs, Dockerfile.worker docling venv + models, railway.worker.json, docs (HANDOVER S7 row, worker runbook, ingestion state machine) | ingestion-worker-reviewer: approve-with-nits (docling_version regex end-anchored in this head; post-commit reclaim window disclosed in runbook). Shadow runs only after commitDocumentIndexGeneration, aggregate numbers-only record via existing metadata merge, no chunk/embedding/index/table-fact/document_index_quality writes, fail-open, bounded 120s/40 pages/1 process, rollback WORKER_DOCUMENT_EXTRACTOR_MODE=legacy; Gate B caveats carried in the PR body | verify:pr-local heavy plan exit 0 (Test Files 673 passed \| 2 skipped, Tests 7292 passed \| 29 skipped, lint+typecheck+build green); focused vitest 9 files 117/117; python unittest 7/7; tsc exit 0; check:production-readiness schema green (only absent local secrets fail); pr-policy offline evaluate ok:true; Docker build not run locally (CI contract) |

docs/ingestion-state-machine.md

@@ -107,12 +107,12 @@ Rows are seeded lazily _inside_ the live `claim_indexing_v3_agent_jobs` from
 
 ## 2. Writers
 
-| Writer | Identity | Touches |
-| --------------------------------------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **W1 — local worker** (`worker/main.ts`) | service-role PostgREST | `claim_ingestion_jobs`, `reset_document_index`, all artifact inserts, image-bucket uploads, `commit_document_index_generation`, `complete_ingestion_job` / `fail_or_retry_ingestion_job`, `complete_strict_enrichment_job`, unconditional `documents` updates, inline enrichment (deep-memory) when `WORKER_INLINE_ENRICHMENT`.  |
-| **W2 — edge agent** (`supabase/functions/indexing-v3-agent/index.ts`) | direct Postgres (postgres.js, pool max 4) | `claim_indexing_v3_agent_jobs`, delete+insert cycles on embedding fields / index units / memory cards / sections / labels / summaries, `document_images.metadata` patches, `update_indexing_v3_agent_job_status` (keyed by document_id, no lock-holder check), `complete_strict_enrichment_job`, jsonb-merge patches of `documents.metadata`. |
-| **W3 — API routes** (`src/app/api/...`) | service-role PostgREST | upload; reindex single/bulk (fence-stamped queue-state write + job insert); **reindex `mode:'enrichment'`** (runs deep-memory/enrichment in-route with **no job row and no fence**); retry (guarded job reset + **unguarded document status write**); delete single (guard → enumerate → ledger → cascade → storage remove); rename / bulk metadata edit (read-modify-write of `documents.metadata`). |
-| **W4 — ops** (scripts + SQL) | service-role | `scripts/recover-ingestion-queue.ts` + `scripts/reindex.ts` (supersede/retry plans, `reset_document_index`, attempt_count=0 re-pends), `cleanup_abandoned_document_index_generations`, `scripts/cleanup-storage.ts` (ledger janitor), cron `invoke_indexing_v3_agent`. |
+| Writer | Identity | Touches |
+| --------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **W1 — local worker** (`worker/main.ts`) | service-role PostgREST | `claim_ingestion_jobs`, `reset_document_index`, all artifact inserts, image-bucket uploads, `commit_document_index_generation`, `complete_ingestion_job` / `fail_or_retry_ingestion_job`, `complete_strict_enrichment_job`, unconditional `documents` updates, inline enrichment (deep-memory) when `WORKER_INLINE_ENRICHMENT`. Packet B4 (`WORKER_DOCUMENT_EXTRACTOR_MODE=shadow`, default `legacy`) adds **no writer and no transition**: docling runs after T5 commit and its aggregate `shadow_extraction` key rides the existing T6 metadata merge (`worker/shadow-extraction.ts`). |
+| **W2 — edge agent** (`supabase/functions/indexing-v3-agent/index.ts`) | direct Postgres (postgres.js, pool max 4) | `claim_indexing_v3_agent_jobs`, delete+insert cycles on embedding fields / index units / memory cards / sections / labels / summaries, `document_images.metadata` patches, `update_indexing_v3_agent_job_status` (keyed by document_id, no lock-holder check), `complete_strict_enrichment_job`, jsonb-merge patches of `documents.metadata`. |
+| **W3 — API routes** (`src/app/api/...`) | service-role PostgREST | upload; reindex single/bulk (fence-stamped queue-state write + job insert); **reindex `mode:'enrichment'`** (runs deep-memory/enrichment in-route with **no job row and no fence**); retry (guarded job reset + **unguarded document status write**); delete single (guard → enumerate → ledger → cascade → storage remove); rename / bulk metadata edit (read-modify-write of `documents.metadata`). |
+| **W4 — ops** (scripts + SQL) | service-role | `scripts/recover-ingestion-queue.ts` + `scripts/reindex.ts` (supersede/retry plans, `reset_document_index`, attempt_count=0 re-pends), `cleanup_abandoned_document_index_generations`, `scripts/cleanup-storage.ts` (ledger janitor), cron `invoke_indexing_v3_agent`. |
 
 ## 3. Legal composite states
 

docs/outstanding-issues-inbox/11e6ac01-c68c-4b5a-ba69-c1380aaabff6.json

@@ -0,0 +1,11 @@
+{
+ "version": 2,
+ "id": "11e6ac01-c68c-4b5a-ba69-c1380aaabff6",
+ "createdOn": "2026-08-19",
+ "action": "done",
+ "payload": {
+ "id": "#9DGA6R",
+ "outcome": "Resolved 2026-08-19 by PR #2170 (branch claude/docling-worker-shadow-mode-b6fa17): typed WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow (default legacy) + WORKER_SHADOW_EXTRACTION_COHORT_PERCENT (1-5, owner-approved 2) + WORKER_DOCLING_PYTHON_BIN; docling runs only after commitDocumentIndexGeneration on an index-quality-selected PDF cohort (tables / OCR / layout proxy), aggregate numbers-only record in documents.metadata.shadow_extraction via the existing metadata merge, no chunk/embedding/index/table-fact/document_index_quality writes, bounded 120 s / 40 pages / one process, fail-open (lost lease swallowed); docling venv + models provisioned in Dockerfile.worker from the Gate B lab lock (CI container build green); rollback WORKER_DOCUMENT_EXTRACTOR_MODE=legacy. Evidence: verify:pr-local heavy plan exit 0 (7292 tests), ingestion-worker-reviewer approve-with-nits (fixed). Both Gate B caveats carried in the PR body; enabling shadow in production is an operator Railway-variable step gated by docs/worker-deploy-runbook.md preconditions.",
+ "baseRowFingerprint": "f32eda79effa39fc7fa81d99bfe744b5204aeb857b98cfd02ec33b1017f39f03"
+ }
+}

docs/outstanding-issues-inbox/dc18b947-7869-4756-9470-d70469749bbd.json

@@ -0,0 +1,10 @@
+{
+ "version": 2,
+ "id": "dc18b947-7869-4756-9470-d70469749bbd",
+ "createdOn": "2026-08-19",
+ "action": "cancel",
+ "payload": {
+ "requestId": "abc21f52-0d39-490d-9676-d106b9af4302",
+ "reason": "Resolved 2026-08-19 by PR #2170 (packet B4, branch claude/docling-worker-shadow-mode-b6fa17): typed WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow (default legacy) + WORKER_SHADOW_EXTRACTION_COHORT_PERCENT (1-5, owner-approved 2) + WORKER_DOCLING_PYTHON_BIN; docling runs only after commitDocumentIndexGeneration on an index-quality-selected PDF cohort (tables/OCR/layout proxy), aggregate numbers-only record in documents.metadata.shadow_extraction via the existing metadata merge, no chunk/embedding/index/table-fact/document_index_quality writes, bounded 120 s / 40 pages / one process, fail-open; docling venv + models provisioned in Dockerfile.worker from the Gate B lab lock; rollback WORKER_DOCUMENT_EXTRACTOR_MODE=legacy. Cancelled rather than done because this add was still unreconciled when the work landed (issues:done needs a canonical row). Both Gate B caveats carried in the PR body."
+ }
+}