From e048dacbafdbfa44eea1314fa043c46453912926 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 21 Aug 2026 18:57:45 +0800 Subject: [PATCH 1/5] docs(worker): answer two of the four B4 operator unknowns from measured evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both were read-only lookups against the live Railway project; no mutation was made. 1. Railway variable-change behaviour, and it was a latent safety trap. Railway's documentation states that containers read environment variables only at startup, so a variable change never restarts a running container by itself and the new value exists only inside the new deployment. The worker parses WORKER_DOCUMENT_EXTRACTOR_MODE once at process start and matches that exactly. Setting the variable is therefore NOT by itself the rollback. Sections 3.5 and 3.7 now state it as two steps — set the variable, then deploy — and say plainly that stopping after the first leaves docling running. 2. Worker memory limit and observed peak. Read from the production worker service over a 7-day window (10,081 samples): limit 24 GB, peak 0.566 GB, average 0.139 GB. That is ~23.4 GB of headroom against the ~1.5 GiB docling needs, so the precondition is satisfied roughly fifteen times over. Section 3.2 records the numbers and reduces the check to "re-run only if the plan or limits change". It also flags a trap the numbers invite: the service reports a 24 vCPU limit while Gate B measured 9-19 s/doc on 2 CPUs. Section 3.4 must NOT be assumed to scale down, because docling runs eager (TORCHDYNAMO_DISABLE=1) and single-process, so extra cores may buy little. The two remaining unknowns need an owner decision rather than investigation and are recorded against #1VFSYF: ratifying the proposed ">10 % of cohort runs timing out" rollback trigger, and whether to build a reader for documents.metadata.shadow_extraction now or when shadow mode is first enabled. Docs only. No code, worker, schema, or provider change; no default changed; shadow mode is still enabled nowhere. Co-Authored-By: Claude Opus 5 --- .../29bf4571-4bc3-42cf-8a4a-7edab14ee031.json | 12 ++++ docs/worker-deploy-runbook.md | 68 +++++++++++++------ 2 files changed, 61 insertions(+), 19 deletions(-) create mode 100644 docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json diff --git a/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json b/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json new file mode 100644 index 0000000000..ff5b92000d --- /dev/null +++ b/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "29bf4571-4bc3-42cf-8a4a-7edab14ee031", + "createdOn": "2026-08-21", + "action": "update", + "payload": { + "id": "#1VFSYF", + "detail": "TWO OF FOUR ANSWERED 2026-08-21 and folded into docs/worker-deploy-runbook.md section 3. (1) RAILWAY VARIABLE-CHANGE BEHAVIOUR - ANSWERED, and it was a latent safety trap: Railway's docs state that containers read environment variables only at startup, so a variable change never restarts a running container by itself and the new value exists only inside the new deployment. The worker parses WORKER_DOCUMENT_EXTRACTOR_MODE once at process start, so setting the variable is NOT by itself the rollback. Sections 3.5 and 3.7 now state the rollback as two steps (set the variable, then deploy) and warn that stopping after the first leaves docling running. (2) MEMORY LIMIT AND PEAK - ANSWERED by a read-only Railway metrics query on the production worker service over a 7-day window, 10081 samples: memory limit 24 GB, peak 0.566 GB, average 0.139 GB, so roughly 23.4 GB of headroom against the ~1.5 GiB docling needs. The precondition is met with about a fifteenfold margin; section 3.2 now records the numbers and says to re-check only if the plan or resource limits change. It also flags that the service reports a 24 vCPU limit while Gate B measured 9-19 s/doc on 2 CPUs, and that the section 3.4 cost model should NOT be assumed to scale down, because docling runs eager and single-process. STILL OPEN, both needing an owner decision rather than investigation: (3) the proposed rollback trigger of more than 10 percent of cohort runs timing out is an unratified operating rule, not a measurement, and nothing in the repository fixes the number; it needs ratifying or replacing, ideally once real wall_ms values exist. (4) there is still no script that reads or aggregates documents.metadata.shadow_extraction, so the first-24-hours watch remains the hand-run SQL query in section 3.6. Recommendation recorded against (4): build the reader when shadow mode is first enabled rather than now, because shadow mode has never run so the table holds zero rows and the tooling cannot be exercised end to end against real data.", + "source": "docs/worker-deploy-runbook.md sections 3.2, 3.5 and 3.7; read-only Railway metrics on service worker (project Database 5deaad0b) 2026-08-21", + "baseRowFingerprint": "05b7833029810d2bae41f47ab1e7ed7f2a23ad3aaad25ae2d1c67db31c168d7a" + } +} diff --git a/docs/worker-deploy-runbook.md b/docs/worker-deploy-runbook.md index 56df5f7b47..955eb6fb4c 100644 --- a/docs/worker-deploy-runbook.md +++ b/docs/worker-deploy-runbook.md @@ -257,17 +257,34 @@ duration of the docling run (up to 120 s) before the final metadata merge and and sets `WORKER_DOCLING_PYTHON_BIN`, `DOCLING_ARTIFACTS_PATH`, `TORCHDYNAMO_DISABLE=1` and `HF_HUB_OFFLINE=1`. On an older image that variable is unset and **every** cohort document records `runtime_unavailable` while legacy indexing continues normally. -2. **Memory headroom of roughly 1.5 GiB above the worker's current peak.** Gate B measured - docling peak RSS **1,504,313,344 B (~1.40 GiB)** against the lab's 6 GiB cap. Docling runs - as a _separate_ process while the Node worker still holds the PDF buffer, so what matters - is headroom above today's peak, not a total figure. **How to check:** Railway → project - `Database` → `worker` service → **Metrics** → memory across a busy ingest window, and the - service's memory limit under the current plan. Neither the limit nor the observed peak is - recorded in this repository; reading them is an operator dashboard action. This is the - precondition that matters most, because a container OOM kill during the docling window is - the one failure the fail-open code cannot catch — the index is already committed, but the - job would sit `processing` until the 45-minute stale reclaim - (`WORKER_STALE_AFTER_MINUTES`) and burn an attempt. +2. **Memory headroom of roughly 1.5 GiB above the worker's current peak — measured 2026-08-21 + and comfortably satisfied.** Gate B measured docling peak RSS **1,504,313,344 B (~1.40 GiB)** + against the lab's 6 GiB cap. Docling runs as a _separate_ process while the Node worker still + holds the PDF buffer, so what matters is headroom above today's peak, not a total figure. + + Read from the Railway `worker` service (production), 7-day window, 10,081 samples: + + | Measure | Value | + | -------------------- | -------------------- | + | Memory limit | **24 GB** | + | Memory peak (7 days) | **0.566 GB** | + | Memory average | 0.139 GB | + | Headroom above peak | **~23.4 GB** | + | vCPU limit | 24 (peak usage 0.40) | + + That is roughly fifteen times the ~1.5 GiB the docling process needs, so this precondition + is met with a very wide margin. Re-check it only if the service plan or resource limits + change — it is the precondition that matters most, because a container OOM kill during the + docling window is the one failure the fail-open code cannot catch: the index is already + committed, but the job would sit `processing` until the 45-minute stale reclaim + (`WORKER_STALE_AFTER_MINUTES`) and burn an attempt. **How to re-check:** Railway → project + `Database` → `worker` service → **Metrics**, memory across a busy ingest window. + + One caveat on the cost model in §3.4: Gate B measured 9–19 s/doc on **2 CPUs**, and this + service reports a 24 vCPU limit. Do not assume the wall-clock will scale down proportionally + — docling runs eager (`TORCHDYNAMO_DISABLE=1`) and single-process, so extra cores may buy + little. Treat §3.4 as unchanged until real `wall_ms` values say otherwise. + 3. **Confirm `WORKER_CONCURRENCY` is 1** (the shipped default). At most one docling process runs per worker regardless of concurrency; with concurrency above 1 the extra cohort hits record `skipped_concurrent` instead of measuring anything. @@ -359,7 +376,10 @@ outside the Gate B authorisation, and they are the cap this section exists to st Leave `WORKER_SHADOW_EXTRACTION_COHORT_PERCENT` unset (it defaults to `2`) or set it to `2` explicitly. Leave `WORKER_DOCLING_PYTHON_BIN` alone — the image sets it. -3. Redeploy or restart the service so the new environment is parsed. +3. **Deploy.** A variable change alone does not restart a running container on Railway, and + the worker reads this mode once at process start — so the setting does nothing until a new + deployment starts. Use Railway's apply/redeploy action. The same two-step rule applies to + the rollback (§3.7). 4. Confirm the startup line in the worker logs: ```text @@ -449,13 +469,23 @@ WORKER_DOCUMENT_EXTRACTOR_MODE=legacy Deleting the variable entirely is equivalent — the schema default is `legacy`. -**Time to take effect.** The mode is parsed once at process start, so it applies from the next -worker restart. The container sets `STOPSIGNAL SIGTERM` and drains its active batch before -exiting 0 (§1), so the worst-case worker-side delay is the current batch plus at most one -120-second docling window. Railway's own behaviour on a variable change — restart versus a -full rebuild — is **not** recorded in this repository; confirm it in the dashboard, and budget -for a rebuild if that is what it does. No migration, no reindex, and no code or image change -is required. +**Time to take effect — and the trap in it.** Setting the variable is **not** by itself the +rollback. Railway's documentation is explicit: _"Containers read environment variables only at +startup, so a variable change never restarts a running container by itself; the new value only +exists inside the new deployment"_ +([rotate-credentials-zero-downtime](https://docs.railway.com/guides/rotate-credentials-zero-downtime#how-variable-changes-apply-on-railway), +read 2026-08-21). The worker parses `WORKER_DOCUMENT_EXTRACTOR_MODE` once at process start and +matches that behaviour exactly. + +So a rollback is **two** steps, and stopping after the first leaves docling still running: + +1. Set `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy` on the `worker` service. +2. **Deploy.** Use Railway's apply/redeploy action so a new deployment starts with the new value. + +Once the new deployment starts, the outgoing container gets `STOPSIGNAL SIGTERM` and drains its +active batch before exiting 0 (§1), so the worst-case worker-side delay is the current batch plus +at most one 120-second docling window. No migration, no reindex, and no code or image change is +required. Existing `shadow_extraction` records stay on their document rows as inert history. Nothing reads them, and nothing in the repository clears them. From f742abf45a4ade215696da3907d4cf40920a304c Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 21 Aug 2026 19:39:39 +0800 Subject: [PATCH 2/5] docs(worker): revalidate headroom and clarify rollback --- docs/worker-deploy-runbook.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/worker-deploy-runbook.md b/docs/worker-deploy-runbook.md index 955eb6fb4c..094cc9b8d1 100644 --- a/docs/worker-deploy-runbook.md +++ b/docs/worker-deploy-runbook.md @@ -186,7 +186,7 @@ the client publishable key (build-time, app bundle only) or `WORKER_DOCUMENT_EXTRACTOR_MODE`, `WORKER_SHADOW_EXTRACTION_COHORT_PERCENT` and `WORKER_DOCLING_PYTHON_BIN` are documented in full in [§3](#3-docling-shadow-extraction-packet-b4--default-off): -preconditions, safe values, the cost cap, what to watch, and the one-step rollback. The +preconditions, safe values, the cost cap, what to watch, and the two-step rollback. The shipped default is `legacy` and nothing in this run recipe enables it. --- @@ -272,13 +272,15 @@ duration of the docling run (up to 120 s) before the final metadata merge and | Headroom above peak | **~23.4 GB** | | vCPU limit | 24 (peak usage 0.40) | - That is roughly fifteen times the ~1.5 GiB the docling process needs, so this precondition - is met with a very wide margin. Re-check it only if the service plan or resource limits - change — it is the precondition that matters most, because a container OOM kill during the - docling window is the one failure the fail-open code cannot catch: the index is already - committed, but the job would sit `processing` until the 45-minute stale reclaim - (`WORKER_STALE_AFTER_MINUTES`) and burn an attempt. **How to re-check:** Railway → project - `Database` → `worker` service → **Metrics**, memory across a busy ingest window. + That is roughly fifteen times the ~1.5 GiB the docling process needs, so the 2026-08-21 + measurement is a baseline, not a standing waiver. **Re-check memory headroom across a busy + ingest window immediately before every shadow enablement, and again after any worker image, + workload, `WORKER_CONCURRENCY`, service plan, or resource-limit change.** It is the precondition + that matters most, because a container OOM kill during the docling window is the one failure + the fail-open code cannot catch: the index is already committed, but the job would sit + `processing` until the 45-minute stale reclaim (`WORKER_STALE_AFTER_MINUTES`) and burn an + attempt. **How to re-check:** Railway → project `Database` → `worker` service → **Metrics**, + memory across a busy ingest window. One caveat on the cost model in §3.4: Gate B measured 9–19 s/doc on **2 CPUs**, and this service reports a 24 vCPU limit. Do not assume the wall-clock will scale down proportionally @@ -442,7 +444,7 @@ a full reindex is run (§3.4). Zero records on a quiet queue is normal. `wall_ms` and `peak_rss_bytes` are the two fields that decide whether the cost model in §3.4 survived contact with the real corpus. Read them first. -### 3.7 Rollback triggers and the one-step rollback +### 3.7 Rollback triggers and the two-step rollback **Roll back immediately, diagnose afterwards, if any of these occur.** @@ -461,7 +463,7 @@ survived contact with the real corpus. Read them first. number; agree it with the owner or replace it. 6. `peak_rss_bytes` sustained above the headroom confirmed in §3.2. -**The one-step rollback.** On the Railway `worker` service, set: +**The two-step rollback.** On the Railway `worker` service, set: ```text WORKER_DOCUMENT_EXTRACTOR_MODE=legacy From 6a698039d906e2d37c2c7a3f376e40a5a15a1bda Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 21 Aug 2026 19:39:41 +0800 Subject: [PATCH 3/5] fix(worker): describe complete shadow rollback --- worker/main.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/worker/main.ts b/worker/main.ts index 2b83ef2e18..3daceb91d1 100644 --- a/worker/main.ts +++ b/worker/main.ts @@ -2067,7 +2067,7 @@ async function main() { if (env.WORKER_DOCUMENT_EXTRACTOR_MODE === "shadow") { console.log( `Docling shadow extraction enabled (packet B4): cohort ${env.WORKER_SHADOW_EXTRACTION_COHORT_PERCENT}% of ` + - "index-quality-selected PDFs after legacy commit; aggregate metadata only. Rollback: WORKER_DOCUMENT_EXTRACTOR_MODE=legacy.", + "index-quality-selected PDFs after legacy commit; aggregate metadata only. Rollback: set WORKER_DOCUMENT_EXTRACTOR_MODE=legacy, then deploy.", ); const doclingPrereqs = await checkDoclingShadowPrerequisites(); if (!doclingPrereqs.ok) { From fa36b5927539ae3adc0415a330a9fbaca3fe67fb Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 21 Aug 2026 20:01:09 +0800 Subject: [PATCH 4/5] docs(worker): align rollback examples with deploy requirement --- docs/worker-deploy-runbook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/worker-deploy-runbook.md b/docs/worker-deploy-runbook.md index 094cc9b8d1..6188cc2ba6 100644 --- a/docs/worker-deploy-runbook.md +++ b/docs/worker-deploy-runbook.md @@ -385,7 +385,7 @@ outside the Gate B authorisation, and they are the cap this section exists to st 4. Confirm the startup line in the worker logs: ```text - Docling shadow extraction enabled (packet B4): cohort 2% of index-quality-selected PDFs after legacy commit; aggregate metadata only. Rollback: WORKER_DOCUMENT_EXTRACTOR_MODE=legacy. + Docling shadow extraction enabled (packet B4): cohort 2% of index-quality-selected PDFs after legacy commit; aggregate metadata only. Rollback: set WORKER_DOCUMENT_EXTRACTOR_MODE=legacy, then deploy. ``` A following `Docling shadow prerequisite warning (shadow extraction will fail open): …` @@ -564,7 +564,7 @@ The second Gate B caveat is already load-bearing above: docling's eager-mode lat ## Rollback This section is the **image** rollback. To turn off docling shadow extraction without -changing the image, use the one-step flag rollback in §3.7 instead. +changing the image, use the two-step flag rollback in §3.7 instead. Redeploy the previous image tag. The worker holds no durable local state; all artifact writes are idempotent per generation/chunk-key, and completion is From a72957f7a585481d4850e6d4fd0396d2dbae567c Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 21 Aug 2026 20:01:10 +0800 Subject: [PATCH 5/5] docs(worker): record shadow-mode headroom rechecks --- .../29bf4571-4bc3-42cf-8a4a-7edab14ee031.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json b/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json index ff5b92000d..eb9166ba91 100644 --- a/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json +++ b/docs/outstanding-issues-inbox/29bf4571-4bc3-42cf-8a4a-7edab14ee031.json @@ -5,7 +5,7 @@ "action": "update", "payload": { "id": "#1VFSYF", - "detail": "TWO OF FOUR ANSWERED 2026-08-21 and folded into docs/worker-deploy-runbook.md section 3. (1) RAILWAY VARIABLE-CHANGE BEHAVIOUR - ANSWERED, and it was a latent safety trap: Railway's docs state that containers read environment variables only at startup, so a variable change never restarts a running container by itself and the new value exists only inside the new deployment. The worker parses WORKER_DOCUMENT_EXTRACTOR_MODE once at process start, so setting the variable is NOT by itself the rollback. Sections 3.5 and 3.7 now state the rollback as two steps (set the variable, then deploy) and warn that stopping after the first leaves docling running. (2) MEMORY LIMIT AND PEAK - ANSWERED by a read-only Railway metrics query on the production worker service over a 7-day window, 10081 samples: memory limit 24 GB, peak 0.566 GB, average 0.139 GB, so roughly 23.4 GB of headroom against the ~1.5 GiB docling needs. The precondition is met with about a fifteenfold margin; section 3.2 now records the numbers and says to re-check only if the plan or resource limits change. It also flags that the service reports a 24 vCPU limit while Gate B measured 9-19 s/doc on 2 CPUs, and that the section 3.4 cost model should NOT be assumed to scale down, because docling runs eager and single-process. STILL OPEN, both needing an owner decision rather than investigation: (3) the proposed rollback trigger of more than 10 percent of cohort runs timing out is an unratified operating rule, not a measurement, and nothing in the repository fixes the number; it needs ratifying or replacing, ideally once real wall_ms values exist. (4) there is still no script that reads or aggregates documents.metadata.shadow_extraction, so the first-24-hours watch remains the hand-run SQL query in section 3.6. Recommendation recorded against (4): build the reader when shadow mode is first enabled rather than now, because shadow mode has never run so the table holds zero rows and the tooling cannot be exercised end to end against real data.", + "detail": "TWO OF FOUR ANSWERED 2026-08-21 and folded into docs/worker-deploy-runbook.md section 3. (1) RAILWAY VARIABLE-CHANGE BEHAVIOUR - ANSWERED, and it was a latent safety trap: Railway's docs state that containers read environment variables only at startup, so a variable change never restarts a running container by itself and the new value exists only inside the new deployment. The worker parses WORKER_DOCUMENT_EXTRACTOR_MODE once at process start, so setting the variable is NOT by itself the rollback. Sections 3.5 and 3.7 now state the rollback as two steps (set the variable, then deploy) and warn that stopping after the first leaves docling running. (2) MEMORY LIMIT AND PEAK - ANSWERED by a read-only Railway metrics query on the production worker service over a 7-day window, 10081 samples: memory limit 24 GB, peak 0.566 GB, average 0.139 GB, so roughly 23.4 GB of headroom against the ~1.5 GiB docling needs. The precondition is met with about a fifteenfold margin; section 3.2 now records the numbers as a baseline and requires a busy-window memory-headroom re-check immediately before every shadow enablement and again after any worker image, workload, WORKER_CONCURRENCY, service-plan, or resource-limit change. It also flags that the service reports a 24 vCPU limit while Gate B measured 9-19 s/doc on 2 CPUs, and that the section 3.4 cost model should NOT be assumed to scale down, because docling runs eager and single-process. STILL OPEN, both needing an owner decision rather than investigation: (3) the proposed rollback trigger of more than 10 percent of cohort runs timing out is an unratified operating rule, not a measurement, and nothing in the repository fixes the number; it needs ratifying or replacing, ideally once real wall_ms values exist. (4) there is still no script that reads or aggregates documents.metadata.shadow_extraction, so the first-24-hours watch remains the hand-run SQL query in section 3.6. Recommendation recorded against (4): build the reader when shadow mode is first enabled rather than now, because shadow mode has never run so the table holds zero rows and the tooling cannot be exercised end to end against real data.", "source": "docs/worker-deploy-runbook.md sections 3.2, 3.5 and 3.7; read-only Railway metrics on service worker (project Database 5deaad0b) 2026-08-21", "baseRowFingerprint": "05b7833029810d2bae41f47ab1e7ed7f2a23ad3aaad25ae2d1c67db31c168d7a" }