[P2] Agent platform: productionize DBOS durability + crash-safe streaming upload

[AndresL230]

Part of the Agent migration epic #152 (platform hardening). Owner: Jack.

Goal

Productionize DBOS durable execution (ADR 0011 shipped it off + unvalidated) and make the upload pipeline's side-effects crash-safe on the path users actually hit — the streaming route.

Background (ADR 0011)

• services/durable.py ships @durable_workflow/@durable_step that no-op unless DBOS_ENABLED=true. @durable_workflow wraps agents.document.process_document, and the workers are wrapped with @durable_step.
• Only /upload/sync gets durability today. The streaming POST /api/documents/upload calls agents inline (to emit SSE) and is intentionally not wrapped — ADR 0011 argues the SSE connection is per-process, so a crash loses the stream and the right semantic is "re-run on client retry, deduped by X-Request-ID."
• ADR 0011's deferred items — test coverage, production validation, resume monitoring — never shipped.

Scope

• Test coverage for durable workflow/step resume (crash mid-_run_workers, resume at last completed step).
• Production validation procedure from ADR 0011 (provision DBOS Postgres, dbos migrate, flip flags, confirm is_durable()), documented and exercised once.
• Streaming-path crash-safety: keep the stream non-durable (per ADR 0011) but ensure the persistence/side-effects (persist doc, graph backstop, syllabus save) are idempotent and resumable on the dedup'd retry — reconcile with audit [P2] Document pipeline robustness: post-result fallback double-fire, empty-OCR persistence, blocking IO, wrong size message #132 (post-result fallback double-fire) so a crash after the result event can't double-fire or strand the doc.
• Resume monitoring wired into observability ([P2] Observability: activate Logfire for ops/error/LLM tracing #119).

Files

backend/services/durable.py, backend/agents/document.py, backend/routes/documents.py (streaming path), requirements-durable.txt.

Acceptance

• Resume test passes with DBOS_ENABLED=true; pipeline behaves identically with it off.
• A crash after the streaming result event leaves exactly one consistent document (no double-fire), verified against the [P2] Document pipeline robustness: post-result fallback double-fire, empty-OCR persistence, blocking IO, wrong size message #132 scenario.
• Validation + monitoring steps documented (update ADR 0011 status).

[AndresL230]

Closed via #474 (merged as 9edfcf5). Evidence against the acceptance criteria:

• Resume test passes with DBOS_ENABLED=true: three opt-in tests (RUN_DBOS_RESUME=1, tests/test_dbos_resume.py) exercised against real dbos==2.28.0 + a real Postgres — a minimal-workflow crash/resume via launch-triggered recovery, a process_document parity run under DBOS, and the real-pipeline proof: crash mid-_run_workers (summary step kills the process), client-style retry with the same workflow id, classify step ran exactly once across both phases (resume at last completed step). Pipeline identical with the flag off: hermetic suite 1480 passed / 32 skipped, plus a full local e2e cycle (29/29 Playwright journeys, oracles exit 0) with the new lifespan wiring in place.
• Crash after the streaming result event leaves exactly one consistent document: the streaming replay test now pins zero agent re-runs AND zero new inserts on an X-Request-ID retry (the [P2] Document pipeline robustness: post-result fallback double-fire, empty-OCR persistence, blocking IO, wrong size message #132 scenario, on top of the fix(backend): close the #136 contract-audit tail + #340 + #72 #464 exactly-one-result guard). Streaming stays intentionally non-durable per ADR 0011's asymmetry.
• Validation + monitoring documented: ADR 0011 → "accepted (shipped + validated, default off)" with the corrected activation procedure (DBOS.launch() runs system-DB migrations itself and auto-recovers PENDING workflows — the old dbos migrate step was stale), resume monitoring (startup INFO line captured by Logfire per [P2] Observability: activate Logfire for ops/error/LLM tracing #119, DBOS's own recovery logs, dbos.workflow_status SQL), and requirements-durable.txt created (dbos>=2.28,<3, outside requirements.txt/lock — verified no CI/Docker path installs it).

Beyond the issue's letter, the review round made durability real for the product: production now pins the workflow id to doc:{user_id}:{request_id} on /upload/sync, so a retry attaches to the same workflow instead of re-running every agent, and the graph merge is a checkpointed durable step. main.py finally constructs + launches DBOS in the lifespan (fail-loud on a misconfigured explicit opt-in) — the shim previously trusted an init that never existed.

Prod flag-flip remains an operator decision; the procedure is documented and was exercised locally, not run against production.