RAG hardening: silent-empty observability, poisoned embedding:None rows, durability inversion

[AndresL230]

Split from the #323 audit (verified @ d1d724f). Three related operational hazards in the delivered document-RAG path:

1. Silent-empty everywhere: retrieve_chunks catches ALL exceptions → [] via a bare print (rag_service.py:137-139) — invisible to the app-logging path; an ungrounded tutor turn is indistinguishable from a grounded one at every layer. → logger + a metric/event.
2. Poisoned rows: a failed embed batch is logged and then upserted anyway with embedding: None (rag_service.py:192,213) — permanently unretrievable, and nothing ever re-indexes them.
3. Durability inversion: DBOS wraps /upload/sync (documents.py:615) which never indexes; the indexing route (streaming /upload) is non-durable by design (ADR 0011:179-181) — a crash between _persist_document and the post-roll task loses chunks permanently behind a healthy-looking documents row. At minimum document the gap; ideally give indexing a retry/backfill trigger (backfill_document_chunks.py exists but nothing invokes it).

[AndresL230]

Items 1 and 3 addressed in #501 (c03c983). Item 2 was already fixed on main.

Re-verified against main before starting

#	Claim	State found
1	Silent-empty: bare print at rag_service.py:137-139	live → fixed
2	Failed batch upserted anyway with embedding: None	already fixed — rag_service.py filters embedding is not None before the upsert, with a comment citing this issue
3	Durability inversion	live → documented

Item 1

rag_service reported everything through bare print, so no failure reached the app-logging path and no rollup could count one. That matters most for retrieve_chunks, which degrades to [] — and [] is also "nothing relevant matched", so an ungrounded turn was indistinguishable from a grounded one at every layer above.

Module logger, all three print sites moved onto it, plus rag.retrieval_failed / rag.chunks_dropped events so the degrade rates are countable.

The non-obvious part: the naive version of this is actively harmful. _require_real_mode() raises _EmbeddingDisabled whenever SAPLING_MODEL_MODE != "real" — every function-mode run — so a blanket except Exception emits an error event on every e2e tutor turn and every e2e upload. The signal would have been worthless the day it shipped. _EmbeddingDisabled is now caught separately (INFO, no event); only genuine failures warn and count.

Code review added two more: the new event types weren't registered in the pinned #117 taxonomy (log_event never enforces membership, so nothing failed — they were just invisible), and one new test depended on ambient SAPLING_MODEL_MODE and passed vacuously with function mode exported. Both fixed.

Item 3 — documented, not fixed

In docs/architecture.md under Known sharp edges: DBOS wraps /upload/sync, which never indexes; the route that does index is the streaming /upload, non-durable by design (ADR 0011). A crash between _persist_document and the post-roll indexing task leaves a healthy-looking documents row whose content is absent from retrieval permanently, and scripts/backfill_document_chunks.py is invoked by nothing.

I did not wire a retry trigger. Where it runs, what re-drives it, and how it interacts with the streaming route's X-Request-ID replay semantic are real design calls, and this issue asks for documentation as the minimum bar. That remainder is the reason I'm leaving this open rather than closing it — say the word if you'd rather track it as its own issue and close this one.

Gates: backend pytest 1531 passed / 32 skipped, ruff clean, full local e2e cycle green (37/37, oracles 0 findings), CI green.