From bbd8a47947ad62c6a6288606e58c95c4b92910fd Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 11:15:25 +0300 Subject: [PATCH 01/21] docs(roadmap): mark 1.AG Done (PR #37) across the status surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1.AG (media output generation, Phase D) merged to main via PR #37 (2026-06-21), including the final pre-merge review's two HIGH fixes (orphaned-vertex cost addend + crash-in-window run:paused). Flip its status everywhere, per the Roadmap-Done-After-Merge rule: - phase-1-engine-and-llm.md: the 1.AG dependency-table row (◇ → ✅ Done, PR #37), the 1.m6 summary row (1.AG ✅), and the narrative entry's Status line. - current.md: 1.AG ✅ Done (PR #37); "only 1.AH remains" on the 1.m6 sub-spine. - deferred-tasks.md: tick [x] the A5 async-media-job ADR + engine-loop obligation (PR #37). - CLAUDE.md: both Status paragraphs note 1.AG Done; remaining Phase-1 work is 1.AH. - README.md: 1.AG ✅; only 1.AH remains. - llm-provider-seam.md: clarify the streaming media triad — non-streaming output landed at 1.AG (generate() + generative endpoint, ADR-0046); the streaming triad is deferred to 1.AH (ADR-0046 §4). Docs-only; format:check clean. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 4 ++-- README.md | 2 +- docs/reference/shared-core/llm-provider-seam.md | 6 ++++-- docs/roadmap/current.md | 4 ++-- docs/roadmap/deferred-tasks.md | 5 ++--- docs/roadmap/phases/phase-1-engine-and-llm.md | 15 ++++++++------- 6 files changed, 19 insertions(+), 17 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fe12df93..9d892808 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ A run executes in one of **three execution modes** behind the one `LLMProvider` engine is identical across all three. See [ADR-0012](docs/decisions/0012-managed-inference-dual-mode.md) to [ADR-0015](docs/decisions/0015-managed-mode-data-handling-and-compliance.md) and [docs/architecture/managed-inference.md](docs/architecture/managed-inference.md). -**Status:** Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2; and the **pre-egress budget governor (1.AC, [ADR-0028](docs/decisions/0028-workflow-resource-governance.md)) + the `AgentSession` agent-first entry point (1.V, [ADR-0024](docs/decisions/0024-agent-first-entry-point-agentsession.md)) landed together (PR #26, 2026-06-16)** — 1.AC was the last 1.m4 component, so **1.m4 is complete** (the full engine stack: node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor), and 1.V opens the Lane-C agent-first sub-spine (1.m5); then the **end-to-end Node harness (1.U) landed (PR #27, 2026-06-16), reaching 🎯 M2** — the engine runs end-to-end (live streaming + per-node-boundary checkpointing + cross-process resume + node retry + provider failover, gap-free), **completing the Phase-1 engine critical path**. The remaining Phase-1 work is additive and off the critical path (Lane C: the **`session:*` namespace (1.W) landed (PR #28, 2026-06-17)** — the `SessionEventSink`→`RunEventBus` adapter + per-session `sequenceNumber`, the `SessionHandle`, and the combined `RunOrSessionEventSchema` gate — and **session persistence (1.X) landed (PR #29, 2026-06-17)** — the `agent_sessions`/`session_messages` tables + migration, `SessionMessageSchema`/`AgentSessionSchema`, and the `SessionStore` + domain↔row mappers (data-layer only); then **session checkpoint/resume (1.Y) + export-to-workflow (1.Z) landed (PR #30, 2026-06-17)** — `reconstructSessionState`/`AgentSession.resume` (reload-not-replay; preload the text-only transcript, re-seed turnCount/cost, no `session:started` re-emit) + the `serializeWorkflow`/`sessionToWorkflow` pair (one agent node per completed turn, transcript in `metadata`, secret/signature exclusion structural) — leaving only the **1.AA** chat-regression harness ‖ the 1.m6 multimodal sub-spine, whose first step — **media-input adapters + the shared SSRF policy primitive (1.AE) — landed (PR #32, 2026-06-18)** (base64 image/audio across the three adapters behind per-modality `assertMediaCapabilities`, and the one `@relavium/shared` SSRF policy primitive reused by the provider-`baseURL` + `http_request` callers; the SSRF *mechanism* half + per-modality FallbackChain gating moved into **1.AF**). **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20):** the `MediaStore`/`deInlineMedia` choke point (the active I3 enforcement), per-modality capability gating (`mediaSupportReason`/`requestSupportReason` — replacing coarse vision), the `media_objects`/`media_references` tables, and ADR-0042/0043/0044 (P1+P2, PR #33); the binary media-egress + the **SSRF mechanism half** (D9 — the `MediaUrlFetch` re-host hook + the SSRF-validated `fetchMediaBytes` wired at the choke point), the `FallbackChain` resolve-before-egress + the **byte-free re-materialization sidecar** (D8/D7), and the byte-delivery **`Range` gate** (D13) (P3 + P4/D13, PR #34); `read_media` + the scope-set authz (D12), the terminal sweep + grace-window GC (D11), `save_to` (D16), the `output_modalities` load-check (D15), and the per-modality media cost governor (D17) (P4 remainder, PR #35); plus a **13-dimension end-to-end review + an external Sonar/CodeRabbit pass** (PR #36 — H1 `agent:tool_call.toolInput` byte-redaction, H2 the unified exact-membership output-combination gate, the GC grace-basis fix, the `nodeMediaEgressDeps` egress-mechanism test). Each phase passed a dedicated adversarial security-review (0 blockers/highs). The deferred **host-wiring** half (D12 `MediaReadAccess` + session-scope population, the D15 loader, the D17/`resolveForEgress` config) + the keychain no-raw-key IPC test are owned by **1.AH** (recorded in deferred-tasks.md). The remaining Phase-1 work is **1.AG/1.AH**; **Phase 2 (CLI, M3) is unblocked**. +**Status:** Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2; and the **pre-egress budget governor (1.AC, [ADR-0028](docs/decisions/0028-workflow-resource-governance.md)) + the `AgentSession` agent-first entry point (1.V, [ADR-0024](docs/decisions/0024-agent-first-entry-point-agentsession.md)) landed together (PR #26, 2026-06-16)** — 1.AC was the last 1.m4 component, so **1.m4 is complete** (the full engine stack: node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor), and 1.V opens the Lane-C agent-first sub-spine (1.m5); then the **end-to-end Node harness (1.U) landed (PR #27, 2026-06-16), reaching 🎯 M2** — the engine runs end-to-end (live streaming + per-node-boundary checkpointing + cross-process resume + node retry + provider failover, gap-free), **completing the Phase-1 engine critical path**. The remaining Phase-1 work is additive and off the critical path (Lane C: the **`session:*` namespace (1.W) landed (PR #28, 2026-06-17)** — the `SessionEventSink`→`RunEventBus` adapter + per-session `sequenceNumber`, the `SessionHandle`, and the combined `RunOrSessionEventSchema` gate — and **session persistence (1.X) landed (PR #29, 2026-06-17)** — the `agent_sessions`/`session_messages` tables + migration, `SessionMessageSchema`/`AgentSessionSchema`, and the `SessionStore` + domain↔row mappers (data-layer only); then **session checkpoint/resume (1.Y) + export-to-workflow (1.Z) landed (PR #30, 2026-06-17)** — `reconstructSessionState`/`AgentSession.resume` (reload-not-replay; preload the text-only transcript, re-seed turnCount/cost, no `session:started` re-emit) + the `serializeWorkflow`/`sessionToWorkflow` pair (one agent node per completed turn, transcript in `metadata`, secret/signature exclusion structural) — leaving only the **1.AA** chat-regression harness ‖ the 1.m6 multimodal sub-spine, whose first step — **media-input adapters + the shared SSRF policy primitive (1.AE) — landed (PR #32, 2026-06-18)** (base64 image/audio across the three adapters behind per-modality `assertMediaCapabilities`, and the one `@relavium/shared` SSRF policy primitive reused by the provider-`baseURL` + `http_request` callers; the SSRF *mechanism* half + per-modality FallbackChain gating moved into **1.AF**). **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20):** the `MediaStore`/`deInlineMedia` choke point (the active I3 enforcement), per-modality capability gating (`mediaSupportReason`/`requestSupportReason` — replacing coarse vision), the `media_objects`/`media_references` tables, and ADR-0042/0043/0044 (P1+P2, PR #33); the binary media-egress + the **SSRF mechanism half** (D9 — the `MediaUrlFetch` re-host hook + the SSRF-validated `fetchMediaBytes` wired at the choke point), the `FallbackChain` resolve-before-egress + the **byte-free re-materialization sidecar** (D8/D7), and the byte-delivery **`Range` gate** (D13) (P3 + P4/D13, PR #34); `read_media` + the scope-set authz (D12), the terminal sweep + grace-window GC (D11), `save_to` (D16), the `output_modalities` load-check (D15), and the per-modality media cost governor (D17) (P4 remainder, PR #35); plus a **13-dimension end-to-end review + an external Sonar/CodeRabbit pass** (PR #36 — H1 `agent:tool_call.toolInput` byte-redaction, H2 the unified exact-membership output-combination gate, the GC grace-basis fix, the `nodeMediaEgressDeps` egress-mechanism test). Each phase passed a dedicated adversarial security-review (0 blockers/highs). The deferred **host-wiring** half (D12 `MediaReadAccess` + session-scope population, the D15 loader, the D17/`resolveForEgress` config) + the keychain no-raw-key IPC test are owned by **1.AH** (recorded in deferred-tasks.md). **1.AG (media output generation) is ✅ Done (PR #37, 2026-06-21):** inline media-out via `generate()` (Section B, [ADR-0046](docs/decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)), sync `generateMedia` + `media_surface` routing (C), the engine-owned async media-job **poll/checkpoint/resume/cancel LRO** (D, [ADR-0045](docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)), and acceptance + generative conformance (E); the five sections were each dual-reviewed (Opus+Sonnet), then a final Opus-consolidated pre-merge review caught + fixed two HIGH cross-fix defects (the orphaned-vertex cost addend + the crash-in-window `run:paused`). The genuinely-remaining vendor-adapter (Sora/Veo/OpenAI-TTS/Imagen) + host-wiring work is **1.AH**. The remaining Phase-1 work is **1.AH**; **Phase 2 (CLI, M3) is unblocked**. Phase 0 (M0, 2026-06-04) landed the monorepo, strict toolchain + CI, `@relavium/shared` (the full Zod contract set), the no-vendor-type seam fence, and `@relavium/db`. Phase 1 has since landed `@relavium/llm` — the `LLMProvider` seam + all three adapters (Anthropic, OpenAI/DeepSeek, @@ -64,7 +64,7 @@ The pre-egress budget governor (1.AC) + the agent-first `AgentSession` (1.V) lan **completing 1.m4**; then the end-to-end Node harness (1.U) landed (PR #27, 2026-06-16) **reaching M2** — the Phase-1 engine critical path is complete. The additive Lane-C agent-first sub-spine is now **complete** (session events **1.W ✅ (PR #28)** + persistence **1.X ✅ (PR #29)** + checkpoint/resume **1.Y** & export **1.Z ✅ (PR #30, 2026-06-17)** + the **1.AA** chat-regression harness ✅ (2026-06-17), closing **1.m5**); -on the 1.m6 multimodal sub-spine, **media-input adapters + the shared SSRF policy primitive (1.AE) landed (PR #32, 2026-06-18)** — after a multi-round + final 8-dimension adversarial review (no SSRF bypass found); the SSRF *mechanism* half + per-modality gating moved into 1.AF. **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20)** — P1+P2 (PR #33); the media-egress + SSRF mechanism half (D9), the `FallbackChain` resolve-before-egress + byte-free re-materialization sidecar (D8/D7), the byte-delivery `Range` gate (D13), and a clean P3 egress/SSRF security-review (P3 + P4/D13, PR #34); D12 `read_media` + authz, D11 GC sweep, D16 `save_to`, D15 load-check, D17 cost governor (P4 remainder, PR #35); and a 13-dimension end-to-end + external Sonar/CodeRabbit review (PR #36 — toolInput byte-redaction, the unified exact-membership output gate, the GC grace-basis fix, the egress-mechanism test). The host-wiring half (D12/D15/D17 mechanism) + the keychain IPC test are deferred to **1.AH**. The remaining Phase-1 work is 1.AG/1.AH; Phase 2 (CLI) is unblocked. See +on the 1.m6 multimodal sub-spine, **media-input adapters + the shared SSRF policy primitive (1.AE) landed (PR #32, 2026-06-18)** — after a multi-round + final 8-dimension adversarial review (no SSRF bypass found); the SSRF *mechanism* half + per-modality gating moved into 1.AF. **1.AF (engine media plumbing) is ✅ Done (PR #33/#34/#35/#36, 2026-06-20)** — P1+P2 (PR #33); the media-egress + SSRF mechanism half (D9), the `FallbackChain` resolve-before-egress + byte-free re-materialization sidecar (D8/D7), the byte-delivery `Range` gate (D13), and a clean P3 egress/SSRF security-review (P3 + P4/D13, PR #34); D12 `read_media` + authz, D11 GC sweep, D16 `save_to`, D15 load-check, D17 cost governor (P4 remainder, PR #35); and a 13-dimension end-to-end + external Sonar/CodeRabbit review (PR #36 — toolInput byte-redaction, the unified exact-membership output gate, the GC grace-basis fix, the egress-mechanism test). The host-wiring half (D12/D15/D17 mechanism) + the keychain IPC test are deferred to **1.AH**. **1.AG (output generation) is ✅ Done (PR #37, 2026-06-21)** — inline media-out (B) + sync `generateMedia`/routing (C) + the async media-job LRO (D, ADR-0045) + acceptance/conformance (E), each section dual-reviewed plus a final Opus-consolidated pre-merge review (two HIGH cross-fix defects caught + fixed). The remaining Phase-1 work is 1.AH; Phase 2 (CLI) is unblocked. See [docs/roadmap/current.md](docs/roadmap/current.md). See [README.md](README.md) for the public overview. ## Non-negotiable rules for AI agents diff --git a/README.md b/README.md index 49751460..f5f49617 100644 --- a/README.md +++ b/README.md @@ -53,5 +53,5 @@ both landed in **PR #26 (2026-06-16)**. With the budget governor in, **milestone end-to-end (live streaming + checkpoint + cross-process resume + retry + provider failover), completing the Phase-1 engine critical path. The remaining Phase-1 work is additive (the agent-first sub-spine — **session events 1.W ✅ (PR #28)** + **persistence 1.X ✅ (PR #29)** + **session checkpoint/resume 1.Y & export-to-workflow 1.Z ✅ (PR #30, 2026-06-17)** + **the 1.AA chat-regression harness ✅ (2026-06-17)**, completing **1.m5** — and the -multimodal sub-spine — **media-input adapters + the shared SSRF policy primitive 1.AE ✅ (PR #32)**, then **engine media plumbing 1.AF ✅ (PR #33/#34/#35/#36, 2026-06-20)** — landed; 1.AG/1.AH remain); **Phase 2 (the CLI) is unblocked**. See +multimodal sub-spine — **media-input adapters + the shared SSRF policy primitive 1.AE ✅ (PR #32)**, **engine media plumbing 1.AF ✅ (PR #33/#34/#35/#36, 2026-06-20)**, and **media output generation 1.AG ✅ (PR #37, 2026-06-21)** — landed; only 1.AH remains); **Phase 2 (the CLI) is unblocked**. See [docs/roadmap/current.md](docs/roadmap/current.md) for live status. diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index d5696d06..e73f3ee8 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -499,8 +499,10 @@ discriminated union: Tool-argument JSON deltas are concatenated into `argsJsonDelta` across `tool_call_delta` chunks and **parsed once at `tool_call_end`**. The final `stop` chunk always carries `stopReason` + `usage`. The `media_start/delta/end` -triad is part of the frozen union but **emitted by no adapter at 1.AD** — media -output wiring is 1.AG/1.AH, consistent with the all-false `media` matrices. +triad is part of the frozen union but **emitted by no adapter** — non-streaming +media output landed at **1.AG** (inline via `generate()` + the generative +endpoint, [ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)); the +**streaming triad is deferred to 1.AH** (ADR-0046 §4). ### 5. Stop reasons diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 58e5bbf5..741b547c 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -184,7 +184,7 @@ harness is now ✅ Done (2026-06-17), completing **1.m5**; cost-event persistenc > non-breaking; the seam doc carries the full amendment section. **1.AE–1.AH (media > input/engine/output + surfaces) are additive and do NOT gate M2** — the seam lane ran straight to > **1.K** (✅ Done, PR #13), which closed it. **1.AE (media-input adapters + the shared SSRF policy -> primitive) is now ✅ Done (PR #32, 2026-06-18)**; **1.AF (engine media plumbing) is now ✅ Done (PR #33/#34/#35/#36, 2026-06-20)**; **1.AG (output generation) — Sections A–E landed on `development`, pending PR merge** (ADR-0045/0046; each section dual-reviewed); 1.AH remains. +> primitive) is now ✅ Done (PR #32, 2026-06-18)**; **1.AF (engine media plumbing) is now ✅ Done (PR #33/#34/#35/#36, 2026-06-20)**; **1.AG (output generation) is now ✅ Done (PR #37, 2026-06-21)** (ADR-0045/0046; each section dual-reviewed + a final Opus-consolidated pre-merge review); **only 1.AH remains** on the 1.m6 sub-spine. > **Review-pass follow-ups landed (PR #12, merged 2026-06-11).** The 2026-06-10 engine/tooling > review pass landed as docs/decisions only — no Phase-1 workstream changed: **MCP client scheduling** @@ -247,7 +247,7 @@ harness is now ✅ Done (2026-06-17), completing **1.m5**; cost-event persistenc > pass — H1 `agent:tool_call.toolInput` byte-redaction, H2 the unified exact-membership `output_modalities` gate > (+ bidirectional set-equality), the GC grace basis bumped to de-reference time, an egress-mechanism test for > `nodeMediaEgressDeps`, and the canonical-home doc-drift cluster. All green + Leakwatch-clean. The remaining -> Phase-1 work is **additive and off the critical path**: only **1.AG–1.AH** (output, surfaces) remain before +> Phase-1 work is **additive and off the critical path**: only **1.AH** (surfaces & managed mode) remains before > Phase 1 closes. **Phase 2 (CLI, milestone M3) is unblocked.** Carry-over hardening is tracked in [deferred-tasks.md](deferred-tasks.md) — pick items up as Phase 1 diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 86fcb2f7..8e421d80 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -123,7 +123,7 @@ Severity is the review's verified rating. Check an item off in the PR that resol hook, it must apply these runtime checks. The current `assertHttpsBaseUrl` and `refineInFlightMediaPart` URL validation are construction-time / seam-ingestion-time policy; they catch malformed URLs but cannot catch DNS rebinding or a public hostname resolving to a private IP. **Scope split (resolving the earlier "Phase 2" framing):** the **media** url-carrier mechanism is **pulled into 1.AF** on a new bytes-shaped media-egress capability ([ADR-0043](../decisions/0043-media-egress-failover-rematerialization-ssrf.md)); the **general tool/MCP** `EgressCapability.fetch` enforcement still lands when the desktop/CLI surface implements that fetch hook. *(packages/core/src/tools/types.ts; security-review.md; media → 1.AF/ADR-0043; tool/MCP → surface fetch hook)* -- [ ] **Async media-job ADR + engine loop (`generateMedia`/`pollMediaJob` behavior, A5) — landed on `development`, pending PR merge (1.AG Sections A–E; the A5 obligation itself discharged in A/C/D).** +- [x] **Async media-job ADR + engine loop (`generateMedia`/`pollMediaJob` behavior, A5) — ✅ DONE (1.AG, PR #37, 2026-06-21; the A5 obligation discharged in Sections A/C/D).** [ADR-0045](../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) is written + Accepted, and the engine-owned **poll / checkpoint / resume / cancel loop** for minute-scale LROs (Sora/Veo) landed in 1.AG Section D (the run loop 1.N + checkpointer 1.R, reusing `LlmError` classification): `media_job:submitted` park, @@ -134,8 +134,7 @@ Severity is the review's verified rating. Check an item off in the PR that resol model dispatches to `generateMedia` BEFORE the turn loop), so the `requestsMediaOutput` guard in [`agent-turn.ts`](../../packages/core/src/engine/agent-turn.ts) only ever sees `'chat'` models — ADR-0046 §1's `media_surface: 'chat'` conjunct holds STRUCTURALLY (the guard's JSDoc `NOTE` points at the fork). The remaining - VENDOR-adapter + host-wiring work is tracked in the 1.AH entries below. **Tick `[x]` with the merged PR number - when the 1.AG PR lands** (this file's rule: check off in the PR that resolves it). *(1.AG Sections A–E — on development, pending merge)* + VENDOR-adapter + host-wiring work is tracked in the 1.AH entries below. *(1.AG Sections A–E — ✅ merged, PR #37)* - [ ] **Streaming media triad (`media_start`/`media_delta`/`media_end`) — host-deferred ([ADR-0046](../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §4).** 1.AG Section B delivers inline media-out through the non-streaming `generate()` path (the in-flight `media` `ContentPart` is de-inlined at `#emitDurable`). The **streaming** triad stays RESERVED: its Node diff --git a/docs/roadmap/phases/phase-1-engine-and-llm.md b/docs/roadmap/phases/phase-1-engine-and-llm.md index d1c9fe69..4e3211a8 100644 --- a/docs/roadmap/phases/phase-1-engine-and-llm.md +++ b/docs/roadmap/phases/phase-1-engine-and-llm.md @@ -1056,11 +1056,12 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. node produces a handle; an async video job checkpoints, survives a simulated restart, and resumes to completion; **a CANCEL aborts the in-flight `pollMediaJob` (via `AbortSignal`), emits `run:cancelled`, and triggers the terminal-state media sweep**; a content-policy job failure maps to `content_filter`. - *Status (pending merge):* **Sections A–E landed on `development`** — ADR-0045 (async media-job LRO) + - ADR-0046 (inline media-out via `generate()`), the five engine/seam slices (contracts, inline media-out, - sync `generateMedia` + routing, the async poll/checkpoint/resume/cancel loop, acceptance + generative - conformance), each dual-reviewed (0 blocker/high). Flips to ✅ Done on PR merge; the genuinely-remaining - vendor-adapter + host-wiring work is **1.AH** (deferred-tasks.md). All four acceptance criteria are tested. + *Status:* ✅ **Done (PR #37, 2026-06-21).** Sections A–E — ADR-0045 (async media-job LRO) + ADR-0046 (inline + media-out via `generate()`), the five engine/seam slices (contracts, inline media-out, sync `generateMedia` + + routing, the async poll/checkpoint/resume/cancel loop, acceptance + generative conformance) — each + dual-reviewed (0 blocker/high), then a final Opus-consolidated pre-merge review that caught + fixed two HIGH + cross-fix defects (the orphaned-vertex cost addend + the crash-in-window `run:paused`). All four acceptance + criteria are tested. The genuinely-remaining vendor-adapter + host-wiring work is **1.AH** (deferred-tasks.md). - **1.AH — Surfaces & managed mode (Phase E, spans Phases 2–6).** Desktop: **[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)** Rust-side media de-inline on egress + session-scoped `read_media` command + the Rust CAS — **must land before any @@ -1087,7 +1088,7 @@ the latter being the critical-path milestone for the whole product. | 1.m4 ✅ | Agent + non-agent node handlers, gate, checkpoint/resume, retry, tools, **expression sandbox** + pre-egress budget (**all components landed — 1.AC closed it, PR #26, 2026-06-16**) | 1.O, 1.P, 1.Q, 1.R, 1.S, 1.T, **1.AB**, **1.AC** | | **M2 ✅** | **Engine end-to-end from a Node harness (stream + checkpoint + retry + fallback) — CRITICAL-PATH MILESTONE** (**reached — 1.U landed, PR #27, 2026-06-16**) | **1.U** | | **1.m5 ✅** | Agent-first sub-spine: `AgentSession` + session events + persistence + checkpoint/resume + export, proven by its own harness (**additive, parallel — does NOT gate M2**) | 1.V, 1.W, 1.X, 1.Y, 1.Z, **1.AA** | -| 1.m6 | Multimodal I/O: seam amendment (**1.AD ✅ Done, PR #11 — landed before 1.K/1.O so the union members are non-breaking**), then media input/engine/output behavior (**additive — does NOT gate M2**) + surfaces threaded into Phases 2–6 ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)) | **1.AD ✅, 1.AE ✅, 1.AF ✅**, 1.AG (Sections A–E landed on development, pending merge), 1.AH | +| 1.m6 | Multimodal I/O: seam amendment (**1.AD ✅ Done, PR #11 — landed before 1.K/1.O so the union members are non-breaking**), then media input/engine/output behavior (**additive — does NOT gate M2**) + surfaces threaded into Phases 2–6 ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)) | **1.AD ✅, 1.AE ✅, 1.AF ✅, 1.AG ✅**, 1.AH | ## Sequencing & parallelization @@ -1249,7 +1250,7 @@ flowchart LR | 1.AD | D | 1.A (seam types) | **must precede 1.K, 1.O** (non-breaking union members); 1.AE | ⬤ shape-only — ✅ **Done (PR #11)** | | 1.AE | D | 1.AD, 1.G/1.H (adapters) | 1.AF | ✅ — **Done (PR #32, 2026-06-18)** | | 1.AF | D | 1.AE, 1.K, 1.N, 1.R | 1.AG | ✅ — **Done (PR #33/#34/#35/#36, 2026-06-20)** | -| 1.AG | D | 1.AF | 1.AH | ◇ — Sections A–E landed on `development` (pending PR merge) | +| 1.AG | D | 1.AF | 1.AH | ✅ — **Done (PR #37, 2026-06-21)** | | 1.AH | D | 1.AG | Phases 2–6 surfaces | ◇ (spans phases) | > The matrix `Depends on` column is **authoritative** for cross-lane feeder edges (e.g. `1.N → 1.W` From 2b202ca367169c841a619ba199393b1aca743400 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 12:44:47 +0300 Subject: [PATCH 02/21] =?UTF-8?q?feat(llm):=201.AH=20A1=20=E2=80=94=20Open?= =?UTF-8?q?AI-TTS=20audio=20generateMedia=20(sync=20media-out)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first 1.AH vendor-adapter section (package-level, no surface — provable offline). Refactors OpenAI `generateMedia` into a modality dispatcher and wires the audio arm: - generateMedia now dispatches by modality: image → gpt-image-1 (extracted into openAiGenerateImage), audio → TTS (new openAiGenerateSpeech), video → typed UnsupportedCapabilityError (the async Sora path is a later section); DeepSeek → typed capability error (generates no media). The extraction also keeps each arm small (no cognitive-complexity smell). - openAiGenerateSpeech: `client.audio.speech` returns BINARY audio → base64-encoded into an in-flight MediaPart (the engine de-inlines to a handle). `req.mimeType` selects the vendor response_format (default mp3); providerOptions.audio.voice selects the voice (default alloy). The raw audio bytes NEVER cross the seam — `raw` carries only a non-byte `{ responseFormat }` diagnostic (strip-on-sink anyway, I3). - TTS_FORMAT_TO_MIME + ttsResponseFormat (mimeType↔format) + ttsVoice helpers. - No new runtime dependency (openai SDK already imported); Buffer base64 mirrors packages/db's media-store (the adapter tsconfig carries `types: ["node"]`; the seam itself stays pure under tsconfig.seam.json `types: []`). Tests: TTS base64 + voice + format-default round-trip; req.mimeType → response_format mapping (audio/opus → opus); the modality-reject test updated (OpenAI video + DeepSeek reject; audio no longer rejected). Toolchain 16/16 green, format clean, Leakwatch 0. Docs: deferred-tasks + llm-provider-seam — OpenAI-TTS now wired (1.AH A1); Gemini-Imagen remains (A2). Pricing rows + count>1 deferred per the 1.AH scope decisions. Refs: ADR-0045, ADR-0031 Co-Authored-By: Claude Opus 4.8 --- .../shared-core/llm-provider-seam.md | 21 +- docs/roadmap/deferred-tasks.md | 13 +- packages/llm/src/adapters/openai.test.ts | 56 ++++- packages/llm/src/adapters/openai.ts | 195 ++++++++++++++---- 4 files changed, 222 insertions(+), 63 deletions(-) diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index e73f3ee8..32ebda5b 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -382,16 +382,17 @@ managed mode) are recorded in the ADR — this section is the dry shape referenc `MediaGenRequest` / `MediaGenResult` / `MediaJobStatus`: a sync generator resolves `{ media }`; an async one (Sora, Veo) resolves a **Relavium-opaque** `jobId` (no vendor operation name crosses the seam); `failed` carries the existing classified `LlmError` (content-policy → - `content_filter`). **Wired (1.AG Sections C/D):** `generateMedia` SYNC — the OpenAI adapter - implements gpt-image-1 image generation (`images.generate` → base64 `media`); a `media_surface: - 'generative'` agent node routes here instead of the inline `generate()`/`stream()` (the engine - resolves the per-model surface). The ASYNC `pollMediaJob` poll/checkpoint/resume/cancel loop is - WIRED in the engine (Section D — `media_job:submitted` park, the derived `pendingMediaJobs` slot, - re-attach-on-resume, a host-timer poll cadence, deadline→retryable-timeout, cancel→abort→terminal - sweep; failed→`content_filter`), proven against a conforming stub async provider. The remaining - work is **1.AH host-wiring**: the Sora/Veo `generateMedia(→jobId)`/`pollMediaJob` adapters, - OpenAI-TTS + Gemini-Imagen sync adapters, the per-model `media_surface` host lookup, and verified - generative pricing rows. + `content_filter`). **Wired (1.AG Sections C/D + 1.AH adapters):** `generateMedia` SYNC — the OpenAI + adapter implements gpt-image-1 image generation (`images.generate` → base64 `media`) and + **OpenAI-TTS audio** (`audio.speech` binary → base64 + `response_format`↔MIME, 1.AH A1); a + `media_surface: 'generative'` agent node routes here instead of the inline `generate()`/`stream()` + (the engine resolves the per-model surface). The ASYNC `pollMediaJob` poll/checkpoint/resume/cancel + loop is WIRED in the engine (Section D — `media_job:submitted` park, the derived `pendingMediaJobs` + slot, re-attach-on-resume, a host-timer poll cadence, deadline→retryable-timeout, + cancel→abort→terminal sweep; failed→`content_filter`), proven against a conforming stub async + provider. The remaining work is **1.AH**: the Gemini-Imagen sync + Sora/Veo async + `generateMedia(→jobId)`/`pollMediaJob` adapters, the per-model `media_surface` host lookup, and + verified generative pricing rows. ```ts // Seam shape (A5; ADR-0045) — behavior WIRED at 1.AG (sync generateMedia Section C, async poll loop Section D). diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 8e421d80..d70c2cb0 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -166,13 +166,12 @@ Severity is the review's verified rating. Check an item off in the PR that resol (no mis-bill). **Test follow-up:** the realized-cost vertical (`realizedMediaCost` → a non-zero `cost:updated`) cannot be exercised end-to-end until a generative model carries a rate; the cost MATH is unit-tested via `mediaCost` against a constructed rate, and a non-zero dispatch assertion lands with these rows. *(packages/llm/src/pricing.ts; verified rates → 1.AH)* -- [ ] **`generateMedia` for OpenAI-TTS audio + Gemini-Imagen — adapter wires deferred (1.AG Section C → 1.AH).** - Section C wires `generateMedia` SYNC for **OpenAI image** (gpt-image-1 `images.generate` → base64), proving the - full engine→adapter→de-inline vertical. The remaining generators are bounded follow-ups: **OpenAI-TTS audio** - (`audio.speech` returns raw bytes → the adapter must base64-encode them + map the requested `response_format` → - MIME) and **Gemini-Imagen** (`generateImages` → `generatedImages[].image.imageBytes`, which needs a - `GeminiTransport.generateImages` extension to keep conformance vendor-free). Neither is runtime-reachable until the - per-model surface lookup is host-wired (above), so they land with that 1.AH wiring. Two bounded image follow-ups +- [ ] **`generateMedia` for Gemini-Imagen — adapter wire deferred (1.AG Section C → 1.AH A2).** + Section C wired `generateMedia` SYNC for **OpenAI image** (gpt-image-1 `images.generate` → base64); **1.AH A1 + wired OpenAI-TTS audio** (`audio.speech` binary → base64 + `response_format`↔MIME map, no new dep). The + remaining sync generator is **Gemini-Imagen** (`generateImages` → `generatedImages[].image.imageBytes`, which + needs a `GeminiTransport.generateImages` extension to keep conformance vendor-free). Not runtime-reachable until the + per-model surface lookup is host-wired (above), so it lands written-now/E2E-at-1.AH-host-wiring. Two bounded image follow-ups ride along: (a) **multi-image `count > 1`** — the SYNC `MediaGenResult.media` carries a SINGLE part, so the OpenAI adapter currently rejects `count > 1` (never bill-N-deliver-1); delivering N needs an additive `media: MediaPart[]` seam amendment (ADR-0031). (b) **image-gen knobs** (`size`/`quality` via `MediaGenRequest.providerOptions`) — the diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index bab310b4..b89b15cf 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -315,12 +315,64 @@ describe('OpenAI-compatible adapter', () => { }); }); - it('generateMedia rejects a non-image modality + DeepSeek with a typed capability error', async () => { + it('generateMedia (audio/TTS) base64-encodes audio.speech bytes into a media part + maps the format MIME (1.AH)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + // audio.speech returns BINARY audio bytes; the replay string stands in for them. + return Promise.resolve( + new Response('FAKE-AUDIO-BYTES', { + status: 200, + headers: { 'content-type': 'audio/mpeg' }, + }), + ); + }, + }); + const result = await genMedia( + adapter, + { + model: 'gpt-4o-mini-tts', + prompt: 'hello world', + modality: 'audio', + providerOptions: { audio: { voice: 'verse' } }, + }, + 'k', + ); + expect(result.jobId).toBeUndefined(); // SYNC arm + expect(sent['input']).toBe('hello world'); + expect(sent['voice']).toBe('verse'); // from providerOptions.audio.voice + expect(sent['response_format']).toBe('mp3'); // default when no req.mimeType + expect(result.media?.mimeType).toBe('audio/mpeg'); + expect(result.media?.source).toEqual({ + kind: 'base64', + data: Buffer.from('FAKE-AUDIO-BYTES').toString('base64'), + }); + }); + + it('generateMedia (audio) maps req.mimeType → response_format + result MIME (audio/opus → opus)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(new Response('x', { status: 200 })); + }, + }); + const result = await genMedia( + adapter, + { model: 'gpt-4o-mini-tts', prompt: 'hi', modality: 'audio', mimeType: 'audio/opus' }, + 'k', + ); + expect(sent['response_format']).toBe('opus'); + expect(result.media?.mimeType).toBe('audio/opus'); + }); + + it('generateMedia rejects OpenAI video (no sync surface) + DeepSeek any modality with a typed capability error', async () => { const oai = createOpenAiAdapter({ fetch: () => Promise.reject(new Error('must fail fast before any egress')), }); await expect( - genMedia(oai, { model: 'gpt-4o-mini-tts', prompt: 'x', modality: 'audio' }, 'k'), + genMedia(oai, { model: 'sora-2', prompt: 'x', modality: 'video' }, 'k'), ).rejects.toThrowError(UnsupportedCapabilityError); const ds = createOpenAiAdapter({ providerId: 'deepseek', diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 460e1708..b4ef836e 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -605,6 +605,43 @@ function resolveOpenAiAudio(providerOptions: LlmRequest['providerOptions']): { return { voice, format }; } +/** The TTS `response_format` values `audio.speech` accepts, mapped to the bare MIME the seam admits (1.AH). */ +const TTS_FORMAT_TO_MIME = { + mp3: 'audio/mpeg', + opus: 'audio/opus', + aac: 'audio/aac', + flac: 'audio/flac', + wav: 'audio/wav', + pcm: 'audio/L16', +} as const satisfies Record; + +/** Map a requested output `mimeType` to a TTS `response_format` — the inverse of {@link TTS_FORMAT_TO_MIME}. + * Default mp3 (the API default + the canonical fallback for an unspecified/unknown request). */ +function ttsResponseFormat(mimeType: string | undefined): keyof typeof TTS_FORMAT_TO_MIME { + switch (mimeType) { + case 'audio/opus': + return 'opus'; + case 'audio/aac': + return 'aac'; + case 'audio/flac': + return 'flac'; + case 'audio/wav': + return 'wav'; + case 'audio/L16': + return 'pcm'; + default: + return 'mp3'; // 'audio/mpeg' or anything unspecified + } +} + +/** The TTS voice from `providerOptions.audio.voice` (any string the vendor accepts), else OpenAI's `alloy`. */ +function ttsVoice(providerOptions: MediaGenRequest['providerOptions']): string { + const audioOpts = isRecord(providerOptions) ? providerOptions['audio'] : undefined; + return isRecord(audioOpts) && typeof audioOpts['voice'] === 'string' + ? audioOpts['voice'] + : 'alloy'; +} + function buildRequestOptions(req: LlmRequest): { signal?: AbortSignal } { return isAbortSignal(req.signal) ? { signal: req.signal } : {}; } @@ -876,58 +913,128 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { * the result is a normalized `MediaGenResult` whose `raw` is strip-discarded by sinks. */ async generateMedia(req: MediaGenRequest, key: string): Promise { - if (providerId !== 'openai' || req.modality !== 'image') { + // DeepSeek (the same adapter pointed at a different baseURL) generates no media. + if (providerId !== 'openai') { throw new UnsupportedCapabilityError( providerId, 'media', - `${providerId} generateMedia supports only OpenAI image generation, not '${req.modality}' (audio/video deferred)`, - ); - } - if (req.count !== undefined && req.count > 1) { - // The SYNC seam carries a SINGLE MediaPart. Rather than generate (and have the engine bill) N images - // and silently drop N-1, reject count > 1 loudly until a multi-part media-array result lands (a future - // additive ADR-0031 seam amendment, deferred-tasks.md) — never deliver fewer artifacts than billed. - throw new LlmProviderError( - makeLlmError({ - provider: providerId, - kind: 'bad_request', - message: `OpenAI image generateMedia delivers a single image; count ${String(req.count)} > 1 is not supported on the SYNC seam`, - }), + `${providerId} generates no media (only OpenAI generateMedia is wired)`, ); } - // Honor a requested output format (req.mimeType → png/jpeg/webp); the result MIME reflects what we asked - // for (or gpt-image-1's PNG default). (providerOptions image knobs — size/quality — are not set by the - // engine in Section C; threading them is a bounded follow-up with the image-knob work, deferred-tasks.md.) - const outputFormat = imageOutputFormat(req.mimeType); - let response: OpenAI.ImagesResponse; - try { - response = await createClient(key).images.generate( - { - model: req.model, - prompt: req.prompt, - ...(outputFormat === undefined ? {} : { output_format: outputFormat }), - }, - isAbortSignal(req.signal) ? { signal: req.signal } : {}, - ); - } catch (err) { - throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + const client = createClient(key); + // SYNC separate-endpoint generation, dispatched by modality (1.AG/1.AH, ADR-0045 §1): image → + // gpt-image-1 (images.generate); audio → TTS (audio.speech). Video is the ASYNC Sora path (a separate + // section) — not a sync surface here. + if (req.modality === 'image') { + return openAiGenerateImage(client, req, providerId); } - const b64 = response.data?.[0]?.b64_json; - if (b64 === undefined || b64.length === 0) { - throw new LlmProviderError( - makeLlmError({ - provider: providerId, - kind: 'bad_request', - message: 'OpenAI image generation returned no base64 image data', - }), - ); + if (req.modality === 'audio') { + return openAiGenerateSpeech(client, req, providerId); } - const mimeType = outputFormat === undefined ? 'image/png' : `image/${outputFormat}`; - return { - media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, - raw: response, - }; + throw new UnsupportedCapabilityError( + providerId, + 'media', + `OpenAI generateMedia has no SYNC surface for '${req.modality}' (video is the async Sora path)`, + ); + }, + }; +} + +/** + * SYNC image generation (gpt-image-1 via `images.generate` → base64). Honors a requested output format + * (`req.mimeType` → png/jpeg/webp; else gpt-image-1's PNG default). The single-`MediaPart` SYNC seam carries + * one image, so `count > 1` is rejected loud rather than billing N and dropping N-1 (a multi-image array is a + * deferred ADR-0031 amendment). No vendor type crosses the seam — the normalized `MediaGenResult.raw` is + * strip-discarded by sinks. + */ +async function openAiGenerateImage( + client: OpenAI, + req: MediaGenRequest, + providerId: ProviderId, +): Promise { + if (req.count !== undefined && req.count > 1) { + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: `OpenAI image generateMedia delivers a single image; count ${String(req.count)} > 1 is not supported on the SYNC seam`, + }), + ); + } + const outputFormat = imageOutputFormat(req.mimeType); + let response: OpenAI.ImagesResponse; + try { + response = await client.images.generate( + { + model: req.model, + prompt: req.prompt, + ...(outputFormat === undefined ? {} : { output_format: outputFormat }), + }, + isAbortSignal(req.signal) ? { signal: req.signal } : {}, + ); + } catch (err) { + throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + } + const b64 = response.data?.[0]?.b64_json; + if (b64 === undefined || b64.length === 0) { + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'OpenAI image generation returned no base64 image data', + }), + ); + } + const mimeType = outputFormat === undefined ? 'image/png' : `image/${outputFormat}`; + return { + media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, + raw: response, + }; +} + +/** + * SYNC text-to-speech (1.AH): `audio.speech` returns BINARY audio bytes, so the adapter base64-encodes them + * into an in-flight `MediaPart` (the engine de-inlines it to a handle). `req.mimeType` selects the vendor + * `response_format` (default mp3); `providerOptions.audio.voice` selects the voice (default `alloy`). The raw + * audio bytes NEVER cross the seam — `raw` carries only a tiny non-byte diagnostic (and sinks strip it anyway). + */ +async function openAiGenerateSpeech( + client: OpenAI, + req: MediaGenRequest, + providerId: ProviderId, +): Promise { + const format = ttsResponseFormat(req.mimeType); + let response: Response; + try { + response = await client.audio.speech.create( + { + model: req.model, + voice: ttsVoice(req.providerOptions), + input: req.prompt, + response_format: format, + }, + isAbortSignal(req.signal) ? { signal: req.signal } : {}, + ); + } catch (err) { + throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + } + const bytes = new Uint8Array(await response.arrayBuffer()); + if (bytes.length === 0) { + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'OpenAI TTS returned no audio bytes', + }), + ); + } + return { + media: { + type: 'media', + mimeType: TTS_FORMAT_TO_MIME[format], + source: { kind: 'base64', data: Buffer.from(bytes).toString('base64') }, }, + raw: { responseFormat: format }, // diagnostic only — never the audio bytes (I3 / strip-on-sink) }; } From dc8731b72bbb11fdfff1519aaca909169676e286 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 12:55:03 +0300 Subject: [PATCH 03/21] =?UTF-8?q?fix(llm):=201.AH=20A1=20=E2=80=94=20Opus-?= =?UTF-8?q?review=20follow-ups=20(TTS=20body-read=20in=20try,=20pcm/count?= =?UTF-8?q?=20notes,=20empty-bytes=20test)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies the verified findings from the A1 Opus review (0 blocker/high; 1 medium, 3 low — all confirmed): - M1 (medium): move the TTS body download (`await response.arrayBuffer()`) INSIDE the try/catch. `audio.speech` is a __binaryResponse — create() returns the raw Response unconsumed, so the multi-MB download happens at the read; a mid-download socket reset / abort was escaping unclassified and flattening to opaque `internal` instead of a classified LlmError (and an abort → `internal` not `cancelled`). Now routed through openaiErrorToLlmError, matching the image/egress-error contract. - L1 (low): document the `pcm → audio/L16` sample-rate loss — OpenAI pcm is 24 kHz but the seam's MediaMimeTypeSchema forbids the `;rate=` parameter (RFC 2586 default 8 kHz), so a consumer must assume 24 kHz. Mirrors the pre-existing chat pcm16 convention; recorded as a low deferred-tasks known-limitation. - L2 (low): comment that `count` is intentionally a no-op for TTS (per-character billing, single stream — no bill-N-deliver-1 hazard, unlike the image path). - L3 (low): test the empty-audio-bytes → bad_request rejection branch. Toolchain 16/16 green, format clean, Leakwatch 0. Refs: ADR-0045 Co-Authored-By: Claude Opus 4.8 --- docs/roadmap/deferred-tasks.md | 6 ++++++ packages/llm/src/adapters/openai.test.ts | 9 +++++++++ packages/llm/src/adapters/openai.ts | 18 ++++++++++++++---- 3 files changed, 29 insertions(+), 4 deletions(-) diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index d70c2cb0..e1dd4081 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -177,6 +177,12 @@ Severity is the review's verified rating. Check an item off in the PR that resol seam amendment (ADR-0031). (b) **image-gen knobs** (`size`/`quality` via `MediaGenRequest.providerOptions`) — the engine does not yet populate `providerOptions` for a generative call, so the adapter threads only the output format (`req.mimeType`); a typed per-knob passthrough lands when the engine wires image knobs. *(packages/llm/src/adapters/{openai,gemini}.ts; types.ts MediaGenResult; 1.AH)* +- [ ] **Rate-carrying media representation for raw PCM (1.AH A1 known-limitation).** OpenAI TTS `pcm` is headerless + 16-bit LE PCM at 24 kHz, but the seam's `MediaMimeTypeSchema` forbids MIME parameters, so the bare `audio/L16` + cannot carry `;rate=24000` (RFC 2586's default is 8 kHz) — a consumer of an `audio/L16` part must assume 24 kHz. + Mirrors the pre-existing chat-audio `pcm16 → audio/L16` convention; the self-describing containers (mp3/opus/aac/ + flac/wav) are unaffected. A rate-carrying media representation (or dropping bare-PCM from the offered set) is the + fix. Low priority — opt-in niche format only. *(packages/shared/src/content.ts MediaMimeTypeSchema; packages/llm/src/adapters/openai.ts; low · 1.AH/Phase-2)* - [ ] **Sora/Veo async-media ADAPTERS (`generateMedia`→`{ jobId }` + `pollMediaJob`) — deferred (1.AG Section D → 1.AH).** 1.AG Section D landed the ENGINE-owned async-job loop (park on `media_job:submitted`, the `host.setTimer` poll cadence, exp-backoff to `pollMaxMs`, deadline→retryable timeout, cross-process re-attach (MJ-1), gate-vs-media diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index b89b15cf..61d6b0a7 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -367,6 +367,15 @@ describe('OpenAI-compatible adapter', () => { expect(result.media?.mimeType).toBe('audio/opus'); }); + it('generateMedia (audio) maps an empty audio body to a typed bad_request LlmProviderError', async () => { + const adapter = createOpenAiAdapter({ + fetch: () => Promise.resolve(new Response('', { status: 200 })), + }); + await expect( + genMedia(adapter, { model: 'gpt-4o-mini-tts', prompt: 'x', modality: 'audio' }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'bad_request', retryable: false } }); + }); + it('generateMedia rejects OpenAI video (no sync surface) + DeepSeek any modality with a typed capability error', async () => { const oai = createOpenAiAdapter({ fetch: () => Promise.reject(new Error('must fail fast before any egress')), diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index b4ef836e..5592e337 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -605,7 +605,12 @@ function resolveOpenAiAudio(providerOptions: LlmRequest['providerOptions']): { return { voice, format }; } -/** The TTS `response_format` values `audio.speech` accepts, mapped to the bare MIME the seam admits (1.AH). */ +/** The TTS `response_format` values `audio.speech` accepts, mapped to the bare MIME the seam admits (1.AH). + * NOTE on `pcm`: OpenAI's pcm output is headerless 16-bit LE PCM at **24 kHz**, but the seam's + * `MediaMimeTypeSchema` forbids MIME parameters (no `;rate=24000`), so the bare `audio/L16` cannot carry the + * rate (RFC 2586's default is 8 kHz) — a downstream consumer of an `audio/L16` part MUST assume 24 kHz. This + * mirrors the pre-existing chat-audio `pcm16 → audio/L16` convention; the self-describing containers (mp3/ + * opus/aac/flac/wav) carry their own rate. Tracked: deferred-tasks.md (a rate-carrying media representation). */ const TTS_FORMAT_TO_MIME = { mp3: 'audio/mpeg', opus: 'audio/opus', @@ -1003,10 +1008,12 @@ async function openAiGenerateSpeech( req: MediaGenRequest, providerId: ProviderId, ): Promise { + // `count` (images-per-call) is a no-op for TTS — `audio.speech` is billed per input character and yields a + // single audio stream, so there is no bill-N-deliver-1 hazard (unlike the image path's loud count>1 reject). const format = ttsResponseFormat(req.mimeType); - let response: Response; + let bytes: Uint8Array; try { - response = await client.audio.speech.create( + const response = await client.audio.speech.create( { model: req.model, voice: ttsVoice(req.providerOptions), @@ -1015,10 +1022,13 @@ async function openAiGenerateSpeech( }, isAbortSignal(req.signal) ? { signal: req.signal } : {}, ); + // The BINARY body download happens HERE (audio.speech is a __binaryResponse — create() returns the raw + // Response unconsumed), so the read MUST be inside the try: a mid-download socket reset / abort would + // otherwise escape unclassified and flatten to an opaque `internal` instead of a classified LlmError. + bytes = new Uint8Array(await response.arrayBuffer()); } catch (err) { throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); } - const bytes = new Uint8Array(await response.arrayBuffer()); if (bytes.length === 0) { throw new LlmProviderError( makeLlmError({ From 3488329adeb3f666de260b68fbaa92f94675549c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 13:12:03 +0300 Subject: [PATCH 04/21] =?UTF-8?q?fix(llm):=201.AH=20A1=20=E2=80=94=20OpenA?= =?UTF-8?q?I-TTS=20Sonnet-review=20follow-ups?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the verified findings from the A1 Sonnet adversarial review (0 blocker/high): - doc(nit): refresh the `generateMedia` method JSDoc — it still claimed audio (TTS) was NOT wired; audio dispatches to `openAiGenerateSpeech` since A1. Video remains the async Sora path (correctly throws `UnsupportedCapabilityError`). - test(low): assert `result.raw === { responseFormat: 'mp3' }` on the TTS happy-path — pins the I3 structural guarantee (no audio bytes leak into `raw`, which is `z.unknown()`) at the test level. - test(low): add a TTS-call-failure test — a rejected `audio.speech.create` surfaces as a typed `LlmProviderError` (transport, retryable) via `openaiErrorToLlmError`, proving the try/catch is wired rather than raw. Toolchain 16/16 green; prettier clean; Leakwatch 0. Refs: ADR-0045, ADR-0046 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/openai.test.ts | 15 +++++++++++++++ packages/llm/src/adapters/openai.ts | 8 ++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 61d6b0a7..4b65c0f0 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -348,6 +348,21 @@ describe('OpenAI-compatible adapter', () => { kind: 'base64', data: Buffer.from('FAKE-AUDIO-BYTES').toString('base64'), }); + // I3 structural guard: `raw` carries ONLY a non-byte diagnostic — never the audio bytes. + expect(result.raw).toEqual({ responseFormat: 'mp3' }); + }); + + it('generateMedia (audio) routes a TTS-call failure through the classifier (the catch is wired, not raw)', async () => { + // Prove the try/catch around `audio.speech.create` + the body read routes ANY thrown provider + // error through `openaiErrorToLlmError` — a rejected request surfaces as a typed `LlmProviderError`, + // never a raw SDK error. A connection failure classifies as `transport` (retryable). + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: () => Promise.reject(new APIConnectionError({ message: 'socket hang up' })), + }); + await expect( + genMedia(adapter, { model: 'gpt-4o-mini-tts', prompt: 'x', modality: 'audio' }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'transport', retryable: true } }); }); it('generateMedia (audio) maps req.mimeType → response_format + result MIME (audio/opus → opus)', async () => { diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 5592e337..96ff968f 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -911,10 +911,10 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { return streamChunks(createClient(key), req, providerId); }, /** - * Separate-endpoint media generation (1.AG Section C, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). - * SYNC image generation via gpt-image-1 (`client.images.generate` → base64). Audio (TTS via - * `audio.speech`) and video are NOT wired here yet — they fail loud with a typed capability error, never a - * silent drop (deferred — deferred-tasks.md). DeepSeek generates no media. No vendor type crosses the seam: + * Separate-endpoint media generation (1.AG/1.AH, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * SYNC image (gpt-image-1 `client.images.generate` → base64) and SYNC TTS audio (`audio.speech` → base64, + * 1.AH A1) are wired; **video** is the ASYNC Sora path and is NOT a sync surface here — it fails loud with a + * typed capability error, never a silent drop. DeepSeek generates no media. No vendor type crosses the seam: * the result is a normalized `MediaGenResult` whose `raw` is strip-discarded by sinks. */ async generateMedia(req: MediaGenRequest, key: string): Promise { From 74673d7d180f92fd2dc699dd608bde635fbfd315 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 13:26:18 +0300 Subject: [PATCH 05/21] =?UTF-8?q?feat(llm):=201.AH=20A2=20=E2=80=94=20Gemi?= =?UTF-8?q?ni=20Imagen=20sync=20image=20generateMedia?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the Gemini adapter's separate-endpoint image generation behind the seam (ADR-0045 §1, ADR-0046), mirroring the OpenAI image arm: - `GeminiTransport.generateImages` — a new required transport method (a transport honestly declares the generative surface it services; image is Gemini's only Phase-1 generative arm, Veo video is 1.AH A4). The default `sdkTransport` wraps `ai.models.generateImages`; the conformance harness replays it. - `GeminiImageRequest`/`GeminiImageResponse` — hand-rolled structural subsets so no vendor type crosses the seam (parity with `GeminiRequest`/`GeminiResponse`). - `generateMedia` dispatches by modality: image → `geminiGenerateImage` (`generatedImages[].image.imageBytes` → a base64 `media` part the engine de-inlines to a `media://` handle — I3); audio/video fail loud with a typed `UnsupportedCapabilityError`. `count > 1` is rejected before egress; a safety-filtered candidate (`raiFilteredReason`, no image) maps to `content_filter`; `numberOfImages` is pinned to 1 AFTER any providerOptions spread so an author count can't smuggle past the single-artifact seam. - Conformance: a recorded Imagen fixture drives the existing generative seam-contract scenario against the real fold; 9 adapter unit tests cover the happy path, MIME default, providerOptions threading, count>1, content_filter, no-image, transport rejection, and audio/video capability rejection. Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0046, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 147 +++++++++++++++++- packages/llm/src/adapters/gemini.ts | 120 ++++++++++++++ .../llm/src/conformance/fixtures/gemini.ts | 12 ++ .../conformance/gemini.conformance.test.ts | 13 ++ 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index 7a4de3e9..0260f7cd 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -3,7 +3,13 @@ import { describe, expect, it } from 'vitest'; import { UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; import { GeminiToolCallIds } from '../tool-normalizer.js'; -import type { LlmRequest, StreamChunk } from '../types.js'; +import type { + LlmProvider, + LlmRequest, + MediaGenRequest, + MediaGenResult, + StreamChunk, +} from '../types.js'; import { buildGeminiRequest, createGeminiAdapter, @@ -12,6 +18,8 @@ import { mapContent, mapStopReason, mapUsage, + type GeminiImageRequest, + type GeminiImageResponse, type GeminiRequest, type GeminiResponse, type GeminiTransport, @@ -46,10 +54,41 @@ function fakeTransport( })(), ); }, + // The text-fold tests never call generateImages; the dedicated generateMedia tests build their own + // image transport (fakeImageTransport). Reject loud if a text test ever reaches it. + generateImages: () => Promise.reject(new Error('fakeTransport.generateImages not configured')), }; return holder; } +/** A transport whose `generateImages` returns a fixed Imagen response and captures the request. */ +function fakeImageTransport( + response: GeminiImageResponse, +): GeminiTransport & { lastImageRequest?: GeminiImageRequest } { + const holder: GeminiTransport & { lastImageRequest?: GeminiImageRequest } = { + generate: () => Promise.reject(new Error('unused')), + stream: () => Promise.reject(new Error('unused')), + generateImages: (request) => { + holder.lastImageRequest = request; + return Promise.resolve(response); + }, + }; + return holder; +} + +/** Call the adapter's optional `generateMedia` via `?.()` — a call (binds `this`), never an extraction, + * so the unbound-method lint stays happy; the `??` branch asserts the method is implemented. */ +function genMedia( + adapter: LlmProvider, + req: MediaGenRequest, + key: string, +): Promise { + return ( + adapter.generateMedia?.(req, key) ?? + Promise.reject(new Error('adapter implements no generateMedia')) + ); +} + const REQ: LlmRequest = { model: 'gemini-2.5-flash', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], @@ -500,6 +539,7 @@ describe('Gemini adapter — generate / stream via injected transport', () => { const transport: GeminiTransport = { generate: () => Promise.reject(Object.assign(new Error(`rl: ${SECRET}`), { status: 429 })), stream: () => Promise.reject(new Error('unused')), + generateImages: () => Promise.reject(new Error('unused')), }; const adapter = createGeminiAdapter({ transport }); let caught: unknown; @@ -543,6 +583,7 @@ describe('Gemini adapter — generate / stream via injected transport', () => { const transport: GeminiTransport = { generate: () => Promise.reject(new Error('unused')), stream: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), + generateImages: () => Promise.reject(new Error('unused')), }; const adapter = createGeminiAdapter({ transport }); const chunks = await collect(adapter.stream(REQ, 'k')); @@ -569,6 +610,7 @@ describe('Gemini adapter — remaining branches', () => { candidates: [{ content: { parts: [{ text: 'hi' }] }, finishReason: 'STOP' }], }), stream: () => Promise.reject(new Error('unused')), + generateImages: () => Promise.reject(new Error('unused')), }, }); const result = await adapter.generate(REQ, 'k'); @@ -587,6 +629,7 @@ describe('Gemini adapter — remaining branches', () => { throw Object.assign(new Error('mid-stream'), { status: 500 }); })(), ), + generateImages: () => Promise.reject(new Error('unused')), }, }); const chunks = await collect(adapter.stream(REQ, 'k')); @@ -747,3 +790,105 @@ describe('Gemini adapter — usage, truncation, refusal, malformed tool (review expect(chunks.at(-1)?.type).toBe('stop'); }); }); + +describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { + const IMG_REQ: MediaGenRequest = { + model: 'imagen-4.0-generate-001', + prompt: 'a red circle on a white background', + modality: 'image', + }; + const B64 = 'aGVsbG8taW1hZ2Vu'; // "hello-imagen" + + it('normalizes a generated image to a base64 media part (no jobId, raw retained)', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], + }); + const result = await genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k'); + expect(result.jobId).toBeUndefined(); // SYNC arm + expect(result.media?.type).toBe('media'); + expect(result.media?.mimeType).toBe('image/png'); + expect(result.media?.source).toEqual({ kind: 'base64', data: B64 }); + expect(result.raw).toBeDefined(); // internal diagnostic — sinks strip it (I3) + // The single-artifact seam: numberOfImages is pinned to 1, and the prompt/model are threaded. + expect(transport.lastImageRequest?.config['numberOfImages']).toBe(1); + expect(transport.lastImageRequest?.prompt).toBe(IMG_REQ.prompt); + }); + + it('defaults the MIME to image/png when the vendor omits it', async () => { + const transport = fakeImageTransport({ generatedImages: [{ image: { imageBytes: B64 } }] }); + const result = await genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k'); + expect(result.media?.mimeType).toBe('image/png'); + }); + + it('threads providerOptions into config but pins numberOfImages to 1 (count can not be smuggled)', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], + }); + await genMedia( + createGeminiAdapter({ transport }), + { ...IMG_REQ, providerOptions: { aspectRatio: '16:9', numberOfImages: 5 } }, + 'k', + ); + expect(transport.lastImageRequest?.config['aspectRatio']).toBe('16:9'); + expect(transport.lastImageRequest?.config['numberOfImages']).toBe(1); // the pin wins over providerOptions + }); + + it('threads an AbortSignal into the Imagen config so a run cancel reaches the in-flight call', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], + }); + const controller = new AbortController(); + await genMedia( + createGeminiAdapter({ transport }), + { ...IMG_REQ, signal: controller.signal }, + 'k', + ); + expect(transport.lastImageRequest?.config['abortSignal']).toBe(controller.signal); + }); + + it('rejects count > 1 (single-artifact SYNC seam) with a typed bad_request before any egress', async () => { + const transport = fakeImageTransport({ generatedImages: [] }); + await expect( + genMedia(createGeminiAdapter({ transport }), { ...IMG_REQ, count: 3 }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'bad_request' } }); + expect(transport.lastImageRequest).toBeUndefined(); // rejected before the transport call + }); + + it('maps a safety-filtered candidate (raiFilteredReason, no image) to content_filter', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ raiFilteredReason: 'Unsafe content detected' }], + }); + await expect(genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k')).rejects.toMatchObject({ + llmError: { kind: 'content_filter' }, + }); + }); + + it('maps a no-image response to a typed bad_request LlmProviderError', async () => { + const transport = fakeImageTransport({ generatedImages: [] }); + await expect(genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k')).rejects.toMatchObject({ + llmError: { kind: 'bad_request' }, + }); + }); + + it('surfaces a transport rejection as a classified LlmProviderError', async () => { + const transport: GeminiTransport = { + generate: () => Promise.reject(new Error('unused')), + stream: () => Promise.reject(new Error('unused')), + generateImages: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), + }; + await expect(genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k')).rejects.toMatchObject({ + llmError: { kind: 'overloaded' }, + }); + }); + + it('rejects audio + video modalities with a typed capability error (image is the only sync surface)', async () => { + const transport = fakeImageTransport({ generatedImages: [] }); + const adapter = createGeminiAdapter({ transport }); + await expect(genMedia(adapter, { ...IMG_REQ, modality: 'audio' }, 'k')).rejects.toBeInstanceOf( + UnsupportedCapabilityError, + ); + await expect(genMedia(adapter, { ...IMG_REQ, modality: 'video' }, 'k')).rejects.toBeInstanceOf( + UnsupportedCapabilityError, + ); + }); +}); diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 163d87eb..ab02c046 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -6,6 +6,7 @@ import type { ContentPart, OutputModality, StopReason } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; import { GeminiToolCallIds, normalizeToolCall, toWire } from '../tool-normalizer.js'; +import { UnsupportedCapabilityError } from '../errors.js'; import type { CapabilityFlags, LlmError, @@ -13,6 +14,8 @@ import type { LlmProvider, LlmRequest, LlmResult, + MediaGenRequest, + MediaGenResult, StreamChunk, ToolChoice, ToolDef, @@ -129,6 +132,25 @@ export interface GeminiRequest { config: Record; } +/** The lowered Imagen request the transport sends (1.AH A2; a plain object the SDK accepts). */ +export interface GeminiImageRequest { + model: string; + prompt: string; + config: Record; +} + +/** + * The minimal slice of the Gemini Imagen response the fold consumes — a hand-rolled structural subset so + * no vendor type crosses the seam (parity with {@link GeminiResponse}). `imageBytes` is base64; + * `raiFilteredReason` is set (with no `image`) when a candidate is dropped by the safety filter. + */ +export interface GeminiImageResponse { + generatedImages?: Array<{ + image?: { imageBytes?: string; mimeType?: string }; + raiFilteredReason?: string; + }>; +} + /** * The injected network seam. The default wraps `@google/genai`; the conformance harness injects a * replay implementation. Keeping it here lets the one adapter run on every host (ADR-0018) and lets @@ -137,6 +159,12 @@ export interface GeminiRequest { export interface GeminiTransport { generate(request: GeminiRequest, key: string): Promise; stream(request: GeminiRequest, key: string): Promise>; + /** + * Imagen generative image endpoint (1.AH A2). The default wraps `ai.models.generateImages`; the + * conformance harness injects a replay. Required so a transport honestly declares the generative + * surface it services (sync image is Gemini's only Phase-1 generative arm; Veo video is 1.AH A4). + */ + generateImages(request: GeminiImageRequest, key: string): Promise; } // --- Normalization: Gemini wire → canonical -------------------------------------------------- @@ -466,6 +494,10 @@ const sdkTransport: GeminiTransport = { const client = new GoogleGenAI({ apiKey: key }); return client.models.generateContentStream(request); }, + async generateImages(request: GeminiImageRequest, key: string): Promise { + const client = new GoogleGenAI({ apiKey: key }); + return client.models.generateImages(request); + }, }; /* v8 ignore stop */ @@ -612,6 +644,81 @@ async function* streamChunks( yield { type: 'stop', stopReason, usage: state.usage }; } +// --- Generative media (Imagen, sync) --------------------------------------------------------- + +/** + * Imagen sync image generation (1.AH A2) — mirrors the OpenAI image arm: ONE artifact per call, base64 + * in flight (the engine de-inlines it to a `media://` handle at the durable boundary — I3). `count > 1` + * is rejected loud rather than billing N and dropping N-1 (a multi-image `MediaGenResult` is a deferred + * seam amend). A safety-filtered candidate (`raiFilteredReason`, no `image`) maps to `content_filter`, + * reusing the one failure vocabulary instead of a generic empty-bytes error. `numberOfImages` is pinned + * to 1 AFTER any `providerOptions` spread so an author-supplied count can't smuggle past the guard. + */ +async function geminiGenerateImage( + transport: GeminiTransport, + req: MediaGenRequest, + key: string, +): Promise { + if (req.count !== undefined && req.count > 1) { + throw new LlmProviderError( + makeLlmError({ + provider: PROVIDER, + kind: 'bad_request', + message: `Gemini image generateMedia delivers a single image; count ${String(req.count)} > 1 is not supported on the SYNC seam`, + }), + ); + } + let response: GeminiImageResponse; + try { + response = await transport.generateImages( + { + model: req.model, + prompt: req.prompt, + // numberOfImages pinned AFTER the providerOptions spread (the single-artifact seam); abortSignal + // threaded so a run cancel reaches the in-flight Imagen call (parity with the generate() path). + config: { + ...(req.providerOptions ?? {}), + numberOfImages: 1, + ...(isAbortSignal(req.signal) ? { abortSignal: req.signal } : {}), + }, + }, + key, + ); + } catch (err) { + throw new LlmProviderError(geminiErrorToLlmError(err)); + } + const first = response.generatedImages?.[0]; + const b64 = first?.image?.imageBytes; + if (b64 === undefined || b64.length === 0) { + // A dropped candidate carries the RAI reason but no bytes — surface it as content_filter, not the + // generic no-data path below (which is a genuine provider-contract violation). + if (first?.raiFilteredReason !== undefined && first.raiFilteredReason.length > 0) { + throw new LlmProviderError( + makeLlmError({ + provider: PROVIDER, + kind: 'content_filter', + message: `Gemini image generation was filtered: ${first.raiFilteredReason}`, + }), + ); + } + throw new LlmProviderError( + makeLlmError({ + provider: PROVIDER, + kind: 'bad_request', + message: 'Gemini image generation returned no base64 image data', + }), + ); + } + // The vendor reports the ACTUAL bytes' MIME; default to png (Imagen's default) when absent. The bare + // MIME is validated downstream by MediaPartSchema before anything durable is written. (`first?.` only + // narrows the type — the non-empty-b64 guard above already proves `first` exists at runtime.) + const mimeType = first?.image?.mimeType ?? 'image/png'; + return { + media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, + raw: response, + }; +} + // --- The adapter ----------------------------------------------------------------------------- /** Dependencies the conformance replayer / tests inject (the network transport). */ @@ -657,6 +764,19 @@ export function createGeminiAdapter(deps: GeminiAdapterDeps = {}): LlmProvider { assertNoStreamingMediaOutput(PROVIDER, req); // media-out is generate()-only; streaming triad deferred (ADR-0046 §4) return streamChunks(transport, buildGeminiRequest(req), key); }, + async generateMedia(req: MediaGenRequest, key: string): Promise { + // SYNC separate-endpoint generation, dispatched by modality (ADR-0045 §1): image → Imagen + // (generateImages). Audio is not a Gemini generative surface; video is the ASYNC Veo path (1.AH A4) + // — both fail loud with a typed capability error, never a silent drop. + if (req.modality === 'image') { + return geminiGenerateImage(transport, req, key); + } + throw new UnsupportedCapabilityError( + PROVIDER, + 'media', + `Gemini generateMedia has no SYNC surface for '${req.modality}' (image is Imagen; video is the async Veo path)`, + ); + }, }; } diff --git a/packages/llm/src/conformance/fixtures/gemini.ts b/packages/llm/src/conformance/fixtures/gemini.ts index 97992d5c..642286d5 100644 --- a/packages/llm/src/conformance/fixtures/gemini.ts +++ b/packages/llm/src/conformance/fixtures/gemini.ts @@ -85,6 +85,16 @@ const structuredOutput = JSON.stringify({ usageMetadata: { promptTokenCount: 8, candidatesTokenCount: 4, totalTokenCount: 12 }, }); +// A recorded Imagen (generateImages) reply — a single base64 image, the SYNC generative seam path +// (1.AH A2). The Gemini conformance harness replays it through the fake transport's generateImages, so +// it drives the real fold/normalization (generatedImages[].image.imageBytes → a base64 `media` part) +// with no vendor import. The base64 matches the OpenAI fixture for cross-adapter parity. +const imageGenerate = JSON.stringify({ + generatedImages: [ + { image: { imageBytes: 'aGVsbG8tY29uZm9ybWFuY2UtaW1hZ2U=', mimeType: 'image/png' } }, + ], +}); + export const GEMINI_FIXTURES: ConformanceFixtures = { textGenerate: { status: 200, body: textResponse }, toolGenerate: { status: 200, body: toolResponse }, @@ -94,6 +104,7 @@ export const GEMINI_FIXTURES: ConformanceFixtures = { streamError: { status: 503, body: overloadedError }, reasoningStream: { status: 200, body: reasoningStream }, structuredOutput: { status: 200, body: structuredOutput }, + mediaGenerate: { status: 200, body: imageGenerate }, toolLoop: { turn1: { status: 200, body: toolResponse }, turn2: { status: 200, body: textResponse }, @@ -107,5 +118,6 @@ export const GEMINI_FIXTURES: ConformanceFixtures = { streamErrorKind: 'overloaded', reasoningStream: { text: 'let me think', reasoningTokens: 2 }, structuredOutput: { text: '{"ok":true}' }, + mediaGenerate: { mimeType: 'image/png', data: 'aGVsbG8tY29uZm9ybWFuY2UtaW1hZ2U=' }, }, }; diff --git a/packages/llm/src/conformance/gemini.conformance.test.ts b/packages/llm/src/conformance/gemini.conformance.test.ts index 83e937c1..19c8b81f 100644 --- a/packages/llm/src/conformance/gemini.conformance.test.ts +++ b/packages/llm/src/conformance/gemini.conformance.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from 'vitest'; import { createGeminiAdapter, geminiAdapter, + type GeminiImageResponse, type GeminiResponse, type GeminiTransport, } from '../adapters/gemini.js'; @@ -24,6 +25,10 @@ const isGeminiResponse = (value: unknown): value is GeminiResponse => typeof value === 'object' && value !== null && !Array.isArray(value); const isGeminiResponseArray = (value: unknown): value is GeminiResponse[] => Array.isArray(value) && value.every(isGeminiResponse); +// GeminiImageResponse's only field is optional, so any object is structurally valid; the helper reads +// generatedImages defensively. An object/non-array check fails loud on a malformed fixture. +const isGeminiImageResponse = (value: unknown): value is GeminiImageResponse => + typeof value === 'object' && value !== null && !Array.isArray(value); // Gemini has no `fetch` hook, so the conformance harness replays at the transport level: a recorded // SDK-output JSON (single response or an array of streamed responses) is parsed and served through a @@ -60,6 +65,14 @@ const makeReplayAdapter: MakeReplayAdapter = (recorded) => { ? Promise.resolve(toAsyncIterable(parsed)) : Promise.reject(new Error('replay fixture is not a GeminiResponse[] array')); }, + generateImages: () => { + const current = nextRecording(); + if (current.status >= 400) return rejection(current.status); + const parsed: unknown = JSON.parse(current.body); + return isGeminiImageResponse(parsed) + ? Promise.resolve(parsed) + : Promise.reject(new Error('replay fixture is not a GeminiImageResponse object')); + }, }; return createGeminiAdapter({ transport }); }; From 5b9676932b71139ea41d1ae3e29f32d97bc4e027 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 13:35:46 +0300 Subject: [PATCH 06/21] =?UTF-8?q?fix(llm):=201.AH=20A2=20=E2=80=94=20Gemin?= =?UTF-8?q?i=20Imagen=20Opus-review=20follow-ups?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the two verified nits from the A2 Opus adversarial review (0 blocker/high; the raw:response I3 question was assessed and confirmed a defensible parity choice — raw is a sink-stripped diagnostic by contract, types.ts:442-444): - doc(nit): document the content_filter taxonomy in geminiGenerateImage — only the in-body raiFilteredReason channel maps to content_filter; an HTTP-level Imagen safety 4xx normalizes to bad_request (Gemini's ApiError carries no structured content-policy code, unlike OpenAI's isContentPolicyCode, so a status-only block has no reliable signal to key on; message-sniffing would be fragile). A documented, behaviorally-inert divergence from the OpenAI sibling. - test(nit): assert a present-but-empty raiFilteredReason falls through to bad_request, not content_filter (the previously-uncovered guard sub-branch). Toolchain 16/16 green; prettier clean; Leakwatch 0. Refs: ADR-0045, ADR-0046, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 9 +++++++++ packages/llm/src/adapters/gemini.ts | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index 0260f7cd..a81ec39d 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -870,6 +870,15 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { }); }); + it('does NOT mistake a present-but-empty raiFilteredReason for content_filter (→ bad_request)', async () => { + // The content_filter guard requires a NON-empty reason; an empty one is a contract violation, not a + // safety block, so it must fall through to the generic no-data bad_request. + const transport = fakeImageTransport({ generatedImages: [{ raiFilteredReason: '' }] }); + await expect(genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k')).rejects.toMatchObject({ + llmError: { kind: 'bad_request' }, + }); + }); + it('surfaces a transport rejection as a classified LlmProviderError', async () => { const transport: GeminiTransport = { generate: () => Promise.reject(new Error('unused')), diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index ab02c046..1161eed6 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -653,6 +653,13 @@ async function* streamChunks( * seam amend). A safety-filtered candidate (`raiFilteredReason`, no `image`) maps to `content_filter`, * reusing the one failure vocabulary instead of a generic empty-bytes error. `numberOfImages` is pinned * to 1 AFTER any `providerOptions` spread so an author-supplied count can't smuggle past the guard. + * + * Taxonomy note: only the IN-BODY safety channel (`raiFilteredReason` on a 200) maps to `content_filter`. + * An HTTP-level Imagen safety rejection (a moderation 4xx) routes through `geminiErrorToLlmError` → + * `kindFromHttpStatus` and normalizes to `bad_request` — Gemini's `ApiError` carries no structured + * content-policy code (unlike OpenAI's `isContentPolicyCode`), so a status-only block has no reliable + * content-policy signal to key on; message-sniffing would be fragile. The OpenAI sibling can map both + * paths; this is a documented, behaviorally-inert divergence (both kinds are fatal/non-retryable). */ async function geminiGenerateImage( transport: GeminiTransport, From 669dcbc78bd6b50325a76cb395fcf2e729650fce Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 13:54:37 +0300 Subject: [PATCH 07/21] =?UTF-8?q?fix(llm):=201.AH=20A2=20=E2=80=94=20Gemin?= =?UTF-8?q?i=20Imagen=20Sonnet-review=20follow-ups=20(SSRF=20blocker)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the A2 Sonnet second-pass adversarial review (1 blocker, 1 medium, 3 low, 3 nit — the SSRF blocker was missed entirely by the first Opus pass): - BLOCKER (B1) — SSRF / API-key exfiltration on the Imagen path: geminiGenerateImage spread providerOptions into the Imagen config WITHOUT stripTransportKeys, so a caller-supplied httpOptions.baseUrl could redirect egress + the live API key to an attacker host (the text generate() path strips it precisely for this reason). Fix: apply stripTransportKeys to the spread; extend stripTransportKeys to also drop an author-injected abortSignal (L1 — the signal is the engine's to set, never providerOptions), hardening both paths. + an SSRF regression test. - MEDIUM (M1) — vendor MIME not parameter-stripped: a parameterized Imagen MIME (image/png; q=1.0) reached MediaStore.put / media_objects.mimeType un-canonicalized (no parameter-stripping CHECK in the production de-inline path). Fix: strip ;-parameters to the bare MIME, mirroring the generate() arm (mapContent). + test. - LOW: assert model threading (L2); cover an entirely-absent generatedImages field (L3). NIT: strengthen isGeminiImageResponse to reject a non-array generatedImages (N2); assert raw is the exact transport response (N3). Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0046, ADR-0011, ADR-0043 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 45 +++++++++++++++++-- packages/llm/src/adapters/gemini.ts | 26 ++++++----- .../conformance/gemini.conformance.test.ts | 13 ++++-- 3 files changed, 67 insertions(+), 17 deletions(-) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index a81ec39d..ed2cb020 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -800,18 +800,20 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { const B64 = 'aGVsbG8taW1hZ2Vu'; // "hello-imagen" it('normalizes a generated image to a base64 media part (no jobId, raw retained)', async () => { - const transport = fakeImageTransport({ + const fixture: GeminiImageResponse = { generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], - }); + }; + const transport = fakeImageTransport(fixture); const result = await genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k'); expect(result.jobId).toBeUndefined(); // SYNC arm expect(result.media?.type).toBe('media'); expect(result.media?.mimeType).toBe('image/png'); expect(result.media?.source).toEqual({ kind: 'base64', data: B64 }); - expect(result.raw).toBeDefined(); // internal diagnostic — sinks strip it (I3) + expect(result.raw).toBe(fixture); // internal diagnostic (the exact transport response) — sinks strip it (I3) // The single-artifact seam: numberOfImages is pinned to 1, and the prompt/model are threaded. expect(transport.lastImageRequest?.config['numberOfImages']).toBe(1); expect(transport.lastImageRequest?.prompt).toBe(IMG_REQ.prompt); + expect(transport.lastImageRequest?.model).toBe(IMG_REQ.model); }); it('defaults the MIME to image/png when the vendor omits it', async () => { @@ -820,6 +822,36 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { expect(result.media?.mimeType).toBe('image/png'); }); + it('strips MIME parameters to the canonical bare MIME (image/png; q=1.0 → image/png)', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png; q=1.0' } }], + }); + const result = await genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k'); + expect(result.media?.mimeType).toBe('image/png'); + }); + + it('strips a caller-supplied httpOptions from the Imagen config (SSRF: no baseUrl/key redirect)', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], + }); + await genMedia( + createGeminiAdapter({ transport }), + { + ...IMG_REQ, + providerOptions: { + httpOptions: { baseUrl: 'https://attacker.example' }, + abortSignal: new AbortController().signal, + aspectRatio: '16:9', + }, + }, + 'k', + ); + // httpOptions (SSRF) + an author-injected abortSignal are stripped; benign knobs survive. + expect(transport.lastImageRequest?.config['httpOptions']).toBeUndefined(); + expect(transport.lastImageRequest?.config['abortSignal']).toBeUndefined(); + expect(transport.lastImageRequest?.config['aspectRatio']).toBe('16:9'); + }); + it('threads providerOptions into config but pins numberOfImages to 1 (count can not be smuggled)', async () => { const transport = fakeImageTransport({ generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], @@ -870,6 +902,13 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { }); }); + it('maps an entirely-absent generatedImages field to bad_request (optional-chain short-circuit)', async () => { + const transport = fakeImageTransport({}); + await expect(genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k')).rejects.toMatchObject({ + llmError: { kind: 'bad_request' }, + }); + }); + it('does NOT mistake a present-but-empty raiFilteredReason for content_filter (→ bad_request)', async () => { // The content_filter guard requires a NON-empty reason; an empty one is a contract violation, not a // safety block, so it must fall through to the generic no-data bad_request. diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 1161eed6..ec10eb39 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -80,12 +80,13 @@ function isPromptBlocked(promptFeedback: { blockReason?: string } | undefined): return reason !== undefined && reason !== 'BLOCKED_REASON_UNSPECIFIED'; } -/** Remove transport-level keys that the SDK exposes for URL/header override — SSRF guard. */ +/** Remove transport-level keys the SDK exposes for URL/header/cancellation override — SSRF + signal guard. + * `httpOptions.baseUrl`/`headers` would redirect egress (and the API key) to an arbitrary host; an + * author-supplied `abortSignal` is the engine's to set (from `req.signal`), never `providerOptions`. */ function stripTransportKeys(opts: Record): Record { const rest: Record = {}; for (const [k, v] of Object.entries(opts)) { - // `httpOptions.baseUrl`/`headers` would redirect egress (and the API key) to an arbitrary host. - if (k !== 'httpOptions') { + if (k !== 'httpOptions' && k !== 'abortSignal') { rest[k] = v; } } @@ -681,10 +682,12 @@ async function geminiGenerateImage( { model: req.model, prompt: req.prompt, - // numberOfImages pinned AFTER the providerOptions spread (the single-artifact seam); abortSignal - // threaded so a run cancel reaches the in-flight Imagen call (parity with the generate() path). + // stripTransportKeys: a caller-supplied httpOptions.baseUrl would redirect egress + the API key to + // an arbitrary host (SSRF) — strip it before the spread, exactly as the generate() path does. + // numberOfImages pinned AFTER the spread (the single-artifact seam); abortSignal threaded so a run + // cancel reaches the in-flight Imagen call (parity with the generate() path). config: { - ...(req.providerOptions ?? {}), + ...stripTransportKeys(req.providerOptions ?? {}), numberOfImages: 1, ...(isAbortSignal(req.signal) ? { abortSignal: req.signal } : {}), }, @@ -716,10 +719,13 @@ async function geminiGenerateImage( }), ); } - // The vendor reports the ACTUAL bytes' MIME; default to png (Imagen's default) when absent. The bare - // MIME is validated downstream by MediaPartSchema before anything durable is written. (`first?.` only - // narrows the type — the non-empty-b64 guard above already proves `first` exists at runtime.) - const mimeType = first?.image?.mimeType ?? 'image/png'; + // The vendor reports the ACTUAL bytes' MIME; default to png (Imagen's default) when absent. Strip any + // `;`-parameters (e.g. `image/png; q=1.0`) to the canonical bare MIME — the production de-inline path + // (MediaStore.put / media_objects.mimeType) has no parameter-stripping CHECK, and the generate() arm + // (mapContent) strips identically. (`first?.` only narrows the type — the non-empty-b64 guard above + // already proves `first` exists at runtime.) + const rawMime = first?.image?.mimeType ?? ''; + const mimeType = rawMime.split(';')[0]?.trim() || 'image/png'; return { media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, raw: response, diff --git a/packages/llm/src/conformance/gemini.conformance.test.ts b/packages/llm/src/conformance/gemini.conformance.test.ts index 19c8b81f..775e0fb1 100644 --- a/packages/llm/src/conformance/gemini.conformance.test.ts +++ b/packages/llm/src/conformance/gemini.conformance.test.ts @@ -25,10 +25,15 @@ const isGeminiResponse = (value: unknown): value is GeminiResponse => typeof value === 'object' && value !== null && !Array.isArray(value); const isGeminiResponseArray = (value: unknown): value is GeminiResponse[] => Array.isArray(value) && value.every(isGeminiResponse); -// GeminiImageResponse's only field is optional, so any object is structurally valid; the helper reads -// generatedImages defensively. An object/non-array check fails loud on a malformed fixture. -const isGeminiImageResponse = (value: unknown): value is GeminiImageResponse => - typeof value === 'object' && value !== null && !Array.isArray(value); +// GeminiImageResponse's only field (generatedImages) is optional; reject non-objects, arrays, and a +// present-but-non-array generatedImages so a malformed fixture fails the guard rather than the fold. +const isGeminiImageResponse = (value: unknown): value is GeminiImageResponse => { + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + return false; + } + const gen = (value as { generatedImages?: unknown }).generatedImages; + return gen === undefined || Array.isArray(gen); +}; // Gemini has no `fetch` hook, so the conformance harness replays at the transport level: a recorded // SDK-output JSON (single response or an array of streamed responses) is parsed and served through a From 4fca8a7c2deea077e033adb42e3d8eeb703816ad Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 14:18:17 +0300 Subject: [PATCH 08/21] =?UTF-8?q?feat(llm):=201.AH=20A3=20=E2=80=94=20Open?= =?UTF-8?q?AI=20Sora=20async=20video=20generateMedia/pollMediaJob?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the OpenAI adapter's ASYNC video LRO behind the seam (ADR-0045 §3/§7), the first real async generative adapter (design validated by a dedicated research pass): - generateMedia(video) → client.videos.create → ALWAYS returns an opaque { jobId } (even on instant completion — completion is the engine poll loop's job; returning { media } would orphan the LRO). raw carries only { id, status }, never bytes. - pollMediaJob(jobId, key, signal) → decode → videos.retrieve → status map: queued/ in_progress → pending(clamped progress); completed → videos.downloadContent → base64 video/mp4 media part (downloaded in the same try as retrieve so a mid- download abort classifies as cancelled); failed → mapVideoCreateError (content- policy code → content_filter, else unknown). A malformed jobId returns a FATAL `failed` (not a throw — a throw is engine-classified retryable and would loop). - Opaque jobId encoding (ADR-0045 §7): rlv-mediajob:1: — a STATELESS reversible bijection (encodeVideoJobId/decodeVideoJobId), NOT an in-memory Map, so a cold-process re-attach (§3) resolves the vendor id with no adapter state. Relavium-namespaced + base64url'd: not the bare vendor op-name/URL, engine never parses it (I1/ADR-0011), the non-secret resource id is safe to persist. NO new ADR (within ADR-0045 §7; rationale recorded in the adapter doc-comment). - providerOptions is NOT spread into the SDK call — only explicit soraSeconds (reject non-{4,8,12}, never round) + soraSize knobs are extracted (no transport-key surface, per the A2-verification sibling-sweep note). - 11 unit tests (encoding round-trip/opacity, foreign-token decode, generateMedia always-jobId, duration reject, full status mapping, content_filter/null-error, malformed-token, abort threading, DeepSeek guards); the generative-seam conformance caveat + the llm-provider-seam.md Wired list updated. Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011, ADR-0031 Co-Authored-By: Claude Opus 4.8 --- .../shared-core/llm-provider-seam.md | 12 +- packages/llm/src/adapters/openai.test.ts | 212 +++++++++++++++- packages/llm/src/adapters/openai.ts | 240 +++++++++++++++++- .../generative-seam.conformance.test.ts | 9 +- 4 files changed, 447 insertions(+), 26 deletions(-) diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index 32ebda5b..ba7eb2b2 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -384,15 +384,17 @@ managed mode) are recorded in the ADR — this section is the dry shape referenc crosses the seam); `failed` carries the existing classified `LlmError` (content-policy → `content_filter`). **Wired (1.AG Sections C/D + 1.AH adapters):** `generateMedia` SYNC — the OpenAI adapter implements gpt-image-1 image generation (`images.generate` → base64 `media`) and - **OpenAI-TTS audio** (`audio.speech` binary → base64 + `response_format`↔MIME, 1.AH A1); a + **OpenAI-TTS audio** (`audio.speech` binary → base64 + `response_format`↔MIME, 1.AH A1), and the + **Gemini-Imagen** adapter implements image generation (`generateImages` → base64 `media`, 1.AH A2); a `media_surface: 'generative'` agent node routes here instead of the inline `generate()`/`stream()` (the engine resolves the per-model surface). The ASYNC `pollMediaJob` poll/checkpoint/resume/cancel loop is WIRED in the engine (Section D — `media_job:submitted` park, the derived `pendingMediaJobs` slot, re-attach-on-resume, a host-timer poll cadence, deadline→retryable-timeout, - cancel→abort→terminal sweep; failed→`content_filter`), proven against a conforming stub async - provider. The remaining work is **1.AH**: the Gemini-Imagen sync + Sora/Veo async - `generateMedia(→jobId)`/`pollMediaJob` adapters, the per-model `media_surface` host lookup, and - verified generative pricing rows. + cancel→abort→terminal sweep; failed→`content_filter`), and the **OpenAI/Sora** async adapter + implements it (`videos.create` → an opaque `jobId`, `pollMediaJob` → `videos.retrieve`/`downloadContent`, + 1.AH A3) — the opaque jobId reversibly encodes the vendor id (`rlv-mediajob:1:`) so a + cold-process re-attach resolves it statelessly (ADR-0045 §7). The remaining work is **1.AH**: the + **Veo** async adapter (A4), the per-model `media_surface` host lookup, and verified generative pricing rows. ```ts // Seam shape (A5; ADR-0045) — behavior WIRED at 1.AG (sync generateMedia Section C, async poll loop Section D). diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 4b65c0f0..e77f3e37 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -1,18 +1,25 @@ import { APIConnectionError, APIConnectionTimeoutError, APIError, APIUserAbortError } from 'openai'; import { describe, expect, it } from 'vitest'; +import type { AbortSignalLike } from '@relavium/shared'; + import { InvalidBaseUrlError, UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; -import type { - LlmProvider, - LlmRequest, - MediaGenRequest, - MediaGenResult, - StreamChunk, +import { + MediaGenResultSchema, + MediaJobStatusSchema, + type LlmProvider, + type LlmRequest, + type MediaGenRequest, + type MediaGenResult, + type MediaJobStatus, + type StreamChunk, } from '../types.js'; import { createOpenAiAdapter, + decodeVideoJobId, deepseekAdapter, + encodeVideoJobId, mapContent, mapStopReason, mapUsage, @@ -34,6 +41,19 @@ function genMedia( ); } +/** Call the adapter's optional `pollMediaJob` via `?.()` (same unbound-method-safe pattern as genMedia). */ +function pollMedia( + adapter: LlmProvider, + jobId: string, + key: string, + signal?: AbortSignalLike, +): Promise { + return ( + adapter.pollMediaJob?.(jobId, key, signal) ?? + Promise.reject(new Error('adapter implements no pollMediaJob')) + ); +} + const isRecord = (value: unknown): value is Record => typeof value === 'object' && value !== null && !Array.isArray(value); @@ -391,13 +411,7 @@ describe('OpenAI-compatible adapter', () => { ).rejects.toMatchObject({ llmError: { kind: 'bad_request', retryable: false } }); }); - it('generateMedia rejects OpenAI video (no sync surface) + DeepSeek any modality with a typed capability error', async () => { - const oai = createOpenAiAdapter({ - fetch: () => Promise.reject(new Error('must fail fast before any egress')), - }); - await expect( - genMedia(oai, { model: 'sora-2', prompt: 'x', modality: 'video' }, 'k'), - ).rejects.toThrowError(UnsupportedCapabilityError); + it('generateMedia rejects DeepSeek any modality with a typed capability error (OpenAI video is the async Sora arm — tested separately)', async () => { const ds = createOpenAiAdapter({ providerId: 'deepseek', fetch: () => Promise.reject(new Error('must fail fast before any egress')), @@ -405,6 +419,9 @@ describe('OpenAI-compatible adapter', () => { await expect( genMedia(ds, { model: 'm', prompt: 'x', modality: 'image' }, 'k'), ).rejects.toThrowError(UnsupportedCapabilityError); + await expect( + genMedia(ds, { model: 'm', prompt: 'x', modality: 'video' }, 'k'), + ).rejects.toThrowError(UnsupportedCapabilityError); }); it('generateMedia maps a no-data image response to a typed bad_request LlmProviderError', async () => { @@ -1473,3 +1490,172 @@ describe('OpenAI-compatible adapter — truncation + refusal normalization', () expect(result.stopReason).toBe('content_filter'); }); }); + +describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMediaJob, 1.AH A3)', () => { + const VIDEO_REQ: MediaGenRequest = { + model: 'sora-2', + prompt: 'a wave breaking on a beach', + modality: 'video', + durationSeconds: 4, + }; + + /** Route the SDK's videos.* HTTP calls: GET …/content → the MP4 bytes; create(POST)/retrieve(GET) → a + * Video JSON. */ + function soraFetch( + video: Record, + bytes = 'FAKE-MP4-BYTES', + ): (input: string | URL | Request, init?: RequestInit) => Promise { + return (input) => { + const url = typeof input === 'string' ? input : input instanceof URL ? input.href : input.url; + if (url.endsWith('/content')) { + return Promise.resolve( + new Response(bytes, { status: 200, headers: { 'content-type': 'video/mp4' } }), + ); + } + return Promise.resolve( + new Response(JSON.stringify(video), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ); + }; + } + + it('encodeVideoJobId mints a Relavium-opaque token that is NOT the raw vendor id (round-trips via decode)', () => { + const jobId = encodeVideoJobId('video_abc123'); + expect(jobId.startsWith('rlv-mediajob:1:')).toBe(true); + expect(jobId).not.toContain('video_abc123'); // base64url-encoded — the raw vendor id never appears verbatim + expect(decodeVideoJobId(jobId)).toBe('video_abc123'); + }); + + it('decodeVideoJobId returns undefined for foreign/malformed tokens (never throws)', () => { + expect( + decodeVideoJobId('rlv-mediajob:2:' + Buffer.from('x').toString('base64url')), + ).toBeUndefined(); // future version + expect(decodeVideoJobId('vendor-op-7f3a')).toBeUndefined(); // non-Relavium + expect(decodeVideoJobId('')).toBeUndefined(); + expect(decodeVideoJobId('rlv-mediajob:1:')).toBeUndefined(); // empty payload + }); + + it('generateMedia (video) ALWAYS returns an opaque jobId (no media), schema-valid, even on instant completion', async () => { + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }), + }); + const result = await genMedia(adapter, VIDEO_REQ, 'k'); + expect(result.media).toBeUndefined(); // ASYNC arm — completion is the engine poll loop's job + expect(result.jobId).toBe(encodeVideoJobId('video_xyz')); + expect(MediaGenResultSchema.safeParse(result).success).toBe(true); + expect(result.raw).toEqual({ id: 'video_xyz', status: 'completed' }); // no bytes in raw (I3) + }); + + it('generateMedia (video) rejects a non-{4,8,12} durationSeconds with a typed bad_request before egress', async () => { + let called = false; + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: (input, init) => { + called = true; + return soraFetch({ id: 'v', status: 'queued' })(input, init); + }, + }); + await expect( + genMedia(adapter, { ...VIDEO_REQ, durationSeconds: 5 }, 'k'), + ).rejects.toMatchObject({ llmError: { kind: 'bad_request' } }); + expect(called).toBe(false); // rejected before any SDK call + }); + + it('pollMediaJob maps queued → pending and in_progress → pending with clamped 0-1 progress', async () => { + const jobId = encodeVideoJobId('video_xyz'); + const queued = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }), + }); + expect(await pollMedia(queued, jobId, 'k')).toEqual({ state: 'pending' }); + + const inProgress = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'in_progress', progress: 50 }), + }); + expect(await pollMedia(inProgress, jobId, 'k')).toEqual({ state: 'pending', progress: 0.5 }); + }); + + it('pollMediaJob completed → downloads the MP4 and returns a base64 video/mp4 media part (done)', async () => { + const jobId = encodeVideoJobId('video_xyz'); + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, 'MP4-BYTES'), + }); + const status = await pollMedia(adapter, jobId, 'k'); + expect(MediaJobStatusSchema.safeParse(status).success).toBe(true); + expect(status).toEqual({ + state: 'done', + media: { + type: 'media', + mimeType: 'video/mp4', + source: { kind: 'base64', data: Buffer.from('MP4-BYTES').toString('base64') }, + }, + }); + }); + + it('pollMediaJob failed → content_filter for a content-policy code, unknown for a null error', async () => { + const jobId = encodeVideoJobId('video_xyz'); + const blocked = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ + id: 'video_xyz', + status: 'failed', + progress: 0, + error: { code: 'content_policy_violation', message: 'blocked' }, + }), + }); + expect(await pollMedia(blocked, jobId, 'k')).toMatchObject({ + state: 'failed', + error: { kind: 'content_filter', retryable: false }, + }); + + const nullErr = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'failed', progress: 0, error: null }), + }); + expect(await pollMedia(nullErr, jobId, 'k')).toMatchObject({ + state: 'failed', + error: { kind: 'unknown' }, + }); + }); + + it('pollMediaJob returns a FATAL failed (not a throw) for an unrecognized jobId token', async () => { + let called = false; + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: (input, init) => { + called = true; + return soraFetch({ id: 'v', status: 'queued' })(input, init); + }, + }); + const status = await pollMedia(adapter, 'not-a-relavium-token', 'k'); + expect(status).toMatchObject({ state: 'failed', error: { kind: 'bad_request' } }); + expect(called).toBe(false); // never reached the SDK — decode failed first + }); + + it('pollMediaJob threads the AbortSignal into the SDK request', async () => { + let sawSignal = false; + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: (input, init) => { + sawSignal = init?.signal instanceof AbortSignal; + return soraFetch({ id: 'video_xyz', status: 'queued' })(input, init); + }, + }); + const controller = new AbortController(); + await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k', controller.signal); + expect(sawSignal).toBe(true); + }); + + it('DeepSeek has no async video: generateMedia(video) → capability error, pollMediaJob → failed', async () => { + await expect(genMedia(deepseekAdapter, VIDEO_REQ, 'k')).rejects.toBeInstanceOf( + UnsupportedCapabilityError, + ); + const status = await pollMedia(deepseekAdapter, encodeVideoJobId('video_xyz'), 'k'); + expect(status).toMatchObject({ state: 'failed' }); + }); +}); diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 96ff968f..e9cb06ce 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -8,6 +8,7 @@ import OpenAI, { import { isPrivateOrLocalHost, urlHasCredentials, + type AbortSignalLike, type ContentPart, mediaModalityOf, type StopReason, @@ -27,6 +28,7 @@ import type { LlmResult, MediaGenRequest, MediaGenResult, + MediaJobStatus, MediaUnitsEntry, ProviderId, StreamChunk, @@ -927,21 +929,53 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { ); } const client = createClient(key); - // SYNC separate-endpoint generation, dispatched by modality (1.AG/1.AH, ADR-0045 §1): image → - // gpt-image-1 (images.generate); audio → TTS (audio.speech). Video is the ASYNC Sora path (a separate - // section) — not a sync surface here. + // Separate-endpoint generation, dispatched by modality (1.AG/1.AH, ADR-0045 §1): image → gpt-image-1 + // (images.generate, SYNC); audio → TTS (audio.speech, SYNC); video → Sora (videos.create, ASYNC LRO — + // returns an opaque jobId the engine polls via pollMediaJob below). if (req.modality === 'image') { return openAiGenerateImage(client, req, providerId); } if (req.modality === 'audio') { return openAiGenerateSpeech(client, req, providerId); } + if (req.modality === 'video') { + return openAiGenerateVideo(client, req, providerId); + } + // Exhaustiveness: MEDIA_BILLED_MODALITIES is image|audio|video, so `modality` is `never` here. A new + // member makes this assignment a COMPILE error — a future modality fails at build, never silently at + // runtime; the throw is the runtime backstop. + const unhandled: never = req.modality; throw new UnsupportedCapabilityError( providerId, 'media', - `OpenAI generateMedia has no SYNC surface for '${req.modality}' (video is the async Sora path)`, + `OpenAI generateMedia has no surface for modality '${String(unhandled)}'`, ); }, + /** + * Poll one async media job (Sora video LRO, 1.AH A3, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * The engine drives this loop and re-delivers the persisted opaque jobId on resume (re-attach, §3); the + * adapter recovers the vendor video id by DECODING the token (stateless bijection — no instance Map — so + * a cold-process restart resolves it). `signal` aborts the in-flight poll/download so a run cancel reaches + * the open request, not just the next schedule. + */ + async pollMediaJob( + jobId: string, + key: string, + signal?: AbortSignalLike, + ): Promise { + if (providerId !== 'openai') { + // DeepSeek (same adapter, different baseURL) has no async media jobs. + return { + state: 'failed', + error: makeLlmError({ + provider: providerId, + kind: 'unknown', + message: `${providerId} has no async media jobs (only OpenAI/Sora is wired)`, + }), + }; + } + return pollMediaJobSora(createClient(key), jobId, providerId, signal); + }, }; } @@ -1048,6 +1082,204 @@ async function openAiGenerateSpeech( }; } +// --- Generative media: Sora video (ASYNC LRO, 1.AH A3, ADR-0045) ----------------------------- + +/** + * The opaque media-job id namespace+version. The engine persists this token in the durable + * `media_job:submitted` event and re-delivers it to `pollMediaJob` on resume (ADR-0045 §3 re-attach, never + * re-submit). The adapter has NO durable store and is rebuilt each process, so the vendor video id is + * reversibly ENCODED into the token (base64url so any vendor id is `:`-split-safe) rather than held in an + * in-memory `Map` — this IS the "vendor↔opaque map internal" of ADR-0045 §7, realized as a STATELESS + * bijection. The token is adapter-minted + Relavium-namespaced (NOT the bare vendor op-name / poll-URL), + * the engine never parses it (I1/[ADR-0011]), and the (non-secret) resource id is safe to persist. A future + * format change bumps the version slot (`:2:`) without an engine or schema change. Managed mode (ADR-0015) + * swaps the `pollMediaJob` body and keeps this prefix as the decode convention. + */ +const SORA_JOB_PREFIX = 'rlv-mediajob:1:'; + +/** Mint the opaque jobId from Sora's `Video.id` (base64url-encoded). Exported for the N4 opacity test. */ +export function encodeVideoJobId(vendorId: string): string { + return SORA_JOB_PREFIX + Buffer.from(vendorId, 'utf8').toString('base64url'); +} + +/** + * Recover the vendor `Video.id` from an opaque jobId; `undefined` on any foreign/malformed token (never + * throws — a throw from `pollMediaJob` is engine-classified as retryable and would loop forever on a + * structurally-dead token). Exported for the opacity/round-trip test. + */ +export function decodeVideoJobId(jobId: string): string | undefined { + if (!jobId.startsWith(SORA_JOB_PREFIX)) { + return undefined; // a non-Relavium id or a future rlv-mediajob:2: token + } + const payload = jobId.slice(SORA_JOB_PREFIX.length); + if (payload.length === 0) { + return undefined; + } + const decoded = Buffer.from(payload, 'base64url').toString('utf8'); + return decoded.length === 0 ? undefined : decoded; +} + +/** Sora accepts only 4/8/12s clips — reject anything else loud (never round; cost-integrity, ADR-0045 §5). */ +const SORA_SECONDS: Readonly> = { 4: '4', 8: '8', 12: '12' }; +function soraSeconds(durationSeconds: number | undefined): OpenAI.VideoSeconds { + const n = durationSeconds ?? 4; // Sora's default clip length + const seconds = SORA_SECONDS[n]; + if (seconds === undefined) { + throw new LlmProviderError( + makeLlmError({ + provider: 'openai', + kind: 'bad_request', + message: `Sora durationSeconds must be 4, 8, or 12; got ${String(n)}`, + }), + ); + } + return seconds; +} + +/** Optional Sora frame size from `providerOptions.video.size` (mirrors ttsVoice's `providerOptions.audio` + * nesting); an unrecognized value is dropped so Sora's default applies. */ +function soraSize( + providerOptions: MediaGenRequest['providerOptions'], +): OpenAI.VideoSize | undefined { + const videoOpts = isRecord(providerOptions) ? providerOptions['video'] : undefined; + const size = isRecord(videoOpts) ? videoOpts['size'] : undefined; + return size === '720x1280' || size === '1280x720' || size === '1024x1792' || size === '1792x1024' + ? size + : undefined; +} + +/** + * ASYNC video generation (Sora via `videos.create` → an opaque jobId the engine polls; ADR-0045 §3). ALWAYS + * returns `{ jobId }` — even if Sora reports `status:'completed'` synchronously — so completion is owned by + * the engine's poll loop (returning `{ media }` would put it on the SYNC path and orphan the LRO). `raw` + * carries only a non-byte diagnostic. providerOptions is NOT spread into the SDK call — only the explicit + * size/seconds knobs are extracted (no transport-key surface, unlike the genai config path). + */ +async function openAiGenerateVideo( + client: OpenAI, + req: MediaGenRequest, + providerId: ProviderId, +): Promise { + const seconds = soraSeconds(req.durationSeconds); // throws bad_request if not 4/8/12 + const size = soraSize(req.providerOptions); + let video: OpenAI.Video; + try { + video = await client.videos.create( + { + model: req.model, + prompt: req.prompt, + seconds, + ...(size === undefined ? {} : { size }), + }, + isAbortSignal(req.signal) ? { signal: req.signal } : {}, + ); + } catch (err) { + throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + } + if (video.id.length === 0) { + throw new LlmProviderError( + makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'Sora video creation returned no job id', + }), + ); + } + return { jobId: encodeVideoJobId(video.id), raw: { id: video.id, status: video.status } }; +} + +/** Map a Sora `VideoCreateError` VALUE object (not a thrown `APIError`) to a normalized `LlmError`; a + * content-policy code → `content_filter`, else the fatal `unknown` (ADR-0045 §6). */ +function mapVideoCreateError(err: OpenAI.VideoCreateError | null, provider: ProviderId): LlmError { + if (err === null) { + return makeLlmError({ + provider, + kind: 'unknown', + message: 'Sora video job failed with no error detail', + }); + } + const kind: LlmErrorKind = isContentPolicyCode(err.code) ? 'content_filter' : 'unknown'; + return makeLlmError({ + provider, + kind, + message: err.message, + ...(err.code.length === 0 ? {} : { code: err.code }), + }); +} + +/** + * Poll one Sora job by its opaque jobId (ADR-0045 §3). Decode → `videos.retrieve`; on `completed` download + * the MP4 bytes in the SAME try (a mid-download abort must classify as `cancelled`, never escape raw) → a + * base64 `media` part the engine de-inlines to a `media://` handle (I3 permits base64 IN FLIGHT; the durable + * boundary gets the handle). A malformed jobId returns a FATAL `failed` (not a throw — a throw is engine- + * classified retryable and would loop on a structurally-dead token). + */ +async function pollMediaJobSora( + client: OpenAI, + jobId: string, + providerId: ProviderId, + signal: AbortSignalLike | undefined, +): Promise { + const vendorId = decodeVideoJobId(jobId); + if (vendorId === undefined) { + return { + state: 'failed', + error: makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'unrecognized Sora media job token', + }), + }; + } + let video: OpenAI.Video; + let bytes: Uint8Array | undefined; + try { + video = await client.videos.retrieve(vendorId, isAbortSignal(signal) ? { signal } : {}); + if (video.status === 'completed') { + // downloadContent is a __binaryResponse (raw Response) — the body read happens HERE, so it must be + // inside this try or a mid-download abort/reset would escape unclassified. + const response = await client.videos.downloadContent( + vendorId, + undefined, + isAbortSignal(signal) ? { signal } : {}, + ); + bytes = new Uint8Array(await response.arrayBuffer()); + } + } catch (err) { + throw new LlmProviderError(openaiErrorToLlmError(err, providerId)); + } + switch (video.status) { + case 'queued': + return { state: 'pending' }; + case 'in_progress': + // Sora reports 0-100; the seam expects 0-1. Clamp defends against an out-of-range value that would + // fail the engine's z.number().min(0).max(1) boundary. + return { state: 'pending', progress: Math.min(1, Math.max(0, video.progress / 100)) }; + case 'completed': { + if (bytes === undefined || bytes.length === 0) { + return { + state: 'failed', + error: makeLlmError({ + provider: providerId, + kind: 'bad_request', + message: 'Sora downloadContent returned no video bytes', + }), + }; + } + return { + state: 'done', + media: { + type: 'media', + mimeType: 'video/mp4', + source: { kind: 'base64', data: Buffer.from(bytes).toString('base64') }, + }, + }; + } + case 'failed': + return { state: 'failed', error: mapVideoCreateError(video.error, providerId) }; + } +} + /** The production OpenAI adapter. */ export const openaiAdapter: LlmProvider = createOpenAiAdapter(); diff --git a/packages/llm/src/conformance/generative-seam.conformance.test.ts b/packages/llm/src/conformance/generative-seam.conformance.test.ts index 4c83ca6a..ec1ae6a7 100644 --- a/packages/llm/src/conformance/generative-seam.conformance.test.ts +++ b/packages/llm/src/conformance/generative-seam.conformance.test.ts @@ -7,10 +7,11 @@ import type { CapabilityFlags, LlmProvider, MediaGenRequest, MediaJobStatus } fr /** * The GENERATIVE-surface seam-contract conformance suite (1.AG Section E, A5 / [ADR-0045]). The chat * conformance suite replays recorded vendor responses through each real adapter; the ASYNC generative arm - * (a `generateMedia` that returns an opaque `jobId`, then `pollMediaJob` driving pending→done/failed) has - * NO production adapter yet — the Sora/Veo adapters are deferred to 1.AH. So this suite asserts the SEAM - * CONTRACT itself against a CONFORMING STUB provider: the exact shape any future async generative adapter - * must satisfy — the `MediaGenResult` exactly-one-of refine, an opaque non-empty `jobId`, the discriminated + * (a `generateMedia` that returns an opaque `jobId`, then `pollMediaJob` driving pending→done/failed) is + * wired for OpenAI/Sora at 1.AH A3 (exercised against the real adapter in `openai.test.ts`); the Gemini/Veo + * async adapter remains deferred (1.AH A4). This suite asserts the SEAM CONTRACT itself against a CONFORMING + * STUB provider — the exact shape every async generative adapter must satisfy, independent of any one vendor: + * the `MediaGenResult` exactly-one-of refine, an opaque non-empty `jobId`, the discriminated * `MediaJobStatus` states, and a content-policy refusal classified as the `content_filter` LlmError kind. * (The SYNC arm is covered against the real OpenAI adapter by the per-provider suite's `mediaGenerate` * scenario; the engine-side poll/checkpoint/resume/cancel loop is covered in `@relavium/core`.) From 0b1e7565326ea621546adc1e17ee28cad3870047 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 14:37:49 +0300 Subject: [PATCH 09/21] =?UTF-8?q?fix(llm):=201.AH=20A3=20=E2=80=94=20Sora?= =?UTF-8?q?=20Opus-review=20follow-ups=20(decode=20hardening=20+=20coverag?= =?UTF-8?q?e)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the A3 Opus adversarial review (0 blocker/high; "no behavioral defects" — all 9 confirmed items were a doc overstatement or a coverage gap on correct code): - decodeVideoJobId now round-trip-validates (re-encode === jobId): base64url decode is lenient (drops invalid chars), so a corrupted/non-canonical right-prefix token would otherwise decode to a wrong-but-non-empty vendor id and be polled. The strict check rejects every such token; every minted token still round-trips exactly. - pollMediaJobSora: a defensive `default` arm on the status switch returns a FATAL `unknown` failed instead of falling through to `undefined` (which would break the Promise contract) if the SDK ever returns an out-of-union status. - Tests (the rest were coverage gaps on correct code): assert the create FormData body forwards model/prompt/seconds (videos.create is multipart — capture FormData fields); seconds defaults to 4 + a valid size knob is forwarded; an invalid size is dropped; progress clamps (150→1); completed-but-empty-bytes → bad_request; empty error code → unknown + omitted; the abort signal reaches BOTH retrieve and downloadContent; the non-canonical-token decode rejection. Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011, ADR-0031 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/openai.test.ts | 127 +++++++++++++++++++---- packages/llm/src/adapters/openai.ts | 22 +++- 2 files changed, 127 insertions(+), 22 deletions(-) diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index e77f3e37..0c9cf734 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -1499,15 +1499,40 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe durationSeconds: 4, }; + interface SoraCapture { + createBody?: Record; + readonly signalByCall: Record<'create' | 'retrieve' | 'content', boolean>; + } + type SoraFetch = ((input: string | URL | Request, init?: RequestInit) => Promise) & { + capture: SoraCapture; + }; + /** Route the SDK's videos.* HTTP calls: GET …/content → the MP4 bytes; create(POST)/retrieve(GET) → a - * Video JSON. */ - function soraFetch( - video: Record, - bytes = 'FAKE-MP4-BYTES', - ): (input: string | URL | Request, init?: RequestInit) => Promise { - return (input) => { + * Video JSON. Records the create POST body + per-call AbortSignal presence on `.capture`. */ + function soraFetch(video: Record, bytes = 'FAKE-MP4-BYTES'): SoraFetch { + const capture: SoraCapture = { + signalByCall: { create: false, retrieve: false, content: false }, + }; + const fn = (input: string | URL | Request, init?: RequestInit): Promise => { const url = typeof input === 'string' ? input : input instanceof URL ? input.href : input.url; - if (url.endsWith('/content')) { + const call = url.endsWith('/content') + ? 'content' + : init?.method === 'POST' + ? 'create' + : 'retrieve'; + capture.signalByCall[call] = init?.signal instanceof AbortSignal; + if (call === 'create') { + // videos.create sends multipart/form-data (it supports file uploads), so the body is a FormData, + // not a JSON string — read its text fields. + const body = init?.body; + capture.createBody = + body instanceof FormData + ? Object.fromEntries( + [...body.entries()].map(([k, v]) => [k, typeof v === 'string' ? v : '']), + ) + : parseJsonBody(init); + } + if (call === 'content') { return Promise.resolve( new Response(bytes, { status: 200, headers: { 'content-type': 'video/mp4' } }), ); @@ -1519,6 +1544,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe }), ); }; + return Object.assign(fn, { capture }); } it('encodeVideoJobId mints a Relavium-opaque token that is NOT the raw vendor id (round-trips via decode)', () => { @@ -1535,18 +1561,47 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe expect(decodeVideoJobId('vendor-op-7f3a')).toBeUndefined(); // non-Relavium expect(decodeVideoJobId('')).toBeUndefined(); expect(decodeVideoJobId('rlv-mediajob:1:')).toBeUndefined(); // empty payload + // Round-trip validation: a right-prefix but NON-CANONICAL payload (trailing junk base64url drops) is + // rejected — it would otherwise decode to a wrong-but-non-empty vendor id and be polled. + expect(decodeVideoJobId(encodeVideoJobId('video_x') + '!')).toBeUndefined(); }); it('generateMedia (video) ALWAYS returns an opaque jobId (no media), schema-valid, even on instant completion', async () => { - const adapter = createOpenAiAdapter({ - maxRetries: 0, - fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }), - }); + const fetch = soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); const result = await genMedia(adapter, VIDEO_REQ, 'k'); expect(result.media).toBeUndefined(); // ASYNC arm — completion is the engine poll loop's job expect(result.jobId).toBe(encodeVideoJobId('video_xyz')); expect(MediaGenResultSchema.safeParse(result).success).toBe(true); expect(result.raw).toEqual({ id: 'video_xyz', status: 'completed' }); // no bytes in raw (I3) + // The create body forwards model/prompt/seconds (NOT a spread of providerOptions). + expect(fetch.capture.createBody).toMatchObject({ + model: 'sora-2', + prompt: VIDEO_REQ.prompt, + seconds: '4', + }); + }); + + it('generateMedia (video) defaults seconds to 4 (no durationSeconds) and forwards a valid size knob', async () => { + const fetch = soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); + await genMedia( + adapter, + { + ...VIDEO_REQ, + durationSeconds: undefined, + providerOptions: { video: { size: '1280x720' } }, + }, + 'k', + ); + expect(fetch.capture.createBody).toMatchObject({ seconds: '4', size: '1280x720' }); + }); + + it('generateMedia (video) drops an unrecognized size from the create body (input filter)', async () => { + const fetch = soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); + await genMedia(adapter, { ...VIDEO_REQ, providerOptions: { video: { size: '99x99' } } }, 'k'); + expect(fetch.capture.createBody?.['size']).toBeUndefined(); }); it('generateMedia (video) rejects a non-{4,8,12} durationSeconds with a typed bad_request before egress', async () => { @@ -1577,6 +1632,24 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe fetch: soraFetch({ id: 'video_xyz', status: 'in_progress', progress: 50 }), }); expect(await pollMedia(inProgress, jobId, 'k')).toEqual({ state: 'pending', progress: 0.5 }); + + // Clamp an out-of-range progress to [0,1] (defends the engine's z.number().min(0).max(1) boundary). + const over = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'in_progress', progress: 150 }), + }); + expect(await pollMedia(over, jobId, 'k')).toEqual({ state: 'pending', progress: 1 }); + }); + + it('pollMediaJob completed-but-empty-bytes → a typed bad_request failed (defensive)', async () => { + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, ''), + }); + expect(await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k')).toMatchObject({ + state: 'failed', + error: { kind: 'bad_request' }, + }); }); it('pollMediaJob completed → downloads the MP4 and returns a base64 video/mp4 media part (done)', async () => { @@ -1623,6 +1696,23 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe }); }); + it('pollMediaJob failed with an empty error code → unknown kind, omits the empty code field', async () => { + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ + id: 'video_xyz', + status: 'failed', + progress: 0, + error: { code: '', message: 'x' }, + }), + }); + const status = await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k'); + expect(status).toMatchObject({ state: 'failed', error: { kind: 'unknown' } }); + if (status.state === 'failed') { + expect(status.error.code).toBeUndefined(); // empty code is omitted, not surfaced as '' + } + }); + it('pollMediaJob returns a FATAL failed (not a throw) for an unrecognized jobId token', async () => { let called = false; const adapter = createOpenAiAdapter({ @@ -1637,18 +1727,13 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe expect(called).toBe(false); // never reached the SDK — decode failed first }); - it('pollMediaJob threads the AbortSignal into the SDK request', async () => { - let sawSignal = false; - const adapter = createOpenAiAdapter({ - maxRetries: 0, - fetch: (input, init) => { - sawSignal = init?.signal instanceof AbortSignal; - return soraFetch({ id: 'video_xyz', status: 'queued' })(input, init); - }, - }); + it('pollMediaJob threads the AbortSignal into BOTH retrieve and downloadContent (completed path)', async () => { + const fetch = soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, 'MP4'); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); const controller = new AbortController(); await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k', controller.signal); - expect(sawSignal).toBe(true); + expect(fetch.capture.signalByCall.retrieve).toBe(true); + expect(fetch.capture.signalByCall.content).toBe(true); // the download honors the run cancel too }); it('DeepSeek has no async video: generateMedia(video) → capability error, pollMediaJob → failed', async () => { diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index e9cb06ce..ae5efea4 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -1116,7 +1116,15 @@ export function decodeVideoJobId(jobId: string): string | undefined { return undefined; } const decoded = Buffer.from(payload, 'base64url').toString('utf8'); - return decoded.length === 0 ? undefined : decoded; + // Round-trip validation: ONLY a canonically-minted token decodes back to itself. base64url decoding is + // lenient (it silently drops invalid chars), so a corrupted/non-canonical payload would otherwise decode + // to a wrong-but-non-empty vendor id and be forwarded to videos.retrieve. Re-encoding and comparing + // rejects every such token here, so pollMediaJob never polls a junk vendor id. Every real minted token + // round-trips exactly (encode is the inverse of decode). + if (decoded.length === 0 || encodeVideoJobId(decoded) !== jobId) { + return undefined; + } + return decoded; } /** Sora accepts only 4/8/12s clips — reject anything else loud (never round; cost-integrity, ADR-0045 §5). */ @@ -1277,6 +1285,18 @@ async function pollMediaJobSora( } case 'failed': return { state: 'failed', error: mapVideoCreateError(video.error, providerId) }; + default: + // The SDK types status as a closed union, but a future/unknown status string must NOT fall through + // to `undefined` (which would break the Promise contract). Fail fatal (non-retryable + // `unknown`), mirroring the decode-failure and engine `#applyMediaJobStatus` defaults. + return { + state: 'failed', + error: makeLlmError({ + provider: providerId, + kind: 'unknown', + message: `Sora returned an unrecognized job status '${String(video.status)}'`, + }), + }; } } From d5ede94a008dc3bf6e444c35f98418450633b5f8 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 14:54:55 +0300 Subject: [PATCH 10/21] =?UTF-8?q?fix(llm):=201.AH=20A3=20=E2=80=94=20Sora?= =?UTF-8?q?=20Sonnet-review=20follow-ups=20(abort=20classification=20+=20c?= =?UTF-8?q?overage)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the A3 Sonnet second-pass review (0 blocker/high; 6 distinct items — 2 low, 4 nit): - abort classification (nit, one-canonical-home fix): a NATIVE abort during a binary body read (TTS audio.speech / Sora downloadContent arrayBuffer) throws a DOMException/ Error named 'AbortError', NOT the SDK's APIUserAbortError, so it previously classified as `unknown`. openaiErrorToLlmError now detects an 'AbortError'-named error → `cancelled` — benefits every binary-read arm (A1 TTS + A3 Sora) consistently. + a classifier test. - soraSeconds now takes providerId (was hardcoded 'openai'), matching every other error helper in the file. - comment: the `bytes === undefined` half of the completed-arm guard is the TS narrowing guard (bytes is control-flow-conditionally assigned), not dead code — clarified. - doc: types.ts pollMediaJob JSDoc no longer claims "no Phase-1 adapter implements it" (Sora wired at A3; Veo at A4). - tests (the rest were coverage gaps on correct code): the videos.create AbortSignal is threaded; durationSeconds 8/12 forward correctly; the pollMediaJobSora default arm maps an unrecognized SDK status to a fatal unknown failed. Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/openai.test.ts | 34 ++++++++++++++++++++++++ packages/llm/src/adapters/openai.ts | 20 +++++++++++--- packages/llm/src/types.ts | 4 +-- 3 files changed, 53 insertions(+), 5 deletions(-) diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 0c9cf734..1bcbc5c6 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -760,6 +760,11 @@ describe('openaiErrorToLlmError — classification', () => { expect( openaiErrorToLlmError(new APIConnectionError({ message: 'down' }), 'deepseek'), ).toMatchObject({ kind: 'transport', retryable: true, provider: 'deepseek' }); + // A NATIVE abort (DOMException/Error named 'AbortError') from a binary body read (TTS/Sora + // arrayBuffer) bypasses APIUserAbortError → must still classify as cancelled, not unknown. + expect( + openaiErrorToLlmError(Object.assign(new Error('aborted'), { name: 'AbortError' }), 'openai'), + ).toMatchObject({ kind: 'cancelled', retryable: false }); }); it('classifies an APIError by HTTP status; status-less → unknown', () => { @@ -1604,6 +1609,23 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe expect(fetch.capture.createBody?.['size']).toBeUndefined(); }); + it('generateMedia (video) forwards each valid durationSeconds (8, 12) to the create body', async () => { + for (const n of [8, 12] as const) { + const fetch = soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); + await genMedia(adapter, { ...VIDEO_REQ, durationSeconds: n }, 'k'); + expect(fetch.capture.createBody?.['seconds']).toBe(String(n)); + } + }); + + it('generateMedia (video) threads the AbortSignal into videos.create', async () => { + const fetch = soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }); + const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); + const controller = new AbortController(); + await genMedia(adapter, { ...VIDEO_REQ, signal: controller.signal }, 'k'); + expect(fetch.capture.signalByCall.create).toBe(true); + }); + it('generateMedia (video) rejects a non-{4,8,12} durationSeconds with a typed bad_request before egress', async () => { let called = false; const adapter = createOpenAiAdapter({ @@ -1713,6 +1735,18 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe } }); + it('pollMediaJob maps an unrecognized SDK status to a FATAL unknown failed (the default arm)', async () => { + // The SDK types status as a closed union; a future/unknown status must not fall through to undefined. + const adapter = createOpenAiAdapter({ + maxRetries: 0, + fetch: soraFetch({ id: 'video_xyz', status: 'rendering', progress: 0 }), + }); + expect(await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k')).toMatchObject({ + state: 'failed', + error: { kind: 'unknown' }, + }); + }); + it('pollMediaJob returns a FATAL failed (not a throw) for an unrecognized jobId token', async () => { let called = false; const adapter = createOpenAiAdapter({ diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index ae5efea4..d9a89382 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -331,6 +331,13 @@ export function openaiErrorToLlmError(err: unknown, provider: ProviderId): LlmEr if (err instanceof APIError) { return mapOpenAiApiError(err, provider); } + // A NATIVE abort — a `DOMException`/`Error` named 'AbortError' thrown by `fetch`/`Response.arrayBuffer()` + // when the signal fires DURING a binary body read (TTS `audio.speech`, Sora `downloadContent`). These + // bypass the SDK's `APIUserAbortError` wrapper (the read is outside the SDK request), so classify by name + // → `cancelled`, not the catch-all `unknown`. + if (err instanceof Error && err.name === 'AbortError') { + return makeLlmError({ provider, kind: 'cancelled', message: 'request aborted' }); + } return makeLlmError({ provider, kind: 'unknown', @@ -1129,13 +1136,16 @@ export function decodeVideoJobId(jobId: string): string | undefined { /** Sora accepts only 4/8/12s clips — reject anything else loud (never round; cost-integrity, ADR-0045 §5). */ const SORA_SECONDS: Readonly> = { 4: '4', 8: '8', 12: '12' }; -function soraSeconds(durationSeconds: number | undefined): OpenAI.VideoSeconds { +function soraSeconds( + durationSeconds: number | undefined, + providerId: ProviderId, +): OpenAI.VideoSeconds { const n = durationSeconds ?? 4; // Sora's default clip length const seconds = SORA_SECONDS[n]; if (seconds === undefined) { throw new LlmProviderError( makeLlmError({ - provider: 'openai', + provider: providerId, kind: 'bad_request', message: `Sora durationSeconds must be 4, 8, or 12; got ${String(n)}`, }), @@ -1168,7 +1178,7 @@ async function openAiGenerateVideo( req: MediaGenRequest, providerId: ProviderId, ): Promise { - const seconds = soraSeconds(req.durationSeconds); // throws bad_request if not 4/8/12 + const seconds = soraSeconds(req.durationSeconds, providerId); // throws bad_request if not 4/8/12 const size = soraSize(req.providerOptions); let video: OpenAI.Video; try { @@ -1264,6 +1274,10 @@ async function pollMediaJobSora( // fail the engine's z.number().min(0).max(1) boundary. return { state: 'pending', progress: Math.min(1, Math.max(0, video.progress / 100)) }; case 'completed': { + // `bytes` is assigned in the SAME `if (status === 'completed')` block above, so at runtime it is + // always set here; the `=== undefined` half is the TS narrowing guard (bytes is typed + // `Uint8Array | undefined` because the assignment is control-flow-conditional) — `length === 0` is + // the live defensive check for an empty download. if (bytes === undefined || bytes.length === 0) { return { state: 'failed', diff --git a/packages/llm/src/types.ts b/packages/llm/src/types.ts index 05be1e14..98d004b1 100644 --- a/packages/llm/src/types.ts +++ b/packages/llm/src/types.ts @@ -485,8 +485,8 @@ export interface LlmProvider { /** * Poll an async media job by its Relavium-opaque id (A5, [ADR-0045](../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). * `signal` aborts the IN-FLIGHT poll so a run cancel reaches the open provider request, not just the - * next schedule. The engine drives this loop (1.AG Section D); no Phase-1 vendor adapter implements it - * yet (the async Sora/Veo adapters are 1.AH). + * next schedule. The engine drives this loop (1.AG Section D); the OpenAI/Sora adapter implements it at + * 1.AH A3 (`pollMediaJobSora`), the Gemini/Veo adapter at 1.AH A4. */ pollMediaJob?(jobId: string, key: string, signal?: AbortSignalLike): Promise; } From 0580c4fa9b2216aa40b8d9dca4ff4900f6f1d5c0 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 15:19:00 +0300 Subject: [PATCH 11/21] =?UTF-8?q?feat(llm):=201.AH=20A4=20=E2=80=94=20Gemi?= =?UTF-8?q?ni=20Veo=20async=20video=20+=20shared=20opaque-jobId=20encoding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the Gemini adapter's ASYNC Veo video LRO behind the seam (ADR-0045 §3/§7), completing the four 1.AH generative adapters, and factor the opaque-jobId encoding into one shared home: - adapters/shared.ts: encodeMediaJobId/decodeMediaJobId (moved from openai.ts A3, generalized — provider-agnostic, the engine routes pollMediaJob by the bound provider so the shared rlv-mediajob:1: prefix is collision-safe). The Sora arm now imports them; the round-trip-validated bijection + its tests live in shared.test.ts. - GeminiTransport gains generateVideos + pollVideo (required) — plain Relavium subsets (GeminiVideoRequest/Operation/Poll), no @google/genai type crosses the seam. The default sdkTransport wraps models.generateVideos / operations.getVideosOperation; getVideosOperation reads only operation.name, so pollVideo reconstructs a fresh GenerateVideosOperation carrying the persisted name → stateless cold-process re-attach. - generateMedia(video) → geminiGenerateVideo: ALWAYS returns an opaque { jobId } from the operation name (never { media }); numberOfVideos pinned to 1 AFTER stripTransportKeys (single-artifact + the A2 SSRF lesson — genai co-mingles httpOptions in the config); durationSeconds threaded; raw byte-free. - pollMediaJob → geminiPollVideo: pending (done:false); done delivers inline videoBytes as base64 OR a re-hostable url source (ADR-0045 §7 — the engine de-inlines via the single fetchMediaBytes, never a second fetch site); raiFilteredCount>0 → content_filter; operation error → unknown; empty → bad_request; malformed jobId → fatal failed. - 14 Veo unit tests (opaque jobId, SSRF strip, status mapping incl. url-source + base64, content_filter/error/empty, malformed token, abort threading); conformance replay + seam-doc Wired list updated. Veo end-to-end completion of a url-source result needs the host MediaUrlFetch re-host hook (1.AH host-wiring, deferred). Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011, ADR-0031, ADR-0043 Co-Authored-By: Claude Opus 4.8 --- .../shared-core/llm-provider-seam.md | 13 +- packages/llm/src/adapters/gemini.test.ts | 220 +++++++++++++++- packages/llm/src/adapters/gemini.ts | 235 +++++++++++++++++- packages/llm/src/adapters/openai.test.ts | 40 +-- packages/llm/src/adapters/openai.ts | 51 +--- packages/llm/src/adapters/shared.test.ts | 33 ++- packages/llm/src/adapters/shared.ts | 42 ++++ .../conformance/gemini.conformance.test.ts | 5 + 8 files changed, 542 insertions(+), 97 deletions(-) diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index ba7eb2b2..54481e8a 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -390,11 +390,14 @@ managed mode) are recorded in the ADR — this section is the dry shape referenc (the engine resolves the per-model surface). The ASYNC `pollMediaJob` poll/checkpoint/resume/cancel loop is WIRED in the engine (Section D — `media_job:submitted` park, the derived `pendingMediaJobs` slot, re-attach-on-resume, a host-timer poll cadence, deadline→retryable-timeout, - cancel→abort→terminal sweep; failed→`content_filter`), and the **OpenAI/Sora** async adapter - implements it (`videos.create` → an opaque `jobId`, `pollMediaJob` → `videos.retrieve`/`downloadContent`, - 1.AH A3) — the opaque jobId reversibly encodes the vendor id (`rlv-mediajob:1:`) so a - cold-process re-attach resolves it statelessly (ADR-0045 §7). The remaining work is **1.AH**: the - **Veo** async adapter (A4), the per-model `media_surface` host lookup, and verified generative pricing rows. + cancel→abort→terminal sweep; failed→`content_filter`), and the async adapters implement it — **OpenAI/Sora** + (`videos.create` → an opaque `jobId`, `pollMediaJob` → `videos.retrieve`/`downloadContent` → base64, 1.AH A3) + and **Gemini/Veo** (`models.generateVideos` → an operation, `pollMediaJob` → `operations.getVideosOperation` + → inline `videoBytes` base64 OR a re-hostable `url` source the engine de-inlines via `fetchMediaBytes`, + 1.AH A4). The opaque jobId reversibly encodes the vendor id/op-name (`rlv-mediajob:1:`, the + shared `encodeMediaJobId`/`decodeMediaJobId`) so a cold-process re-attach resolves it statelessly (ADR-0045 + §7). The remaining work is **1.AH host-wiring**: the per-model `media_surface` lookup, the `MediaUrlFetch` + re-host hook (a Veo `url` result needs it end-to-end), and verified generative pricing rows. ```ts // Seam shape (A5; ADR-0045) — behavior WIRED at 1.AG (sync generateMedia Section C, async poll loop Section D). diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index ed2cb020..ac546c0e 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -1,15 +1,20 @@ import { describe, expect, it } from 'vitest'; +import type { AbortSignalLike } from '@relavium/shared'; + import { UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; import { GeminiToolCallIds } from '../tool-normalizer.js'; +import { MediaGenResultSchema, MediaJobStatusSchema } from '../types.js'; import type { LlmProvider, LlmRequest, MediaGenRequest, MediaGenResult, + MediaJobStatus, StreamChunk, } from '../types.js'; +import { encodeMediaJobId } from './shared.js'; import { buildGeminiRequest, createGeminiAdapter, @@ -23,6 +28,9 @@ import { type GeminiRequest, type GeminiResponse, type GeminiTransport, + type GeminiVideoOperation, + type GeminiVideoPoll, + type GeminiVideoRequest, } from './gemini.js'; async function collect(stream: AsyncIterable): Promise { @@ -33,12 +41,21 @@ async function collect(stream: AsyncIterable): Promise = { + generateImages: () => Promise.reject(new Error('unused')), + generateVideos: () => Promise.reject(new Error('unused')), + pollVideo: () => Promise.reject(new Error('unused')), +}; + /** A transport that returns a fixed response and captures the request it was handed. */ function fakeTransport( response: GeminiResponse, stream: readonly GeminiResponse[] = [response], ): GeminiTransport & { lastRequest?: GeminiRequest } { const holder: GeminiTransport & { lastRequest?: GeminiRequest } = { + ...unusedGenerative, // text-fold tests never call the generative endpoints generate: (request) => { holder.lastRequest = request; return Promise.resolve(response); @@ -54,9 +71,6 @@ function fakeTransport( })(), ); }, - // The text-fold tests never call generateImages; the dedicated generateMedia tests build their own - // image transport (fakeImageTransport). Reject loud if a text test ever reaches it. - generateImages: () => Promise.reject(new Error('fakeTransport.generateImages not configured')), }; return holder; } @@ -66,6 +80,7 @@ function fakeImageTransport( response: GeminiImageResponse, ): GeminiTransport & { lastImageRequest?: GeminiImageRequest } { const holder: GeminiTransport & { lastImageRequest?: GeminiImageRequest } = { + ...unusedGenerative, generate: () => Promise.reject(new Error('unused')), stream: () => Promise.reject(new Error('unused')), generateImages: (request) => { @@ -76,6 +91,40 @@ function fakeImageTransport( return holder; } +/** A transport whose `generateVideos`/`pollVideo` drive a Veo flow; other methods reject. `poll` may be a + * single status or a sequence served by call index (pending → done). */ +function fakeVideoTransport(opts: { + operation?: GeminiVideoOperation; + poll?: GeminiVideoPoll | readonly GeminiVideoPoll[]; +}): GeminiTransport & { + lastVideoRequest?: GeminiVideoRequest; + lastPollSignal?: AbortSignalLike | undefined; +} { + const polls = opts.poll === undefined ? [] : 'done' in opts.poll ? [opts.poll] : opts.poll; + let call = 0; + const holder: GeminiTransport & { + lastVideoRequest?: GeminiVideoRequest; + lastPollSignal?: AbortSignalLike | undefined; + } = { + ...unusedGenerative, + generate: () => Promise.reject(new Error('unused')), + stream: () => Promise.reject(new Error('unused')), + generateVideos: (request) => { + holder.lastVideoRequest = request; + return Promise.resolve(opts.operation ?? { name: 'operations/veo-1' }); + }, + pollVideo: (_operationName, _key, signal) => { + holder.lastPollSignal = signal; + const status = polls[Math.min(call, polls.length - 1)]; + call += 1; + return status === undefined + ? Promise.reject(new Error('no poll status scripted')) + : Promise.resolve(status); + }, + }; + return holder; +} + /** Call the adapter's optional `generateMedia` via `?.()` — a call (binds `this`), never an extraction, * so the unbound-method lint stays happy; the `??` branch asserts the method is implemented. */ function genMedia( @@ -89,6 +138,19 @@ function genMedia( ); } +/** Call the adapter's optional `pollMediaJob` via `?.()` (same unbound-method-safe pattern as genMedia). */ +function pollMedia( + adapter: LlmProvider, + jobId: string, + key: string, + signal?: AbortSignalLike, +): Promise { + return ( + adapter.pollMediaJob?.(jobId, key, signal) ?? + Promise.reject(new Error('adapter implements no pollMediaJob')) + ); +} + const REQ: LlmRequest = { model: 'gemini-2.5-flash', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], @@ -539,7 +601,7 @@ describe('Gemini adapter — generate / stream via injected transport', () => { const transport: GeminiTransport = { generate: () => Promise.reject(Object.assign(new Error(`rl: ${SECRET}`), { status: 429 })), stream: () => Promise.reject(new Error('unused')), - generateImages: () => Promise.reject(new Error('unused')), + ...unusedGenerative, }; const adapter = createGeminiAdapter({ transport }); let caught: unknown; @@ -583,7 +645,7 @@ describe('Gemini adapter — generate / stream via injected transport', () => { const transport: GeminiTransport = { generate: () => Promise.reject(new Error('unused')), stream: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), - generateImages: () => Promise.reject(new Error('unused')), + ...unusedGenerative, }; const adapter = createGeminiAdapter({ transport }); const chunks = await collect(adapter.stream(REQ, 'k')); @@ -610,7 +672,7 @@ describe('Gemini adapter — remaining branches', () => { candidates: [{ content: { parts: [{ text: 'hi' }] }, finishReason: 'STOP' }], }), stream: () => Promise.reject(new Error('unused')), - generateImages: () => Promise.reject(new Error('unused')), + ...unusedGenerative, }, }); const result = await adapter.generate(REQ, 'k'); @@ -629,7 +691,7 @@ describe('Gemini adapter — remaining branches', () => { throw Object.assign(new Error('mid-stream'), { status: 500 }); })(), ), - generateImages: () => Promise.reject(new Error('unused')), + ...unusedGenerative, }, }); const chunks = await collect(adapter.stream(REQ, 'k')); @@ -920,6 +982,7 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { it('surfaces a transport rejection as a classified LlmProviderError', async () => { const transport: GeminiTransport = { + ...unusedGenerative, generate: () => Promise.reject(new Error('unused')), stream: () => Promise.reject(new Error('unused')), generateImages: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), @@ -929,14 +992,151 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { }); }); - it('rejects audio + video modalities with a typed capability error (image is the only sync surface)', async () => { + it('rejects audio modality with a typed capability error (image is Imagen; video is the async Veo arm)', async () => { const transport = fakeImageTransport({ generatedImages: [] }); const adapter = createGeminiAdapter({ transport }); await expect(genMedia(adapter, { ...IMG_REQ, modality: 'audio' }, 'k')).rejects.toBeInstanceOf( UnsupportedCapabilityError, ); - await expect(genMedia(adapter, { ...IMG_REQ, modality: 'video' }, 'k')).rejects.toBeInstanceOf( - UnsupportedCapabilityError, + }); +}); + +describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1.AH A4)', () => { + const VIDEO_REQ: MediaGenRequest = { + model: 'veo-3.0-generate-001', + prompt: 'a wave breaking on a beach', + modality: 'video', + durationSeconds: 6, + }; + const B64 = 'dmVvLWJ5dGVz'; // "veo-bytes" + + it('generateMedia (video) ALWAYS returns an opaque jobId from the operation name (no media, raw byte-free)', async () => { + const transport = fakeVideoTransport({ operation: { name: 'operations/veo-42' } }); + const result = await genMedia(createGeminiAdapter({ transport }), VIDEO_REQ, 'k'); + expect(result.media).toBeUndefined(); // ASYNC arm + expect(result.jobId).toBe(encodeMediaJobId('operations/veo-42')); + expect(result.jobId).not.toContain('operations/veo-42'); // base64url-opaque + expect(MediaGenResultSchema.safeParse(result).success).toBe(true); + expect(result.raw).toEqual({ name: 'operations/veo-42' }); // no bytes in raw (I3) + // Single-artifact pin + durationSeconds threaded into the typed config. + expect(transport.lastVideoRequest?.config['numberOfVideos']).toBe(1); + expect(transport.lastVideoRequest?.config['durationSeconds']).toBe(6); + expect(transport.lastVideoRequest?.prompt).toBe(VIDEO_REQ.prompt); + }); + + it('generateMedia (video) strips a caller-supplied httpOptions from the Veo config (SSRF guard)', async () => { + const transport = fakeVideoTransport({ operation: { name: 'operations/veo-42' } }); + await genMedia( + createGeminiAdapter({ transport }), + { + ...VIDEO_REQ, + providerOptions: { + httpOptions: { baseUrl: 'https://attacker.example' }, + aspectRatio: '16:9', + }, + }, + 'k', ); + expect(transport.lastVideoRequest?.config['httpOptions']).toBeUndefined(); + expect(transport.lastVideoRequest?.config['aspectRatio']).toBe('16:9'); // benign knob survives + }); + + it('generateMedia (video) maps a missing operation name to a typed bad_request', async () => { + const transport = fakeVideoTransport({ operation: { name: '' } }); + await expect( + genMedia(createGeminiAdapter({ transport }), VIDEO_REQ, 'k'), + ).rejects.toMatchObject({ + llmError: { kind: 'bad_request' }, + }); + }); + + it('pollMediaJob maps an in-progress operation (done:false) to pending', async () => { + const transport = fakeVideoTransport({ poll: { done: false } }); + expect( + await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), + ).toEqual({ + state: 'pending', + }); + }); + + it('pollMediaJob delivers inline videoBytes as a base64 video/mp4 media part (done)', async () => { + const transport = fakeVideoTransport({ + poll: { done: true, video: { videoBytes: B64, mimeType: 'video/mp4' } }, + }); + const status = await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'); + expect(MediaJobStatusSchema.safeParse(status).success).toBe(true); + expect(status).toEqual({ + state: 'done', + media: { type: 'media', mimeType: 'video/mp4', source: { kind: 'base64', data: B64 } }, + }); + }); + + it('pollMediaJob delivers a uri-only result as a re-hostable url media source (engine de-inlines it)', async () => { + const transport = fakeVideoTransport({ + poll: { done: true, video: { uri: 'https://generativelanguage.googleapis.com/v1/files/x' } }, + }); + const status = await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'); + expect(status).toEqual({ + state: 'done', + media: { + type: 'media', + mimeType: 'video/mp4', + source: { kind: 'url', url: 'https://generativelanguage.googleapis.com/v1/files/x' }, + }, + }); + }); + + it('pollMediaJob maps a safety-filtered completion (raiFilteredCount > 0, no video) to content_filter', async () => { + const transport = fakeVideoTransport({ poll: { done: true, raiFilteredCount: 1 } }); + expect( + await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), + ).toMatchObject({ state: 'failed', error: { kind: 'content_filter' } }); + }); + + it('pollMediaJob maps an operation error to a fatal unknown failed', async () => { + const transport = fakeVideoTransport({ poll: { done: true, error: { message: 'quota' } } }); + expect( + await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), + ).toMatchObject({ state: 'failed', error: { kind: 'unknown' } }); + }); + + it('pollMediaJob maps a completion with no video/error/rai to bad_request', async () => { + const transport = fakeVideoTransport({ poll: { done: true } }); + expect( + await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), + ).toMatchObject({ state: 'failed', error: { kind: 'bad_request' } }); + }); + + it('pollMediaJob returns a FATAL failed (not a throw) for an unrecognized jobId token', async () => { + const transport = fakeVideoTransport({ poll: { done: false } }); + const status = await pollMedia(createGeminiAdapter({ transport }), 'not-a-relavium-token', 'k'); + expect(status).toMatchObject({ state: 'failed', error: { kind: 'bad_request' } }); + expect(transport.lastPollSignal).toBeUndefined(); // never reached the transport — decode failed first + }); + + it('pollMediaJob threads the AbortSignal into the Veo poll', async () => { + const transport = fakeVideoTransport({ poll: { done: false } }); + const controller = new AbortController(); + await pollMedia( + createGeminiAdapter({ transport }), + encodeMediaJobId('op'), + 'k', + controller.signal, + ); + expect(transport.lastPollSignal).toBe(controller.signal); + }); + + it('surfaces a generateVideos transport rejection as a classified LlmProviderError', async () => { + const transport: GeminiTransport = { + ...unusedGenerative, + generate: () => Promise.reject(new Error('unused')), + stream: () => Promise.reject(new Error('unused')), + generateVideos: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), + }; + await expect( + genMedia(createGeminiAdapter({ transport }), VIDEO_REQ, 'k'), + ).rejects.toMatchObject({ + llmError: { kind: 'overloaded' }, + }); }); }); diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index ec10eb39..8b618a8a 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -1,7 +1,7 @@ -import { GoogleGenAI } from '@google/genai'; +import { GenerateVideosOperation, GoogleGenAI } from '@google/genai'; import { mediaModalityOf } from '@relavium/shared'; -import type { ContentPart, OutputModality, StopReason } from '@relavium/shared'; +import type { AbortSignalLike, ContentPart, OutputModality, StopReason } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; @@ -16,6 +16,7 @@ import type { LlmResult, MediaGenRequest, MediaGenResult, + MediaJobStatus, StreamChunk, ToolChoice, ToolDef, @@ -26,6 +27,8 @@ import { REASONING_ID, assertMediaCapabilities, assertNoStreamingMediaOutput, + decodeMediaJobId, + encodeMediaJobId, isAbortSignal, } from './shared.js'; @@ -152,6 +155,36 @@ export interface GeminiImageResponse { }>; } +/** The lowered Veo request the transport sends (1.AH A4; a plain object the SDK accepts). */ +export interface GeminiVideoRequest { + model: string; + prompt: string; + config: Record; +} + +/** The minimal slice of a Veo `generateVideos` operation the adapter mints a jobId from (1.AH A4): just + * the server-assigned operation name. No vendor type crosses the seam. */ +export interface GeminiVideoOperation { + name?: string | undefined; +} + +/** + * One poll of a Veo operation, normalized to a vendor-type-free subset (1.AH A4). `videoBytes` is base64 + * (inline); `uri` is a re-hostable provider URL the engine de-inlines via the one SSRF-validated + * `fetchMediaBytes` (ADR-0045 §7 — never a second fetch site); `raiFilteredCount > 0` signals a safety + * block (no video, no error). `error` is the Google operation error, normalized to a message. + */ +export interface GeminiVideoPoll { + done: boolean; + error?: { message?: string | undefined }; + video?: { + videoBytes?: string | undefined; + uri?: string | undefined; + mimeType?: string | undefined; + }; + raiFilteredCount?: number; +} + /** * The injected network seam. The default wraps `@google/genai`; the conformance harness injects a * replay implementation. Keeping it here lets the one adapter run on every host (ADR-0018) and lets @@ -166,6 +199,13 @@ export interface GeminiTransport { * surface it services (sync image is Gemini's only Phase-1 generative arm; Veo video is 1.AH A4). */ generateImages(request: GeminiImageRequest, key: string): Promise; + /** + * Veo async video LRO (1.AH A4): `generateVideos` mints an operation, `pollVideo` re-polls it by name + * (the engine re-delivers the persisted opaque jobId on resume — re-attach, ADR-0045 §3). The default + * wraps `ai.models.generateVideos` / `ai.operations.getVideosOperation`; the harness injects a replay. + */ + generateVideos(request: GeminiVideoRequest, key: string): Promise; + pollVideo(operationName: string, key: string, signal?: AbortSignalLike): Promise; } // --- Normalization: Gemini wire → canonical -------------------------------------------------- @@ -499,6 +539,49 @@ const sdkTransport: GeminiTransport = { const client = new GoogleGenAI({ apiKey: key }); return client.models.generateImages(request); }, + async generateVideos(request: GeminiVideoRequest, key: string): Promise { + const client = new GoogleGenAI({ apiKey: key }); + const op = await client.models.generateVideos(request); + return { name: op.name }; + }, + async pollVideo( + operationName: string, + key: string, + signal?: AbortSignalLike, + ): Promise { + const client = new GoogleGenAI({ apiKey: key }); + // getVideosOperation reads ONLY operation.name (verified in the SDK), so a fresh operation carrying + // just the persisted name re-attaches across a process restart with no in-memory handle (ADR-0045 §3). + const operation = new GenerateVideosOperation(); + operation.name = operationName; + const op = await client.operations.getVideosOperation({ + operation, + ...(isAbortSignal(signal) ? { config: { abortSignal: signal } } : {}), + }); + if (op.done !== true) { + return { done: false }; + } + if (op.error !== undefined && op.error !== null) { + const message = op.error['message']; + return { done: true, error: { message: typeof message === 'string' ? message : undefined } }; + } + const video = op.response?.generatedVideos?.[0]?.video; + return { + done: true, + ...(video === undefined + ? {} + : { + video: { + videoBytes: video.videoBytes, + uri: video.uri, + mimeType: video.mimeType, + }, + }), + ...(op.response?.raiMediaFilteredCount === undefined + ? {} + : { raiFilteredCount: op.response.raiMediaFilteredCount }), + }; + }, }; /* v8 ignore stop */ @@ -732,6 +815,130 @@ async function geminiGenerateImage( }; } +// --- Generative media: Veo video (ASYNC LRO, 1.AH A4, ADR-0045) ------------------------------ + +/** + * Veo async video generation (1.AH A4) — `generateVideos` mints an operation; the adapter ALWAYS returns + * an opaque `{ jobId }` (encoding the operation name) the engine polls, never `{ media }` (even if Veo + * somehow completed instantly — completion is the poll loop's job, ADR-0045 §3). `numberOfVideos` is + * pinned to 1 AFTER the stripTransportKeys spread (single-artifact seam + SSRF guard, the A2 lesson); + * `durationSeconds` rides the typed config. `raw` carries only the non-byte operation name. + */ +async function geminiGenerateVideo( + transport: GeminiTransport, + req: MediaGenRequest, + key: string, +): Promise { + let op: GeminiVideoOperation; + try { + op = await transport.generateVideos( + { + model: req.model, + prompt: req.prompt, + config: { + ...stripTransportKeys(req.providerOptions ?? {}), + numberOfVideos: 1, + ...(req.durationSeconds === undefined ? {} : { durationSeconds: req.durationSeconds }), + ...(isAbortSignal(req.signal) ? { abortSignal: req.signal } : {}), + }, + }, + key, + ); + } catch (err) { + throw new LlmProviderError(geminiErrorToLlmError(err)); + } + if (op.name === undefined || op.name.length === 0) { + throw new LlmProviderError( + makeLlmError({ + provider: PROVIDER, + kind: 'bad_request', + message: 'Veo generateVideos returned no operation name', + }), + ); + } + return { jobId: encodeMediaJobId(op.name), raw: { name: op.name } }; +} + +/** + * Poll one Veo job by its opaque jobId (ADR-0045 §3). Decode → `pollVideo`; on completion deliver the + * video as base64 (if Veo returned inline `videoBytes`) or as a re-hostable `url` source the engine + * de-inlines via the single SSRF-validated `fetchMediaBytes` (ADR-0045 §7 — never a second fetch site). A + * safety block (`raiFilteredCount > 0`, no video) maps to `content_filter`; an operation error to the + * fatal `unknown`. A malformed jobId returns a FATAL `failed` (not a throw — a throw is engine-classified + * retryable and would loop on a structurally-dead token). + */ +async function geminiPollVideo( + transport: GeminiTransport, + jobId: string, + key: string, + signal: AbortSignalLike | undefined, +): Promise { + const operationName = decodeMediaJobId(jobId); + if (operationName === undefined) { + return { + state: 'failed', + error: makeLlmError({ + provider: PROVIDER, + kind: 'bad_request', + message: 'unrecognized Veo media job token', + }), + }; + } + let poll: GeminiVideoPoll; + try { + poll = await transport.pollVideo(operationName, key, signal); + } catch (err) { + throw new LlmProviderError(geminiErrorToLlmError(err)); + } + if (!poll.done) { + return { state: 'pending' }; + } + if (poll.error !== undefined) { + return { + state: 'failed', + error: makeLlmError({ + provider: PROVIDER, + kind: 'unknown', + message: poll.error.message ?? 'Veo video job failed', + }), + }; + } + const video = poll.video; + // Strip `;`-parameters to the canonical bare MIME (vendor-reported), default video/mp4 — same as the + // image arm (the shared bareMime consolidation is the A6 N1 carry-forward). + const mimeType = (video?.mimeType ?? '').split(';')[0]?.trim() || 'video/mp4'; + if (video?.videoBytes !== undefined && video.videoBytes.length > 0) { + return { + state: 'done', + media: { type: 'media', mimeType, source: { kind: 'base64', data: video.videoBytes } }, + }; + } + if (video?.uri !== undefined && video.uri.length > 0) { + return { + state: 'done', + media: { type: 'media', mimeType, source: { kind: 'url', url: video.uri } }, + }; + } + if ((poll.raiFilteredCount ?? 0) > 0) { + return { + state: 'failed', + error: makeLlmError({ + provider: PROVIDER, + kind: 'content_filter', + message: `Veo filtered ${String(poll.raiFilteredCount)} video(s) by safety policy`, + }), + }; + } + return { + state: 'failed', + error: makeLlmError({ + provider: PROVIDER, + kind: 'bad_request', + message: 'Veo completed with no video output', + }), + }; +} + // --- The adapter ----------------------------------------------------------------------------- /** Dependencies the conformance replayer / tests inject (the network transport). */ @@ -778,18 +985,34 @@ export function createGeminiAdapter(deps: GeminiAdapterDeps = {}): LlmProvider { return streamChunks(transport, buildGeminiRequest(req), key); }, async generateMedia(req: MediaGenRequest, key: string): Promise { - // SYNC separate-endpoint generation, dispatched by modality (ADR-0045 §1): image → Imagen - // (generateImages). Audio is not a Gemini generative surface; video is the ASYNC Veo path (1.AH A4) - // — both fail loud with a typed capability error, never a silent drop. + // Separate-endpoint generation, dispatched by modality (ADR-0045 §1): image → Imagen (generateImages, + // SYNC); video → Veo (generateVideos, ASYNC LRO — returns an opaque jobId the engine polls via + // pollMediaJob below). Audio is not a Gemini generative surface — fail loud, never a silent drop. if (req.modality === 'image') { return geminiGenerateImage(transport, req, key); } + if (req.modality === 'video') { + return geminiGenerateVideo(transport, req, key); + } throw new UnsupportedCapabilityError( PROVIDER, 'media', - `Gemini generateMedia has no SYNC surface for '${req.modality}' (image is Imagen; video is the async Veo path)`, + `Gemini generateMedia has no surface for '${req.modality}' (image is Imagen; video is Veo; audio is unsupported)`, ); }, + /** + * Poll one async media job (Veo video LRO, 1.AH A4, [ADR-0045](../../../../docs/decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md)). + * The engine re-delivers the persisted opaque jobId on resume (re-attach, §3); the adapter recovers the + * Veo operation name by DECODING the token (stateless bijection — no instance Map — so a cold-process + * restart resolves it). `signal` aborts the in-flight poll so a run cancel reaches the open request. + */ + async pollMediaJob( + jobId: string, + key: string, + signal?: AbortSignalLike, + ): Promise { + return geminiPollVideo(transport, jobId, key, signal); + }, }; } diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 1bcbc5c6..411f86cc 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -15,11 +15,10 @@ import { type MediaJobStatus, type StreamChunk, } from '../types.js'; +import { encodeMediaJobId } from './shared.js'; import { createOpenAiAdapter, - decodeVideoJobId, deepseekAdapter, - encodeVideoJobId, mapContent, mapStopReason, mapUsage, @@ -1552,31 +1551,12 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe return Object.assign(fn, { capture }); } - it('encodeVideoJobId mints a Relavium-opaque token that is NOT the raw vendor id (round-trips via decode)', () => { - const jobId = encodeVideoJobId('video_abc123'); - expect(jobId.startsWith('rlv-mediajob:1:')).toBe(true); - expect(jobId).not.toContain('video_abc123'); // base64url-encoded — the raw vendor id never appears verbatim - expect(decodeVideoJobId(jobId)).toBe('video_abc123'); - }); - - it('decodeVideoJobId returns undefined for foreign/malformed tokens (never throws)', () => { - expect( - decodeVideoJobId('rlv-mediajob:2:' + Buffer.from('x').toString('base64url')), - ).toBeUndefined(); // future version - expect(decodeVideoJobId('vendor-op-7f3a')).toBeUndefined(); // non-Relavium - expect(decodeVideoJobId('')).toBeUndefined(); - expect(decodeVideoJobId('rlv-mediajob:1:')).toBeUndefined(); // empty payload - // Round-trip validation: a right-prefix but NON-CANONICAL payload (trailing junk base64url drops) is - // rejected — it would otherwise decode to a wrong-but-non-empty vendor id and be polled. - expect(decodeVideoJobId(encodeVideoJobId('video_x') + '!')).toBeUndefined(); - }); - it('generateMedia (video) ALWAYS returns an opaque jobId (no media), schema-valid, even on instant completion', async () => { const fetch = soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }); const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); const result = await genMedia(adapter, VIDEO_REQ, 'k'); expect(result.media).toBeUndefined(); // ASYNC arm — completion is the engine poll loop's job - expect(result.jobId).toBe(encodeVideoJobId('video_xyz')); + expect(result.jobId).toBe(encodeMediaJobId('video_xyz')); expect(MediaGenResultSchema.safeParse(result).success).toBe(true); expect(result.raw).toEqual({ id: 'video_xyz', status: 'completed' }); // no bytes in raw (I3) // The create body forwards model/prompt/seconds (NOT a spread of providerOptions). @@ -1642,7 +1622,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe }); it('pollMediaJob maps queued → pending and in_progress → pending with clamped 0-1 progress', async () => { - const jobId = encodeVideoJobId('video_xyz'); + const jobId = encodeMediaJobId('video_xyz'); const queued = createOpenAiAdapter({ maxRetries: 0, fetch: soraFetch({ id: 'video_xyz', status: 'queued', progress: 0 }), @@ -1668,14 +1648,14 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe maxRetries: 0, fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, ''), }); - expect(await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k')).toMatchObject({ + expect(await pollMedia(adapter, encodeMediaJobId('video_xyz'), 'k')).toMatchObject({ state: 'failed', error: { kind: 'bad_request' }, }); }); it('pollMediaJob completed → downloads the MP4 and returns a base64 video/mp4 media part (done)', async () => { - const jobId = encodeVideoJobId('video_xyz'); + const jobId = encodeMediaJobId('video_xyz'); const adapter = createOpenAiAdapter({ maxRetries: 0, fetch: soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, 'MP4-BYTES'), @@ -1693,7 +1673,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe }); it('pollMediaJob failed → content_filter for a content-policy code, unknown for a null error', async () => { - const jobId = encodeVideoJobId('video_xyz'); + const jobId = encodeMediaJobId('video_xyz'); const blocked = createOpenAiAdapter({ maxRetries: 0, fetch: soraFetch({ @@ -1728,7 +1708,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe error: { code: '', message: 'x' }, }), }); - const status = await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k'); + const status = await pollMedia(adapter, encodeMediaJobId('video_xyz'), 'k'); expect(status).toMatchObject({ state: 'failed', error: { kind: 'unknown' } }); if (status.state === 'failed') { expect(status.error.code).toBeUndefined(); // empty code is omitted, not surfaced as '' @@ -1741,7 +1721,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe maxRetries: 0, fetch: soraFetch({ id: 'video_xyz', status: 'rendering', progress: 0 }), }); - expect(await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k')).toMatchObject({ + expect(await pollMedia(adapter, encodeMediaJobId('video_xyz'), 'k')).toMatchObject({ state: 'failed', error: { kind: 'unknown' }, }); @@ -1765,7 +1745,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe const fetch = soraFetch({ id: 'video_xyz', status: 'completed', progress: 100 }, 'MP4'); const adapter = createOpenAiAdapter({ maxRetries: 0, fetch }); const controller = new AbortController(); - await pollMedia(adapter, encodeVideoJobId('video_xyz'), 'k', controller.signal); + await pollMedia(adapter, encodeMediaJobId('video_xyz'), 'k', controller.signal); expect(fetch.capture.signalByCall.retrieve).toBe(true); expect(fetch.capture.signalByCall.content).toBe(true); // the download honors the run cancel too }); @@ -1774,7 +1754,7 @@ describe('OpenAI-compatible adapter — Sora async video (generateMedia + pollMe await expect(genMedia(deepseekAdapter, VIDEO_REQ, 'k')).rejects.toBeInstanceOf( UnsupportedCapabilityError, ); - const status = await pollMedia(deepseekAdapter, encodeVideoJobId('video_xyz'), 'k'); + const status = await pollMedia(deepseekAdapter, encodeMediaJobId('video_xyz'), 'k'); expect(status).toMatchObject({ state: 'failed' }); }); }); diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index d9a89382..790a8632 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -41,6 +41,8 @@ import { REASONING_ID, assertMediaCapabilities, assertNoStreamingMediaOutput, + decodeMediaJobId, + encodeMediaJobId, isAbortSignal, } from './shared.js'; @@ -1090,49 +1092,8 @@ async function openAiGenerateSpeech( } // --- Generative media: Sora video (ASYNC LRO, 1.AH A3, ADR-0045) ----------------------------- - -/** - * The opaque media-job id namespace+version. The engine persists this token in the durable - * `media_job:submitted` event and re-delivers it to `pollMediaJob` on resume (ADR-0045 §3 re-attach, never - * re-submit). The adapter has NO durable store and is rebuilt each process, so the vendor video id is - * reversibly ENCODED into the token (base64url so any vendor id is `:`-split-safe) rather than held in an - * in-memory `Map` — this IS the "vendor↔opaque map internal" of ADR-0045 §7, realized as a STATELESS - * bijection. The token is adapter-minted + Relavium-namespaced (NOT the bare vendor op-name / poll-URL), - * the engine never parses it (I1/[ADR-0011]), and the (non-secret) resource id is safe to persist. A future - * format change bumps the version slot (`:2:`) without an engine or schema change. Managed mode (ADR-0015) - * swaps the `pollMediaJob` body and keeps this prefix as the decode convention. - */ -const SORA_JOB_PREFIX = 'rlv-mediajob:1:'; - -/** Mint the opaque jobId from Sora's `Video.id` (base64url-encoded). Exported for the N4 opacity test. */ -export function encodeVideoJobId(vendorId: string): string { - return SORA_JOB_PREFIX + Buffer.from(vendorId, 'utf8').toString('base64url'); -} - -/** - * Recover the vendor `Video.id` from an opaque jobId; `undefined` on any foreign/malformed token (never - * throws — a throw from `pollMediaJob` is engine-classified as retryable and would loop forever on a - * structurally-dead token). Exported for the opacity/round-trip test. - */ -export function decodeVideoJobId(jobId: string): string | undefined { - if (!jobId.startsWith(SORA_JOB_PREFIX)) { - return undefined; // a non-Relavium id or a future rlv-mediajob:2: token - } - const payload = jobId.slice(SORA_JOB_PREFIX.length); - if (payload.length === 0) { - return undefined; - } - const decoded = Buffer.from(payload, 'base64url').toString('utf8'); - // Round-trip validation: ONLY a canonically-minted token decodes back to itself. base64url decoding is - // lenient (it silently drops invalid chars), so a corrupted/non-canonical payload would otherwise decode - // to a wrong-but-non-empty vendor id and be forwarded to videos.retrieve. Re-encoding and comparing - // rejects every such token here, so pollMediaJob never polls a junk vendor id. Every real minted token - // round-trips exactly (encode is the inverse of decode). - if (decoded.length === 0 || encodeVideoJobId(decoded) !== jobId) { - return undefined; - } - return decoded; -} +// The opaque-jobId encode/decode bijection (ADR-0045 §7) is provider-agnostic and lives in ./shared.ts +// (encodeMediaJobId/decodeMediaJobId), shared with the Gemini/Veo adapter (1.AH A4). /** Sora accepts only 4/8/12s clips — reject anything else loud (never round; cost-integrity, ADR-0045 §5). */ const SORA_SECONDS: Readonly> = { 4: '4', 8: '8', 12: '12' }; @@ -1203,7 +1164,7 @@ async function openAiGenerateVideo( }), ); } - return { jobId: encodeVideoJobId(video.id), raw: { id: video.id, status: video.status } }; + return { jobId: encodeMediaJobId(video.id), raw: { id: video.id, status: video.status } }; } /** Map a Sora `VideoCreateError` VALUE object (not a thrown `APIError`) to a normalized `LlmError`; a @@ -1238,7 +1199,7 @@ async function pollMediaJobSora( providerId: ProviderId, signal: AbortSignalLike | undefined, ): Promise { - const vendorId = decodeVideoJobId(jobId); + const vendorId = decodeMediaJobId(jobId); if (vendorId === undefined) { return { state: 'failed', diff --git a/packages/llm/src/adapters/shared.test.ts b/packages/llm/src/adapters/shared.test.ts index 6919c5ba..ea151e4a 100644 --- a/packages/llm/src/adapters/shared.test.ts +++ b/packages/llm/src/adapters/shared.test.ts @@ -4,7 +4,12 @@ import { describe, expect, it } from 'vitest'; import { UnsupportedCapabilityError } from '../errors.js'; import type { CapabilityFlags, LlmRequest } from '../types.js'; -import { assertMediaCapabilities, isAbortSignal } from './shared.js'; +import { + assertMediaCapabilities, + decodeMediaJobId, + encodeMediaJobId, + isAbortSignal, +} from './shared.js'; const textOnly: LlmRequest = { model: 'm', @@ -412,3 +417,29 @@ describe('isAbortSignal', () => { ).toBe(false); }); }); + +describe('encodeMediaJobId / decodeMediaJobId (opaque async-media-job bijection, ADR-0045 §7)', () => { + it('mints a Relavium-opaque token that is NOT the raw vendor id (round-trips via decode)', () => { + const jobId = encodeMediaJobId('video_abc123'); + expect(jobId.startsWith('rlv-mediajob:1:')).toBe(true); + expect(jobId).not.toContain('video_abc123'); // base64url-encoded — the raw id never appears verbatim + expect(decodeMediaJobId(jobId)).toBe('video_abc123'); + }); + + it('round-trips an operation-name-style vendor id (Veo) with no information loss', () => { + const opName = 'operations/abc-123_xyz'; + expect(decodeMediaJobId(encodeMediaJobId(opName))).toBe(opName); + }); + + it('returns undefined for foreign/malformed tokens (never throws)', () => { + expect( + decodeMediaJobId('rlv-mediajob:2:' + Buffer.from('x').toString('base64url')), + ).toBeUndefined(); // future version + expect(decodeMediaJobId('vendor-op-7f3a')).toBeUndefined(); // non-Relavium + expect(decodeMediaJobId('')).toBeUndefined(); + expect(decodeMediaJobId('rlv-mediajob:1:')).toBeUndefined(); // empty payload + // Round-trip validation: a right-prefix but NON-CANONICAL payload (trailing junk base64url drops) is + // rejected — it would otherwise decode to a wrong-but-non-empty vendor id and be polled. + expect(decodeMediaJobId(encodeMediaJobId('video_x') + '!')).toBeUndefined(); + }); +}); diff --git a/packages/llm/src/adapters/shared.ts b/packages/llm/src/adapters/shared.ts index 2c39eb59..e166e719 100644 --- a/packages/llm/src/adapters/shared.ts +++ b/packages/llm/src/adapters/shared.ts @@ -67,3 +67,45 @@ export function assertNoStreamingMediaOutput(provider: ProviderId, req: LlmReque * multiple reasoning streams must move to an index-keyed id like the Anthropic adapter (`reasoning-${index}`). */ export const REASONING_ID = 'reasoning-0'; + +/** + * The Relavium-opaque async media-job id namespace+version, shared by every async generative adapter + * (Sora `Video.id`, Veo operation `name`, …). The engine persists this token in the durable + * `media_job:submitted` event and re-delivers it to `pollMediaJob` on resume (ADR-0045 §3 re-attach, + * never re-submit). An adapter has NO durable store and is rebuilt each process, so the vendor id is + * reversibly ENCODED into the token (base64url so any vendor id is `:`-split-safe) rather than held in an + * in-memory `Map` — this IS the "vendor↔opaque map internal" of ADR-0045 §7, realized as a STATELESS + * bijection. The token is adapter-minted + Relavium-namespaced (NOT the bare vendor op-name / poll-URL), + * the engine never parses it (I1/ADR-0011), and the (non-secret) resource id is safe to persist. A future + * format change bumps the version slot (`:2:`). The engine routes `pollMediaJob` by the BOUND provider, so + * each adapter only ever decodes its own jobs — the shared prefix is collision-safe across providers. + */ +const MEDIA_JOB_PREFIX = 'rlv-mediajob:1:'; + +/** Mint the opaque media-job id from a vendor job/operation id (base64url-encoded). */ +export function encodeMediaJobId(vendorId: string): string { + return MEDIA_JOB_PREFIX + Buffer.from(vendorId, 'utf8').toString('base64url'); +} + +/** + * Recover the vendor id from an opaque media-job id; `undefined` on any foreign/malformed token (never + * throws — a throw from `pollMediaJob` is engine-classified as retryable and would loop forever on a + * structurally-dead token). Round-trip-validated: base64url decoding is lenient (it silently drops invalid + * chars), so a corrupted/non-canonical payload would otherwise decode to a wrong-but-non-empty vendor id; + * re-encoding and comparing rejects every such token, so a caller never polls a junk vendor id. Every real + * minted token round-trips exactly. + */ +export function decodeMediaJobId(jobId: string): string | undefined { + if (!jobId.startsWith(MEDIA_JOB_PREFIX)) { + return undefined; // a non-Relavium id or a future rlv-mediajob:2: token + } + const payload = jobId.slice(MEDIA_JOB_PREFIX.length); + if (payload.length === 0) { + return undefined; + } + const decoded = Buffer.from(payload, 'base64url').toString('utf8'); + if (decoded.length === 0 || encodeMediaJobId(decoded) !== jobId) { + return undefined; + } + return decoded; +} diff --git a/packages/llm/src/conformance/gemini.conformance.test.ts b/packages/llm/src/conformance/gemini.conformance.test.ts index 775e0fb1..c23781f3 100644 --- a/packages/llm/src/conformance/gemini.conformance.test.ts +++ b/packages/llm/src/conformance/gemini.conformance.test.ts @@ -78,6 +78,11 @@ const makeReplayAdapter: MakeReplayAdapter = (recorded) => { ? Promise.resolve(parsed) : Promise.reject(new Error('replay fixture is not a GeminiImageResponse object')); }, + // The Veo async arm (1.AH A4) has no recorded-fixture conformance scenario yet; the seam contract is + // covered by generative-seam.conformance.test.ts (stub) + the gemini.test.ts unit tests. + generateVideos: () => + Promise.reject(new Error('veo not exercised by the chat conformance replay')), + pollVideo: () => Promise.reject(new Error('veo not exercised by the chat conformance replay')), }; return createGeminiAdapter({ transport }); }; From 0353f214546d104d35f308d08322f8fb19e44ac6 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 15:32:10 +0300 Subject: [PATCH 12/21] =?UTF-8?q?fix(llm):=201.AH=20A4=20=E2=80=94=20Veo?= =?UTF-8?q?=20Opus-review=20follow-ups=20(coverage=20+=20style)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the A4 Opus adversarial review (0 blocker/high; "every finding is a test-coverage or style item on correct code — A3/Sora behavior fully preserved by the refactor"): - GeminiVideoPoll.raiFilteredCount → `number | undefined` for sibling parity (N1). - tests (coverage gaps on correct code): assert the Veo create-path threads the AbortSignal (L1); assert numberOfVideos:5 in providerOptions is force-pinned back to 1 (the A2 single-artifact guard, L3); a pollVideo transport-rejection classifies (L2 — the only previously-uncovered new lines); a pending→done poll sequence transition (L4 — also exercises fakeVideoTransport's sequence path); schema-validate the url-source done result for parity with the base64 case (N2). Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 40 +++++++++++++++++++++++- packages/llm/src/adapters/gemini.ts | 2 +- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index ac546c0e..aa550f3f 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -1024,7 +1024,7 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 expect(transport.lastVideoRequest?.prompt).toBe(VIDEO_REQ.prompt); }); - it('generateMedia (video) strips a caller-supplied httpOptions from the Veo config (SSRF guard)', async () => { + it('generateMedia (video) strips httpOptions + pins numberOfVideos to 1 (SSRF + single-artifact guards)', async () => { const transport = fakeVideoTransport({ operation: { name: 'operations/veo-42' } }); await genMedia( createGeminiAdapter({ transport }), @@ -1032,15 +1032,28 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 ...VIDEO_REQ, providerOptions: { httpOptions: { baseUrl: 'https://attacker.example' }, + numberOfVideos: 5, // must be force-pinned back to 1 (the A2 single-artifact lesson) aspectRatio: '16:9', }, }, 'k', ); expect(transport.lastVideoRequest?.config['httpOptions']).toBeUndefined(); + expect(transport.lastVideoRequest?.config['numberOfVideos']).toBe(1); // pin wins over providerOptions expect(transport.lastVideoRequest?.config['aspectRatio']).toBe('16:9'); // benign knob survives }); + it('generateMedia (video) threads the AbortSignal into the Veo create config', async () => { + const transport = fakeVideoTransport({ operation: { name: 'operations/veo-42' } }); + const controller = new AbortController(); + await genMedia( + createGeminiAdapter({ transport }), + { ...VIDEO_REQ, signal: controller.signal }, + 'k', + ); + expect(transport.lastVideoRequest?.config['abortSignal']).toBe(controller.signal); + }); + it('generateMedia (video) maps a missing operation name to a typed bad_request', async () => { const transport = fakeVideoTransport({ operation: { name: '' } }); await expect( @@ -1059,6 +1072,16 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 }); }); + it('pollMediaJob walks a pending → done sequence across successive polls', async () => { + const transport = fakeVideoTransport({ + poll: [{ done: false }, { done: true, video: { videoBytes: B64, mimeType: 'video/mp4' } }], + }); + const adapter = createGeminiAdapter({ transport }); + const jobId = encodeMediaJobId('op'); + expect(await pollMedia(adapter, jobId, 'k')).toEqual({ state: 'pending' }); + expect(await pollMedia(adapter, jobId, 'k')).toMatchObject({ state: 'done' }); + }); + it('pollMediaJob delivers inline videoBytes as a base64 video/mp4 media part (done)', async () => { const transport = fakeVideoTransport({ poll: { done: true, video: { videoBytes: B64, mimeType: 'video/mp4' } }, @@ -1076,6 +1099,7 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 poll: { done: true, video: { uri: 'https://generativelanguage.googleapis.com/v1/files/x' } }, }); const status = await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'); + expect(MediaJobStatusSchema.safeParse(status).success).toBe(true); // url source is a valid done state expect(status).toEqual({ state: 'done', media: { @@ -1139,4 +1163,18 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 llmError: { kind: 'overloaded' }, }); }); + + it('surfaces a pollVideo transport rejection as a classified LlmProviderError', async () => { + const transport: GeminiTransport = { + ...unusedGenerative, + generate: () => Promise.reject(new Error('unused')), + stream: () => Promise.reject(new Error('unused')), + pollVideo: () => Promise.reject(Object.assign(new Error('overloaded'), { status: 503 })), + }; + await expect( + pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), + ).rejects.toMatchObject({ + llmError: { kind: 'overloaded' }, + }); + }); }); diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 8b618a8a..483fd8fe 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -182,7 +182,7 @@ export interface GeminiVideoPoll { uri?: string | undefined; mimeType?: string | undefined; }; - raiFilteredCount?: number; + raiFilteredCount?: number | undefined; } /** From 6b127db1721296fc2dfd0f6ffac9f4c143b6586e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 15:54:53 +0300 Subject: [PATCH 13/21] =?UTF-8?q?fix(llm):=201.AH=20A4=20=E2=80=94=20Veo?= =?UTF-8?q?=20Sonnet-review=20follow-ups=20(coverage=20parity=20+=20doc)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the A4 Sonnet second-pass review (0 blocker/high/medium; "the Opus GO holds — every finding is test-coverage/doc completeness on correct code"): - test: pollVideo threads the DECODED op-name to the transport (fakeVideoTransport now captures lastOperationName) — proves the re-attach decode→thread invariant (N1). - test: a parameterized vendor video MIME (video/mp4; codecs=h264) strips to the bare video/mp4 — parity with the Imagen arm (L1/N5). - test: durationSeconds is omitted from the Veo config when not requested (N2). - test: the pending→done poll sequence clamps on the last status (a 3rd poll stays done) — locks the fakeVideoTransport sequence contract (N9). - doc: the generative-seam conformance header no longer says Veo "remains deferred" (Veo is wired at A4, exercised via fakeVideoTransport) (N7). Skipped by design: a cross-provider token-rejection test (N3 — the opaque token is provider-agnostic by the maintainer-approved design; the engine routes pollMediaJob by the bound provider) and tightening GeminiVideoRequest.config typing (N4 — the Record escape hatch is the established pattern for all 3 gemini requests). Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0045, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 40 +++++++++++++++---- .../generative-seam.conformance.test.ts | 7 ++-- 2 files changed, 37 insertions(+), 10 deletions(-) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index aa550f3f..fe557389 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -98,12 +98,14 @@ function fakeVideoTransport(opts: { poll?: GeminiVideoPoll | readonly GeminiVideoPoll[]; }): GeminiTransport & { lastVideoRequest?: GeminiVideoRequest; + lastOperationName?: string; lastPollSignal?: AbortSignalLike | undefined; } { const polls = opts.poll === undefined ? [] : 'done' in opts.poll ? [opts.poll] : opts.poll; let call = 0; const holder: GeminiTransport & { lastVideoRequest?: GeminiVideoRequest; + lastOperationName?: string; lastPollSignal?: AbortSignalLike | undefined; } = { ...unusedGenerative, @@ -113,7 +115,8 @@ function fakeVideoTransport(opts: { holder.lastVideoRequest = request; return Promise.resolve(opts.operation ?? { name: 'operations/veo-1' }); }, - pollVideo: (_operationName, _key, signal) => { + pollVideo: (operationName, _key, signal) => { + holder.lastOperationName = operationName; // the decoded op-name the adapter threads (re-attach, §3) holder.lastPollSignal = signal; const status = polls[Math.min(call, polls.length - 1)]; call += 1; @@ -1054,6 +1057,16 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 expect(transport.lastVideoRequest?.config['abortSignal']).toBe(controller.signal); }); + it('generateMedia (video) omits durationSeconds from the config when not requested', async () => { + const transport = fakeVideoTransport({ operation: { name: 'operations/veo-42' } }); + await genMedia( + createGeminiAdapter({ transport }), + { ...VIDEO_REQ, durationSeconds: undefined }, + 'k', + ); + expect(transport.lastVideoRequest?.config['durationSeconds']).toBeUndefined(); + }); + it('generateMedia (video) maps a missing operation name to a typed bad_request', async () => { const transport = fakeVideoTransport({ operation: { name: '' } }); await expect( @@ -1063,16 +1076,20 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 }); }); - it('pollMediaJob maps an in-progress operation (done:false) to pending', async () => { + it('pollMediaJob maps an in-progress operation (done:false) to pending + threads the DECODED op-name', async () => { const transport = fakeVideoTransport({ poll: { done: false } }); expect( - await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'), - ).toEqual({ - state: 'pending', - }); + await pollMedia( + createGeminiAdapter({ transport }), + encodeMediaJobId('operations/veo-42'), + 'k', + ), + ).toEqual({ state: 'pending' }); + // The opaque jobId is decoded back to the vendor op-name and threaded to the transport (re-attach, §3). + expect(transport.lastOperationName).toBe('operations/veo-42'); }); - it('pollMediaJob walks a pending → done sequence across successive polls', async () => { + it('pollMediaJob walks a pending → done sequence across successive polls (clamps on the last status)', async () => { const transport = fakeVideoTransport({ poll: [{ done: false }, { done: true, video: { videoBytes: B64, mimeType: 'video/mp4' } }], }); @@ -1080,6 +1097,7 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 const jobId = encodeMediaJobId('op'); expect(await pollMedia(adapter, jobId, 'k')).toEqual({ state: 'pending' }); expect(await pollMedia(adapter, jobId, 'k')).toMatchObject({ state: 'done' }); + expect(await pollMedia(adapter, jobId, 'k')).toMatchObject({ state: 'done' }); // clamps on the last }); it('pollMediaJob delivers inline videoBytes as a base64 video/mp4 media part (done)', async () => { @@ -1094,6 +1112,14 @@ describe('Gemini adapter — generateMedia/pollMediaJob (Veo video, async LRO, 1 }); }); + it('pollMediaJob strips MIME parameters from the vendor video MIME (video/mp4;codecs=h264 → video/mp4)', async () => { + const transport = fakeVideoTransport({ + poll: { done: true, video: { videoBytes: B64, mimeType: 'video/mp4; codecs=h264' } }, + }); + const status = await pollMedia(createGeminiAdapter({ transport }), encodeMediaJobId('op'), 'k'); + expect(status).toMatchObject({ media: { mimeType: 'video/mp4' } }); + }); + it('pollMediaJob delivers a uri-only result as a re-hostable url media source (engine de-inlines it)', async () => { const transport = fakeVideoTransport({ poll: { done: true, video: { uri: 'https://generativelanguage.googleapis.com/v1/files/x' } }, diff --git a/packages/llm/src/conformance/generative-seam.conformance.test.ts b/packages/llm/src/conformance/generative-seam.conformance.test.ts index ec1ae6a7..8f790ecc 100644 --- a/packages/llm/src/conformance/generative-seam.conformance.test.ts +++ b/packages/llm/src/conformance/generative-seam.conformance.test.ts @@ -8,9 +8,10 @@ import type { CapabilityFlags, LlmProvider, MediaGenRequest, MediaJobStatus } fr * The GENERATIVE-surface seam-contract conformance suite (1.AG Section E, A5 / [ADR-0045]). The chat * conformance suite replays recorded vendor responses through each real adapter; the ASYNC generative arm * (a `generateMedia` that returns an opaque `jobId`, then `pollMediaJob` driving pending→done/failed) is - * wired for OpenAI/Sora at 1.AH A3 (exercised against the real adapter in `openai.test.ts`); the Gemini/Veo - * async adapter remains deferred (1.AH A4). This suite asserts the SEAM CONTRACT itself against a CONFORMING - * STUB provider — the exact shape every async generative adapter must satisfy, independent of any one vendor: + * wired for OpenAI/Sora (1.AH A3, exercised against the real adapter in `openai.test.ts`) and Gemini/Veo + * (1.AH A4, exercised in `gemini.test.ts` via `fakeVideoTransport`). This suite asserts the SEAM CONTRACT + * itself against a CONFORMING STUB provider — the exact shape every async generative adapter must satisfy, + * independent of any one vendor: * the `MediaGenResult` exactly-one-of refine, an opaque non-empty `jobId`, the discriminated * `MediaJobStatus` states, and a content-policy refusal classified as the `content_filter` LlmError kind. * (The SYNC arm is covered against the real OpenAI adapter by the per-provider suite's `mediaGenerate` From d2d7b437b7f90dd10ccbbad71605488cc88dfedc Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 16:03:29 +0300 Subject: [PATCH 14/21] =?UTF-8?q?feat(llm):=201.AH=20A6=20=E2=80=94=20shar?= =?UTF-8?q?ed=20bareMime=20validation=20+=20gpt-image-1=20size/quality=20k?= =?UTF-8?q?nobs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two adapter-level improvements (A5 save_to double-fetch stays deferred — it's a rare-case perf optimization needing a risky engine-flow refactor; correctness is already fine since the CAS put dedupes the bytes): - bareMimeType (gemini.ts): a shared helper that strips ;-parameters AND validates the vendor MIME via the shared MediaMimeTypeSchema (returns '' on absent/parameter-only/ ILLEGAL incl. a CR/LF-injected value that split(';') alone would pass). Replaces the three split-only sites (mapContent inline media-out, geminiGenerateImage, geminiPollVideo) — closes the A2-N1 cross-arm gap where a CR/LF MIME could reach media_objects.mimeType (the production de-inline path runs no MIME regex). Callers fall back ('' → default) or skip the part. + a CR/LF-rejection test. - gpt-image-1 size/quality knobs (openai.ts): openAiGenerateImage now threads optional size/quality from providerOptions.image.{size,quality}, narrowed to the valid gpt-image-1 values (unrecognized dropped — no 400-inducing passthrough), in the request BODY only (never the SDK RequestOptions — the A2 SSRF lesson). E2E-reachable once the engine populates providerOptions for a generative call (host-wiring). + forward/drop tests. Toolchain 16/16 green; prettier clean; Leakwatch 0; seam fence clean. Refs: ADR-0044, ADR-0031, ADR-0011 Co-Authored-By: Claude Opus 4.8 --- packages/llm/src/adapters/gemini.test.ts | 8 ++++ packages/llm/src/adapters/gemini.ts | 43 +++++++++++------- packages/llm/src/adapters/openai.test.ts | 55 ++++++++++++++++++++++++ packages/llm/src/adapters/openai.ts | 31 +++++++++++++ 4 files changed, 121 insertions(+), 16 deletions(-) diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index fe557389..b623357a 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -895,6 +895,14 @@ describe('Gemini adapter — generateMedia (Imagen, sync, 1.AH A2)', () => { expect(result.media?.mimeType).toBe('image/png'); }); + it('rejects an illegal (CR/LF-injected) vendor MIME → falls back to image/png (bareMimeType validates)', async () => { + const transport = fakeImageTransport({ + generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png\nX-Injected: 1' } }], + }); + const result = await genMedia(createGeminiAdapter({ transport }), IMG_REQ, 'k'); + expect(result.media?.mimeType).toBe('image/png'); // the injected MIME never reaches media_objects.mimeType + }); + it('strips a caller-supplied httpOptions from the Imagen config (SSRF: no baseUrl/key redirect)', async () => { const transport = fakeImageTransport({ generatedImages: [{ image: { imageBytes: B64, mimeType: 'image/png' } }], diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 483fd8fe..12c5e505 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -1,6 +1,6 @@ import { GenerateVideosOperation, GoogleGenAI } from '@google/genai'; -import { mediaModalityOf } from '@relavium/shared'; +import { MediaMimeTypeSchema, mediaModalityOf } from '@relavium/shared'; import type { AbortSignalLike, ContentPart, OutputModality, StopReason } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; @@ -73,6 +73,20 @@ const ZERO_USAGE: Usage = { inputTokens: 0, outputTokens: 0 }; const isRecord = (value: unknown): value is Record => typeof value === 'object' && value !== null && !Array.isArray(value); +/** + * The canonical BARE media MIME (no RFC-2045 parameters) from a vendor-reported type, validated against the + * shared {@link MediaMimeTypeSchema}; returns `''` when the value is absent, parameter-only, or illegal — + * including a CR/LF-injected MIME that a `split(';')` alone would pass through (the chars fail the regex). + * The vendor MIME is the de-facto guard: the production de-inline path (`MediaStore.put` / + * `media_objects.mimeType`) runs NO MIME regex, so callers MUST fall back (`'' → default`) or skip the part + * rather than persist an unvalidated MIME. Shared by all three Gemini media-MIME sites (inline media-out, + * Imagen image-out, Veo video-out). + */ +function bareMimeType(raw: string | undefined): string { + const bare = (raw ?? '').split(';')[0]?.trim() ?? ''; + return MediaMimeTypeSchema.safeParse(bare).success ? bare : ''; +} + /** * True when the prompt was actually blocked. Gemini's `blockReason` enum includes the * `BLOCKED_REASON_UNSPECIFIED` sentinel that does **not** mean "blocked" — treat only a real, @@ -310,11 +324,11 @@ function mapGeminiPart(part: GeminiPart, ids: GeminiToolCallIds): ContentPart | // (1.AG/ADR-0046). Emit an IN-FLIGHT media part; the engine de-inlines it to a handle at #emitDurable // (1.AF). No vendor shape escapes (I1) — only the normalized media ContentPart. Gemini AUDIO output carries // a PARAMETERIZED mime (e.g. `audio/L16;codec=pcm;rate=24000`), but the seam's MediaMimeTypeSchema admits - // only a BARE type/subtype — strip parameters to the bare prefix (the modality derives from it; the durable - // media part cannot carry parameters anyway). A pathological `;…`-only value strips to empty and is SKIPPED - // (returns undefined — still consumed, never re-interpreted as text), symmetric with the empty-data skip, - // never emitted as a doomed `application/octet-stream` that would HARD-FAIL the de-inline. - const bareMime = inline.mimeType.split(';')[0]?.trim() ?? ''; + // only a BARE type/subtype — bareMimeType strips parameters + validates (the modality derives from it; the + // durable part cannot carry parameters). A pathological `;…`-only OR illegal (CR/LF) value strips/validates + // to empty and is SKIPPED (returns undefined — still consumed, never re-interpreted as text), symmetric with + // the empty-data skip, never emitted as a doomed part that would HARD-FAIL the de-inline. + const bareMime = bareMimeType(inline.mimeType); return bareMime.length > 0 ? { type: 'media', mimeType: bareMime, source: { kind: 'base64', data: inline.data } } : undefined; @@ -802,13 +816,10 @@ async function geminiGenerateImage( }), ); } - // The vendor reports the ACTUAL bytes' MIME; default to png (Imagen's default) when absent. Strip any - // `;`-parameters (e.g. `image/png; q=1.0`) to the canonical bare MIME — the production de-inline path - // (MediaStore.put / media_objects.mimeType) has no parameter-stripping CHECK, and the generate() arm - // (mapContent) strips identically. (`first?.` only narrows the type — the non-empty-b64 guard above - // already proves `first` exists at runtime.) - const rawMime = first?.image?.mimeType ?? ''; - const mimeType = rawMime.split(';')[0]?.trim() || 'image/png'; + // The vendor reports the ACTUAL bytes' MIME; bareMimeType strips `;`-parameters + validates (the + // production de-inline path runs no MIME regex), defaulting to png (Imagen's default) on an absent/illegal + // value. (`first?.` only narrows the type — the non-empty-b64 guard above already proves `first` exists.) + const mimeType = bareMimeType(first?.image?.mimeType) || 'image/png'; return { media: { type: 'media', mimeType, source: { kind: 'base64', data: b64 } }, raw: response, @@ -904,9 +915,9 @@ async function geminiPollVideo( }; } const video = poll.video; - // Strip `;`-parameters to the canonical bare MIME (vendor-reported), default video/mp4 — same as the - // image arm (the shared bareMime consolidation is the A6 N1 carry-forward). - const mimeType = (video?.mimeType ?? '').split(';')[0]?.trim() || 'video/mp4'; + // bareMimeType strips `;`-parameters + validates the vendor MIME, defaulting to video/mp4 — same shared + // helper as the inline + Imagen arms. + const mimeType = bareMimeType(video?.mimeType) || 'video/mp4'; if (video?.videoBytes !== undefined && video.videoBytes.length > 0) { return { state: 'done', diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 411f86cc..d606ea96 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -469,6 +469,61 @@ describe('OpenAI-compatible adapter', () => { expect(result.media?.mimeType).toBe('image/webp'); }); + it('generateMedia (image) forwards valid size/quality knobs from providerOptions.image; drops invalid (1.AH A6)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response(JSON.stringify({ created: 0, data: [{ b64_json: 'aW1n' }] }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ); + }, + }); + await genMedia( + adapter, + { + model: 'gpt-image-1', + prompt: 'x', + modality: 'image', + providerOptions: { image: { size: '1536x1024', quality: 'high', bogus: 'x' } }, + }, + 'k', + ); + expect(sent['size']).toBe('1536x1024'); + expect(sent['quality']).toBe('high'); + expect(sent['bogus']).toBeUndefined(); // only the recognized knobs are forwarded (no spread) + }); + + it('generateMedia (image) drops an unrecognized size/quality (no 400-inducing passthrough)', async () => { + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response(JSON.stringify({ created: 0, data: [{ b64_json: 'aW1n' }] }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ); + }, + }); + await genMedia( + adapter, + { + model: 'gpt-image-1', + prompt: 'x', + modality: 'image', + providerOptions: { image: { size: '99x99', quality: 'ultra' } }, + }, + 'k', + ); + expect(sent['size']).toBeUndefined(); + expect(sent['quality']).toBeUndefined(); + }); + it('generateMedia maps an image content-policy refusal to content_filter (the documented taxonomy)', async () => { const adapter = createOpenAiAdapter({ fetch: () => diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index 790a8632..c4b66700 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -199,6 +199,33 @@ function imageOutputFormat(mimeType: string | undefined): 'png' | 'jpeg' | 'webp } } +/** gpt-image-1 frame sizes (1.AH A6). */ +type ImageSize = '1024x1024' | '1024x1536' | '1536x1024' | 'auto'; +/** gpt-image-1 quality tiers (1.AH A6). */ +type ImageQuality = 'low' | 'medium' | 'high' | 'auto'; + +/** Optional gpt-image-1 `size` knob from `providerOptions.image.size` (mirrors ttsVoice's providerOptions + * nesting); an unrecognized value is dropped so the model default applies. providerOptions rides the + * REQUEST BODY only — never the SDK RequestOptions (which holds baseURL/headers) — so no SSRF surface. */ +function imageSize(providerOptions: MediaGenRequest['providerOptions']): ImageSize | undefined { + const imageOpts = isRecord(providerOptions) ? providerOptions['image'] : undefined; + const size = isRecord(imageOpts) ? imageOpts['size'] : undefined; + return size === '1024x1024' || size === '1024x1536' || size === '1536x1024' || size === 'auto' + ? size + : undefined; +} + +/** Optional gpt-image-1 `quality` knob from `providerOptions.image.quality`; an unrecognized value is dropped. */ +function imageQuality( + providerOptions: MediaGenRequest['providerOptions'], +): ImageQuality | undefined { + const imageOpts = isRecord(providerOptions) ? providerOptions['image'] : undefined; + const quality = isRecord(imageOpts) ? imageOpts['quality'] : undefined; + return quality === 'low' || quality === 'medium' || quality === 'high' || quality === 'auto' + ? quality + : undefined; +} + /** Map a requested output-audio `format` (providerOptions.audio.format) to its MIME — OpenAI's response * echoes no format, so the requested one types the media part. Defaults to `audio/wav` (OpenAI's default). */ export function outputAudioMime(req: LlmRequest): string { @@ -1010,6 +1037,8 @@ async function openAiGenerateImage( ); } const outputFormat = imageOutputFormat(req.mimeType); + const size = imageSize(req.providerOptions); + const quality = imageQuality(req.providerOptions); let response: OpenAI.ImagesResponse; try { response = await client.images.generate( @@ -1017,6 +1046,8 @@ async function openAiGenerateImage( model: req.model, prompt: req.prompt, ...(outputFormat === undefined ? {} : { output_format: outputFormat }), + ...(size === undefined ? {} : { size }), + ...(quality === undefined ? {} : { quality }), }, isAbortSignal(req.signal) ? { signal: req.signal } : {}, ); From 8f09ac73e9923a3d32af21c2f2783c7a05ee3a0e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 21 Jun 2026 16:34:01 +0300 Subject: [PATCH 15/21] =?UTF-8?q?docs(roadmap):=201.AH=20PART=20C=20?= =?UTF-8?q?=E2=80=94=20record=20later-phase=20media=20work=20in=20the=20ph?= =?UTF-8?q?ase=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Analyze the media host-wiring that 1.AH deliberately left to later phases and record it as tasks in each phase doc (rule 8: task + canonical ADR/deferred-tasks pointer, never a spec restatement). No HW/S labels invented — each task is named by its mechanism + the canonical D-series / ADR pointer (matching deferred-tasks.md): - phase-1 §1.AH: expand the narrative bullet into the landed A1–A6 adapter list (the four generative adapters + shared opaque-jobId codec + bareMime, behind the seam, not yet runtime-reachable), the deferred decisions (count>1 ADR, pricing rows, the Responses-API path), and the host-wiring-spans-2–6 surface map + the build-with-the- first-surface warning. A5 noted deferred. - phase-2 §2.S (NEW): the CANONICAL home for the cross-surface host-wiring task list — resolveMediaSurface, the read_media MediaReadAccess factory, session-scope media_references, the load-check, the cost governor, resolveForEgress, the save_to write port, and the EgressCapability.fetch SSRF mechanism (security-critical) + milestone/dependency rows. - phase-3 §3.B: split into the desktop Rust CAS + handle-only de-inline integ test + the read_media Tauri byte gate + save_to MediaWritePort + the keychain no-raw-key IPC test + canvas rendering + a points-to-§2.S shared-wiring bullet + the reserved-triad note + exit criterion #9 + a Risks row + the media ADR Related links. - phase-4 §4.N (NEW): VS Code FS-CAS host-wiring (points to §2.S) + webview-resource rendering + the SSRF mechanism + a P4.M7 milestone + ADR links. - phase-5: managed gateway media poll-through (pass-through, not a store), mediaUnits metering + reserve→settle, durable settle of a billed-but-failed async job, the no-logging-extends-to-media-bytes guarantee, the per-modality $ dashboard axis, ADRs. - phase-6: object-storage MediaStore + BullMQ sweep, media-table tenancy/RLS + cross-org CAS dedup, GET /media/{handle} byte gate, durable failed/cancelled cost, the reserved {kind:'workspace'} read_media scope, and the record-only portal preview triad. Prettier clean; Leakwatch 0; all referenced ADR/phase links verified to exist. Refs: ADR-0031, ADR-0032, ADR-0042, ADR-0043, ADR-0044, ADR-0045, ADR-0046, ADR-0015 Co-Authored-By: Claude Opus 4.8 --- docs/roadmap/phases/phase-1-engine-and-llm.md | 48 +++++-- docs/roadmap/phases/phase-2-cli.md | 118 ++++++++++++++++++ docs/roadmap/phases/phase-3-desktop.md | 13 +- docs/roadmap/phases/phase-4-vscode.md | 47 ++++++- .../phases/phase-5-managed-inference.md | 6 + .../phases/phase-6-cloud-execution-portal.md | 12 ++ 6 files changed, 232 insertions(+), 12 deletions(-) diff --git a/docs/roadmap/phases/phase-1-engine-and-llm.md b/docs/roadmap/phases/phase-1-engine-and-llm.md index 4e3211a8..253adea7 100644 --- a/docs/roadmap/phases/phase-1-engine-and-llm.md +++ b/docs/roadmap/phases/phase-1-engine-and-llm.md @@ -1062,16 +1062,44 @@ phases (2–6). Each phase below maps to the design doc's Phase A–E. dual-reviewed (0 blocker/high), then a final Opus-consolidated pre-merge review that caught + fixed two HIGH cross-fix defects (the orphaned-vertex cost addend + the crash-in-window `run:paused`). All four acceptance criteria are tested. The genuinely-remaining vendor-adapter + host-wiring work is **1.AH** (deferred-tasks.md). -- **1.AH — Surfaces & managed mode (Phase E, spans Phases 2–6).** Desktop: - **[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)** Rust-side media - de-inline on egress + session-scoped `read_media` command + the Rust CAS — **must land before any - desktop media-output** (phase-3-desktop). CLI/VS Code media rendering (phase-2-cli / phase-4-vscode). - Managed-mode gateway media materialization-to-user-store + `mediaUnits` metering, reconciled with - [ADR-0015](../../decisions/0015-managed-mode-data-handling-and-compliance.md) counts-not-content - (phase-5-managed-inference). *Acceptance:* on desktop, media bytes never transit the WebView↔Rust - channel (only handles do); **`read_media` is the only byte path out — no second raw static mount, - `follow_symlink` off, `realpath`+`commonpath` fail-closed**; each surface renders a produced media - handle; managed mode meters counts and stores no artifact. +- **1.AH — Generative media-output adapters + surfaces & managed mode (Phase E, spans Phases 2–6).** The + Phase-1-doable half landed (unmerged): the **four generative media-output adapters** behind the + `@relavium/llm` seam — OpenAI-TTS (audio), Gemini-Imagen (image), OpenAI/Sora (async video), Gemini/Veo + (async video) — plus the shared **opaque-`jobId` encode/decode** (ADR-0045 §7) and the shared **bare-MIME + validator**, all proven with stubs + the generative conformance suite. They are **not yet runtime-reachable**: + each needs the host per-model `media_surface` lookup (the deferred wiring below). Adapter pieces, behind the + seam ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md) §4, + [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md), + [ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)): + - **A1 — OpenAI-TTS audio** (`audio.speech` binary → base64 + `response_format`↔MIME map, no new dep). + - **A2 — Gemini-Imagen image** (`generateImages` → `generatedImages[].image.imageBytes`, via a vendor-free + `GeminiTransport.generateImages` extension). + - **A3 — OpenAI/Sora async video** (`generateMedia`→`{ jobId }` + `pollMediaJob`, the engine LRO loop, 1.AG D). + - **A4 — Gemini/Veo async video** (`generateMedia`→`{ jobId }` + `pollMediaJob`, same LRO loop). + - **A5 — `save_to` url double-fetch** stays **DEFERRED** (rare-case perf; correctness fine) — deferred-tasks.md §1.AF. + - **A6 — bare-MIME validator + gpt-image-1 `size`/`quality` knobs** (the per-knob `MediaGenRequest.providerOptions` passthrough). + - *Deferred decisions (each needs its own future record — do NOT pre-empt here):* **multi-artifact `count > 1`** + needs a future media-array ADR amending [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md) + (the SYNC `MediaGenResult.media` carries a SINGLE part — never bill-N-deliver-1); **verified generative + pricing rows** are deferred (adapters ship at **0-cost**, gated/folded correctly — never fabricate a rate); + **OpenAI agentic image-gen via the Responses API** is deferred ([ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §3). All three are recorded in deferred-tasks.md §Multimodal / §1.AF. + - **Surfaces & managed mode — the HOST-WIRING half spans Phases 2–6** (the deferred mechanism for D5/D8/D12/D15/D17 + + the per-model `media_surface` lookup + verified rates; deferred-tasks.md §1.AF-P4 / §Multimodal). Desktop: + **[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)** Rust-side media de-inline on + egress + session-scoped `read_media` command + the Rust CAS — **must land before any desktop media-output** + ([phase-3-desktop.md §3.B](phase-3-desktop.md)). CLI media rendering + the host `media_surface` resolver + + `EgressCapability.fetch` SSRF mechanism ([phase-2-cli.md §2.S](phase-2-cli.md)); VS Code media rendering + ([phase-4-vscode.md §4.N](phase-4-vscode.md)). Managed-mode gateway media materialization-to-user-store + + `mediaUnits` metering, reconciled with [ADR-0015](../../decisions/0015-managed-mode-data-handling-and-compliance.md) + counts-not-content ([phase-5-managed-inference.md](phase-5-managed-inference.md) media bullets); cloud-execution + media handling ([phase-6-cloud-execution-portal.md](phase-6-cloud-execution-portal.md) media tasks). + - **⚠ Sequencing warning (deferred-tasks.md):** do **NOT** build the host media-surface resolver (or the + D12 `MediaReadAccess`/scope-population) **in isolation** — build them **WITH the first surface (CLI, §2.S)**, + designed to fit desktop/VS Code, so the inert `read_media` result-shape + base64-vs-handle contract is + resolved coherently once (fixing it standalone risks conflicting with the 1.AH host design). + *Acceptance:* on desktop, media bytes never transit the WebView↔Rust channel (only handles do); **`read_media` + is the only byte path out — no second raw static mount, `follow_symlink` off, `realpath`+`commonpath` + fail-closed**; each surface renders a produced media handle; managed mode meters counts and stores no artifact. ## Milestones diff --git a/docs/roadmap/phases/phase-2-cli.md b/docs/roadmap/phases/phase-2-cli.md index 3e443386..54c60534 100644 --- a/docs/roadmap/phases/phase-2-cli.md +++ b/docs/roadmap/phases/phase-2-cli.md @@ -467,6 +467,117 @@ is deliberately not in this slot. reach the server only via its spawn-time `env`; a private-range MCP `url` is rejected without the explicit opt-in; the import-zone check confirms no SDK type leaks past the integration layer. +### 2.S — Media host-wiring (1.AH / Phase-2) + +The surface half of the multimodal sub-spine. The engine-pure media **policy** landed +in Phase 1 (1.AF/1.AG: the `MediaStore`/`deInlineMedia` choke point, `read_media` + +scope-set authz, the byte-delivery `Range` gate, the `output_modalities` load-check, +the per-modality cost governor, the `save_to` write port, the inline media-out path, +and the engine-owned async media-job LRO); 1.AH then landed the four generative +media-output adapters behind the `@relavium/llm` seam. None of it is reachable +end-to-end until a **host** wires it. The CLI is the first surface to do so, so this +is the canonical home for the host-wiring task list — desktop ([phase-3-desktop.md](phase-3-desktop.md)) +and VS Code ([phase-4-vscode.md](phase-4-vscode.md)) point here for the shared +obligations and add only their surface-specific deltas (the desktop Rust de-inline + +Rust CAS per [ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)). +Each task is the deferred **mechanism/wiring** half of a 1.AF/1.AG decision-point; +the canonical pointers are the media ADRs ([ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md)–[ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)) +and the D-series entries in [../deferred-tasks.md](../deferred-tasks.md) (§1.AF P4 + +Multimodal forward-obligations). + +**Off the M3 critical path** — built *with* the CLI but **designed to fit +desktop/VS Code**: each port lives behind an injectable seam (`AgentRunnerDeps`, +`MediaReadAccess`, the write port) so the desktop's Rust CAS and the VS Code host can +reuse it. The warning: do not let a CLI-shaped shortcut leak into a port the other +surfaces inherit (treat any such friction as a Phase-1 amendment, per Risks). Media +runs at **0 cost** until verified generative pricing rows land (never fabricate a +rate); multi-image `count > 1` and the OpenAI Responses-API image-gen surface stay +deferred (each needs a future seam amendment — see +[deferred-tasks.md](../deferred-tasks.md)). + +**Tasks:** + +- **`resolveMediaSurface` (`AgentRunnerDeps`) — the top routing gate.** Wire the host + lookup that reads `model_catalog.media_surface` and supplies + `resolveMediaSurface?(model) → MediaSurface`, the discriminator that routes a + generative-surface model to `generateMedia` / `pollMediaJob` (default `'chat'`). + Until wired, every model routes inline and no generative model is runtime-reachable. + *([ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §1; deferred-tasks.md 1.AG Section C → 1.AH.)* +- **The `read_media` host `MediaReadAccess` factory.** Build the host factory that + bridges `MediaReferenceStore.describe()` + `MediaStore.readRange()` into the + `MediaReadAccess` the tool needs; it must return a **handle-bearing** result (durable + form, resolved on egress by the seam), not inline base64, so the next LLM call's + `containsInlineMediaBytes` refine does not reject the turn — and **narrow** + `readRange` to that chosen form (not the wide `MediaSource`) and **thread** + `AbortSignalLike` into `describe`/`readRange`. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1; deferred-tasks.md D12.)* +- **Session-kind `media_references` population (D12 authz data).** Write + `session` (and, when a shared-asset consumer lands, `workspace`) `media_references` + rows at the node/session input-transfer boundary so `describe().allowedScopes` is + non-empty; today the only writer emits `run` refs and every read denies. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1; deferred-tasks.md D12.)* +- **Thread `ctx.mediaRead` + `requestingScope` into `ToolDispatchContext`.** The + AgentRunner + AgentSession build the dispatch context without these, so `read_media` + currently throws `ToolUnavailableError` (fail-closed). Inject the `MediaReadAccess` + impl and the requesting scope so the tool resolves in the engine path. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1; deferred-tasks.md D12.)* +- **`validateWorkflowWithCatalog` post-parse (the D15 load-check).** Call the exported, + tested load-check from the CLI load path with the DB `model_catalog` so authored + `output_modalities` are validated at parse time, not only by the runtime + FallbackChain pre-skip. + *([ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md); deferred-tasks.md D15.)* +- **`[defaults].media_cost_estimate` → `mediaCostEstimate` (the D17 cost governor).** + Read the config key and thread it into `createAgentNodeExecutor` so the pre-egress + governor uses the configured per-modality estimate instead of the built-in + `DEFAULT_MEDIA_UNIT_ESTIMATE`. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3; deferred-tasks.md D17.)* +- **`resolveForEgress` (the D8 FallbackChain resolve-before-egress).** Inject the + re-materialization hook on the AgentRunner so a durable handle in a transcript + message is resolved before egress; without it the D7/D8 failover re-materialization + is inert (a handle is sent unchanged). + *([ADR-0043](../../decisions/0043-media-egress-failover-rematerialization-ssrf.md); deferred-tasks.md D8.)* +- **`createFilesystemMediaWrite` — the `save_to` write port.** Implement the host + filesystem write port with `realpath` + `commonpath` **fail-closed** containment + (no symlink escape) so an output node's `save_to` lands bytes in the workspace. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §2.)* +- **`save_to` `run.id` namespace.** Scope the written artifact path under the run id + namespace (so concurrent/repeat runs do not collide), and surface the load-time + `run.id`-only restriction so a non-`run.id` `save_to` ref errors at load, not runtime. + *([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §2; deferred-tasks.md §1.AF-P4 save_to.)* +- **The `EgressCapability.fetch` SSRF mechanism (DNS-resolve + connect-by-IP + + redirect re-validate) — SECURITY-CRITICAL.** Implement the host-side runtime half of + the SSRF defense: resolve the hostname, validate the IP against the shared range + block, pin the connection to that validated IP, and re-validate on every redirect + hop. The shared 1.AE primitive covers only construction-time **policy**; this is the + mechanism that catches DNS rebinding / a public hostname resolving to a private IP. + **Flag for explicit security review.** + *([ADR-0043](../../decisions/0043-media-egress-failover-rematerialization-ssrf.md); deferred-tasks.md §Multimodal — host-side SSRF enforcement in `EgressCapability.fetch`.)* +- **Render a produced media handle on `node:completed` (CLI surface).** Render the + produced media **handle** (never inline bytes) in the TUI and `--json` paths when a + node emits media output — the CLI's leaf of the cross-surface + "each surface renders a produced media handle" acceptance. + *([ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md); deferred-tasks.md 1.AG.)* +- **`[chat].max_turns` → `SessionDeps.maxTurns`.** Read the config turn cap and thread + it into the `AgentSession` deps so a media (or any) chat turn budget is + host-configurable. *(config-spec.md; 2.M–2.Q.)* +- **Host media GC + durable fail-cost.** Run the host media garbage collection — the + CAS-orphan sweep (row-less `save_to` bytes from a crash window) + the + clean-terminal reclaim retry keyed on terminal run events — and snapshot durable + realized cost onto `node:failed` / `run:cancelled` for a billed-but-failed paid media + job (the live `cost:updated` stream is the only carrier today). + *([ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md) §4, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; deferred-tasks.md §1.AF-P4 GC + review M1.)* + +**Acceptance:** a fixture workflow with a generative media-output node runs end-to-end +via `relavium run`, routed by a host `resolveMediaSurface` to `generateMedia` / +`pollMediaJob`; `read_media` resolves a scoped handle (and denies an out-of-scope one) +with no inline bytes crossing into the next turn; `save_to` writes a containment-checked +artifact under the `run.id` namespace and rejects an escape attempt; the SSRF +mechanism rejects a hostname that resolves to a private IP (and re-validates on +redirect) — covered by a dedicated security review; the load-check rejects an +unsupported authored `output_modalities`; media cost folds at **0** until a verified +rate lands; and the CLI renders the produced media handle in both the TUI and `--json` +streams. Built behind injectable ports so desktop (§3.B) and VS Code (§4.N) reuse them. + ## Milestones | In-phase milestone | Completed by | Global milestone | @@ -481,6 +592,7 @@ explicit opt-in; the import-zone check confirms no SDK type leaks past the integ | Published, installable binary verified on all OSes | 2.L | — | | **Agent-first CLI** — `relavium chat` + session commands (resume / list / export / `agent run` / `gate list`): the **first user-facing `AgentSession` surface**, a committed build-phase-2 deliverable (off the M3 critical path, but a phase exit item — the agent-first headline is demonstrable here) | 2.M, 2.N, 2.O, 2.P, 2.Q | — | | **MCP client live** — a fixture agent completes a real stdio MCP tool round-trip behind the `ToolRegistry`, per [ADR-0034](../../decisions/0034-mcp-client-sdk-dependency.md) (off the M3 critical path) | 2.R | — | +| **Media host-wiring** — a generative media-output fixture runs end-to-end on the CLI (host `resolveMediaSurface` routing, scoped `read_media`, containment-checked `save_to`, the `EgressCapability.fetch` SSRF mechanism), the shared ports designed to fit desktop/VS Code, per the media ADRs ([ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md)–[ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md)) (off the M3 critical path) | 2.S | — | ## Dependencies @@ -500,6 +612,12 @@ explicit opt-in; the import-zone check confirms no SDK type leaks past the integ [`RunEvent` schema](../../reference/contracts/sse-event-schema.md), the [config spec](../../reference/contracts/config-spec.md), and the [keychain/secret model](../../reference/desktop/keychain-and-secrets.md). +- **Media engine policy (1.AF/1.AG)** complete behind the `@relavium/llm` seam — the + `MediaStore`/`deInlineMedia` choke point, `read_media` authz, the `save_to` write + port, the load-check, the cost governor, and the async media-job LRO — per + [ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md)–[ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md); + the host-wiring tasks (2.S) discharge the deferred mechanism half tracked in + [deferred-tasks.md](../deferred-tasks.md) (§1.AF P4 + Multimodal forward-obligations). - **`@relavium/shared`** Zod schemas for workflow/agent/config validation (Phase 0). ## Exit criteria (go / no-go) diff --git a/docs/roadmap/phases/phase-3-desktop.md b/docs/roadmap/phases/phase-3-desktop.md index 620eb6f5..ec0eb289 100644 --- a/docs/roadmap/phases/phase-3-desktop.md +++ b/docs/roadmap/phases/phase-3-desktop.md @@ -2,7 +2,7 @@ > Status: Not started (Product Phase 1). Blocked on Phase 2. -- **Related**: [../README.md](../README.md), [phase-2-cli.md](phase-2-cli.md), [phase-4-vscode.md](phase-4-vscode.md), [../../architecture/desktop-architecture.md](../../architecture/desktop-architecture.md), [../../architecture/state-management.md](../../architecture/state-management.md), [../../reference/desktop/routes-and-screens.md](../../reference/desktop/routes-and-screens.md), [../../reference/desktop/tauri-plugins.md](../../reference/desktop/tauri-plugins.md), [../../reference/desktop/database-schema.md](../../reference/desktop/database-schema.md), [../../reference/desktop/keychain-and-secrets.md](../../reference/desktop/keychain-and-secrets.md), [../../reference/contracts/ipc-contract.md](../../reference/contracts/ipc-contract.md), [../../reference/contracts/sse-event-schema.md](../../reference/contracts/sse-event-schema.md), [../../reference/shared-core/store-shapes.md](../../reference/shared-core/store-shapes.md), [../../reference/shared-core/node-types.md](../../reference/shared-core/node-types.md), [../../decisions/0001-tauri-v2-over-electron.md](../../decisions/0001-tauri-v2-over-electron.md), [../../decisions/0007-desktop-is-not-an-ide.md](../../decisions/0007-desktop-is-not-an-ide.md), [../../decisions/0010-zustand-direct-subscriptions-for-reactflow.md](../../decisions/0010-zustand-direct-subscriptions-for-reactflow.md), [../../decisions/0018-desktop-execution-and-rust-egress.md](../../decisions/0018-desktop-execution-and-rust-egress.md), [../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md) +- **Related**: [../README.md](../README.md), [phase-2-cli.md](phase-2-cli.md), [phase-4-vscode.md](phase-4-vscode.md), [../../architecture/desktop-architecture.md](../../architecture/desktop-architecture.md), [../../architecture/state-management.md](../../architecture/state-management.md), [../../reference/desktop/routes-and-screens.md](../../reference/desktop/routes-and-screens.md), [../../reference/desktop/tauri-plugins.md](../../reference/desktop/tauri-plugins.md), [../../reference/desktop/database-schema.md](../../reference/desktop/database-schema.md), [../../reference/desktop/keychain-and-secrets.md](../../reference/desktop/keychain-and-secrets.md), [../../reference/contracts/ipc-contract.md](../../reference/contracts/ipc-contract.md), [../../reference/contracts/sse-event-schema.md](../../reference/contracts/sse-event-schema.md), [../../reference/shared-core/store-shapes.md](../../reference/shared-core/store-shapes.md), [../../reference/shared-core/node-types.md](../../reference/shared-core/node-types.md), [../../decisions/0001-tauri-v2-over-electron.md](../../decisions/0001-tauri-v2-over-electron.md), [../../decisions/0007-desktop-is-not-an-ide.md](../../decisions/0007-desktop-is-not-an-ide.md), [../../decisions/0010-zustand-direct-subscriptions-for-reactflow.md](../../decisions/0010-zustand-direct-subscriptions-for-reactflow.md), [../../decisions/0018-desktop-execution-and-rust-egress.md](../../decisions/0018-desktop-execution-and-rust-egress.md), [../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md), [../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md), [../../decisions/0043-media-egress-failover-rematerialization-ssrf.md](../../decisions/0043-media-egress-failover-rematerialization-ssrf.md), [../../decisions/0044-media-access-governance-read-media-save-to-cost.md](../../decisions/0044-media-access-governance-read-media-save-to-cost.md), [../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) ## Goal @@ -187,6 +187,12 @@ and receive its event stream. requesting session is in the handle's scope-set, bounds returned size) that serves display bytes **off** the hot channel. Text/tool/reasoning chunks are still framed verbatim; this is the one narrow case where Rust understands chunk content. +- **De-inline acceptance — handle-only on the WebView↔Rust channel ([ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md); invariant I3).** Add an integration test over the `Channel` that runs a media-bearing `llm_stream` and asserts **no** base64/byte payload ever frames on the channel — only a Relavium `media://sha256-` handle — proving I3 is literally true across the boundary. (The CAS write itself is ADR-0042; this is its on-the-wire proof.) +- **`read_media` Tauri command — the full byte gate ([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1; the [deferred-tasks](../deferred-tasks.md) D12 mechanism half).** Implement the off-channel display-bytes command with every gate the engine policy presumes: `Range` validated against `byteLength` (never an unbounded read), `realpath` + common-path containment fail-closed, `follow_symlink` OFF, **no** second raw static mount, scope-set membership before any byte is served, a **secret-free `media_scope_denied`** on a miss, and a `mimeType`-bound response. This is the desktop concretion of the D12 `MediaReadAccess` factory + session-scope population — wire the host factory per the §2.S shared host-wiring tasks. +- **`save_to` `MediaWritePort` + the CAS-orphan sweep ([ADR-0042](../../decisions/0042-engine-media-storage-substrate-mediastore-deinline-retention.md) §4; the [deferred-tasks](../deferred-tasks.md) `save_to` GC prose).** Implement the desktop `MediaWritePort` so an output node's `save_to` materializes bytes to the CAS, and add the **CAS-orphan sweep** that reclaims row-less bytes left by a crash between the CAS put and the `media_objects` row write (the deferred-tasks CAS-orphan / clean-terminal-reclaim-retry items name this host sweep). +- **Keychain no-raw-key IPC test ([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §4 acceptance gate; the [deferred-tasks](../deferred-tasks.md) deferred 1.AF keychain-bridge test).** This phase introduces the keychain/media IPC command surface, so it owns the ADR-0044 §4 deliverable deferred from 1.AF: a direct test asserting **no IPC command ever returns a raw key** (status + last-4 hint only), bundled with the media byte-delivery review surface. +- **Shared engine host-wiring — apply through the desktop engine bridge.** The cross-surface host-wiring mechanisms — `resolveMediaSurface`, the `read_media` `MediaReadAccess` factory, the `validateWorkflowWithCatalog` load-check, the media cost governor, and `resolveForEgress` — have their canonical home in the **CLI §2.S host-wiring tasks** ([phase-2-cli.md](phase-2-cli.md)). Wire the same set into the WebView-resident engine bridge here (do not re-list them — Rule 8); their decision points are the [deferred-tasks](../deferred-tasks.md) D-series (D8/D12/D15/D17) under §1.AF-P4 / §Multimodal. +- **Reserved — record-only, do NOT build ([ADR-0046](../../decisions/0046-inline-media-out-via-generate-streaming-triad-deferred.md) §4).** The streaming media triad (`media_start`/`media_delta`/`media_end`) and `partialRef` progressive-write semantics touch the **frozen `StreamChunk` seam** and are host-deferred. Acknowledge them here for forward-compat; their desktop Rust-CAS wiring is the later progressive-preview surface, not this phase. - Honor backpressure on the egress channel: when the WebView consumer lags, the `Channel` buffer fills and the Rust sender awaits (no dropped chunks), per the channel semantics in the IPC contract. @@ -327,6 +333,7 @@ path.** - Implement the **human-gate overlay** at the root layout: render on `human_gate:paused`, block the canvas, collect the decision, and call `resume_run` with the `GateDecision`; reflect `human_gate:resumed`. +- **Render media output on the canvas ([ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1; via `read_media`).** When a `node:completed` carries a media handle, fetch its display bytes through the `read_media` command (off the hot channel) and render an `` / `