diff --git a/.changeset/anthropic-opus-5-tool-map.md b/.changeset/anthropic-opus-5-tool-map.md new file mode 100644 index 0000000000..d91649d8ae --- /dev/null +++ b/.changeset/anthropic-opus-5-tool-map.md @@ -0,0 +1,5 @@ +--- +'@tanstack/ai-anthropic': patch +--- + +Register `claude-opus-5` and `claude-opus-5-fast` on the per-model tool-capabilities map so they match `ANTHROPIC_MODELS`. diff --git a/.changeset/grok-imagine-catch-up.md b/.changeset/grok-imagine-catch-up.md new file mode 100644 index 0000000000..b54bdbac65 --- /dev/null +++ b/.changeset/grok-imagine-catch-up.md @@ -0,0 +1,9 @@ +--- +'@tanstack/ai-grok': minor +--- + +Catch up with the current xAI Imagine / Voice catalog: + +- **Image**: add `grok-imagine-image-2.0` (xAI's recommended model, $0.04/image) with its 2.0-only `quality: 'low' | 'medium'` provider option. +- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference' | 'character'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3) — 1.5-only, typed per model and gated at runtime. Image-to-video and reference-to-video cannot be combined. Video editing and extension land on `grok-imagine-video` only, via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). Because edit/extend outputs inherit the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` (and `duration` in edit mode) in those modes instead of sending fields the API ignores. +- **Voice**: add `grok-voice-think-fast-2.0` (current recommended) and the `grok-voice-latest` alias to the realtime models; the realtime token and adapter defaults move off the deprecated 1.0 ids to `grok-voice-think-fast-2.0`. diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 832ac99866..c95c1a9f60 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -181,18 +181,21 @@ const result = await generateImage({ console.log(result.images); ``` -The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-quality`) -are aspect-ratio sized — `size` takes an `aspectRatio_resolution` template -like `"16:9_2k"` (the `_2k` suffix is optional): +The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-2.0`, +`grok-imagine-image-quality`) are aspect-ratio sized — `size` takes an +`aspectRatio_resolution` template like `"16:9_2k"` (the `_2k` suffix is +optional). `grok-imagine-image-2.0` is xAI's recommended model and adds a +2.0-only `quality` provider option (`'low' | 'medium'`, default `'medium'`): ```typescript import { generateImage } from "@tanstack/ai"; import { grokImage } from "@tanstack/ai-grok"; const result = await generateImage({ - adapter: grokImage("grok-imagine-image"), + adapter: grokImage("grok-imagine-image-2.0"), prompt: "A futuristic cityscape at sunset", size: "16:9_2k", + modelOptions: { quality: "medium" }, }); ``` @@ -238,16 +241,16 @@ Generate short video clips (1–15 seconds, with audio) with the Grok Imagine vi Available models: -- `grok-imagine-video` (v1.0) — text-to-video and image-to-video, $0.05 per second of video. -- `grok-imagine-video-1.5` — **image-to-video only**, $0.08 per second of video. A text-only prompt is rejected by the API; the adapter fails fast with a clear error telling you to add a starting-frame image or use `grok-imagine-video`. +- `grok-imagine-video` (v1.0) — text-to-video, image-to-video, and source-video edit / extend, $0.05 per second of video. +- `grok-imagine-video-1.5` — xAI's recommended default, $0.08 per second of video. Supports text-to-video (with native 1080p), image-to-video, and reference-to-video. It does not accept a source video. -Text-to-video with the base `grok-imagine-video` model: +Text-to-video: ```typescript import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; -const adapter = grokVideo("grok-imagine-video"); +const adapter = grokVideo("grok-imagine-video-1.5"); // 1. Create the job const { jobId } = await generateVideo({ @@ -267,7 +270,7 @@ while (status.status !== "completed" && status.status !== "failed") { console.log(status.url); // hosted .mp4 URL ``` -For image-to-video (required for `grok-imagine-video-1.5`, optional for `grok-imagine-video`), include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: +For image-to-video, include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -290,7 +293,78 @@ const { jobId } = await generateVideo({ }); ``` -Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. +Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and (on `grok-imagine-video-1.5` text-to-video / image-to-video only) `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. + +### Reference-to-Video + +On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` (or `'character'`) become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p. A starting-frame image and reference inputs cannot be combined — xAI rejects that mix with 400. Reference inputs are a 1.5-only feature — the adapter rejects them on `grok-imagine-video`: + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video-1.5"), + prompt: [ + { + type: "text", + content: " walks through a neon-lit alley while narrates", + }, + { + type: "image", + source: { type: "url", value: "https://example.com/character.png" }, + metadata: { role: "reference" }, + }, + ], + size: "16:9_720p", + modelOptions: { + reference_audios: [{ voice_id: "eve" }], + }, +}); +``` + +### Video Editing and Extension + +`grok-imagine-video` (v1.0) can rewrite or continue an existing clip. `grok-imagine-video-1.5` has no video input — the adapter rejects a source-video part or `mode` on that model. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`: + +- `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p), so the adapter rejects `size`, `aspect_ratio`, `resolution`, and `duration` in this mode rather than sending fields the API ignores. +- `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. Output geometry is still inherited from the source, so `size` / `aspect_ratio` / `resolution` are rejected here too. + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const adapter = grokVideo("grok-imagine-video"); + +// Edit: change the clip in place +const edit = await generateVideo({ + adapter, + prompt: [ + { type: "text", content: "Make the sky stormy with distant lightning" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + modelOptions: { mode: "edit" }, +}); + +// Extend: append 5 more seconds +const extension = await generateVideo({ + adapter, + prompt: [ + { type: "text", content: "The camera keeps panning right across the bay" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + duration: 5, // added seconds, not the total + modelOptions: { mode: "extend" }, +}); +``` + +Both return the usual `{ jobId }` and are polled like any other Grok video job. When the job completes, the adapter reports usage on the result: `usage.unitsBilled` carries the billed seconds of video and `usage.cost` the exact cost in USD, both as returned by the xAI API. @@ -333,7 +407,7 @@ console.log(result.text); ## Realtime Voice -Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. +Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. The default model is `grok-voice-think-fast-2.0` (xAI's current recommended speech-to-speech model); `grok-voice-latest` always points at the newest model. The 1.0 ids remain accepted for compatibility, but xAI has deprecated `grok-voice-think-fast-1.0`. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. ## Environment Variables diff --git a/docs/config.json b/docs/config.json index e5c98bc877..b8a9c09fe1 100644 --- a/docs/config.json +++ b/docs/config.json @@ -443,7 +443,7 @@ "label": "Video Generation", "to": "media/video-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-13" + "updatedAt": "2026-08-18" }, { "label": "Generation Hooks", @@ -808,7 +808,7 @@ "label": "Grok (xAI)", "to": "adapters/grok", "addedAt": "2026-04-15", - "updatedAt": "2026-06-24" + "updatedAt": "2026-08-18" }, { "label": "Groq", diff --git a/docs/media/video-generation.md b/docs/media/video-generation.md index a75f0f7909..4b83af2f92 100644 --- a/docs/media/video-generation.md +++ b/docs/media/video-generation.md @@ -49,7 +49,7 @@ Currently supported: - **OpenAI**: Sora-2 and Sora-2-Pro models (when available) - **Google Gemini**: Veo 3.1 models (via the long-running operations API), and Gemini Omni Flash (via the Interactions API) -- **Grok (xAI)**: grok-imagine-video (text-to-video + image-to-video) and grok-imagine-video-1.5 (image-to-video only) models +- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5 (text-to-video, image-to-video; 1.5 adds reference-to-video; v1.0 adds editing and extension) - **BytePlus**: Seedance 2.0, 1.5-pro and 1.0-pro models (text-to-video, first/last frame, and multimodal references on 2.0) - **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models - **OpenRouter**: Seedance, Veo 3.1, Wan, Kling, Sora 2 Pro and others via the dedicated async video API (`POST /api/v1/videos`) @@ -691,16 +691,16 @@ instead of letting the model infer the task mode). #### Grok (xAI Imagine) Model Options -Based on the [xAI video generation API](https://docs.x.ai/docs/guides/video-generations). Two models are available: `grok-imagine-video` (v1.0) supports **text-to-video and image-to-video**, while `grok-imagine-video-1.5` is **image-to-video only** (a text-only prompt is rejected by the API; the adapter throws a clear error pointing you at `grok-imagine-video`). Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. +Based on the [xAI video generation API](https://docs.x.ai/developers/model-capabilities/video/generation). Two models are available: `grok-imagine-video` (v1.0) and `grok-imagine-video-1.5` (xAI's recommended default, with native 1080p text-to-video). Both support **text-to-video and image-to-video**; 1.5 adds **reference-to-video**. **Video editing and extension** are `grok-imagine-video` only — 1.5 has no video input. Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. -Text-to-video with the base model: +Text-to-video: ```typescript import { generateVideo } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; const { jobId } = await generateVideo({ - adapter: grokVideo("grok-imagine-video"), + adapter: grokVideo("grok-imagine-video-1.5"), prompt: "A beautiful sunset over the ocean", size: "16:9_720p", // aspect ratio: '1:1' | '16:9' | '9:16' | '4:3' | '3:4' | '3:2' | '2:3' // resolution (optional suffix): '480p' | '720p' | '1080p' @@ -713,7 +713,7 @@ const { jobId } = await generateVideo({ }); ``` -Image-to-video (required for `grok-imagine-video-1.5`) — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: +Image-to-video — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -733,12 +733,53 @@ const { jobId } = await generateVideo({ }); ``` +Reference-to-video (`grok-imagine-video-1.5` only, output capped at 720p) — image prompt parts with `metadata.role: 'reference'` or `'character'` become `reference_images` (addressed from the prompt as ``, ``, …), and up to 3 preset TTS voices can be referenced via `modelOptions.reference_audios` (addressed as ``, …): + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video-1.5"), + prompt: [ + { type: "text", content: " waves at the camera while says hello" }, + { + type: "image", + source: { type: "url", value: "https://example.com/character.png" }, + metadata: { role: "reference" }, + }, + ], + size: "16:9_720p", + modelOptions: { reference_audios: [{ voice_id: "eve" }] }, +}); +``` + +Video editing and extension (`grok-imagine-video` only) — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total. Because the output inherits the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` in both modes (and `duration` in edit mode) instead of sending fields the API ignores. The adapter rejects a source-video part or `mode` on `grok-imagine-video-1.5`. + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video"), + prompt: [ + { type: "text", content: "The camera keeps panning right across the bay" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + duration: 5, // 'extend' mode: seconds added to the clip, not the total + modelOptions: { mode: "extend" }, +}); +``` + Both models accept any whole second in the **1–15** range. A raw `duration` is coerced into that range rather than rejected — values are clamped to `[1, 15]` and rounded to the nearest second. Inspect or pre-snap the range the same way as Veo: ```typescript import { grokVideo } from "@tanstack/ai-grok"; -const adapter = grokVideo("grok-imagine-video"); +const adapter = grokVideo("grok-imagine-video-1.5"); adapter.availableDurations(); // { kind: 'range', min: 1, max: 15, step: 1, unit: 'seconds' } adapter.snapDuration(2.5); // 3 — clamped/rounded into range diff --git a/examples/ts-react-chat/src/lib/use-realtime.ts b/examples/ts-react-chat/src/lib/use-realtime.ts index 44874ef0da..3c3038f577 100644 --- a/examples/ts-react-chat/src/lib/use-realtime.ts +++ b/examples/ts-react-chat/src/lib/use-realtime.ts @@ -42,7 +42,7 @@ const getRealtimeTokenFn = createServerFn({ method: 'POST' }) if (data.provider === 'grok') { return realtimeToken({ - adapter: grokRealtimeToken({ model: 'grok-voice-fast-1.0' }), + adapter: grokRealtimeToken({ model: 'grok-voice-think-fast-2.0' }), }) } diff --git a/examples/ts-react-media/src/lib/models.ts b/examples/ts-react-media/src/lib/models.ts index 633f8da8bd..4c42c741e9 100644 --- a/examples/ts-react-media/src/lib/models.ts +++ b/examples/ts-react-media/src/lib/models.ts @@ -23,6 +23,14 @@ export const IMAGE_MODELS = [ sizeType: 'aspect_ratio' as const, provider: 'xai' as const, }, + { + id: 'grok-imagine-image-2.0', + name: 'Grok Imagine 2.0 (xAI Direct)', + description: 'xAI recommended Imagine model with the quality option', + defaultSize: '16:9' as const, + sizeType: 'aspect_ratio' as const, + provider: 'xai' as const, + }, { id: 'grok-imagine-image-quality', name: 'Grok Imagine Quality (xAI Direct)', @@ -149,11 +157,18 @@ export const VIDEO_MODELS = [ mode: 'text-to-video' as const, provider: 'xai' as const, }, + { + id: 'grok-imagine-video-1.5', + name: 'Grok Imagine Video 1.5 (Text-to-Video)', + description: + 'xAI recommended video model via the native grokVideo adapter (native 1080p text-to-video)', + mode: 'text-to-video' as const, + provider: 'xai' as const, + }, { id: 'grok-imagine-video-1.5/image-to-video', name: 'Grok Imagine Video 1.5 (Image-to-Video)', - description: - 'Animate a starting frame via the native grokVideo adapter (1.5 is image-to-video only)', + description: 'Animate a starting frame via the native grokVideo adapter', mode: 'image-to-video' as const, provider: 'xai' as const, }, diff --git a/examples/ts-react-media/src/lib/server-functions.ts b/examples/ts-react-media/src/lib/server-functions.ts index aa0f12c07a..7b08ce8646 100644 --- a/examples/ts-react-media/src/lib/server-functions.ts +++ b/examples/ts-react-media/src/lib/server-functions.ts @@ -167,6 +167,17 @@ export const generateImageFn = createServerFn({ method: 'POST' }) size: '16:9', }) } + case 'grok-imagine-image-2.0': { + // xAI's recommended Imagine model; `quality` is a 2.0-only option + // ('low' | 'medium', default 'medium'). + return generateImage({ + adapter: grokImage('grok-imagine-image-2.0'), + prompt: asImagePrompt(data.prompt), + numberOfImages: 1, + size: '16:9', + modelOptions: { quality: 'medium' }, + }) + } case 'grok-imagine-image-quality': { return generateImage({ adapter: grokImage('grok-imagine-image-quality'), @@ -339,6 +350,18 @@ function videoStreamForModel(data: VideoRequest): AsyncIterable { duration: 5, }) } + case 'grok-imagine-video-1.5': { + // Direct xAI Imagine API — grok-imagine-video-1.5 is xAI's recommended + // default and supports text-to-video with native 1080p. + return generateVideo({ + stream: true, + pollingInterval: VIDEO_POLL_INTERVAL_MS, + adapter: grokVideo('grok-imagine-video-1.5'), + prompt: asTextPrompt(data.prompt), + size: '16:9_1080p', + duration: 5, + }) + } case 'dreamina-seedance-2-0-260128': { // BytePlus Seedance via ModelArk (ARK_API_KEY). `size` is a "ratio" or // "ratio_resolution" template; durations are 4-15 integer seconds. diff --git a/packages/ai-anthropic/src/model-meta.ts b/packages/ai-anthropic/src/model-meta.ts index b3b248d27e..5dc0c80f18 100644 --- a/packages/ai-anthropic/src/model-meta.ts +++ b/packages/ai-anthropic/src/model-meta.ts @@ -819,6 +819,8 @@ export type AnthropicChatModelToolCapabilitiesByName = { [CLAUDE_OPUS_4_8.id]: typeof CLAUDE_OPUS_4_8.supports.tools [CLAUDE_FABLE_5.id]: typeof CLAUDE_FABLE_5.supports.tools [CLAUDE_SONNET_5.id]: typeof CLAUDE_SONNET_5.supports.tools + [CLAUDE_OPUS_5.id]: typeof CLAUDE_OPUS_5.supports.tools + [CLAUDE_OPUS_5_FAST.id]: typeof CLAUDE_OPUS_5_FAST.supports.tools } /** diff --git a/packages/ai-grok/src/adapters/image.ts b/packages/ai-grok/src/adapters/image.ts index 50bd38e51c..28932c83df 100644 --- a/packages/ai-grok/src/adapters/image.ts +++ b/packages/ai-grok/src/adapters/image.ts @@ -129,7 +129,8 @@ export class GrokImageAdapter< throw new Error( `grok: model "${model}" does not support image prompt parts. ` + `Image-conditioned generation requires an Imagine API model ` + - `('grok-imagine-image' or 'grok-imagine-image-quality').`, + `('grok-imagine-image', 'grok-imagine-image-2.0' or ` + + `'grok-imagine-image-quality').`, ) } return await this.editImages(options, resolved) diff --git a/packages/ai-grok/src/adapters/video.ts b/packages/ai-grok/src/adapters/video.ts index 21807360e3..90709fdd8f 100644 --- a/packages/ai-grok/src/adapters/video.ts +++ b/packages/ai-grok/src/adapters/video.ts @@ -3,8 +3,11 @@ import { BaseVideoAdapter, snapToDurationOption } from '@tanstack/ai/adapters' import { toRunErrorPayload } from '@tanstack/ai/adapter-internals' import { getGrokApiKeyFromEnv, withGrokDefaults } from '../utils/client' import { + GROK_VIDEO_MAX_REFERENCE_AUDIOS, + GROK_VIDEO_MAX_REFERENCE_IMAGES, getGrokVideoDurationOptions, - isImageToVideoOnlyModel, + isGrokVideoReferenceModel, + isGrokVideoSourceModel, parseGrokVideoSize, validateVideoSize, } from '../video/video-provider-options' @@ -15,6 +18,7 @@ import type { TokenUsage, VideoGenerationOptions, VideoJobResult, + VideoPart, VideoStatusResult, VideoUrlResult, } from '@tanstack/ai' @@ -24,7 +28,7 @@ import type { GrokVideoModelInputModalitiesByName, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, - GrokVideoProviderOptions, + GrokVideoRuntimeOptions, } from '../video/video-provider-options' import type { GrokClientConfig } from '../utils/client' @@ -41,7 +45,7 @@ export interface GrokVideoConfig extends GrokClientConfig {} */ const USD_TICKS_PER_DOLLAR = 10_000_000_000 -/** Response of POST /v1/videos/generations. */ +/** Response of the POST /v1/videos/{generations,edits,extensions} endpoints. */ interface GrokVideoCreateResponse { request_id?: string } @@ -62,11 +66,13 @@ interface GrokVideoStatusResponse { } /** - * Convert a TanStack ImagePart to the URL string accepted by xAI's Imagine - * video endpoint: public URLs pass through (fetched by xAI's servers), data - * sources become base64 data URIs. + * Convert a TanStack image / video part to the URL string accepted by xAI's + * Imagine video endpoints: public URLs pass through (fetched by xAI's + * servers), data sources become base64 data URIs. */ -function imagePartToUrl(part: ImagePart): string { +function mediaPartToUrl( + part: ImagePart | VideoPart, +): string { if (part.source.type === 'url') return part.source.value return `data:${part.source.mimeType};base64,${part.source.value}` } @@ -93,10 +99,10 @@ function buildGrokVideoUsage( * async jobs/polling architecture: create a generation request, poll it, * then read the completed video URL. * - * `grok-imagine-video` (v1.0) supports text-to-video and image-to-video. - * `grok-imagine-video-1.5` is image-to-video only — every request needs an - * image prompt part as the starting frame, and the adapter rejects a - * text-only prompt with a clear error rather than a raw API 400. + * Both models support text-to-video and image-to-video; + * `grok-imagine-video-1.5` is xAI's documented default and adds native + * 1080p generation plus reference-to-video inputs. Source-video edit + * and extend are `grok-imagine-video` only. * * The Imagine video endpoints are not part of the OpenAI SDK surface (and * xAI rejects the SDK's multipart paths), so requests are plain JSON calls @@ -109,13 +115,21 @@ function buildGrokVideoUsage( * - Aspect-ratio sizing via the "aspectRatio_resolution" size template * (e.g. '16:9_720p'), consistent with the grok-imagine image models * - Image-to-video via an `image` prompt part (starting frame URL or data URI) + * - Reference-to-video via image prompt parts with + * `metadata.role: 'reference'` or `'character'` (→ `reference_images`) + * and preset voices via `modelOptions.reference_audios` + * (grok-imagine-video-1.5 only) + * - Video editing / extension on `grok-imagine-video` via a source + * `video` prompt part and `modelOptions.mode: 'edit' | 'extend'` + * (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode + * `duration` is the added tail) * - Usage reporting: billed seconds (`unitsBilled`) and exact cost */ export class GrokVideoAdapter< TModel extends GrokVideoModel, > extends BaseVideoAdapter< TModel, - GrokVideoProviderOptions, + GrokVideoModelProviderOptionsByName[TModel], GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, GrokVideoModelInputModalitiesByName, @@ -174,98 +188,350 @@ export class GrokVideoAdapter< async createVideoJob( options: VideoGenerationOptions< - GrokVideoProviderOptions, + GrokVideoModelProviderOptionsByName[TModel], GrokVideoModelSizeByName[TModel], GrokVideoModelDurationByName[TModel] >, ): Promise { const { model, size, modelOptions, logger } = options + // `mode` is a routing hint for this adapter, not an API field — strip it + // before the remaining options are spread onto the request body. The + // per-model map narrows what callers can pass, but modelOptions often + // arrives as deserialized JSON, so the adapter handles the widest option + // surface (the 1.5 shape) uniformly and gates by model at runtime. + const { mode, ...wireOptions } = (modelOptions ?? + {}) as GrokVideoRuntimeOptions + + // `mode` is typed 'edit' | 'extend' but reaches us untrusted from JSON + // callers. An unrecognised value must not fall through to the + // generations endpoint with a source-video body — that would silently + // run (and bill) a generation the caller never asked for. + if (mode !== undefined && mode !== 'edit' && mode !== 'extend') { + throw new Error( + `${this.name}: unknown modelOptions.mode '${String(mode)}'. ` + + `Expected 'edit' or 'extend'.`, + ) + } + + // The interleaved prompt decomposes into verbatim text plus typed media + // buckets. Reference audio is voice-id based (not an audio file), so + // audio prompt parts have no request field to land in. + const resolved = resolveMediaPrompt(options.prompt) + if (resolved.audios.length > 0) { + throw new Error( + `${this.name}.createVideoJob does not support audio prompt parts (model: ${model}). ` + + `To reference a preset voice, pass modelOptions.reference_audios ` + + `(e.g. [{ voice_id: 'eve' }]).`, + ) + } + + // A video prompt part is the source clip for edit / extension mode. + // Those endpoints are grok-imagine-video only — 1.5 has no video input. + if ( + !isGrokVideoSourceModel(model) && + (mode !== undefined || resolved.videos.length > 0) + ) { + throw new Error( + `${this.name}: ${model} does not support video editing or extension. ` + + `Use 'grok-imagine-video' for /v1/videos/edits and /v1/videos/extensions.`, + ) + } + + // The mode must be chosen explicitly because the two endpoints have + // different semantics (edit rewrites the clip, extend appends + // `duration` seconds). + if (resolved.videos.length > 1) { + throw new Error( + `${this.name}: ${model} accepts at most one source video; received ${resolved.videos.length}.`, + ) + } + const [sourceVideo] = resolved.videos + if (sourceVideo && mode === undefined) { + throw new Error( + `${this.name}: a video prompt part needs modelOptions.mode set to ` + + `'edit' (rewrite the clip) or 'extend' (append to it).`, + ) + } + if (!sourceVideo && mode !== undefined) { + throw new Error( + `${this.name}: modelOptions.mode '${mode}' requires a video prompt ` + + `part carrying the source clip.`, + ) + } + + if (mode !== undefined && sourceVideo) { + return await this.createSourceVideoJob({ + model, + mode, + sourceVideo, + resolved, + wireOptions, + size, + genericDuration: options.duration, + logger, + }) + } + validateVideoSize(model, size) + // Pull the specially-handled keys out of the wire options: `duration` + // is folded into the snapped value below, and the reference fields are + // re-added explicitly so a JSON-serialized `null` or empty array reads + // as "unset" instead of leaking onto the wire. + const { + duration: rawOptionDuration, + reference_images: explicitReferenceImages, + reference_audios: referenceAudios, + ...generationOptions + } = wireOptions + // Coerce the requested duration into the model's valid range (1–15s, // integer) instead of rejecting it — `snapDuration` clamps and rounds. // modelOptions wins over the generic `duration`, mirroring the size // precedence below. - const rawDuration = modelOptions?.duration ?? options.duration + const rawDuration = rawOptionDuration ?? options.duration const duration = - rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined - - // The interleaved prompt decomposes into verbatim text plus typed media - // buckets. The Imagine video endpoint takes a text prompt and an optional - // starting frame; reject the modalities it can't consume. - const resolved = resolveMediaPrompt(options.prompt) - if (resolved.videos.length > 0) { + rawDuration != null ? this.snapDuration(rawDuration) : undefined + + // Image parts split by role: un-roled / 'start_frame' images become the + // starting frame (image-to-video); 'reference' / 'character' images + // become reference_images (reference-to-video). The Imagine API has no + // mask / control / end-frame inputs. Unknown role strings (possible via + // JSON callers) throw rather than silently dropping the part. + const startFrames: Array> = [] + const referenceImages: Array<{ url: string }> = [] + for (const part of resolved.images) { + const role = part.metadata?.role + switch (role) { + case 'mask': + case 'control': + case 'end_frame': + throw new Error( + `${this.name}: the Imagine video API has no '${role}' image ` + + `input on model ${model}. Use an un-roled / 'start_frame' ` + + `image as the starting frame, or 'reference' images.`, + ) + case 'reference': + case 'character': + referenceImages.push({ url: mediaPartToUrl(part) }) + break + case 'start_frame': + case undefined: + startFrames.push(part) + break + default: + throw new Error( + `${this.name}: unknown image metadata.role '${String(role)}'. ` + + `Expected 'start_frame', 'reference', or 'character'.`, + ) + } + } + if (startFrames.length > 1) { throw new Error( - `${this.name}.createVideoJob does not support video prompt parts (model: ${model}).`, + `${this.name}: ${model} accepts at most one starting-frame image; received ${startFrames.length}. ` + + `Use metadata.role: 'reference' for reference-to-video inputs.`, ) } - if (resolved.audios.length > 0) { + // Explicit modelOptions.reference_images replaces the part-derived list + // (an explicit empty array means "none"). + const finalReferenceImages = + explicitReferenceImages ?? + (referenceImages.length > 0 ? referenceImages : undefined) + const referenceImageCount = finalReferenceImages?.length ?? 0 + const referenceAudioCount = referenceAudios?.length ?? 0 + const hasReference = referenceImageCount > 0 || referenceAudioCount > 0 + + // Reference inputs are a grok-imagine-video-1.5 feature. The per-model + // options map already hides the fields from other models at compile + // time; this runtime gate covers prompt-part roles and untyped callers. + if (!isGrokVideoReferenceModel(model) && hasReference) { throw new Error( - `${this.name}.createVideoJob does not support audio prompt parts (model: ${model}).`, + `${this.name}: ${model} does not support reference-to-video inputs. ` + + `Use 'grok-imagine-video-1.5' for reference_images / reference_audios.`, ) } - // grok-imagine-video-1.5 is image-to-video only — text-to-video is - // rejected by the API, so fail fast with a clear, actionable message - // pointing at the model that does support text-to-video. - if (resolved.images.length === 0 && isImageToVideoOnlyModel(model)) { + if (referenceAudioCount > GROK_VIDEO_MAX_REFERENCE_AUDIOS) { throw new Error( - `${this.name}: ${model} does not support text-to-video — it is image-to-video only. ` + - `Include an image prompt part as the starting frame, or use 'grok-imagine-video' for text-to-video.`, + `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_AUDIOS} reference voices; received ${referenceAudioCount}.`, ) } - if (resolved.images.length > 1) { + if (referenceImageCount > GROK_VIDEO_MAX_REFERENCE_IMAGES) { throw new Error( - `${this.name}: ${model} accepts at most one starting-frame image; received ${resolved.images.length}.`, + `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_IMAGES} reference images; received ${referenceImageCount}.`, ) } // Image-to-video: the single image prompt part becomes the starting frame // and the prompt text describes the desired motion. URL sources are // fetched by xAI's servers; data sources are sent as base64 data URIs. - const [startFrame] = resolved.images + const [startFrame] = startFrames + + // xAI rejects `image` + `reference_images` / `reference_audios` as a + // 400: only one of image-to-video or reference-to-video can be active. + if (startFrame && hasReference) { + throw new Error( + `${this.name}: image-to-video and reference-to-video cannot be combined. ` + + `Use a starting-frame image, or reference images / voices, not both.`, + ) + } // The generic `size` option carries an "aspectRatio_resolution" template // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / - // `resolution` parameters; explicit modelOptions win over the template. + // `resolution` parameters; explicit modelOptions win over the template + // (including `reference_images`, which replaces the part-derived list). const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined + const resolvedResolution = + generationOptions.resolution ?? parsedSize?.resolution + if (hasReference && resolvedResolution === '1080p') { + throw new Error( + `${this.name}: reference-to-video is capped at 720p on ${model}.`, + ) + } const request = { model, prompt: resolved.text, - ...(startFrame && { image: { url: imagePartToUrl(startFrame) } }), + ...(startFrame && { image: { url: mediaPartToUrl(startFrame) } }), + ...(referenceImageCount > 0 && { + reference_images: finalReferenceImages, + }), + ...(referenceAudioCount > 0 && { + reference_audios: referenceAudios, + }), ...(parsedSize && { aspect_ratio: parsedSize.aspectRatio, ...(parsedSize.resolution !== undefined && { resolution: parsedSize.resolution, }), }), - ...modelOptions, - // Spread after modelOptions so the snapped duration is authoritative - // (modelOptions.duration is folded into `duration` via snapDuration above). + // The remaining options spread after the size template so explicit + // aspect_ratio / resolution win over it; duration and the reference + // fields were destructured out above and re-added normalized. + ...generationOptions, ...(duration !== undefined && { duration }), } - try { - logger.request( - `activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`, - { provider: this.name, model }, + return await this.postVideoJob('/videos/generations', request, { + model, + logger, + logLine: `activity=video.create provider=${this.name} model=${model} mode=generate size=${size ?? 'default'} duration=${duration ?? 'default'}`, + }) + } + + /** + * Build and post an edit / extension request. Both endpoints take only + * `model`, `prompt`, and the source `video` (plus `duration` — the length + * of the added tail — for extensions): output geometry is inherited from + * the source clip, capped at 720p, and edit outputs also inherit the + * source length. Rather than sending fields the API documents as ignored, + * the inapplicable options are rejected with actionable errors. + */ + private async createSourceVideoJob(args: { + model: string + mode: 'edit' | 'extend' + sourceVideo: VideoPart + resolved: ReturnType + wireOptions: Omit + size: string | undefined + genericDuration: number | undefined + logger: VideoGenerationOptions['logger'] + }): Promise { + const { model, mode, sourceVideo, resolved, wireOptions, logger } = args + const endpoint = mode === 'edit' ? '/videos/edits' : '/videos/extensions' + + if (resolved.images.length > 0) { + throw new Error( + `${this.name}: '${mode}' mode takes only the source video — image ` + + `prompt parts are not supported by ${endpoint}.`, ) + } - const response = await this.request('/videos/generations', { + // Pull every generation-only key out of the wire options so nothing can + // leak into the edit/extend body via the spread below. JSON-serialized + // `null` values (a common "unset" encoding) are treated as absent; + // actual values are rejected with actionable errors. + const { + aspect_ratio: aspectRatio, + resolution, + duration: modeDuration, + reference_images: referenceImagesOption, + reference_audios: referenceAudiosOption, + ...passthrough + } = wireOptions + if ( + (referenceImagesOption?.length ?? 0) > 0 || + (referenceAudiosOption?.length ?? 0) > 0 + ) { + throw new Error( + `${this.name}: reference inputs are only supported by video ` + + `generation, not '${mode}' mode.`, + ) + } + if (args.size !== undefined || aspectRatio != null || resolution != null) { + throw new Error( + `${this.name}: '${mode}' mode does not accept size / aspect_ratio / ` + + `resolution — the output inherits the source clip's geometry ` + + `(capped at 720p).`, + ) + } + const rawDuration = modeDuration ?? args.genericDuration + if (mode === 'edit' && rawDuration != null) { + throw new Error( + `${this.name}: 'edit' mode does not accept a duration — the output ` + + `inherits the source clip's length. Use mode 'extend' to append ` + + `seconds to the clip.`, + ) + } + // Extend: the snapped duration is the added-tail length (1–15s). + const duration = + rawDuration != null ? this.snapDuration(rawDuration) : undefined + + const request = { + model, + prompt: resolved.text, + video: { url: mediaPartToUrl(sourceVideo) }, + ...passthrough, + ...(duration !== undefined && { duration }), + } + + return await this.postVideoJob(endpoint, request, { + model, + logger, + logLine: `activity=video.create provider=${this.name} model=${model} mode=${mode} duration=${duration ?? 'default'}`, + }) + } + + /** + * POST a create-job request body to one of the Imagine video endpoints + * (`/videos/generations`, `/videos/edits`, `/videos/extensions`) and read + * the `request_id` out of the shared response shape. + */ + private async postVideoJob( + endpoint: string, + request: Record, + context: { + model: string + logger: VideoGenerationOptions['logger'] + logLine: string + }, + ): Promise { + const { model, logger, logLine } = context + try { + logger.request(logLine, { provider: this.name, model }) + + const response = await this.request(endpoint, { method: 'POST', body: JSON.stringify(request), }) if (!response.ok) { throw new Error( - `grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, + `grok: ${endpoint} request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, ) } const result = (await response.json()) as GrokVideoCreateResponse if (!result.request_id) { - throw new Error( - 'grok: video generation response contained no request_id', - ) + throw new Error(`grok: ${endpoint} response contained no request_id`) } return { jobId: result.request_id, model } } catch (error: unknown) { @@ -394,15 +660,14 @@ export class GrokVideoAdapter< * * @experimental Video generation is an experimental feature and may change. * - * @param model - The model name (e.g., 'grok-imagine-video') + * @param model - The model name (e.g., 'grok-imagine-video-1.5') * @param apiKey - Your xAI API key * @param config - Optional additional configuration * @returns Configured Grok video adapter instance with resolved types * * @example * ```typescript - * // grok-imagine-video (v1.0) supports text-to-video. - * const adapter = createGrokVideo('grok-imagine-video', 'xai-...'); + * const adapter = createGrokVideo('grok-imagine-video-1.5', 'xai-...'); * * const { jobId } = await generateVideo({ * adapter, @@ -440,7 +705,7 @@ export function createGrokVideo( * // Automatically uses XAI_API_KEY from environment * const adapter = grokVideo('grok-imagine-video-1.5'); * - * // Image-to-video only: the prompt must carry a starting-frame image part. + * // Image-to-video: an optional image prompt part is the starting frame. * const { jobId } = await generateVideo({ * adapter, * prompt: [ diff --git a/packages/ai-grok/src/image/image-provider-options.ts b/packages/ai-grok/src/image/image-provider-options.ts index a433534444..19cffb4b79 100644 --- a/packages/ai-grok/src/image/image-provider-options.ts +++ b/packages/ai-grok/src/image/image-provider-options.ts @@ -1,8 +1,9 @@ /** * Grok Image Generation Provider Options * - * These are provider-specific options for Grok image generation. - * Grok uses the grok-2-image-1212 model for image generation. + * Provider-specific options for Grok image generation: the aspect-ratio + * sized Imagine API models (grok-imagine-image, grok-imagine-image-2.0, + * grok-imagine-image-quality) and the legacy pixel-sized grok-2-image-1212. */ /** @@ -141,12 +142,25 @@ export interface GrokImagineImageProviderOptions extends GrokImageBaseProviderOp service_tier?: 'default' | 'priority' } +/** + * Provider options for grok-imagine-image-2.0, which adds a generation + * `quality` knob on top of the shared Imagine options. + */ +export interface GrokImagineImage2ProviderOptions extends GrokImagineImageProviderOptions { + /** + * Generation quality. Only supported by grok-imagine-image-2.0. + * @default 'medium' + */ + quality?: 'low' | 'medium' +} + /** * Type-only map from model name to its specific provider options. */ export type GrokImageModelProviderOptionsByName = { 'grok-2-image-1212': GrokImageProviderOptions 'grok-imagine-image': GrokImagineImageProviderOptions + 'grok-imagine-image-2.0': GrokImagineImage2ProviderOptions 'grok-imagine-image-quality': GrokImagineImageProviderOptions } @@ -156,6 +170,7 @@ export type GrokImageModelProviderOptionsByName = { export type GrokImageModelSizeByName = { 'grok-2-image-1212': GrokImageSize 'grok-imagine-image': GrokImagineImageSize + 'grok-imagine-image-2.0': GrokImagineImageSize 'grok-imagine-image-quality': GrokImagineImageSize } @@ -167,6 +182,7 @@ export type GrokImageModelSizeByName = { export type GrokImageModelInputModalitiesByName = { 'grok-2-image-1212': readonly [] 'grok-imagine-image': readonly ['image'] + 'grok-imagine-image-2.0': readonly ['image'] 'grok-imagine-image-quality': readonly ['image'] } diff --git a/packages/ai-grok/src/index.ts b/packages/ai-grok/src/index.ts index e342645caa..e1c935f734 100644 --- a/packages/ai-grok/src/index.ts +++ b/packages/ai-grok/src/index.ts @@ -28,6 +28,8 @@ export { } from './adapters/image' export type { GrokImageProviderOptions, + GrokImagineImageProviderOptions, + GrokImagineImage2ProviderOptions, GrokImageModelProviderOptionsByName, } from './image/image-provider-options' @@ -43,7 +45,11 @@ export { getGrokVideoDurationOptions, } from './video/video-provider-options' export type { + GrokVideoMode, + GrokVideoBaseProviderOptions, + GrokVideoSourceProviderOptions, GrokVideoProviderOptions, + GrokVideoRuntimeOptions, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, GrokVideoModelDurationByName, @@ -101,6 +107,7 @@ export { GROK_TTS_MODELS, GROK_TRANSCRIPTION_MODELS, GROK_REALTIME_MODELS, + GROK_DEFAULT_REALTIME_MODEL, } from './model-meta' export type { GrokTextMetadata, diff --git a/packages/ai-grok/src/model-meta.ts b/packages/ai-grok/src/model-meta.ts index ea476fe998..35feac479f 100644 --- a/packages/ai-grok/src/model-meta.ts +++ b/packages/ai-grok/src/model-meta.ts @@ -131,17 +131,38 @@ const GROK_IMAGINE_IMAGE_QUALITY = { }, } as const satisfies ModelMeta +// xAI's recommended Imagine image model. Supports the 2.0-only `quality` +// provider option ('low' | 'medium', default 'medium'). +const GROK_IMAGINE_IMAGE_2_0 = { + name: 'grok-imagine-image-2.0', + supports: { + input: ['text', 'image'], + output: ['image'], + }, + pricing: { + input: { + normal: 0, + }, + output: { + normal: 0.04, + }, + }, +} as const satisfies ModelMeta + // Imagine API video models. Pricing is per second of generated video // (output only); generated videos carry an audio track. // -// grok-imagine-video (v1.0) supports both text-to-video (a starting image is -// optional) and image-to-video. grok-imagine-video-1.5 is image-to-video -// only: a starting-frame image is required (the text prompt describes the -// desired motion) — its text-to-video is rejected by the API. +// Both models support text-to-video and image-to-video (a starting-frame +// image is optional). grok-imagine-video-1.5 is the documented default: it +// adds native 1080p for text-to-video / image-to-video plus +// reference-to-video (`reference_images` / `reference_audios`; reference +// output is capped at 720p). Source-video edit (`/v1/videos/edits`) and +// extend (`/v1/videos/extensions`) are grok-imagine-video only — xAI's +// 1.5 model page lists text+image input, not video. const GROK_IMAGINE_VIDEO = { name: 'grok-imagine-video', supports: { - input: ['text', 'image'], + input: ['text', 'image', 'video'], output: ['video', 'audio'], }, pricing: { @@ -228,6 +249,7 @@ export const GROK_CHAT_MODELS = [ export const GROK_IMAGE_MODELS = [ GROK_2_IMAGE.name, GROK_IMAGINE_IMAGE.name, + GROK_IMAGINE_IMAGE_2_0.name, GROK_IMAGINE_IMAGE_QUALITY.name, ] as const @@ -273,6 +295,7 @@ const GROK_VOICE_FAST_1 = { }, } as const satisfies ModelMeta +/** @deprecated xAI has deprecated grok-voice-think-fast-1.0 — use grok-voice-think-fast-2.0. */ const GROK_VOICE_THINK_FAST_1 = { name: 'grok-voice-think-fast-1.0', supports: { @@ -283,15 +306,48 @@ const GROK_VOICE_THINK_FAST_1 = { }, } as const satisfies ModelMeta +// xAI's current recommended speech-to-speech model. +const GROK_VOICE_THINK_FAST_2 = { + name: 'grok-voice-think-fast-2.0', + supports: { + input: ['audio', 'text'], + output: ['audio', 'text'], + capabilities: ['reasoning', 'tool_calling'], + tools: [] as const, + }, +} as const satisfies ModelMeta + +// Rolling alias used by xAI's realtime docs examples; always points at the +// latest speech-to-speech model. +const GROK_VOICE_LATEST = { + name: 'grok-voice-latest', + supports: { + input: ['audio', 'text'], + output: ['audio', 'text'], + capabilities: ['reasoning', 'tool_calling'], + tools: [] as const, + }, +} as const satisfies ModelMeta + export const GROK_TTS_MODELS = [GROK_TTS.name] as const export const GROK_TRANSCRIPTION_MODELS = [GROK_STT.name] as const export const GROK_REALTIME_MODELS = [ + GROK_VOICE_THINK_FAST_2.name, + GROK_VOICE_LATEST.name, GROK_VOICE_FAST_1.name, GROK_VOICE_THINK_FAST_1.name, ] as const +/** + * Default speech-to-speech model used by the realtime token issuer and the + * realtime client adapter when no model is specified. Single source of truth + * so a future default bump cannot leave the two sides disagreeing. + */ +export const GROK_DEFAULT_REALTIME_MODEL: GrokRealtimeModel = + 'grok-voice-think-fast-2.0' + export type GrokChatModel = (typeof GROK_CHAT_MODELS)[number] export type GrokImageModel = (typeof GROK_IMAGE_MODELS)[number] export type GrokVideoModel = (typeof GROK_VIDEO_MODELS)[number] diff --git a/packages/ai-grok/src/realtime/adapter.ts b/packages/ai-grok/src/realtime/adapter.ts index 59831fa44b..a26456d6cc 100644 --- a/packages/ai-grok/src/realtime/adapter.ts +++ b/packages/ai-grok/src/realtime/adapter.ts @@ -1,4 +1,5 @@ import { resolveDebugOption } from '@tanstack/ai/adapter-internals' +import { GROK_DEFAULT_REALTIME_MODEL } from '../model-meta' import type { AnyClientTool, AudioVisualization, @@ -89,7 +90,7 @@ export function grokRealtime( token: RealtimeToken, _clientTools?: ReadonlyArray, ): Promise { - const model = token.config.model ?? 'grok-voice-fast-1.0' + const model = token.config.model ?? GROK_DEFAULT_REALTIME_MODEL logger.request(`activity=realtime provider=grok model=${model}`, { provider: 'grok', model, @@ -115,7 +116,7 @@ async function createWebRTCConnection( token: RealtimeToken, logger: InternalLogger, ): Promise { - const model = token.config.model ?? 'grok-voice-fast-1.0' + const model = token.config.model ?? GROK_DEFAULT_REALTIME_MODEL const eventHandlers = new Map>>() const pc = new RTCPeerConnection() diff --git a/packages/ai-grok/src/realtime/token.ts b/packages/ai-grok/src/realtime/token.ts index ad0962e26e..b574c50888 100644 --- a/packages/ai-grok/src/realtime/token.ts +++ b/packages/ai-grok/src/realtime/token.ts @@ -1,4 +1,5 @@ import { resolveDebugOption } from '@tanstack/ai/adapter-internals' +import { GROK_DEFAULT_REALTIME_MODEL } from '../model-meta' import { getGrokApiKeyFromEnv } from '../utils' import type { RealtimeToken, RealtimeTokenAdapter } from '@tanstack/ai' import type { GrokRealtimeModel } from '../model-meta' @@ -27,7 +28,7 @@ const DEFAULT_TOKEN_FETCH_TIMEOUT_MS = 15_000 * import { grokRealtimeToken } from '@tanstack/ai-grok' * * const token = await realtimeToken({ - * adapter: grokRealtimeToken({ model: 'grok-voice-fast-1.0' }), + * adapter: grokRealtimeToken({ model: 'grok-voice-think-fast-2.0' }), * }) * ``` */ @@ -41,7 +42,8 @@ export function grokRealtimeToken( provider: 'grok', async generateToken(): Promise { - const model: GrokRealtimeModel = options.model ?? 'grok-voice-fast-1.0' + const model: GrokRealtimeModel = + options.model ?? GROK_DEFAULT_REALTIME_MODEL logger.request(`activity=realtimeToken provider=grok model=${model}`, { provider: 'grok', diff --git a/packages/ai-grok/src/realtime/types.ts b/packages/ai-grok/src/realtime/types.ts index 7c8f1143c2..c8a01d46a3 100644 --- a/packages/ai-grok/src/realtime/types.ts +++ b/packages/ai-grok/src/realtime/types.ts @@ -35,7 +35,7 @@ export type GrokTurnDetection = * Options for the Grok realtime token adapter. */ export interface GrokRealtimeTokenOptions { - /** Model to use (default: 'grok-voice-fast-1.0'). */ + /** Model to use (default: 'grok-voice-think-fast-2.0'). */ model?: GrokRealtimeModel /** * Enable debug logging for token creation. diff --git a/packages/ai-grok/src/video/video-provider-options.ts b/packages/ai-grok/src/video/video-provider-options.ts index b84c03f8b5..ca279e1575 100644 --- a/packages/ai-grok/src/video/video-provider-options.ts +++ b/packages/ai-grok/src/video/video-provider-options.ts @@ -1,7 +1,9 @@ /** * Grok Video Generation Provider Options (xAI Imagine API) * - * Based on https://docs.x.ai/docs/guides/video-generations + * Based on https://docs.x.ai/developers/model-capabilities/video/generation + * (plus the image-to-video, reference-to-video, editing, and extension pages + * under the same section). * * @experimental Video generation is an experimental feature and may change. */ @@ -34,6 +36,14 @@ export type GrokVideoAspectRatio = */ export type GrokVideoResolution = '480p' | '720p' | '1080p' +/** + * Resolutions accepted by grok-imagine-video (v1.0). Native 1080p is a + * grok-imagine-video-1.5 generation feature. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoResolutionV1 = '480p' | '720p' + /** * Size strings for grok-imagine video models. The Imagine API is * aspect-ratio based rather than pixel-size based; like the grok-imagine @@ -47,6 +57,15 @@ export type GrokVideoSize = | GrokVideoAspectRatio | `${GrokVideoAspectRatio}_${GrokVideoResolution}` +/** + * Size strings for grok-imagine-video (v1.0) — 1080p is not in the type. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoSizeV1 = + | GrokVideoAspectRatio + | `${GrokVideoAspectRatio}_${GrokVideoResolutionV1}` + const GROK_VIDEO_ASPECT_RATIOS: ReadonlyArray = [ '1:1', '16:9', @@ -80,6 +99,16 @@ export function parseGrokVideoSize( return { aspectRatio, ...(resolution !== undefined && { resolution }) } } +/** + * Models that accept native 1080p on text-to-video and image-to-video. + * Reference-to-video stays capped at 720p even on these models. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoNative1080pModel(model: string): boolean { + return model === 'grok-imagine-video-1.5' +} + /** * Validate the `size` template for a given grok video model. * @@ -107,6 +136,12 @@ export function validateVideoSize( `Supported resolutions: ${GROK_VIDEO_RESOLUTIONS.join(', ')}`, ) } + if (parsed.resolution === '1080p' && !isGrokVideoNative1080pModel(model)) { + throw new Error( + `Resolution "1080p" is not supported by model "${model}". ` + + `Use 'grok-imagine-video-1.5' for native 1080p text-to-video / image-to-video.`, + ) + } } /** @@ -162,80 +197,209 @@ export function getGrokVideoDurationOptions( } /** - * Provider-specific options for grok video generation. These map directly - * onto the Imagine API request body and take precedence over the generic - * `size` / `duration` options when both are provided. + * Request mode for a source-video job. `'edit'` posts to `/v1/videos/edits` + * (modify the source clip in place); `'extend'` posts to + * `/v1/videos/extensions` (continue the source clip — `duration` is the + * length of the **added tail**, not the total). Both require exactly one + * video prompt part carrying the source clip, and both are + * `grok-imagine-video` (v1.0) only. + * + * Output geometry (aspect ratio / resolution) is inherited from the source + * clip in both modes, capped at 720p, and edit outputs also inherit the + * source length — the adapter rejects `size`, `aspect_ratio`, `resolution`, + * and (in edit mode) `duration` rather than sending fields the API ignores. * * @experimental Video generation is an experimental feature and may change. */ -export interface GrokVideoProviderOptions { +export type GrokVideoMode = 'edit' | 'extend' + +/** + * Provider options shared by both grok-imagine video models. These map + * directly onto the Imagine API request body and take precedence over the + * generic `size` / `duration` options when both are provided. + * + * @experimental Video generation is an experimental feature and may change. + */ +export interface GrokVideoBaseProviderOptions { /** - * Output aspect ratio. + * Output aspect ratio. Generation only — edit / extend outputs inherit + * the source clip's geometry and the adapter rejects this in those modes. */ aspect_ratio?: GrokVideoAspectRatio /** - * Output resolution tier. + * Output resolution tier. Generation only — edit / extend outputs inherit + * the source clip's geometry and the adapter rejects this in those modes. + * `1080p` is grok-imagine-video-1.5 generation only; reference-to-video + * is capped at 720p. */ resolution?: GrokVideoResolution /** - * Video duration in integer seconds (1–15). + * Video duration in integer seconds (1–15). In `'extend'` mode this is + * the length of the added tail only, not the total output length. Not + * valid in `'edit'` mode (the output inherits the source clip's length). */ duration?: number } /** - * Type-only map from model name to its specific provider options. + * Provider options for grok-imagine-video (v1.0), which is the only model + * that accepts a source-video edit / extend job. * * @experimental Video generation is an experimental feature and may change. */ -export type GrokVideoModelProviderOptionsByName = { - 'grok-imagine-video': GrokVideoProviderOptions - 'grok-imagine-video-1.5': GrokVideoProviderOptions +export interface GrokVideoSourceProviderOptions extends GrokVideoBaseProviderOptions { + /** + * Selects the request mode for a source-video prompt part: `'edit'` + * (`/v1/videos/edits`) or `'extend'` (`/v1/videos/extensions`). Required + * when the prompt carries a video part; not valid without one. Omit for + * plain generation (`/v1/videos/generations`). grok-imagine-video only. + */ + mode?: GrokVideoMode } /** - * Type-only map from model name to its supported `size` strings. + * Provider options for grok-imagine-video-1.5, which adds the + * reference-to-video inputs on top of the shared options. * * @experimental Video generation is an experimental feature and may change. */ -export type GrokVideoModelSizeByName = { - 'grok-imagine-video': GrokVideoSize - 'grok-imagine-video-1.5': GrokVideoSize +export interface GrokVideoProviderOptions extends GrokVideoBaseProviderOptions { + /** + * Reference images for reference-to-video generation (output capped at + * 720p). Usually populated from image prompt parts with + * `metadata.role: 'reference'` (or `'character'`); set explicitly to + * replace the part-derived list. Reference images are addressed from the + * prompt text as ``, ``, … in request order, and do not + * lock the first frame. + */ + reference_images?: Array<{ url: string }> + + /** + * Preset TTS voices to reference for generated speech (max 3). Voice ids + * come from the xAI TTS voice roster (e.g. 'eve', 'rex') or a custom + * voice id, and are addressed from the prompt text as ``, + * ``, ``. + */ + reference_audios?: Array<{ voice_id: string }> } /** - * Type-only map from model name to the non-text prompt modalities it accepts. - * Both models accept an `image` prompt part as the starting frame: - * `grok-imagine-video` (v1.0) does text-to-video and image-to-video, while - * `grok-imagine-video-1.5` is image-to-video only (the image is required). + * Widest option surface. Used when `modelOptions` arrives as deserialized + * JSON and the adapter must validate fields the per-model map already + * hides at compile time. * * @experimental Video generation is an experimental feature and may change. */ -export type GrokVideoModelInputModalitiesByName = { - 'grok-imagine-video': readonly ['image'] - 'grok-imagine-video-1.5': readonly ['image'] +export type GrokVideoRuntimeOptions = GrokVideoSourceProviderOptions & + GrokVideoProviderOptions + +/** + * Maximum reference voices accepted by the Imagine video endpoint. + */ +export const GROK_VIDEO_MAX_REFERENCE_AUDIOS = 3 + +/** + * Maximum reference images accepted by the Imagine video endpoint. + */ +export const GROK_VIDEO_MAX_REFERENCE_IMAGES = 7 + +/** + * Model names whose per-model options declare the reference fields. Keeps + * the runtime set below provably in sync with + * {@link GrokVideoModelProviderOptionsByName} — a typo or a new + * reference-capable model missing from the set is a compile error. + */ +type GrokVideoReferenceModel = { + [TModel in GrokVideoModel]: 'reference_images' extends keyof GrokVideoModelProviderOptionsByName[TModel] + ? TModel + : never +}[GrokVideoModel] + +/** + * Models that support reference-to-video inputs (`reference_images` / + * `reference_audios`). The per-model provider-options map hides the fields + * from other models at compile time; this backs the runtime gate for + * untyped callers (e.g. deserialized JSON) so they get a clear error + * instead of a raw API 400. + * + * @experimental Video generation is an experimental feature and may change. + */ +const GROK_VIDEO_REFERENCE_MODELS: ReadonlySet = + new Set(['grok-imagine-video-1.5']) + +/** + * True when the model accepts reference-to-video inputs. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoReferenceModel(model: string): boolean { + return GROK_VIDEO_REFERENCE_MODELS.has(model) +} + +/** + * Model names whose per-model options declare `mode`. Same + * provably-in-sync construction as {@link GrokVideoReferenceModel}. + */ +type GrokVideoSourceModel = { + [TModel in GrokVideoModel]: 'mode' extends keyof GrokVideoModelProviderOptionsByName[TModel] + ? TModel + : never +}[GrokVideoModel] + +/** + * Models that accept a source-video prompt part for `/v1/videos/edits` + * and `/v1/videos/extensions`. xAI lists video input only on + * grok-imagine-video (v1.0). + * + * @experimental Video generation is an experimental feature and may change. + */ +const GROK_VIDEO_SOURCE_MODELS: ReadonlySet = + new Set(['grok-imagine-video']) + +/** + * True when the model accepts edit / extend source-video jobs. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoSourceModel(model: string): boolean { + return GROK_VIDEO_SOURCE_MODELS.has(model) +} + +/** + * Type-only map from model name to its specific provider options. Only + * grok-imagine-video-1.5 exposes the reference-to-video fields. Only + * grok-imagine-video (v1.0) exposes `mode` for edit / extend. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoModelProviderOptionsByName = { + 'grok-imagine-video': GrokVideoSourceProviderOptions + 'grok-imagine-video-1.5': GrokVideoProviderOptions } /** - * Models that only support image-to-video — a starting-frame image is - * required and text-to-video is rejected by the Imagine API. Used by the - * adapter to fail fast with a clear message instead of surfacing the raw - * "Text-to-video is not supported for this model" 400. + * Type-only map from model name to its supported `size` strings. * * @experimental Video generation is an experimental feature and may change. */ -const GROK_VIDEO_IMAGE_TO_VIDEO_ONLY: ReadonlySet = new Set([ - 'grok-imagine-video-1.5', -]) +export type GrokVideoModelSizeByName = { + 'grok-imagine-video': GrokVideoSizeV1 + 'grok-imagine-video-1.5': GrokVideoSize +} /** - * True when the model only supports image-to-video (a starting frame is - * required). + * Type-only map from model name to the non-text prompt modalities it accepts. + * Both models support text-to-video and accept an optional `image` prompt + * part as the starting frame; image parts with `metadata.role: 'reference'` + * or `'character'` become `reference_images` (grok-imagine-video-1.5 only). + * A `video` prompt part carries the source clip for edit / extension mode + * on grok-imagine-video only (`modelOptions.mode: 'edit' | 'extend'`). * * @experimental Video generation is an experimental feature and may change. */ -export function isImageToVideoOnlyModel(model: string): boolean { - return GROK_VIDEO_IMAGE_TO_VIDEO_ONLY.has(model) +export type GrokVideoModelInputModalitiesByName = { + 'grok-imagine-video': readonly ['image', 'video'] + 'grok-imagine-video-1.5': readonly ['image'] } diff --git a/packages/ai-grok/tests/grok-adapter.test.ts b/packages/ai-grok/tests/grok-adapter.test.ts index 408db3d5a9..ca04fef1b8 100644 --- a/packages/ai-grok/tests/grok-adapter.test.ts +++ b/packages/ai-grok/tests/grok-adapter.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { resolveDebugOption } from '@tanstack/ai/adapter-internals' -import { EventType, summarize } from '@tanstack/ai' +import { EventType, generateImage, summarize } from '@tanstack/ai' import { createGrokText, grokText } from '../src/adapters/text' import { createGrokImage, grokImage } from '../src/adapters/image' import { createGrokSummarize, grokSummarize } from '../src/adapters/summarize' @@ -475,6 +475,31 @@ describe('Grok adapters', () => { ) expect(mockGenerate.mock.calls[0]![0]).not.toHaveProperty('size') }) + + it('passes the 2.0-only quality option through for grok-imagine-image-2.0', async () => { + const adapter = createGrokImage('grok-imagine-image-2.0', 'test-api-key') + const mockGenerate = vi.fn().mockResolvedValue({ + data: [{ url: 'https://example.com/out.png' }], + }) + ;(adapter as any).client = { images: { generate: mockGenerate } } + + // Via the public generateImage() entry point so the per-model provider + // options map is exercised: `quality` only type-checks on the 2.0 model. + await generateImage({ + adapter, + prompt: 'A skyline', + size: '16:9', + modelOptions: { quality: 'low' }, + }) + + expect(mockGenerate).toHaveBeenCalledWith( + expect.objectContaining({ + model: 'grok-imagine-image-2.0', + aspect_ratio: '16:9', + quality: 'low', + }), + ) + }) }) describe('Summarize adapter', () => { diff --git a/packages/ai-grok/tests/realtime-token.test.ts b/packages/ai-grok/tests/realtime-token.test.ts index c91c17ce3b..e8c291d237 100644 --- a/packages/ai-grok/tests/realtime-token.test.ts +++ b/packages/ai-grok/tests/realtime-token.test.ts @@ -19,7 +19,10 @@ afterEach(() => { } }) -function makeSessionResponse(expiresAt: number) { +function makeSessionResponse( + expiresAt: number, + model = 'grok-voice-think-fast-2.0', +) { return { ok: true, status: 200, @@ -27,7 +30,7 @@ function makeSessionResponse(expiresAt: number) { Promise.resolve({ id: 'sess_1', object: 'realtime.session', - model: 'grok-voice-fast-1.0', + model, modalities: ['audio', 'text'], instructions: '', voice: 'eve', @@ -52,7 +55,9 @@ describe('grokRealtimeToken request body', () => { it('wraps the model under the `session` key per xAI /v1/realtime/client_secrets schema', async () => { const fetchMock = vi .fn() - .mockResolvedValue(makeSessionResponse(1_700_000_000)) + .mockResolvedValue( + makeSessionResponse(1_700_000_000, 'grok-voice-think-fast-1.0'), + ) globalThis.fetch = fetchMock as unknown as typeof fetch await realtimeToken({ @@ -63,6 +68,20 @@ describe('grokRealtimeToken request body', () => { const body = JSON.parse(init.body as string) as Record expect(body).toEqual({ session: { model: 'grok-voice-think-fast-1.0' } }) }) + + it("defaults to xAI's current recommended model when no model is given", async () => { + const fetchMock = vi + .fn() + .mockResolvedValue(makeSessionResponse(1_700_000_000)) + globalThis.fetch = fetchMock as unknown as typeof fetch + + const token = await realtimeToken({ adapter: grokRealtimeToken() }) + + const init = fetchMock.mock.calls[0]![1]! + const body = JSON.parse(init.body as string) as Record + expect(body).toEqual({ session: { model: 'grok-voice-think-fast-2.0' } }) + expect(token.config.model).toBe('grok-voice-think-fast-2.0') + }) }) describe('grokRealtimeToken expires_at unit-safety', () => { diff --git a/packages/ai-grok/tests/video-adapter.test.ts b/packages/ai-grok/tests/video-adapter.test.ts index a6239adbc9..d31188b07e 100644 --- a/packages/ai-grok/tests/video-adapter.test.ts +++ b/packages/ai-grok/tests/video-adapter.test.ts @@ -45,10 +45,20 @@ function adapterWithFetch( }) } +function sourceAdapterWithFetch( + fetchMock: ( + input: string | URL | Request, + init?: RequestInit, + ) => Promise, +) { + return createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) +} + /** - * grok-imagine-video-1.5 is image-to-video only, so every request needs a - * starting-frame image part. This builds a text + image prompt for the - * request-shape / status / error tests. + * Builds a text + starting-frame-image prompt for the request-shape / + * status / error tests. */ function i2vPrompt(text = 'p') { return [ @@ -221,7 +231,7 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) - it('rejects video and audio prompt parts before calling the API', async () => { + it('rejects audio prompt parts before calling the API', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -231,28 +241,36 @@ describe('Grok Video Adapter', () => { prompt: [ { type: 'text', content: 'p' }, { - type: 'video', - source: { type: 'url', value: 'https://example.com/clip.mp4' }, + type: 'audio', + source: { type: 'url', value: 'https://example.com/voice.mp3' }, }, ], logger: testLogger, }), - ).rejects.toThrow(/does not support video prompt parts/) + ).rejects.toThrow(/does not support audio prompt parts/) expect(fetchMock).not.toHaveBeenCalled() }) - it('rejects a text-only prompt on 1.5 — image-to-video only', async () => { - const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + it('allows a text-only prompt on 1.5 (text-to-video, native 1080p)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'tv-15' })) const adapter = adapterWithFetch(fetchMock) - await expect( - adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', - prompt: 'a red ball bouncing once', - logger: testLogger, - }), - ).rejects.toThrow(/does not support text-to-video/) - expect(fetchMock).not.toHaveBeenCalled() + const result = await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'a red ball bouncing once', + size: '16:9_1080p', + logger: testLogger, + }) + + expect(result).toEqual({ + jobId: 'tv-15', + model: 'grok-imagine-video-1.5', + }) + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.prompt).toBe('a red ball bouncing once') + expect(body.aspect_ratio).toBe('16:9') + expect(body.resolution).toBe('1080p') + expect(body).not.toHaveProperty('image') }) it('allows a text-only prompt on grok-imagine-video (text-to-video)', async () => { @@ -309,6 +327,23 @@ describe('Grok Video Adapter', () => { expect(body.resolution).toBe('1080p') }) + it('rejects 1080p on grok-imagine-video (v1.0)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + // Native 1080p is 1.5-only; the v1 size map excludes the suffix. + // @ts-expect-error 1080p is not a grok-imagine-video size + size: '16:9_1080p', + logger: testLogger, + }), + ).rejects.toThrow(/1080p/) + expect(fetchMock).not.toHaveBeenCalled() + }) + it('rejects unsupported sizes before calling the API', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -390,10 +425,58 @@ describe('Grok Video Adapter', () => { logger: testLogger, }), ).rejects.toThrow( - /video generation request failed \(400.*Duration must be between 1 and 15 seconds/, + /\/videos\/generations request failed \(400.*Duration must be between 1 and 15 seconds/, ) }) + it('strips a JSON-null duration instead of sending it', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: i2vPrompt(), + // Serializers commonly encode "unset" as null; it must read as + // absent, not go on the wire as `"duration": null`. + // @ts-expect-error runtime handling of a JSON-null option + modelOptions: { duration: null }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body).not.toHaveProperty('duration') + }) + + it('treats null / empty reference fields as unset', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + + // Empty array on 1.5: nothing sent. + const adapter15 = adapterWithFetch(fetchMock) + await adapter15.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { reference_audios: [] }, + logger: testLogger, + }) + let body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body).not.toHaveProperty('reference_audios') + + // JSON-null on v1.0: reads as unset — no reference-gate throw, no + // field on the wire. + const adapter10 = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + await adapter10.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + // @ts-expect-error runtime handling of a JSON-null option + modelOptions: { reference_images: null }, + logger: testLogger, + }) + body = JSON.parse(String(fetchMock.mock.calls[1]![1]?.body)) + expect(body).not.toHaveProperty('reference_images') + }) + it('throws when the response carries no request_id', async () => { const fetchMock = mockFetch(() => jsonResponse({})) const adapter = adapterWithFetch(fetchMock) @@ -426,6 +509,591 @@ describe('Grok Video Adapter', () => { }) }) + describe('reference-to-video', () => { + it('maps role:reference image parts to reference_images', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-1' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' walks past ' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + { + type: 'image', + source: { type: 'data', mimeType: 'image/png', value: 'BBBB' }, + metadata: { role: 'character' }, + }, + ], + size: '16:9_720p', + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_images).toEqual([ + { url: 'https://example.com/person.png' }, + { url: 'data:image/png;base64,BBBB' }, + ]) + expect(body).not.toHaveProperty('image') + }) + + it('passes reference_audios voice ids through', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-2' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: ' narrates a sunrise', + modelOptions: { reference_audios: [{ voice_id: 'eve' }] }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_audios).toEqual([{ voice_id: 'eve' }]) + }) + + it('rejects more than 7 reference images before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { + reference_images: Array.from({ length: 8 }, (_, i) => ({ + url: `https://example.com/${i}.png`, + })), + }, + logger: testLogger, + }), + ).rejects.toThrow(/at most 7 reference images/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects more than 3 reference voices before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { + reference_audios: [ + { voice_id: 'eve' }, + { voice_id: 'ara' }, + { voice_id: 'rex' }, + { voice_id: 'sal' }, + ], + }, + logger: testLogger, + }), + ).rejects.toThrow(/at most 3 reference voices/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects combining a starting frame with reference images', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' enters from the left' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/start.png' }, + metadata: { role: 'start_frame' }, + }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/cannot be combined/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects combining a starting frame with reference_audios', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: i2vPrompt(' narrates'), + modelOptions: { reference_audios: [{ voice_id: 'eve' }] }, + logger: testLogger, + }), + ).rejects.toThrow(/cannot be combined/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects 1080p on reference-to-video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' walks' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + ], + size: '16:9_1080p', + logger: testLogger, + }), + ).rejects.toThrow(/capped at 720p/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('lets explicit modelOptions.reference_images replace the part-derived list', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-4' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/from-part.png' }, + metadata: { role: 'reference' }, + }, + ], + modelOptions: { + reference_images: [{ url: 'https://example.com/explicit.png' }], + }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_images).toEqual([ + { url: 'https://example.com/explicit.png' }, + ]) + }) + + it('rejects reference inputs on grok-imagine-video (v1.0)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + metadata: { role: 'reference' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/does not support reference-to-video inputs/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + modelOptions: { + // The per-model options map hides the reference fields from + // v1.0 at compile time; the runtime gate covers JSON callers. + // @ts-expect-error reference_audios is 1.5-only + reference_audios: [{ voice_id: 'eve' }], + }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support reference-to-video inputs/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects an unknown image metadata.role before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + // Roles arrive untrusted from JSON callers; an unrecognised + // value must throw, not silently drop the part. + // @ts-expect-error runtime validation of a non-MediaInputRole string + metadata: { role: 'first_frame' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/unknown image metadata\.role 'first_frame'/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects mask / control / end_frame image roles', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + for (const role of ['mask', 'control', 'end_frame'] as const) { + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + metadata: { role }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(new RegExp(`has no '${role}' image`)) + } + expect(fetchMock).not.toHaveBeenCalled() + }) + }) + + describe('edit and extend modes', () => { + const sourceVideoPart = { + type: 'video' as const, + source: { type: 'url' as const, value: 'https://example.com/clip.mp4' }, + } + + it("posts to /videos/edits in 'edit' mode with the source video", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'edit-1' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + const result = await adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'make the sky stormy' }, + sourceVideoPart, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }) + + expect(result).toEqual({ + jobId: 'edit-1', + model: 'grok-imagine-video', + }) + const [url, init] = fetchMock.mock.calls[0]! + expect(url).toBe('https://api.x.ai/v1/videos/edits') + expect(JSON.parse(String(init?.body))).toEqual({ + model: 'grok-imagine-video', + prompt: 'make the sky stormy', + video: { url: 'https://example.com/clip.mp4' }, + }) + }) + + it("posts to /videos/extensions in 'extend' mode with the added-tail duration", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-1' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'the camera keeps panning right' }, + sourceVideoPart, + ], + duration: 5, + modelOptions: { mode: 'extend' }, + logger: testLogger, + }) + + const [url, init] = fetchMock.mock.calls[0]! + expect(url).toBe('https://api.x.ai/v1/videos/extensions') + expect(JSON.parse(String(init?.body))).toEqual({ + model: 'grok-imagine-video', + prompt: 'the camera keeps panning right', + video: { url: 'https://example.com/clip.mp4' }, + duration: 5, + }) + }) + + it('sends a base64 data source as a data URI source video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'video', + source: { type: 'data', mimeType: 'video/mp4', value: 'CCCC' }, + }, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.video).toEqual({ url: 'data:video/mp4;base64,CCCC' }) + }) + + it('snaps the extend duration passed via modelOptions', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-2' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], + modelOptions: { mode: 'extend', duration: 2.6 }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.duration).toBe(3) + }) + + it('snaps an out-of-range generic extend duration', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-3' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], + duration: 20, + modelOptions: { mode: 'extend' }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.duration).toBe(15) + }) + + it("rejects a duration in 'edit' mode — inherited from the source", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + duration: 5, + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/'edit' mode does not accept a duration/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit', duration: 99 }, + logger: testLogger, + }), + ).rejects.toThrow(/'edit' mode does not accept a duration/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects size / aspect_ratio / resolution in edit and extend modes', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + size: '16:9_720p', + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'extend', aspect_ratio: '9:16' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit', resolution: '720p' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('names the edit endpoint in failure messages', async () => { + const fetchMock = mockFetch(() => + jsonResponse({ code: 'invalid-argument', error: 'bad clip' }, 400), + ) + const adapter = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/\/videos\/edits request failed \(400.*bad clip/) + }) + + it('rejects an unknown mode instead of misrouting to generations', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + // Mode arrives untrusted from JSON callers; a typo must not fall + // through to /videos/generations with a source-video body. + // @ts-expect-error runtime validation of a non-GrokVideoMode string + modelOptions: { mode: 'remix' }, + logger: testLogger, + }), + ).rejects.toThrow(/unknown modelOptions\.mode 'remix'/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects a video prompt part without an explicit mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + logger: testLogger, + }), + ).rejects.toThrow(/needs modelOptions\.mode/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects a mode without a source video part', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + modelOptions: { mode: 'extend' }, + logger: testLogger, + }), + ).rejects.toThrow(/requires a video prompt part/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects image prompt parts in edit / extend mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'p' }, + sourceVideoPart, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + }, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/takes only the source video/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects reference inputs in edit / extend mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { + mode: 'edit', + // Per-model map hides reference fields from v1.0; JSON callers + // still hit the runtime gate. + // @ts-expect-error reference_audios is 1.5-only + reference_audios: [{ voice_id: 'eve' }], + }, + logger: testLogger, + }), + ).rejects.toThrow(/reference inputs are only supported/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects more than one source video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'p' }, + sourceVideoPart, + sourceVideoPart, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/at most one source video/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects edit / extend on grok-imagine-video-1.5', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + // 1.5 has no video input on generateVideo(); createVideoJob still + // accepts the part so JSON callers hit the runtime gate. + sourceVideoPart, + ], + // @ts-expect-error mode is grok-imagine-video only + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support video editing or extension/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + // @ts-expect-error mode is grok-imagine-video only + modelOptions: { mode: 'extend' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support video editing or extension/) + expect(fetchMock).not.toHaveBeenCalled() + }) + }) + describe('getVideoStatus', () => { it('maps a pending job with progress', async () => { const fetchMock = mockFetch(() => @@ -602,7 +1270,12 @@ describe('Grok Video Adapter', () => { it('validates sizes', () => { expect(() => validateVideoSize('m', '16:9')).not.toThrow() - expect(() => validateVideoSize('m', '2:3_1080p')).not.toThrow() + expect(() => + validateVideoSize('grok-imagine-video-1.5', '2:3_1080p'), + ).not.toThrow() + expect(() => + validateVideoSize('grok-imagine-video', '16:9_1080p'), + ).toThrow(/1080p/) expect(() => validateVideoSize('m', undefined)).not.toThrow() expect(() => validateVideoSize('m', '9:19.5')).toThrow(/not supported/) expect(() => validateVideoSize('m', 'auto')).toThrow(/not supported/) diff --git a/packages/ai/skills/ai-core/media-generation/SKILL.md b/packages/ai/skills/ai-core/media-generation/SKILL.md index fdf185a615..15aef21f7e 100644 --- a/packages/ai/skills/ai-core/media-generation/SKILL.md +++ b/packages/ai/skills/ai-core/media-generation/SKILL.md @@ -295,18 +295,19 @@ with `allowUrlFetch: true` on the adapter config **Provider support matrix:** -| Provider | `generateImage` image parts | `generateVideo` image parts | -| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| OpenAI | gpt-image-2 / gpt-image-1 / -mini → `images.edit()` (up to 16). dall-e-2 → edit (1). dall-e-3 throws. | Sora-2 / -pro → `input_reference` (single). Throws if >1. | -| Gemini | Native (gemini-\*-flash-image, "nano-banana") → multimodal `contents`. Imagen throws. | Veo → first un-roled / `'start_frame'` image is the input image; `'end_frame'` → `lastFrame`; `'reference'` / `'character'` → `referenceImages`. Omni Flash sends image/video parts as interaction content blocks (no role routing). | -| fal | Per-endpoint field names from a generated map (`pnpm generate:fal-image-fields`). Defaults: 1 input → `image_url`; >1 → `image_urls`; roles → `mask_url` / `control_image_url` / `reference_image_urls`. | Per-endpoint map (e.g. Kling i2v start frame → `image_url`). Defaults: 1 input → `image_url`; `start_frame`/`end_frame` → `start_image_url`/`end_image_url`; `reference` → `reference_image_urls`. | -| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | n/a | -| OpenRouter | Prompt parts map 1:1 onto multimodal `text` / `image_url` content parts, preserving interleaved order. | Dedicated async API (`openRouterVideo`): `start_frame`/`end_frame` → `frame_images[]` (`first_frame`/`last_frame`); `reference`/`character` → `input_references[]`; an unroled image defaults to the start frame. Frame roles validated against the model's `supported_frame_images` metadata. | -| Anthropic | n/a (no image generation API). | n/a | +| Provider | `generateImage` image parts | `generateVideo` image parts | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| OpenAI | gpt-image-2 / gpt-image-1 / -mini → `images.edit()` (up to 16). dall-e-2 → edit (1). dall-e-3 throws. | Sora-2 / -pro → `input_reference` (single). Throws if >1. | +| Gemini | Native (gemini-\*-flash-image, "nano-banana") → multimodal `contents`. Imagen throws. | Veo → first un-roled / `'start_frame'` image is the input image; `'end_frame'` → `lastFrame`; `'reference'` / `'character'` → `referenceImages`. Omni Flash sends image/video parts as interaction content blocks (no role routing). | +| fal | Per-endpoint field names from a generated map (`pnpm generate:fal-image-fields`). Defaults: 1 input → `image_url`; >1 → `image_urls`; roles → `mask_url` / `control_image_url` / `reference_image_urls`. | Per-endpoint map (e.g. Kling i2v start frame → `image_url`). Defaults: 1 input → `image_url`; `start_frame`/`end_frame` → `start_image_url`/`end_image_url`; `reference` → `reference_image_urls`. | +| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | Un-roled / `'start_frame'` image → starting frame; `'reference'` / `'character'` → `reference_images` (1.5). Starting frame and reference inputs cannot be combined. A `video` part + `modelOptions.mode: 'edit' \| 'extend'` routes to `/videos/edits` / `/videos/extensions` on `grok-imagine-video` only. | +| OpenRouter | Prompt parts map 1:1 onto multimodal `text` / `image_url` content parts, preserving interleaved order. | Dedicated async API (`openRouterVideo`): `start_frame`/`end_frame` → `frame_images[]` (`first_frame`/`last_frame`); `reference`/`character` → `input_references[]`; an unroled image defaults to the start frame. Frame roles validated against the model's `supported_frame_images` metadata. | +| Anthropic | n/a (no image generation API). | n/a | Video and audio prompt parts follow the same `metadata.role` convention -for video-to-video and lipsync flows on fal; other providers throw when -they're passed. +for video-to-video and lipsync flows on fal. Grok accepts one source +`video` part on `grok-imagine-video` with `modelOptions.mode: 'edit' | 'extend'` +and rejects audio parts. Other providers throw when those parts are passed. ### 2. Audio Generation (Music, Sound Effects) @@ -544,10 +545,16 @@ const edited = await generateVideo({ Other video adapters: `openaiVideo('sora-2')` (pixel sizes like `'1280x720'`, durations 4/8/12s, single `input_reference` image prompt part), `grokVideo(...)` -(`grok-imagine-video` does text-to-video + image-to-video; `grok-imagine-video-1.5` is -image-to-video only — needs an `image` prompt part as the starting frame, text-only throws; -aspect-ratio size template like `'16:9_720p'`, integer durations 1-15s, reports -`usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — +(`grok-imagine-video` and `grok-imagine-video-1.5` both do text-to-video + image-to-video; +1.5 adds reference-to-video — `'reference'`/`'character'`-roled image parts → +`reference_images` (max 7), preset voices via `modelOptions.reference_audios` (max 3) — +1.5-only, capped at 720p, and not combinable with a starting-frame image; only +`grok-imagine-video` edits/extends a source `video` prompt part via +`modelOptions.mode: 'edit' | 'extend'` (extend `duration` = added tail). Edit/extend +outputs inherit the source clip's properties, so `size`/`aspect_ratio`/`resolution` +throw in both modes and `duration` throws in edit mode — pass none of them there; +generation uses the aspect-ratio size template like `'16:9_720p'` (1080p is 1.5-only), +integer durations 1-15s, reports `usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — aspect-ratio size template like `'16:9_720p'`, durations 4-15s on the 2.0 family, 4-12s on 1.5-pro, 2-12s on the 1.0-pro models; reads `ARK_API_KEY`), `openRouterVideo(...)` (OpenRouter's dedicated `POST /api/v1/videos` gateway),