From 36424b2408351512bafbf19257bcd621b359cda2 Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:25:26 +1000 Subject: [PATCH 1/5] feat(ai-grok): catch up Grok Imagine image, video, and voice modes - Image: add grok-imagine-image-2.0 (xAI's recommended model) with the 2.0-only quality provider option ('low' | 'medium'). - Video: drop the stale image-to-video-only guard on grok-imagine-video-1.5 (it now supports text-to-video with native 1080p); add reference-to-video (role:'reference' image parts -> reference_images, preset voices via modelOptions.reference_audios, max 3); add edit/extend modes via a source video prompt part + modelOptions.mode ('edit' -> /v1/videos/edits, 'extend' -> /v1/videos/extensions with duration = added tail). - Voice: add grok-voice-think-fast-2.0 (current recommended) and the grok-voice-latest alias; move realtime token/adapter defaults off the deprecated 1.0 ids. - Fix the stale GROK_CHAT_MODELS test expectation broken by the #1048 model-metadata sync. - Update docs, media-generation skill, and example apps accordingly. Closes #1133 --- .changeset/grok-imagine-catch-up.md | 9 + docs/adapters/grok.md | 92 +++++- docs/config.json | 4 +- docs/media/video-generation.md | 53 +++- .../ts-react-chat/src/lib/use-realtime.ts | 2 +- examples/ts-react-media/src/lib/models.ts | 19 +- .../src/lib/server-functions.ts | 23 ++ packages/ai-grok/src/adapters/image.ts | 3 +- packages/ai-grok/src/adapters/video.ts | 189 ++++++++--- .../src/image/image-provider-options.ts | 15 + packages/ai-grok/src/index.ts | 3 + packages/ai-grok/src/model-meta.ts | 59 +++- packages/ai-grok/src/realtime/adapter.ts | 4 +- packages/ai-grok/src/realtime/token.ts | 5 +- packages/ai-grok/src/realtime/types.ts | 2 +- .../src/video/video-provider-options.ts | 83 +++-- packages/ai-grok/tests/grok-adapter.test.ts | 27 +- packages/ai-grok/tests/video-adapter.test.ts | 300 +++++++++++++++++- .../skills/ai-core/media-generation/SKILL.md | 8 +- 19 files changed, 772 insertions(+), 128 deletions(-) create mode 100644 .changeset/grok-imagine-catch-up.md diff --git a/.changeset/grok-imagine-catch-up.md b/.changeset/grok-imagine-catch-up.md new file mode 100644 index 0000000000..7c0c014200 --- /dev/null +++ b/.changeset/grok-imagine-catch-up.md @@ -0,0 +1,9 @@ +--- +'@tanstack/ai-grok': minor +--- + +Catch up with the current xAI Imagine / Voice catalog: + +- **Image**: add `grok-imagine-image-2.0` (xAI's recommended model, $0.04/image) with its 2.0-only `quality: 'low' | 'medium'` provider option. +- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3). Video editing and extension land via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). +- **Voice**: add `grok-voice-think-fast-2.0` (current recommended) and the `grok-voice-latest` alias to the realtime models; the realtime token and adapter defaults move off the deprecated 1.0 ids to `grok-voice-think-fast-2.0`. diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 832ac99866..00040af61c 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -181,18 +181,21 @@ const result = await generateImage({ console.log(result.images); ``` -The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-quality`) -are aspect-ratio sized — `size` takes an `aspectRatio_resolution` template -like `"16:9_2k"` (the `_2k` suffix is optional): +The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-2.0`, +`grok-imagine-image-quality`) are aspect-ratio sized — `size` takes an +`aspectRatio_resolution` template like `"16:9_2k"` (the `_2k` suffix is +optional). `grok-imagine-image-2.0` is xAI's recommended model and adds a +2.0-only `quality` provider option (`'low' | 'medium'`, default `'medium'`): ```typescript import { generateImage } from "@tanstack/ai"; import { grokImage } from "@tanstack/ai-grok"; const result = await generateImage({ - adapter: grokImage("grok-imagine-image"), + adapter: grokImage("grok-imagine-image-2.0"), prompt: "A futuristic cityscape at sunset", size: "16:9_2k", + modelOptions: { quality: "medium" }, }); ``` @@ -239,15 +242,15 @@ Generate short video clips (1–15 seconds, with audio) with the Grok Imagine vi Available models: - `grok-imagine-video` (v1.0) — text-to-video and image-to-video, $0.05 per second of video. -- `grok-imagine-video-1.5` — **image-to-video only**, $0.08 per second of video. A text-only prompt is rejected by the API; the adapter fails fast with a clear error telling you to add a starting-frame image or use `grok-imagine-video`. +- `grok-imagine-video-1.5` — xAI's recommended default, $0.08 per second of video. Supports text-to-video (with native 1080p), image-to-video, and reference-to-video. -Text-to-video with the base `grok-imagine-video` model: +Text-to-video: ```typescript import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; -const adapter = grokVideo("grok-imagine-video"); +const adapter = grokVideo("grok-imagine-video-1.5"); // 1. Create the job const { jobId } = await generateVideo({ @@ -267,7 +270,7 @@ while (status.status !== "completed" && status.status !== "failed") { console.log(status.url); // hosted .mp4 URL ``` -For image-to-video (required for `grok-imagine-video-1.5`, optional for `grok-imagine-video`), include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: +For image-to-video, include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -292,6 +295,77 @@ const { jobId } = await generateVideo({ Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. +### Reference-to-Video + +On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p: + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video-1.5"), + prompt: [ + { + type: "text", + content: " walks through a neon-lit alley while narrates", + }, + { + type: "image", + source: { type: "url", value: "https://example.com/character.png" }, + metadata: { role: "reference" }, + }, + ], + size: "16:9_720p", + modelOptions: { + reference_audios: [{ voice_id: "eve" }], + }, +}); +``` + +### Video Editing and Extension + +Both models can rewrite or continue an existing clip. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`: + +- `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p). +- `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const adapter = grokVideo("grok-imagine-video-1.5"); + +// Edit: change the clip in place +const edit = await generateVideo({ + adapter, + prompt: [ + { type: "text", content: "Make the sky stormy with distant lightning" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + modelOptions: { mode: "edit" }, +}); + +// Extend: append 5 more seconds +const extension = await generateVideo({ + adapter, + prompt: [ + { type: "text", content: "The camera keeps panning right across the bay" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + duration: 5, // added seconds, not the total + modelOptions: { mode: "extend" }, +}); +``` + +Both return the usual `{ jobId }` and are polled like any other Grok video job. + When the job completes, the adapter reports usage on the result: `usage.unitsBilled` carries the billed seconds of video and `usage.cost` the exact cost in USD, both as returned by the xAI API. See [Video Generation](../media/video-generation) for the full jobs/polling flow, streaming mode, and the `useGenerateVideo` hook. @@ -333,7 +407,7 @@ console.log(result.text); ## Realtime Voice -Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. +Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. The default model is `grok-voice-think-fast-2.0` (xAI's current recommended speech-to-speech model); `grok-voice-latest` always points at the newest model. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. ## Environment Variables diff --git a/docs/config.json b/docs/config.json index e5c98bc877..b8a9c09fe1 100644 --- a/docs/config.json +++ b/docs/config.json @@ -443,7 +443,7 @@ "label": "Video Generation", "to": "media/video-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-13" + "updatedAt": "2026-08-18" }, { "label": "Generation Hooks", @@ -808,7 +808,7 @@ "label": "Grok (xAI)", "to": "adapters/grok", "addedAt": "2026-04-15", - "updatedAt": "2026-06-24" + "updatedAt": "2026-08-18" }, { "label": "Groq", diff --git a/docs/media/video-generation.md b/docs/media/video-generation.md index a75f0f7909..e446826ec6 100644 --- a/docs/media/video-generation.md +++ b/docs/media/video-generation.md @@ -49,7 +49,7 @@ Currently supported: - **OpenAI**: Sora-2 and Sora-2-Pro models (when available) - **Google Gemini**: Veo 3.1 models (via the long-running operations API), and Gemini Omni Flash (via the Interactions API) -- **Grok (xAI)**: grok-imagine-video (text-to-video + image-to-video) and grok-imagine-video-1.5 (image-to-video only) models +- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5 (text-to-video, image-to-video, reference-to-video, editing, extension) - **BytePlus**: Seedance 2.0, 1.5-pro and 1.0-pro models (text-to-video, first/last frame, and multimodal references on 2.0) - **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models - **OpenRouter**: Seedance, Veo 3.1, Wan, Kling, Sora 2 Pro and others via the dedicated async video API (`POST /api/v1/videos`) @@ -691,16 +691,16 @@ instead of letting the model infer the task mode). #### Grok (xAI Imagine) Model Options -Based on the [xAI video generation API](https://docs.x.ai/docs/guides/video-generations). Two models are available: `grok-imagine-video` (v1.0) supports **text-to-video and image-to-video**, while `grok-imagine-video-1.5` is **image-to-video only** (a text-only prompt is rejected by the API; the adapter throws a clear error pointing you at `grok-imagine-video`). Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. +Based on the [xAI video generation API](https://docs.x.ai/developers/model-capabilities/video/generation). Two models are available: `grok-imagine-video` (v1.0) and `grok-imagine-video-1.5` (xAI's recommended default, with native 1080p text-to-video). Both support **text-to-video and image-to-video**; 1.5 adds **reference-to-video**, and both drive **video editing and extension**. Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. -Text-to-video with the base model: +Text-to-video: ```typescript import { generateVideo } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; const { jobId } = await generateVideo({ - adapter: grokVideo("grok-imagine-video"), + adapter: grokVideo("grok-imagine-video-1.5"), prompt: "A beautiful sunset over the ocean", size: "16:9_720p", // aspect ratio: '1:1' | '16:9' | '9:16' | '4:3' | '3:4' | '3:2' | '2:3' // resolution (optional suffix): '480p' | '720p' | '1080p' @@ -713,7 +713,7 @@ const { jobId } = await generateVideo({ }); ``` -Image-to-video (required for `grok-imagine-video-1.5`) — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: +Image-to-video — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -733,12 +733,53 @@ const { jobId } = await generateVideo({ }); ``` +Reference-to-video (`grok-imagine-video-1.5`, capped at 720p) — image prompt parts with `metadata.role: 'reference'` become `reference_images` (addressed from the prompt as ``, ``, …), and up to 3 preset TTS voices can be referenced via `modelOptions.reference_audios` (addressed as ``, …): + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video-1.5"), + prompt: [ + { type: "text", content: " waves at the camera while says hello" }, + { + type: "image", + source: { type: "url", value: "https://example.com/character.png" }, + metadata: { role: "reference" }, + }, + ], + size: "16:9_720p", + modelOptions: { reference_audios: [{ voice_id: "eve" }] }, +}); +``` + +Video editing and extension — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total: + +```typescript +import { generateVideo } from "@tanstack/ai"; +import { grokVideo } from "@tanstack/ai-grok"; + +const { jobId } = await generateVideo({ + adapter: grokVideo("grok-imagine-video-1.5"), + prompt: [ + { type: "text", content: "The camera keeps panning right across the bay" }, + { + type: "video", + source: { type: "url", value: "https://example.com/clip.mp4" }, + }, + ], + duration: 5, // 'extend' mode: seconds added to the clip, not the total + modelOptions: { mode: "extend" }, +}); +``` + Both models accept any whole second in the **1–15** range. A raw `duration` is coerced into that range rather than rejected — values are clamped to `[1, 15]` and rounded to the nearest second. Inspect or pre-snap the range the same way as Veo: ```typescript import { grokVideo } from "@tanstack/ai-grok"; -const adapter = grokVideo("grok-imagine-video"); +const adapter = grokVideo("grok-imagine-video-1.5"); adapter.availableDurations(); // { kind: 'range', min: 1, max: 15, step: 1, unit: 'seconds' } adapter.snapDuration(2.5); // 3 — clamped/rounded into range diff --git a/examples/ts-react-chat/src/lib/use-realtime.ts b/examples/ts-react-chat/src/lib/use-realtime.ts index 44874ef0da..3c3038f577 100644 --- a/examples/ts-react-chat/src/lib/use-realtime.ts +++ b/examples/ts-react-chat/src/lib/use-realtime.ts @@ -42,7 +42,7 @@ const getRealtimeTokenFn = createServerFn({ method: 'POST' }) if (data.provider === 'grok') { return realtimeToken({ - adapter: grokRealtimeToken({ model: 'grok-voice-fast-1.0' }), + adapter: grokRealtimeToken({ model: 'grok-voice-think-fast-2.0' }), }) } diff --git a/examples/ts-react-media/src/lib/models.ts b/examples/ts-react-media/src/lib/models.ts index 633f8da8bd..4c42c741e9 100644 --- a/examples/ts-react-media/src/lib/models.ts +++ b/examples/ts-react-media/src/lib/models.ts @@ -23,6 +23,14 @@ export const IMAGE_MODELS = [ sizeType: 'aspect_ratio' as const, provider: 'xai' as const, }, + { + id: 'grok-imagine-image-2.0', + name: 'Grok Imagine 2.0 (xAI Direct)', + description: 'xAI recommended Imagine model with the quality option', + defaultSize: '16:9' as const, + sizeType: 'aspect_ratio' as const, + provider: 'xai' as const, + }, { id: 'grok-imagine-image-quality', name: 'Grok Imagine Quality (xAI Direct)', @@ -149,11 +157,18 @@ export const VIDEO_MODELS = [ mode: 'text-to-video' as const, provider: 'xai' as const, }, + { + id: 'grok-imagine-video-1.5', + name: 'Grok Imagine Video 1.5 (Text-to-Video)', + description: + 'xAI recommended video model via the native grokVideo adapter (native 1080p text-to-video)', + mode: 'text-to-video' as const, + provider: 'xai' as const, + }, { id: 'grok-imagine-video-1.5/image-to-video', name: 'Grok Imagine Video 1.5 (Image-to-Video)', - description: - 'Animate a starting frame via the native grokVideo adapter (1.5 is image-to-video only)', + description: 'Animate a starting frame via the native grokVideo adapter', mode: 'image-to-video' as const, provider: 'xai' as const, }, diff --git a/examples/ts-react-media/src/lib/server-functions.ts b/examples/ts-react-media/src/lib/server-functions.ts index aa0f12c07a..74df551dfe 100644 --- a/examples/ts-react-media/src/lib/server-functions.ts +++ b/examples/ts-react-media/src/lib/server-functions.ts @@ -167,6 +167,17 @@ export const generateImageFn = createServerFn({ method: 'POST' }) size: '16:9', }) } + case 'grok-imagine-image-2.0': { + // xAI's recommended Imagine model; `quality` is a 2.0-only option + // ('low' | 'medium', default 'medium'). + return generateImage({ + adapter: grokImage('grok-imagine-image-2.0'), + prompt: asImagePrompt(data.prompt), + numberOfImages: 1, + size: '16:9', + modelOptions: { quality: 'medium' }, + }) + } case 'grok-imagine-image-quality': { return generateImage({ adapter: grokImage('grok-imagine-image-quality'), @@ -339,6 +350,18 @@ function videoStreamForModel(data: VideoRequest): AsyncIterable { duration: 5, }) } + case 'grok-imagine-video-1.5': { + // Direct xAI Imagine API — grok-imagine-video-1.5 is xAI's recommended + // default and supports text-to-video with native 1080p. + return generateVideo({ + stream: true, + pollingInterval: VIDEO_POLL_INTERVAL_MS, + adapter: grokVideo('grok-imagine-video-1.5'), + prompt: asTextPrompt(data.prompt), + size: '16:9_720p', + duration: 5, + }) + } case 'dreamina-seedance-2-0-260128': { // BytePlus Seedance via ModelArk (ARK_API_KEY). `size` is a "ratio" or // "ratio_resolution" template; durations are 4-15 integer seconds. diff --git a/packages/ai-grok/src/adapters/image.ts b/packages/ai-grok/src/adapters/image.ts index 50bd38e51c..28932c83df 100644 --- a/packages/ai-grok/src/adapters/image.ts +++ b/packages/ai-grok/src/adapters/image.ts @@ -129,7 +129,8 @@ export class GrokImageAdapter< throw new Error( `grok: model "${model}" does not support image prompt parts. ` + `Image-conditioned generation requires an Imagine API model ` + - `('grok-imagine-image' or 'grok-imagine-image-quality').`, + `('grok-imagine-image', 'grok-imagine-image-2.0' or ` + + `'grok-imagine-image-quality').`, ) } return await this.editImages(options, resolved) diff --git a/packages/ai-grok/src/adapters/video.ts b/packages/ai-grok/src/adapters/video.ts index 21807360e3..5c1d95df64 100644 --- a/packages/ai-grok/src/adapters/video.ts +++ b/packages/ai-grok/src/adapters/video.ts @@ -3,8 +3,8 @@ import { BaseVideoAdapter, snapToDurationOption } from '@tanstack/ai/adapters' import { toRunErrorPayload } from '@tanstack/ai/adapter-internals' import { getGrokApiKeyFromEnv, withGrokDefaults } from '../utils/client' import { + GROK_VIDEO_MAX_REFERENCE_AUDIOS, getGrokVideoDurationOptions, - isImageToVideoOnlyModel, parseGrokVideoSize, validateVideoSize, } from '../video/video-provider-options' @@ -15,6 +15,7 @@ import type { TokenUsage, VideoGenerationOptions, VideoJobResult, + VideoPart, VideoStatusResult, VideoUrlResult, } from '@tanstack/ai' @@ -62,11 +63,13 @@ interface GrokVideoStatusResponse { } /** - * Convert a TanStack ImagePart to the URL string accepted by xAI's Imagine - * video endpoint: public URLs pass through (fetched by xAI's servers), data - * sources become base64 data URIs. + * Convert a TanStack image / video part to the URL string accepted by xAI's + * Imagine video endpoints: public URLs pass through (fetched by xAI's + * servers), data sources become base64 data URIs. */ -function imagePartToUrl(part: ImagePart): string { +function mediaPartToUrl( + part: ImagePart | VideoPart, +): string { if (part.source.type === 'url') return part.source.value return `data:${part.source.mimeType};base64,${part.source.value}` } @@ -93,10 +96,9 @@ function buildGrokVideoUsage( * async jobs/polling architecture: create a generation request, poll it, * then read the completed video URL. * - * `grok-imagine-video` (v1.0) supports text-to-video and image-to-video. - * `grok-imagine-video-1.5` is image-to-video only — every request needs an - * image prompt part as the starting frame, and the adapter rejects a - * text-only prompt with a clear error rather than a raw API 400. + * Both models support text-to-video and image-to-video; + * `grok-imagine-video-1.5` is xAI's documented default and adds native + * 1080p text-to-video plus reference-to-video inputs. * * The Imagine video endpoints are not part of the OpenAI SDK surface (and * xAI rejects the SDK's multipart paths), so requests are plain JSON calls @@ -109,6 +111,12 @@ function buildGrokVideoUsage( * - Aspect-ratio sizing via the "aspectRatio_resolution" size template * (e.g. '16:9_720p'), consistent with the grok-imagine image models * - Image-to-video via an `image` prompt part (starting frame URL or data URI) + * - Reference-to-video via image prompt parts with + * `metadata.role: 'reference'` (→ `reference_images`) and preset voices + * via `modelOptions.reference_audios` (grok-imagine-video-1.5) + * - Video editing / extension via a source `video` prompt part and + * `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / + * `/v1/videos/extensions`; in extend mode `duration` is the added tail) * - Usage reporting: billed seconds (`unitsBilled`) and exact cost */ export class GrokVideoAdapter< @@ -183,75 +191,165 @@ export class GrokVideoAdapter< validateVideoSize(model, size) + // `mode` is a routing hint for this adapter, not an API field — strip it + // before the remaining options are spread onto the request body. + const { mode, ...wireOptions } = modelOptions ?? {} + // Coerce the requested duration into the model's valid range (1–15s, // integer) instead of rejecting it — `snapDuration` clamps and rounds. // modelOptions wins over the generic `duration`, mirroring the size - // precedence below. - const rawDuration = modelOptions?.duration ?? options.duration + // precedence below. In extend mode the same range applies to the added + // tail. + const rawDuration = wireOptions.duration ?? options.duration const duration = rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined // The interleaved prompt decomposes into verbatim text plus typed media - // buckets. The Imagine video endpoint takes a text prompt and an optional - // starting frame; reject the modalities it can't consume. + // buckets. Reference audio is voice-id based (not an audio file), so + // audio prompt parts have no request field to land in. const resolved = resolveMediaPrompt(options.prompt) - if (resolved.videos.length > 0) { + if (resolved.audios.length > 0) { throw new Error( - `${this.name}.createVideoJob does not support video prompt parts (model: ${model}).`, + `${this.name}.createVideoJob does not support audio prompt parts (model: ${model}). ` + + `To reference a preset voice, pass modelOptions.reference_audios ` + + `(e.g. [{ voice_id: 'eve' }]).`, ) } - if (resolved.audios.length > 0) { + + // A video prompt part is the source clip for edit / extension mode; the + // mode must be chosen explicitly because the two endpoints have different + // semantics (edit rewrites the clip, extend appends `duration` seconds). + if (resolved.videos.length > 1) { + throw new Error( + `${this.name}: ${model} accepts at most one source video; received ${resolved.videos.length}.`, + ) + } + const [sourceVideo] = resolved.videos + if (sourceVideo && mode === undefined) { + throw new Error( + `${this.name}: a video prompt part needs modelOptions.mode set to ` + + `'edit' (rewrite the clip) or 'extend' (append to it).`, + ) + } + if (!sourceVideo && mode !== undefined) { throw new Error( - `${this.name}.createVideoJob does not support audio prompt parts (model: ${model}).`, + `${this.name}: modelOptions.mode '${mode}' requires a video prompt ` + + `part carrying the source clip.`, ) } - // grok-imagine-video-1.5 is image-to-video only — text-to-video is - // rejected by the API, so fail fast with a clear, actionable message - // pointing at the model that does support text-to-video. - if (resolved.images.length === 0 && isImageToVideoOnlyModel(model)) { + + // Image parts split by role: un-roled / 'start_frame' images become the + // starting frame (image-to-video); 'reference' / 'character' images + // become reference_images (reference-to-video). The Imagine API has no + // mask / control / end-frame inputs. + const startFrames: Array> = [] + const referenceImages: Array<{ url: string }> = [] + for (const part of resolved.images) { + const role = part.metadata?.role + switch (role) { + case 'mask': + case 'control': + case 'end_frame': + throw new Error( + `${this.name}: the Imagine video API has no '${role}' image ` + + `input on model ${model}. Use an un-roled / 'start_frame' ` + + `image as the starting frame, or 'reference' images.`, + ) + case 'reference': + case 'character': + referenceImages.push({ url: mediaPartToUrl(part) }) + break + case 'start_frame': + case undefined: + startFrames.push(part) + break + } + } + if (startFrames.length > 1) { + throw new Error( + `${this.name}: ${model} accepts at most one starting-frame image; received ${startFrames.length}. ` + + `Use metadata.role: 'reference' for reference-to-video inputs.`, + ) + } + if (mode !== undefined && resolved.images.length > 0) { throw new Error( - `${this.name}: ${model} does not support text-to-video — it is image-to-video only. ` + - `Include an image prompt part as the starting frame, or use 'grok-imagine-video' for text-to-video.`, + `${this.name}: '${mode}' mode takes only the source video — image ` + + `prompt parts are not supported by ${mode === 'edit' ? '/videos/edits' : '/videos/extensions'}.`, ) } - if (resolved.images.length > 1) { + const referenceAudioCount = wireOptions.reference_audios?.length ?? 0 + if (referenceAudioCount > GROK_VIDEO_MAX_REFERENCE_AUDIOS) { throw new Error( - `${this.name}: ${model} accepts at most one starting-frame image; received ${resolved.images.length}.`, + `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_AUDIOS} reference voices; received ${referenceAudioCount}.`, + ) + } + if ( + mode !== undefined && + (wireOptions.reference_images !== undefined || + wireOptions.reference_audios !== undefined) + ) { + throw new Error( + `${this.name}: reference inputs are only supported by video ` + + `generation, not '${mode}' mode.`, ) } // Image-to-video: the single image prompt part becomes the starting frame // and the prompt text describes the desired motion. URL sources are // fetched by xAI's servers; data sources are sent as base64 data URIs. - const [startFrame] = resolved.images + const [startFrame] = startFrames // The generic `size` option carries an "aspectRatio_resolution" template // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / // `resolution` parameters; explicit modelOptions win over the template. + // Edit / extend outputs inherit these from the source clip, so the + // template is only mapped for generation requests. const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined - const request = { - model, - prompt: resolved.text, - ...(startFrame && { image: { url: imagePartToUrl(startFrame) } }), - ...(parsedSize && { - aspect_ratio: parsedSize.aspectRatio, - ...(parsedSize.resolution !== undefined && { - resolution: parsedSize.resolution, - }), - }), - ...modelOptions, - // Spread after modelOptions so the snapped duration is authoritative - // (modelOptions.duration is folded into `duration` via snapDuration above). - ...(duration !== undefined && { duration }), - } + const request = + mode !== undefined && sourceVideo + ? { + model, + prompt: resolved.text, + video: { url: mediaPartToUrl(sourceVideo) }, + ...wireOptions, + // Edit outputs also inherit their length from the source clip; + // extend takes the snapped duration as the added-tail length. + ...(mode === 'extend' && duration !== undefined && { duration }), + } + : { + model, + prompt: resolved.text, + ...(startFrame && { image: { url: mediaPartToUrl(startFrame) } }), + ...(referenceImages.length > 0 && { + reference_images: referenceImages, + }), + ...(parsedSize && { + aspect_ratio: parsedSize.aspectRatio, + ...(parsedSize.resolution !== undefined && { + resolution: parsedSize.resolution, + }), + }), + ...wireOptions, + // Spread after wireOptions so the snapped duration is + // authoritative (modelOptions.duration is folded into `duration` + // via snapDuration above). + ...(duration !== undefined && { duration }), + } + + const endpoint = + mode === 'edit' + ? '/videos/edits' + : mode === 'extend' + ? '/videos/extensions' + : '/videos/generations' try { logger.request( - `activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`, + `activity=video.create provider=${this.name} model=${model} mode=${mode ?? 'generate'} size=${size ?? 'default'} duration=${duration ?? 'default'}`, { provider: this.name, model }, ) - const response = await this.request('/videos/generations', { + const response = await this.request(endpoint, { method: 'POST', body: JSON.stringify(request), }) @@ -401,8 +499,7 @@ export class GrokVideoAdapter< * * @example * ```typescript - * // grok-imagine-video (v1.0) supports text-to-video. - * const adapter = createGrokVideo('grok-imagine-video', 'xai-...'); + * const adapter = createGrokVideo('grok-imagine-video-1.5', 'xai-...'); * * const { jobId } = await generateVideo({ * adapter, @@ -440,7 +537,7 @@ export function createGrokVideo( * // Automatically uses XAI_API_KEY from environment * const adapter = grokVideo('grok-imagine-video-1.5'); * - * // Image-to-video only: the prompt must carry a starting-frame image part. + * // Image-to-video: an optional image prompt part is the starting frame. * const { jobId } = await generateVideo({ * adapter, * prompt: [ diff --git a/packages/ai-grok/src/image/image-provider-options.ts b/packages/ai-grok/src/image/image-provider-options.ts index a433534444..1b4a40a8c1 100644 --- a/packages/ai-grok/src/image/image-provider-options.ts +++ b/packages/ai-grok/src/image/image-provider-options.ts @@ -141,12 +141,25 @@ export interface GrokImagineImageProviderOptions extends GrokImageBaseProviderOp service_tier?: 'default' | 'priority' } +/** + * Provider options for grok-imagine-image-2.0, which adds a generation + * `quality` knob on top of the shared Imagine options. + */ +export interface GrokImagineImage2ProviderOptions extends GrokImagineImageProviderOptions { + /** + * Generation quality. Only supported by grok-imagine-image-2.0. + * @default 'medium' + */ + quality?: 'low' | 'medium' +} + /** * Type-only map from model name to its specific provider options. */ export type GrokImageModelProviderOptionsByName = { 'grok-2-image-1212': GrokImageProviderOptions 'grok-imagine-image': GrokImagineImageProviderOptions + 'grok-imagine-image-2.0': GrokImagineImage2ProviderOptions 'grok-imagine-image-quality': GrokImagineImageProviderOptions } @@ -156,6 +169,7 @@ export type GrokImageModelProviderOptionsByName = { export type GrokImageModelSizeByName = { 'grok-2-image-1212': GrokImageSize 'grok-imagine-image': GrokImagineImageSize + 'grok-imagine-image-2.0': GrokImagineImageSize 'grok-imagine-image-quality': GrokImagineImageSize } @@ -167,6 +181,7 @@ export type GrokImageModelSizeByName = { export type GrokImageModelInputModalitiesByName = { 'grok-2-image-1212': readonly [] 'grok-imagine-image': readonly ['image'] + 'grok-imagine-image-2.0': readonly ['image'] 'grok-imagine-image-quality': readonly ['image'] } diff --git a/packages/ai-grok/src/index.ts b/packages/ai-grok/src/index.ts index e342645caa..a5a5495fd6 100644 --- a/packages/ai-grok/src/index.ts +++ b/packages/ai-grok/src/index.ts @@ -28,6 +28,8 @@ export { } from './adapters/image' export type { GrokImageProviderOptions, + GrokImagineImageProviderOptions, + GrokImagineImage2ProviderOptions, GrokImageModelProviderOptionsByName, } from './image/image-provider-options' @@ -43,6 +45,7 @@ export { getGrokVideoDurationOptions, } from './video/video-provider-options' export type { + GrokVideoMode, GrokVideoProviderOptions, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, diff --git a/packages/ai-grok/src/model-meta.ts b/packages/ai-grok/src/model-meta.ts index ea476fe998..c65efe76cf 100644 --- a/packages/ai-grok/src/model-meta.ts +++ b/packages/ai-grok/src/model-meta.ts @@ -131,17 +131,37 @@ const GROK_IMAGINE_IMAGE_QUALITY = { }, } as const satisfies ModelMeta +// xAI's recommended Imagine image model. Supports the 2.0-only `quality` +// provider option ('low' | 'medium', default 'medium'). +const GROK_IMAGINE_IMAGE_2_0 = { + name: 'grok-imagine-image-2.0', + supports: { + input: ['text', 'image'], + output: ['image'], + }, + pricing: { + input: { + normal: 0, + }, + output: { + normal: 0.04, + }, + }, +} as const satisfies ModelMeta + // Imagine API video models. Pricing is per second of generated video // (output only); generated videos carry an audio track. // -// grok-imagine-video (v1.0) supports both text-to-video (a starting image is -// optional) and image-to-video. grok-imagine-video-1.5 is image-to-video -// only: a starting-frame image is required (the text prompt describes the -// desired motion) — its text-to-video is rejected by the API. +// Both models support text-to-video and image-to-video (a starting-frame +// image is optional). grok-imagine-video-1.5 is the documented default: it +// adds native 1080p text-to-video plus reference-to-video inputs +// (`reference_images` / `reference_audios`, capped at 720p). Both models +// also drive video editing (`/v1/videos/edits`) and extension +// (`/v1/videos/extensions`) via a source video prompt part. const GROK_IMAGINE_VIDEO = { name: 'grok-imagine-video', supports: { - input: ['text', 'image'], + input: ['text', 'image', 'video'], output: ['video', 'audio'], }, pricing: { @@ -158,7 +178,7 @@ const GROK_IMAGINE_VIDEO = { const GROK_IMAGINE_VIDEO_1_5 = { name: 'grok-imagine-video-1.5', supports: { - input: ['text', 'image'], + input: ['text', 'image', 'video'], output: ['video', 'audio'], }, pricing: { @@ -228,6 +248,7 @@ export const GROK_CHAT_MODELS = [ export const GROK_IMAGE_MODELS = [ GROK_2_IMAGE.name, GROK_IMAGINE_IMAGE.name, + GROK_IMAGINE_IMAGE_2_0.name, GROK_IMAGINE_IMAGE_QUALITY.name, ] as const @@ -273,6 +294,7 @@ const GROK_VOICE_FAST_1 = { }, } as const satisfies ModelMeta +/** @deprecated xAI has deprecated grok-voice-think-fast-1.0 — use grok-voice-think-fast-2.0. */ const GROK_VOICE_THINK_FAST_1 = { name: 'grok-voice-think-fast-1.0', supports: { @@ -283,11 +305,36 @@ const GROK_VOICE_THINK_FAST_1 = { }, } as const satisfies ModelMeta +// xAI's current recommended speech-to-speech model. +const GROK_VOICE_THINK_FAST_2 = { + name: 'grok-voice-think-fast-2.0', + supports: { + input: ['audio', 'text'], + output: ['audio', 'text'], + capabilities: ['reasoning', 'tool_calling'], + tools: [] as const, + }, +} as const satisfies ModelMeta + +// Rolling alias used by xAI's realtime docs examples; always points at the +// latest speech-to-speech model. +const GROK_VOICE_LATEST = { + name: 'grok-voice-latest', + supports: { + input: ['audio', 'text'], + output: ['audio', 'text'], + capabilities: ['reasoning', 'tool_calling'], + tools: [] as const, + }, +} as const satisfies ModelMeta + export const GROK_TTS_MODELS = [GROK_TTS.name] as const export const GROK_TRANSCRIPTION_MODELS = [GROK_STT.name] as const export const GROK_REALTIME_MODELS = [ + GROK_VOICE_THINK_FAST_2.name, + GROK_VOICE_LATEST.name, GROK_VOICE_FAST_1.name, GROK_VOICE_THINK_FAST_1.name, ] as const diff --git a/packages/ai-grok/src/realtime/adapter.ts b/packages/ai-grok/src/realtime/adapter.ts index 59831fa44b..ae52b98c9f 100644 --- a/packages/ai-grok/src/realtime/adapter.ts +++ b/packages/ai-grok/src/realtime/adapter.ts @@ -89,7 +89,7 @@ export function grokRealtime( token: RealtimeToken, _clientTools?: ReadonlyArray, ): Promise { - const model = token.config.model ?? 'grok-voice-fast-1.0' + const model = token.config.model ?? 'grok-voice-think-fast-2.0' logger.request(`activity=realtime provider=grok model=${model}`, { provider: 'grok', model, @@ -115,7 +115,7 @@ async function createWebRTCConnection( token: RealtimeToken, logger: InternalLogger, ): Promise { - const model = token.config.model ?? 'grok-voice-fast-1.0' + const model = token.config.model ?? 'grok-voice-think-fast-2.0' const eventHandlers = new Map>>() const pc = new RTCPeerConnection() diff --git a/packages/ai-grok/src/realtime/token.ts b/packages/ai-grok/src/realtime/token.ts index ad0962e26e..294ffc6608 100644 --- a/packages/ai-grok/src/realtime/token.ts +++ b/packages/ai-grok/src/realtime/token.ts @@ -27,7 +27,7 @@ const DEFAULT_TOKEN_FETCH_TIMEOUT_MS = 15_000 * import { grokRealtimeToken } from '@tanstack/ai-grok' * * const token = await realtimeToken({ - * adapter: grokRealtimeToken({ model: 'grok-voice-fast-1.0' }), + * adapter: grokRealtimeToken({ model: 'grok-voice-think-fast-2.0' }), * }) * ``` */ @@ -41,7 +41,8 @@ export function grokRealtimeToken( provider: 'grok', async generateToken(): Promise { - const model: GrokRealtimeModel = options.model ?? 'grok-voice-fast-1.0' + const model: GrokRealtimeModel = + options.model ?? 'grok-voice-think-fast-2.0' logger.request(`activity=realtimeToken provider=grok model=${model}`, { provider: 'grok', diff --git a/packages/ai-grok/src/realtime/types.ts b/packages/ai-grok/src/realtime/types.ts index 7c8f1143c2..c8a01d46a3 100644 --- a/packages/ai-grok/src/realtime/types.ts +++ b/packages/ai-grok/src/realtime/types.ts @@ -35,7 +35,7 @@ export type GrokTurnDetection = * Options for the Grok realtime token adapter. */ export interface GrokRealtimeTokenOptions { - /** Model to use (default: 'grok-voice-fast-1.0'). */ + /** Model to use (default: 'grok-voice-think-fast-2.0'). */ model?: GrokRealtimeModel /** * Enable debug logging for token creation. diff --git a/packages/ai-grok/src/video/video-provider-options.ts b/packages/ai-grok/src/video/video-provider-options.ts index b84c03f8b5..6ee1179af7 100644 --- a/packages/ai-grok/src/video/video-provider-options.ts +++ b/packages/ai-grok/src/video/video-provider-options.ts @@ -162,13 +162,34 @@ export function getGrokVideoDurationOptions( } /** - * Provider-specific options for grok video generation. These map directly + * Request mode for a source-video job. `'edit'` posts to `/v1/videos/edits` + * (modify the source clip in place — duration / aspect ratio / resolution + * are inherited from the input, capped at 720p); `'extend'` posts to + * `/v1/videos/extensions` (continue the source clip — `duration` is the + * length of the **added tail**, not the total). Both require exactly one + * video prompt part carrying the source clip. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoMode = 'edit' | 'extend' + +/** + * Provider-specific options for grok video generation. Apart from `mode` + * (a routing hint stripped before the request is sent), these map directly * onto the Imagine API request body and take precedence over the generic * `size` / `duration` options when both are provided. * * @experimental Video generation is an experimental feature and may change. */ export interface GrokVideoProviderOptions { + /** + * Selects the request mode for a source-video prompt part: `'edit'` + * (`/v1/videos/edits`) or `'extend'` (`/v1/videos/extensions`). Required + * when the prompt carries a video part; not valid without one. Omit for + * plain generation (`/v1/videos/generations`). + */ + mode?: GrokVideoMode + /** * Output aspect ratio. */ @@ -180,11 +201,35 @@ export interface GrokVideoProviderOptions { resolution?: GrokVideoResolution /** - * Video duration in integer seconds (1–15). + * Video duration in integer seconds (1–15). In `'extend'` mode this is + * the length of the added tail only, not the total output length. */ duration?: number + + /** + * Reference images for reference-to-video generation + * (grok-imagine-video-1.5, output capped at 720p). Usually populated from + * image prompt parts with `metadata.role: 'reference'`; set explicitly to + * override. Reference images are addressed from the prompt text as + * ``, ``, … in request order, and do not lock the first + * frame. + */ + reference_images?: Array<{ url: string }> + + /** + * Preset TTS voices to reference for generated speech + * (grok-imagine-video-1.5, max 3). Voice ids come from the xAI TTS voice + * roster (e.g. 'eve', 'rex') or a custom voice id, and are addressed from + * the prompt text as ``, ``, ``. + */ + reference_audios?: Array<{ voice_id: string }> } +/** + * Maximum reference voices accepted by the Imagine video endpoint. + */ +export const GROK_VIDEO_MAX_REFERENCE_AUDIOS = 3 + /** * Type-only map from model name to its specific provider options. * @@ -207,35 +252,15 @@ export type GrokVideoModelSizeByName = { /** * Type-only map from model name to the non-text prompt modalities it accepts. - * Both models accept an `image` prompt part as the starting frame: - * `grok-imagine-video` (v1.0) does text-to-video and image-to-video, while - * `grok-imagine-video-1.5` is image-to-video only (the image is required). + * Both models support text-to-video and accept an optional `image` prompt + * part as the starting frame; image parts with `metadata.role: 'reference'` + * become `reference_images` (grok-imagine-video-1.5). A `video` prompt part + * carries the source clip for edit / extension mode + * (`modelOptions.mode: 'edit' | 'extend'`). * * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoModelInputModalitiesByName = { - 'grok-imagine-video': readonly ['image'] - 'grok-imagine-video-1.5': readonly ['image'] -} - -/** - * Models that only support image-to-video — a starting-frame image is - * required and text-to-video is rejected by the Imagine API. Used by the - * adapter to fail fast with a clear message instead of surfacing the raw - * "Text-to-video is not supported for this model" 400. - * - * @experimental Video generation is an experimental feature and may change. - */ -const GROK_VIDEO_IMAGE_TO_VIDEO_ONLY: ReadonlySet = new Set([ - 'grok-imagine-video-1.5', -]) - -/** - * True when the model only supports image-to-video (a starting frame is - * required). - * - * @experimental Video generation is an experimental feature and may change. - */ -export function isImageToVideoOnlyModel(model: string): boolean { - return GROK_VIDEO_IMAGE_TO_VIDEO_ONLY.has(model) + 'grok-imagine-video': readonly ['image', 'video'] + 'grok-imagine-video-1.5': readonly ['image', 'video'] } diff --git a/packages/ai-grok/tests/grok-adapter.test.ts b/packages/ai-grok/tests/grok-adapter.test.ts index 408db3d5a9..ca04fef1b8 100644 --- a/packages/ai-grok/tests/grok-adapter.test.ts +++ b/packages/ai-grok/tests/grok-adapter.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { resolveDebugOption } from '@tanstack/ai/adapter-internals' -import { EventType, summarize } from '@tanstack/ai' +import { EventType, generateImage, summarize } from '@tanstack/ai' import { createGrokText, grokText } from '../src/adapters/text' import { createGrokImage, grokImage } from '../src/adapters/image' import { createGrokSummarize, grokSummarize } from '../src/adapters/summarize' @@ -475,6 +475,31 @@ describe('Grok adapters', () => { ) expect(mockGenerate.mock.calls[0]![0]).not.toHaveProperty('size') }) + + it('passes the 2.0-only quality option through for grok-imagine-image-2.0', async () => { + const adapter = createGrokImage('grok-imagine-image-2.0', 'test-api-key') + const mockGenerate = vi.fn().mockResolvedValue({ + data: [{ url: 'https://example.com/out.png' }], + }) + ;(adapter as any).client = { images: { generate: mockGenerate } } + + // Via the public generateImage() entry point so the per-model provider + // options map is exercised: `quality` only type-checks on the 2.0 model. + await generateImage({ + adapter, + prompt: 'A skyline', + size: '16:9', + modelOptions: { quality: 'low' }, + }) + + expect(mockGenerate).toHaveBeenCalledWith( + expect.objectContaining({ + model: 'grok-imagine-image-2.0', + aspect_ratio: '16:9', + quality: 'low', + }), + ) + }) }) describe('Summarize adapter', () => { diff --git a/packages/ai-grok/tests/video-adapter.test.ts b/packages/ai-grok/tests/video-adapter.test.ts index a6239adbc9..1dffe589ff 100644 --- a/packages/ai-grok/tests/video-adapter.test.ts +++ b/packages/ai-grok/tests/video-adapter.test.ts @@ -46,9 +46,8 @@ function adapterWithFetch( } /** - * grok-imagine-video-1.5 is image-to-video only, so every request needs a - * starting-frame image part. This builds a text + image prompt for the - * request-shape / status / error tests. + * Builds a text + starting-frame-image prompt for the request-shape / + * status / error tests. */ function i2vPrompt(text = 'p') { return [ @@ -221,7 +220,7 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) - it('rejects video and audio prompt parts before calling the API', async () => { + it('rejects audio prompt parts before calling the API', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -231,28 +230,34 @@ describe('Grok Video Adapter', () => { prompt: [ { type: 'text', content: 'p' }, { - type: 'video', - source: { type: 'url', value: 'https://example.com/clip.mp4' }, + type: 'audio', + source: { type: 'url', value: 'https://example.com/voice.mp3' }, }, ], logger: testLogger, }), - ).rejects.toThrow(/does not support video prompt parts/) + ).rejects.toThrow(/does not support audio prompt parts/) expect(fetchMock).not.toHaveBeenCalled() }) - it('rejects a text-only prompt on 1.5 — image-to-video only', async () => { - const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + it('allows a text-only prompt on 1.5 (text-to-video)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'tv-15' })) const adapter = adapterWithFetch(fetchMock) - await expect( - adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', - prompt: 'a red ball bouncing once', - logger: testLogger, - }), - ).rejects.toThrow(/does not support text-to-video/) - expect(fetchMock).not.toHaveBeenCalled() + const result = await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'a red ball bouncing once', + size: '16:9_1080p', + logger: testLogger, + }) + + expect(result).toEqual({ + jobId: 'tv-15', + model: 'grok-imagine-video-1.5', + }) + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.prompt).toBe('a red ball bouncing once') + expect(body).not.toHaveProperty('image') }) it('allows a text-only prompt on grok-imagine-video (text-to-video)', async () => { @@ -426,6 +431,267 @@ describe('Grok Video Adapter', () => { }) }) + describe('reference-to-video', () => { + it('maps role:reference image parts to reference_images', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-1' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' walks past ' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + { + type: 'image', + source: { type: 'data', mimeType: 'image/png', value: 'BBBB' }, + metadata: { role: 'character' }, + }, + ], + size: '16:9_720p', + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_images).toEqual([ + { url: 'https://example.com/person.png' }, + { url: 'data:image/png;base64,BBBB' }, + ]) + expect(body).not.toHaveProperty('image') + }) + + it('passes reference_audios voice ids through', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-2' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: ' narrates a sunrise', + modelOptions: { reference_audios: [{ voice_id: 'eve' }] }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_audios).toEqual([{ voice_id: 'eve' }]) + }) + + it('rejects more than 3 reference voices before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { + reference_audios: [ + { voice_id: 'eve' }, + { voice_id: 'ara' }, + { voice_id: 'rex' }, + { voice_id: 'sal' }, + ], + }, + logger: testLogger, + }), + ).rejects.toThrow(/at most 3 reference voices/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects mask / control / end_frame image roles', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + for (const role of ['mask', 'control', 'end_frame'] as const) { + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + metadata: { role }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(new RegExp(`has no '${role}' image`)) + } + expect(fetchMock).not.toHaveBeenCalled() + }) + }) + + describe('edit and extend modes', () => { + const sourceVideoPart = { + type: 'video' as const, + source: { type: 'url' as const, value: 'https://example.com/clip.mp4' }, + } + + it("posts to /videos/edits in 'edit' mode with the source video", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'edit-1' })) + const adapter = adapterWithFetch(fetchMock) + + const result = await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'make the sky stormy' }, + sourceVideoPart, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }) + + expect(result).toEqual({ + jobId: 'edit-1', + model: 'grok-imagine-video-1.5', + }) + const [url, init] = fetchMock.mock.calls[0]! + expect(url).toBe('https://api.x.ai/v1/videos/edits') + expect(JSON.parse(String(init?.body))).toEqual({ + model: 'grok-imagine-video-1.5', + prompt: 'make the sky stormy', + video: { url: 'https://example.com/clip.mp4' }, + }) + }) + + it("posts to /videos/extensions in 'extend' mode with the added-tail duration", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-1' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'the camera keeps panning right' }, + sourceVideoPart, + ], + duration: 5, + modelOptions: { mode: 'extend' }, + logger: testLogger, + }) + + const [url, init] = fetchMock.mock.calls[0]! + expect(url).toBe('https://api.x.ai/v1/videos/extensions') + expect(JSON.parse(String(init?.body))).toEqual({ + model: 'grok-imagine-video-1.5', + prompt: 'the camera keeps panning right', + video: { url: 'https://example.com/clip.mp4' }, + duration: 5, + }) + }) + + it('sends a base64 data source as a data URI source video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'video', + source: { type: 'data', mimeType: 'video/mp4', value: 'CCCC' }, + }, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.video).toEqual({ url: 'data:video/mp4;base64,CCCC' }) + }) + + it('rejects a video prompt part without an explicit mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + logger: testLogger, + }), + ).rejects.toThrow(/needs modelOptions\.mode/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects a mode without a source video part', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { mode: 'extend' }, + logger: testLogger, + }), + ).rejects.toThrow(/requires a video prompt part/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects image prompt parts in edit / extend mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + sourceVideoPart, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + }, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/takes only the source video/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects reference inputs in edit / extend mode', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { + mode: 'edit', + reference_audios: [{ voice_id: 'eve' }], + }, + logger: testLogger, + }), + ).rejects.toThrow(/reference inputs are only supported/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects more than one source video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + sourceVideoPart, + sourceVideoPart, + ], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/at most one source video/) + expect(fetchMock).not.toHaveBeenCalled() + }) + }) + describe('getVideoStatus', () => { it('maps a pending job with progress', async () => { const fetchMock = mockFetch(() => diff --git a/packages/ai/skills/ai-core/media-generation/SKILL.md b/packages/ai/skills/ai-core/media-generation/SKILL.md index fdf185a615..2e6ac5eb54 100644 --- a/packages/ai/skills/ai-core/media-generation/SKILL.md +++ b/packages/ai/skills/ai-core/media-generation/SKILL.md @@ -300,7 +300,7 @@ with `allowUrlFetch: true` on the adapter config | OpenAI | gpt-image-2 / gpt-image-1 / -mini → `images.edit()` (up to 16). dall-e-2 → edit (1). dall-e-3 throws. | Sora-2 / -pro → `input_reference` (single). Throws if >1. | | Gemini | Native (gemini-\*-flash-image, "nano-banana") → multimodal `contents`. Imagen throws. | Veo → first un-roled / `'start_frame'` image is the input image; `'end_frame'` → `lastFrame`; `'reference'` / `'character'` → `referenceImages`. Omni Flash sends image/video parts as interaction content blocks (no role routing). | | fal | Per-endpoint field names from a generated map (`pnpm generate:fal-image-fields`). Defaults: 1 input → `image_url`; >1 → `image_urls`; roles → `mask_url` / `control_image_url` / `reference_image_urls`. | Per-endpoint map (e.g. Kling i2v start frame → `image_url`). Defaults: 1 input → `image_url`; `start_frame`/`end_frame` → `start_image_url`/`end_image_url`; `reference` → `reference_image_urls`. | -| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | n/a | +| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | Un-roled / `'start_frame'` image → starting frame; `'reference'` / `'character'` → `reference_images` (1.5). A `video` part + `modelOptions.mode: 'edit' \| 'extend'` routes to `/videos/edits` / `/videos/extensions`. | | OpenRouter | Prompt parts map 1:1 onto multimodal `text` / `image_url` content parts, preserving interleaved order. | Dedicated async API (`openRouterVideo`): `start_frame`/`end_frame` → `frame_images[]` (`first_frame`/`last_frame`); `reference`/`character` → `input_references[]`; an unroled image defaults to the start frame. Frame roles validated against the model's `supported_frame_images` metadata. | | Anthropic | n/a (no image generation API). | n/a | @@ -544,8 +544,10 @@ const edited = await generateVideo({ Other video adapters: `openaiVideo('sora-2')` (pixel sizes like `'1280x720'`, durations 4/8/12s, single `input_reference` image prompt part), `grokVideo(...)` -(`grok-imagine-video` does text-to-video + image-to-video; `grok-imagine-video-1.5` is -image-to-video only — needs an `image` prompt part as the starting frame, text-only throws; +(`grok-imagine-video` and `grok-imagine-video-1.5` both do text-to-video + image-to-video; +1.5 adds reference-to-video — `'reference'`-roled image parts → `reference_images`, preset +voices via `modelOptions.reference_audios` (max 3) — and both edit/extend a source `video` +prompt part via `modelOptions.mode: 'edit' | 'extend'` (extend `duration` = added tail); aspect-ratio size template like `'16:9_720p'`, integer durations 1-15s, reports `usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — aspect-ratio size template like `'16:9_720p'`, durations 4-15s on the 2.0 family, From d448898ab2646ee9c5b686de586b8604aa635145 Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:46:02 +1000 Subject: [PATCH 2/5] fix(ai-grok): harden video mode routing and per-model options after review Address /review-pr findings on #1136: - Reject unknown modelOptions.mode values instead of silently misrouting a source-video body to /videos/generations (wrong endpoint, wrong billing). - Stop leaking raw duration/aspect_ratio/resolution from modelOptions into edit/extend bodies: edit rejects any duration (output inherits the source clip's length) and both modes reject size/aspect_ratio/resolution, matching the documented inherit-from-source semantics instead of sending fields the API ignores. Extend still snaps its added-tail duration. - Hoist mode-exclusivity checks into a dedicated edit/extend path so error messages never give generation-path advice for mode requests. - Throw on unknown image metadata.role strings instead of silently dropping the part (roles arrive untrusted from JSON callers). - Gate reference inputs to grok-imagine-video-1.5 at runtime, and split the provider options per model (GrokVideoBaseProviderOptions vs the 1.5 shape) so v1.0 no longer advertises reference fields at compile time. - Extract GROK_DEFAULT_REALTIME_MODEL so the token issuer and client adapter can't drift apart on future default bumps. - Comment/doc accuracy fixes: 'character' role alias documented, stale xAI URL and file headers refreshed, create-response JSDoc covers all three endpoints, deprecated 1.0 voice ids named in docs, example now requests the native 1080p it advertises. - 10 new tests: unknown mode/role rejection, edit duration/size rejection, extend snapping (modelOptions + generic), combined start-frame+reference body, explicit reference_images override, v1.0 reference gating, 1080p T2V body assertion, realtime token default model. --- .changeset/grok-imagine-catch-up.md | 2 +- docs/adapters/grok.md | 8 +- docs/media/video-generation.md | 4 +- .../src/lib/server-functions.ts | 2 +- packages/ai-grok/src/adapters/video.ts | 268 +++++++++++++----- .../src/image/image-provider-options.ts | 5 +- packages/ai-grok/src/index.ts | 2 + packages/ai-grok/src/model-meta.ts | 14 +- packages/ai-grok/src/realtime/adapter.ts | 5 +- packages/ai-grok/src/realtime/token.ts | 3 +- .../src/video/video-provider-options.ts | 90 ++++-- packages/ai-grok/tests/realtime-token.test.ts | 13 + packages/ai-grok/tests/video-adapter.test.ts | 224 ++++++++++++++- 13 files changed, 521 insertions(+), 119 deletions(-) diff --git a/.changeset/grok-imagine-catch-up.md b/.changeset/grok-imagine-catch-up.md index 7c0c014200..d9b58c28a3 100644 --- a/.changeset/grok-imagine-catch-up.md +++ b/.changeset/grok-imagine-catch-up.md @@ -5,5 +5,5 @@ Catch up with the current xAI Imagine / Voice catalog: - **Image**: add `grok-imagine-image-2.0` (xAI's recommended model, $0.04/image) with its 2.0-only `quality: 'low' | 'medium'` provider option. -- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3). Video editing and extension land via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). +- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference' | 'character'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3) — 1.5-only, typed per model and gated at runtime. Video editing and extension land via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). Because edit/extend outputs inherit the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` (and `duration` in edit mode) in those modes instead of sending fields the API ignores. - **Voice**: add `grok-voice-think-fast-2.0` (current recommended) and the `grok-voice-latest` alias to the realtime models; the realtime token and adapter defaults move off the deprecated 1.0 ids to `grok-voice-think-fast-2.0`. diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 00040af61c..6122b258b4 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -297,7 +297,7 @@ Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` opt ### Reference-to-Video -On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p: +On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` (or `'character'`) become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p. Reference inputs are a 1.5-only feature — the adapter rejects them on `grok-imagine-video`: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -327,8 +327,8 @@ const { jobId } = await generateVideo({ Both models can rewrite or continue an existing clip. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`: -- `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p). -- `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. +- `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p), so the adapter rejects `size`, `aspect_ratio`, `resolution`, and `duration` in this mode rather than sending fields the API ignores. +- `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. Output geometry is still inherited from the source, so `size` / `aspect_ratio` / `resolution` are rejected here too. ```typescript import { generateVideo } from "@tanstack/ai"; @@ -407,7 +407,7 @@ console.log(result.text); ## Realtime Voice -Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. The default model is `grok-voice-think-fast-2.0` (xAI's current recommended speech-to-speech model); `grok-voice-latest` always points at the newest model. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. +Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. The default model is `grok-voice-think-fast-2.0` (xAI's current recommended speech-to-speech model); `grok-voice-latest` always points at the newest model. The 1.0 ids remain accepted for compatibility, but xAI has deprecated `grok-voice-think-fast-1.0`. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow. ## Environment Variables diff --git a/docs/media/video-generation.md b/docs/media/video-generation.md index e446826ec6..a555bfe60b 100644 --- a/docs/media/video-generation.md +++ b/docs/media/video-generation.md @@ -733,7 +733,7 @@ const { jobId } = await generateVideo({ }); ``` -Reference-to-video (`grok-imagine-video-1.5`, capped at 720p) — image prompt parts with `metadata.role: 'reference'` become `reference_images` (addressed from the prompt as ``, ``, …), and up to 3 preset TTS voices can be referenced via `modelOptions.reference_audios` (addressed as ``, …): +Reference-to-video (`grok-imagine-video-1.5` only, output capped at 720p) — image prompt parts with `metadata.role: 'reference'` or `'character'` become `reference_images` (addressed from the prompt as ``, ``, …), and up to 3 preset TTS voices can be referenced via `modelOptions.reference_audios` (addressed as ``, …): ```typescript import { generateVideo } from "@tanstack/ai"; @@ -754,7 +754,7 @@ const { jobId } = await generateVideo({ }); ``` -Video editing and extension — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total: +Video editing and extension — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total. Because the output inherits the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` in both modes (and `duration` in edit mode) instead of sending fields the API ignores: ```typescript import { generateVideo } from "@tanstack/ai"; diff --git a/examples/ts-react-media/src/lib/server-functions.ts b/examples/ts-react-media/src/lib/server-functions.ts index 74df551dfe..7b08ce8646 100644 --- a/examples/ts-react-media/src/lib/server-functions.ts +++ b/examples/ts-react-media/src/lib/server-functions.ts @@ -358,7 +358,7 @@ function videoStreamForModel(data: VideoRequest): AsyncIterable { pollingInterval: VIDEO_POLL_INTERVAL_MS, adapter: grokVideo('grok-imagine-video-1.5'), prompt: asTextPrompt(data.prompt), - size: '16:9_720p', + size: '16:9_1080p', duration: 5, }) } diff --git a/packages/ai-grok/src/adapters/video.ts b/packages/ai-grok/src/adapters/video.ts index 5c1d95df64..6b193dcbad 100644 --- a/packages/ai-grok/src/adapters/video.ts +++ b/packages/ai-grok/src/adapters/video.ts @@ -5,6 +5,7 @@ import { getGrokApiKeyFromEnv, withGrokDefaults } from '../utils/client' import { GROK_VIDEO_MAX_REFERENCE_AUDIOS, getGrokVideoDurationOptions, + isGrokVideoReferenceModel, parseGrokVideoSize, validateVideoSize, } from '../video/video-provider-options' @@ -42,7 +43,7 @@ export interface GrokVideoConfig extends GrokClientConfig {} */ const USD_TICKS_PER_DOLLAR = 10_000_000_000 -/** Response of POST /v1/videos/generations. */ +/** Response of the POST /v1/videos/{generations,edits,extensions} endpoints. */ interface GrokVideoCreateResponse { request_id?: string } @@ -112,8 +113,9 @@ function buildGrokVideoUsage( * (e.g. '16:9_720p'), consistent with the grok-imagine image models * - Image-to-video via an `image` prompt part (starting frame URL or data URI) * - Reference-to-video via image prompt parts with - * `metadata.role: 'reference'` (→ `reference_images`) and preset voices - * via `modelOptions.reference_audios` (grok-imagine-video-1.5) + * `metadata.role: 'reference'` or `'character'` (→ `reference_images`) + * and preset voices via `modelOptions.reference_audios` + * (grok-imagine-video-1.5 only) * - Video editing / extension via a source `video` prompt part and * `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / * `/v1/videos/extensions`; in extend mode `duration` is the added tail) @@ -123,7 +125,7 @@ export class GrokVideoAdapter< TModel extends GrokVideoModel, > extends BaseVideoAdapter< TModel, - GrokVideoProviderOptions, + GrokVideoModelProviderOptionsByName[TModel], GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, GrokVideoModelInputModalitiesByName, @@ -182,27 +184,31 @@ export class GrokVideoAdapter< async createVideoJob( options: VideoGenerationOptions< - GrokVideoProviderOptions, + GrokVideoModelProviderOptionsByName[TModel], GrokVideoModelSizeByName[TModel], GrokVideoModelDurationByName[TModel] >, ): Promise { const { model, size, modelOptions, logger } = options - validateVideoSize(model, size) - // `mode` is a routing hint for this adapter, not an API field — strip it - // before the remaining options are spread onto the request body. - const { mode, ...wireOptions } = modelOptions ?? {} - - // Coerce the requested duration into the model's valid range (1–15s, - // integer) instead of rejecting it — `snapDuration` clamps and rounds. - // modelOptions wins over the generic `duration`, mirroring the size - // precedence below. In extend mode the same range applies to the added - // tail. - const rawDuration = wireOptions.duration ?? options.duration - const duration = - rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined + // before the remaining options are spread onto the request body. The + // per-model map narrows what callers can pass, but modelOptions often + // arrives as deserialized JSON, so the adapter handles the widest option + // surface (the 1.5 shape) uniformly and gates by model at runtime. + const { mode, ...wireOptions } = (modelOptions ?? + {}) as GrokVideoProviderOptions + + // `mode` is typed 'edit' | 'extend' but reaches us untrusted from JSON + // callers. An unrecognised value must not fall through to the + // generations endpoint with a source-video body — that would silently + // run (and bill) a generation the caller never asked for. + if (mode !== undefined && mode !== 'edit' && mode !== 'extend') { + throw new Error( + `${this.name}: unknown modelOptions.mode '${String(mode)}'. ` + + `Expected 'edit' or 'extend'.`, + ) + } // The interleaved prompt decomposes into verbatim text plus typed media // buckets. Reference audio is voice-id based (not an audio file), so @@ -238,10 +244,34 @@ export class GrokVideoAdapter< ) } + if (mode !== undefined && sourceVideo) { + return await this.createSourceVideoJob({ + model, + mode, + sourceVideo, + resolved, + wireOptions, + size, + genericDuration: options.duration, + logger, + }) + } + + validateVideoSize(model, size) + + // Coerce the requested duration into the model's valid range (1–15s, + // integer) instead of rejecting it — `snapDuration` clamps and rounds. + // modelOptions wins over the generic `duration`, mirroring the size + // precedence below. + const rawDuration = wireOptions.duration ?? options.duration + const duration = + rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined + // Image parts split by role: un-roled / 'start_frame' images become the // starting frame (image-to-video); 'reference' / 'character' images // become reference_images (reference-to-video). The Imagine API has no - // mask / control / end-frame inputs. + // mask / control / end-frame inputs. Unknown role strings (possible via + // JSON callers) throw rather than silently dropping the part. const startFrames: Array> = [] const referenceImages: Array<{ url: string }> = [] for (const part of resolved.images) { @@ -263,6 +293,11 @@ export class GrokVideoAdapter< case undefined: startFrames.push(part) break + default: + throw new Error( + `${this.name}: unknown image metadata.role '${String(role)}'. ` + + `Expected 'start_frame', 'reference', or 'character'.`, + ) } } if (startFrames.length > 1) { @@ -271,10 +306,18 @@ export class GrokVideoAdapter< `Use metadata.role: 'reference' for reference-to-video inputs.`, ) } - if (mode !== undefined && resolved.images.length > 0) { + // Reference inputs are a grok-imagine-video-1.5 feature. The per-model + // options map already hides the fields from other models at compile + // time; this runtime gate covers prompt-part roles and untyped callers. + if ( + !isGrokVideoReferenceModel(model) && + (referenceImages.length > 0 || + wireOptions.reference_images !== undefined || + wireOptions.reference_audios !== undefined) + ) { throw new Error( - `${this.name}: '${mode}' mode takes only the source video — image ` + - `prompt parts are not supported by ${mode === 'edit' ? '/videos/edits' : '/videos/extensions'}.`, + `${this.name}: ${model} does not support reference-to-video inputs. ` + + `Use 'grok-imagine-video-1.5' for reference_images / reference_audios.`, ) } const referenceAudioCount = wireOptions.reference_audios?.length ?? 0 @@ -283,16 +326,6 @@ export class GrokVideoAdapter< `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_AUDIOS} reference voices; received ${referenceAudioCount}.`, ) } - if ( - mode !== undefined && - (wireOptions.reference_images !== undefined || - wireOptions.reference_audios !== undefined) - ) { - throw new Error( - `${this.name}: reference inputs are only supported by video ` + - `generation, not '${mode}' mode.`, - ) - } // Image-to-video: the single image prompt part becomes the starting frame // and the prompt text describes the desired motion. URL sources are @@ -301,53 +334,134 @@ export class GrokVideoAdapter< // The generic `size` option carries an "aspectRatio_resolution" template // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / - // `resolution` parameters; explicit modelOptions win over the template. - // Edit / extend outputs inherit these from the source clip, so the - // template is only mapped for generation requests. + // `resolution` parameters; explicit modelOptions win over the template + // (including `reference_images`, which replaces the part-derived list). const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined - const request = - mode !== undefined && sourceVideo - ? { - model, - prompt: resolved.text, - video: { url: mediaPartToUrl(sourceVideo) }, - ...wireOptions, - // Edit outputs also inherit their length from the source clip; - // extend takes the snapped duration as the added-tail length. - ...(mode === 'extend' && duration !== undefined && { duration }), - } - : { - model, - prompt: resolved.text, - ...(startFrame && { image: { url: mediaPartToUrl(startFrame) } }), - ...(referenceImages.length > 0 && { - reference_images: referenceImages, - }), - ...(parsedSize && { - aspect_ratio: parsedSize.aspectRatio, - ...(parsedSize.resolution !== undefined && { - resolution: parsedSize.resolution, - }), - }), - ...wireOptions, - // Spread after wireOptions so the snapped duration is - // authoritative (modelOptions.duration is folded into `duration` - // via snapDuration above). - ...(duration !== undefined && { duration }), - } - - const endpoint = - mode === 'edit' - ? '/videos/edits' - : mode === 'extend' - ? '/videos/extensions' - : '/videos/generations' + const request = { + model, + prompt: resolved.text, + ...(startFrame && { image: { url: mediaPartToUrl(startFrame) } }), + ...(referenceImages.length > 0 && { + reference_images: referenceImages, + }), + ...(parsedSize && { + aspect_ratio: parsedSize.aspectRatio, + ...(parsedSize.resolution !== undefined && { + resolution: parsedSize.resolution, + }), + }), + ...wireOptions, + // Spread after wireOptions so the snapped duration is + // authoritative (modelOptions.duration is folded into `duration` + // via snapDuration above). + ...(duration !== undefined && { duration }), + } - try { - logger.request( - `activity=video.create provider=${this.name} model=${model} mode=${mode ?? 'generate'} size=${size ?? 'default'} duration=${duration ?? 'default'}`, - { provider: this.name, model }, + return await this.postVideoJob('/videos/generations', request, { + model, + logger, + logLine: `activity=video.create provider=${this.name} model=${model} mode=generate size=${size ?? 'default'} duration=${duration ?? 'default'}`, + }) + } + + /** + * Build and post an edit / extension request. Both endpoints take only + * `model`, `prompt`, and the source `video` (plus `duration` — the length + * of the added tail — for extensions): output geometry is inherited from + * the source clip, capped at 720p, and edit outputs also inherit the + * source length. Rather than sending fields the API documents as ignored, + * the inapplicable options are rejected with actionable errors. + */ + private async createSourceVideoJob(args: { + model: string + mode: 'edit' | 'extend' + sourceVideo: VideoPart + resolved: ReturnType + wireOptions: Omit + size: string | undefined + genericDuration: number | undefined + logger: VideoGenerationOptions['logger'] + }): Promise { + const { model, mode, sourceVideo, resolved, wireOptions, logger } = args + const endpoint = mode === 'edit' ? '/videos/edits' : '/videos/extensions' + + if (resolved.images.length > 0) { + throw new Error( + `${this.name}: '${mode}' mode takes only the source video — image ` + + `prompt parts are not supported by ${endpoint}.`, + ) + } + if ( + wireOptions.reference_images !== undefined || + wireOptions.reference_audios !== undefined + ) { + throw new Error( + `${this.name}: reference inputs are only supported by video ` + + `generation, not '${mode}' mode.`, ) + } + if ( + args.size !== undefined || + wireOptions.aspect_ratio !== undefined || + wireOptions.resolution !== undefined + ) { + throw new Error( + `${this.name}: '${mode}' mode does not accept size / aspect_ratio / ` + + `resolution — the output inherits the source clip's geometry ` + + `(capped at 720p).`, + ) + } + + const { + aspect_ratio: _aspectRatio, + resolution: _resolution, + duration: modeDuration, + ...passthrough + } = wireOptions + const rawDuration = modeDuration ?? args.genericDuration + if (mode === 'edit' && rawDuration !== undefined) { + throw new Error( + `${this.name}: 'edit' mode does not accept a duration — the output ` + + `inherits the source clip's length. Use mode 'extend' to append ` + + `seconds to the clip.`, + ) + } + // Extend: the snapped duration is the added-tail length (1–15s). + const duration = + rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined + + const request = { + model, + prompt: resolved.text, + video: { url: mediaPartToUrl(sourceVideo) }, + ...passthrough, + ...(duration !== undefined && { duration }), + } + + return await this.postVideoJob(endpoint, request, { + model, + logger, + logLine: `activity=video.create provider=${this.name} model=${model} mode=${mode} duration=${duration ?? 'default'}`, + }) + } + + /** + * POST a create-job request body to one of the Imagine video endpoints + * (`/videos/generations`, `/videos/edits`, `/videos/extensions`) and read + * the `request_id` out of the shared response shape. + */ + private async postVideoJob( + endpoint: string, + request: Record, + context: { + model: string + logger: VideoGenerationOptions['logger'] + logLine: string + }, + ): Promise { + const { model, logger, logLine } = context + try { + logger.request(logLine, { provider: this.name, model }) const response = await this.request(endpoint, { method: 'POST', @@ -492,7 +606,7 @@ export class GrokVideoAdapter< * * @experimental Video generation is an experimental feature and may change. * - * @param model - The model name (e.g., 'grok-imagine-video') + * @param model - The model name (e.g., 'grok-imagine-video-1.5') * @param apiKey - Your xAI API key * @param config - Optional additional configuration * @returns Configured Grok video adapter instance with resolved types diff --git a/packages/ai-grok/src/image/image-provider-options.ts b/packages/ai-grok/src/image/image-provider-options.ts index 1b4a40a8c1..19cffb4b79 100644 --- a/packages/ai-grok/src/image/image-provider-options.ts +++ b/packages/ai-grok/src/image/image-provider-options.ts @@ -1,8 +1,9 @@ /** * Grok Image Generation Provider Options * - * These are provider-specific options for Grok image generation. - * Grok uses the grok-2-image-1212 model for image generation. + * Provider-specific options for Grok image generation: the aspect-ratio + * sized Imagine API models (grok-imagine-image, grok-imagine-image-2.0, + * grok-imagine-image-quality) and the legacy pixel-sized grok-2-image-1212. */ /** diff --git a/packages/ai-grok/src/index.ts b/packages/ai-grok/src/index.ts index a5a5495fd6..6f245ceb7f 100644 --- a/packages/ai-grok/src/index.ts +++ b/packages/ai-grok/src/index.ts @@ -46,6 +46,7 @@ export { } from './video/video-provider-options' export type { GrokVideoMode, + GrokVideoBaseProviderOptions, GrokVideoProviderOptions, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, @@ -104,6 +105,7 @@ export { GROK_TTS_MODELS, GROK_TRANSCRIPTION_MODELS, GROK_REALTIME_MODELS, + GROK_DEFAULT_REALTIME_MODEL, } from './model-meta' export type { GrokTextMetadata, diff --git a/packages/ai-grok/src/model-meta.ts b/packages/ai-grok/src/model-meta.ts index c65efe76cf..fd5ff45f60 100644 --- a/packages/ai-grok/src/model-meta.ts +++ b/packages/ai-grok/src/model-meta.ts @@ -155,9 +155,9 @@ const GROK_IMAGINE_IMAGE_2_0 = { // Both models support text-to-video and image-to-video (a starting-frame // image is optional). grok-imagine-video-1.5 is the documented default: it // adds native 1080p text-to-video plus reference-to-video inputs -// (`reference_images` / `reference_audios`, capped at 720p). Both models -// also drive video editing (`/v1/videos/edits`) and extension -// (`/v1/videos/extensions`) via a source video prompt part. +// (`reference_images` / `reference_audios`; reference output is capped at +// 720p). Both models also drive video editing (`/v1/videos/edits`) and +// extension (`/v1/videos/extensions`) via a source video prompt part. const GROK_IMAGINE_VIDEO = { name: 'grok-imagine-video', supports: { @@ -339,6 +339,14 @@ export const GROK_REALTIME_MODELS = [ GROK_VOICE_THINK_FAST_1.name, ] as const +/** + * Default speech-to-speech model used by the realtime token issuer and the + * realtime client adapter when no model is specified. Single source of truth + * so a future default bump cannot leave the two sides disagreeing. + */ +export const GROK_DEFAULT_REALTIME_MODEL: GrokRealtimeModel = + 'grok-voice-think-fast-2.0' + export type GrokChatModel = (typeof GROK_CHAT_MODELS)[number] export type GrokImageModel = (typeof GROK_IMAGE_MODELS)[number] export type GrokVideoModel = (typeof GROK_VIDEO_MODELS)[number] diff --git a/packages/ai-grok/src/realtime/adapter.ts b/packages/ai-grok/src/realtime/adapter.ts index ae52b98c9f..a26456d6cc 100644 --- a/packages/ai-grok/src/realtime/adapter.ts +++ b/packages/ai-grok/src/realtime/adapter.ts @@ -1,4 +1,5 @@ import { resolveDebugOption } from '@tanstack/ai/adapter-internals' +import { GROK_DEFAULT_REALTIME_MODEL } from '../model-meta' import type { AnyClientTool, AudioVisualization, @@ -89,7 +90,7 @@ export function grokRealtime( token: RealtimeToken, _clientTools?: ReadonlyArray, ): Promise { - const model = token.config.model ?? 'grok-voice-think-fast-2.0' + const model = token.config.model ?? GROK_DEFAULT_REALTIME_MODEL logger.request(`activity=realtime provider=grok model=${model}`, { provider: 'grok', model, @@ -115,7 +116,7 @@ async function createWebRTCConnection( token: RealtimeToken, logger: InternalLogger, ): Promise { - const model = token.config.model ?? 'grok-voice-think-fast-2.0' + const model = token.config.model ?? GROK_DEFAULT_REALTIME_MODEL const eventHandlers = new Map>>() const pc = new RTCPeerConnection() diff --git a/packages/ai-grok/src/realtime/token.ts b/packages/ai-grok/src/realtime/token.ts index 294ffc6608..b574c50888 100644 --- a/packages/ai-grok/src/realtime/token.ts +++ b/packages/ai-grok/src/realtime/token.ts @@ -1,4 +1,5 @@ import { resolveDebugOption } from '@tanstack/ai/adapter-internals' +import { GROK_DEFAULT_REALTIME_MODEL } from '../model-meta' import { getGrokApiKeyFromEnv } from '../utils' import type { RealtimeToken, RealtimeTokenAdapter } from '@tanstack/ai' import type { GrokRealtimeModel } from '../model-meta' @@ -42,7 +43,7 @@ export function grokRealtimeToken( async generateToken(): Promise { const model: GrokRealtimeModel = - options.model ?? 'grok-voice-think-fast-2.0' + options.model ?? GROK_DEFAULT_REALTIME_MODEL logger.request(`activity=realtimeToken provider=grok model=${model}`, { provider: 'grok', diff --git a/packages/ai-grok/src/video/video-provider-options.ts b/packages/ai-grok/src/video/video-provider-options.ts index 6ee1179af7..4a8745577f 100644 --- a/packages/ai-grok/src/video/video-provider-options.ts +++ b/packages/ai-grok/src/video/video-provider-options.ts @@ -1,7 +1,9 @@ /** * Grok Video Generation Provider Options (xAI Imagine API) * - * Based on https://docs.x.ai/docs/guides/video-generations + * Based on https://docs.x.ai/developers/model-capabilities/video/generation + * (plus the image-to-video, reference-to-video, editing, and extension pages + * under the same section). * * @experimental Video generation is an experimental feature and may change. */ @@ -163,25 +165,29 @@ export function getGrokVideoDurationOptions( /** * Request mode for a source-video job. `'edit'` posts to `/v1/videos/edits` - * (modify the source clip in place — duration / aspect ratio / resolution - * are inherited from the input, capped at 720p); `'extend'` posts to + * (modify the source clip in place); `'extend'` posts to * `/v1/videos/extensions` (continue the source clip — `duration` is the * length of the **added tail**, not the total). Both require exactly one * video prompt part carrying the source clip. * + * Output geometry (aspect ratio / resolution) is inherited from the source + * clip in both modes, capped at 720p, and edit outputs also inherit the + * source length — the adapter rejects `size`, `aspect_ratio`, `resolution`, + * and (in edit mode) `duration` rather than sending fields the API ignores. + * * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoMode = 'edit' | 'extend' /** - * Provider-specific options for grok video generation. Apart from `mode` - * (a routing hint stripped before the request is sent), these map directly - * onto the Imagine API request body and take precedence over the generic - * `size` / `duration` options when both are provided. + * Provider options shared by both grok-imagine video models. Apart from + * `mode` (a routing hint stripped before the request is sent), these map + * directly onto the Imagine API request body and take precedence over the + * generic `size` / `duration` options when both are provided. * * @experimental Video generation is an experimental feature and may change. */ -export interface GrokVideoProviderOptions { +export interface GrokVideoBaseProviderOptions { /** * Selects the request mode for a source-video prompt part: `'edit'` * (`/v1/videos/edits`) or `'extend'` (`/v1/videos/extensions`). Required @@ -191,36 +197,47 @@ export interface GrokVideoProviderOptions { mode?: GrokVideoMode /** - * Output aspect ratio. + * Output aspect ratio. Generation only — edit / extend outputs inherit + * the source clip's geometry and the adapter rejects this in those modes. */ aspect_ratio?: GrokVideoAspectRatio /** - * Output resolution tier. + * Output resolution tier. Generation only — edit / extend outputs inherit + * the source clip's geometry and the adapter rejects this in those modes. */ resolution?: GrokVideoResolution /** * Video duration in integer seconds (1–15). In `'extend'` mode this is - * the length of the added tail only, not the total output length. + * the length of the added tail only, not the total output length. Not + * valid in `'edit'` mode (the output inherits the source clip's length). */ duration?: number +} +/** + * Provider options for grok-imagine-video-1.5, which adds the + * reference-to-video inputs on top of the shared options. + * + * @experimental Video generation is an experimental feature and may change. + */ +export interface GrokVideoProviderOptions extends GrokVideoBaseProviderOptions { /** - * Reference images for reference-to-video generation - * (grok-imagine-video-1.5, output capped at 720p). Usually populated from - * image prompt parts with `metadata.role: 'reference'`; set explicitly to - * override. Reference images are addressed from the prompt text as - * ``, ``, … in request order, and do not lock the first - * frame. + * Reference images for reference-to-video generation (output capped at + * 720p). Usually populated from image prompt parts with + * `metadata.role: 'reference'` (or `'character'`); set explicitly to + * replace the part-derived list. Reference images are addressed from the + * prompt text as ``, ``, … in request order, and do not + * lock the first frame. */ reference_images?: Array<{ url: string }> /** - * Preset TTS voices to reference for generated speech - * (grok-imagine-video-1.5, max 3). Voice ids come from the xAI TTS voice - * roster (e.g. 'eve', 'rex') or a custom voice id, and are addressed from - * the prompt text as ``, ``, ``. + * Preset TTS voices to reference for generated speech (max 3). Voice ids + * come from the xAI TTS voice roster (e.g. 'eve', 'rex') or a custom + * voice id, and are addressed from the prompt text as ``, + * ``, ``. */ reference_audios?: Array<{ voice_id: string }> } @@ -231,12 +248,35 @@ export interface GrokVideoProviderOptions { export const GROK_VIDEO_MAX_REFERENCE_AUDIOS = 3 /** - * Type-only map from model name to its specific provider options. + * Models that support reference-to-video inputs (`reference_images` / + * `reference_audios`). The per-model provider-options map hides the fields + * from other models at compile time; this backs the runtime gate for + * untyped callers (e.g. deserialized JSON) so they get a clear error + * instead of a raw API 400. + * + * @experimental Video generation is an experimental feature and may change. + */ +const GROK_VIDEO_REFERENCE_MODELS: ReadonlySet = new Set([ + 'grok-imagine-video-1.5', +]) + +/** + * True when the model accepts reference-to-video inputs. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoReferenceModel(model: string): boolean { + return GROK_VIDEO_REFERENCE_MODELS.has(model) +} + +/** + * Type-only map from model name to its specific provider options. Only + * grok-imagine-video-1.5 exposes the reference-to-video fields. * * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoModelProviderOptionsByName = { - 'grok-imagine-video': GrokVideoProviderOptions + 'grok-imagine-video': GrokVideoBaseProviderOptions 'grok-imagine-video-1.5': GrokVideoProviderOptions } @@ -254,8 +294,8 @@ export type GrokVideoModelSizeByName = { * Type-only map from model name to the non-text prompt modalities it accepts. * Both models support text-to-video and accept an optional `image` prompt * part as the starting frame; image parts with `metadata.role: 'reference'` - * become `reference_images` (grok-imagine-video-1.5). A `video` prompt part - * carries the source clip for edit / extension mode + * or `'character'` become `reference_images` (grok-imagine-video-1.5 only). + * A `video` prompt part carries the source clip for edit / extension mode * (`modelOptions.mode: 'edit' | 'extend'`). * * @experimental Video generation is an experimental feature and may change. diff --git a/packages/ai-grok/tests/realtime-token.test.ts b/packages/ai-grok/tests/realtime-token.test.ts index c91c17ce3b..4aa74fc3db 100644 --- a/packages/ai-grok/tests/realtime-token.test.ts +++ b/packages/ai-grok/tests/realtime-token.test.ts @@ -63,6 +63,19 @@ describe('grokRealtimeToken request body', () => { const body = JSON.parse(init.body as string) as Record expect(body).toEqual({ session: { model: 'grok-voice-think-fast-1.0' } }) }) + + it("defaults to xAI's current recommended model when no model is given", async () => { + const fetchMock = vi + .fn() + .mockResolvedValue(makeSessionResponse(1_700_000_000)) + globalThis.fetch = fetchMock as unknown as typeof fetch + + await realtimeToken({ adapter: grokRealtimeToken() }) + + const init = fetchMock.mock.calls[0]![1]! + const body = JSON.parse(init.body as string) as Record + expect(body).toEqual({ session: { model: 'grok-voice-think-fast-2.0' } }) + }) }) describe('grokRealtimeToken expires_at unit-safety', () => { diff --git a/packages/ai-grok/tests/video-adapter.test.ts b/packages/ai-grok/tests/video-adapter.test.ts index 1dffe589ff..caaa0f051e 100644 --- a/packages/ai-grok/tests/video-adapter.test.ts +++ b/packages/ai-grok/tests/video-adapter.test.ts @@ -240,7 +240,7 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) - it('allows a text-only prompt on 1.5 (text-to-video)', async () => { + it('allows a text-only prompt on 1.5 (text-to-video, native 1080p)', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'tv-15' })) const adapter = adapterWithFetch(fetchMock) @@ -257,6 +257,8 @@ describe('Grok Video Adapter', () => { }) const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) expect(body.prompt).toBe('a red ball bouncing once') + expect(body.aspect_ratio).toBe('16:9') + expect(body.resolution).toBe('1080p') expect(body).not.toHaveProperty('image') }) @@ -500,6 +502,121 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) + it('sends both a starting frame and reference images when combined', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-3' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' enters from the left' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/start.png' }, + metadata: { role: 'start_frame' }, + }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + ], + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.image).toEqual({ url: 'https://example.com/start.png' }) + expect(body.reference_images).toEqual([ + { url: 'https://example.com/person.png' }, + ]) + }) + + it('lets explicit modelOptions.reference_images replace the part-derived list', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-4' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/from-part.png' }, + metadata: { role: 'reference' }, + }, + ], + modelOptions: { + reference_images: [{ url: 'https://example.com/explicit.png' }], + }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.reference_images).toEqual([ + { url: 'https://example.com/explicit.png' }, + ]) + }) + + it('rejects reference inputs on grok-imagine-video (v1.0)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + metadata: { role: 'reference' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/does not support reference-to-video inputs/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + modelOptions: { + // The per-model options map hides the reference fields from + // v1.0 at compile time; the runtime gate covers JSON callers. + // @ts-expect-error reference_audios is 1.5-only + reference_audios: [{ voice_id: 'eve' }], + }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support reference-to-video inputs/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects an unknown image metadata.role before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/a.png' }, + // Roles arrive untrusted from JSON callers; an unrecognised + // value must throw, not silently drop the part. + // @ts-expect-error runtime validation of a non-MediaInputRole string + metadata: { role: 'first_frame' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/unknown image metadata\.role 'first_frame'/) + expect(fetchMock).not.toHaveBeenCalled() + }) + it('rejects mask / control / end_frame image roles', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -603,6 +720,111 @@ describe('Grok Video Adapter', () => { expect(body.video).toEqual({ url: 'data:video/mp4;base64,CCCC' }) }) + it('snaps the extend duration passed via modelOptions', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-2' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], + modelOptions: { mode: 'extend', duration: 2.6 }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.duration).toBe(3) + }) + + it('snaps an out-of-range generic extend duration', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-3' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], + duration: 20, + modelOptions: { mode: 'extend' }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body.duration).toBe(15) + }) + + it("rejects a duration in 'edit' mode — inherited from the source", async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + duration: 5, + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/'edit' mode does not accept a duration/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit', duration: 99 }, + logger: testLogger, + }), + ).rejects.toThrow(/'edit' mode does not accept a duration/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects size / aspect_ratio / resolution in edit and extend modes', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + size: '16:9_1080p', + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'extend', aspect_ratio: '9:16' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit', resolution: '1080p' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects an unknown mode instead of misrouting to generations', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + // Mode arrives untrusted from JSON callers; a typo must not fall + // through to /videos/generations with a source-video body. + // @ts-expect-error runtime validation of a non-GrokVideoMode string + modelOptions: { mode: 'remix' }, + logger: testLogger, + }), + ).rejects.toThrow(/unknown modelOptions\.mode 'remix'/) + expect(fetchMock).not.toHaveBeenCalled() + }) + it('rejects a video prompt part without an explicit mode', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) From bd96ec58b02d01ff3e774ba2169a01aa52cf87d1 Mon Sep 17 00:00:00 2001 From: Alem Tuzlak Date: Tue, 18 Aug 2026 10:58:22 +0200 Subject: [PATCH 3/5] fix(ai-grok): restrict edit/extend to grok-imagine-video and reject mixed modes --- .changeset/grok-imagine-catch-up.md | 2 +- docs/adapters/grok.md | 12 +- docs/media/video-generation.md | 8 +- packages/ai-grok/src/adapters/video.ts | 66 ++++- packages/ai-grok/src/index.ts | 2 + packages/ai-grok/src/model-meta.ts | 11 +- .../src/video/video-provider-options.ts | 111 ++++++-- packages/ai-grok/tests/realtime-token.test.ts | 14 +- packages/ai-grok/tests/video-adapter.test.ts | 237 +++++++++++++----- .../skills/ai-core/media-generation/SKILL.md | 23 +- 10 files changed, 368 insertions(+), 118 deletions(-) diff --git a/.changeset/grok-imagine-catch-up.md b/.changeset/grok-imagine-catch-up.md index d9b58c28a3..b54bdbac65 100644 --- a/.changeset/grok-imagine-catch-up.md +++ b/.changeset/grok-imagine-catch-up.md @@ -5,5 +5,5 @@ Catch up with the current xAI Imagine / Voice catalog: - **Image**: add `grok-imagine-image-2.0` (xAI's recommended model, $0.04/image) with its 2.0-only `quality: 'low' | 'medium'` provider option. -- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference' | 'character'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3) — 1.5-only, typed per model and gated at runtime. Video editing and extension land via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). Because edit/extend outputs inherit the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` (and `duration` in edit mode) in those modes instead of sending fields the API ignores. +- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference' | 'character'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3) — 1.5-only, typed per model and gated at runtime. Image-to-video and reference-to-video cannot be combined. Video editing and extension land on `grok-imagine-video` only, via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). Because edit/extend outputs inherit the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` (and `duration` in edit mode) in those modes instead of sending fields the API ignores. - **Voice**: add `grok-voice-think-fast-2.0` (current recommended) and the `grok-voice-latest` alias to the realtime models; the realtime token and adapter defaults move off the deprecated 1.0 ids to `grok-voice-think-fast-2.0`. diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 6122b258b4..c95c1a9f60 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -241,8 +241,8 @@ Generate short video clips (1–15 seconds, with audio) with the Grok Imagine vi Available models: -- `grok-imagine-video` (v1.0) — text-to-video and image-to-video, $0.05 per second of video. -- `grok-imagine-video-1.5` — xAI's recommended default, $0.08 per second of video. Supports text-to-video (with native 1080p), image-to-video, and reference-to-video. +- `grok-imagine-video` (v1.0) — text-to-video, image-to-video, and source-video edit / extend, $0.05 per second of video. +- `grok-imagine-video-1.5` — xAI's recommended default, $0.08 per second of video. Supports text-to-video (with native 1080p), image-to-video, and reference-to-video. It does not accept a source video. Text-to-video: @@ -293,11 +293,11 @@ const { jobId } = await generateVideo({ }); ``` -Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. +Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and (on `grok-imagine-video-1.5` text-to-video / image-to-video only) `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional. ### Reference-to-Video -On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` (or `'character'`) become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p. Reference inputs are a 1.5-only feature — the adapter rejects them on `grok-imagine-video`: +On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` (or `'character'`) become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as ``, ``, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as ``, ``, ``. Reference-to-video output is capped at 720p. A starting-frame image and reference inputs cannot be combined — xAI rejects that mix with 400. Reference inputs are a 1.5-only feature — the adapter rejects them on `grok-imagine-video`: ```typescript import { generateVideo } from "@tanstack/ai"; @@ -325,7 +325,7 @@ const { jobId } = await generateVideo({ ### Video Editing and Extension -Both models can rewrite or continue an existing clip. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`: +`grok-imagine-video` (v1.0) can rewrite or continue an existing clip. `grok-imagine-video-1.5` has no video input — the adapter rejects a source-video part or `mode` on that model. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`: - `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p), so the adapter rejects `size`, `aspect_ratio`, `resolution`, and `duration` in this mode rather than sending fields the API ignores. - `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. Output geometry is still inherited from the source, so `size` / `aspect_ratio` / `resolution` are rejected here too. @@ -334,7 +334,7 @@ Both models can rewrite or continue an existing clip. Pass the source clip as a import { generateVideo } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; -const adapter = grokVideo("grok-imagine-video-1.5"); +const adapter = grokVideo("grok-imagine-video"); // Edit: change the clip in place const edit = await generateVideo({ diff --git a/docs/media/video-generation.md b/docs/media/video-generation.md index a555bfe60b..4b83af2f92 100644 --- a/docs/media/video-generation.md +++ b/docs/media/video-generation.md @@ -49,7 +49,7 @@ Currently supported: - **OpenAI**: Sora-2 and Sora-2-Pro models (when available) - **Google Gemini**: Veo 3.1 models (via the long-running operations API), and Gemini Omni Flash (via the Interactions API) -- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5 (text-to-video, image-to-video, reference-to-video, editing, extension) +- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5 (text-to-video, image-to-video; 1.5 adds reference-to-video; v1.0 adds editing and extension) - **BytePlus**: Seedance 2.0, 1.5-pro and 1.0-pro models (text-to-video, first/last frame, and multimodal references on 2.0) - **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models - **OpenRouter**: Seedance, Veo 3.1, Wan, Kling, Sora 2 Pro and others via the dedicated async video API (`POST /api/v1/videos`) @@ -691,7 +691,7 @@ instead of letting the model infer the task mode). #### Grok (xAI Imagine) Model Options -Based on the [xAI video generation API](https://docs.x.ai/developers/model-capabilities/video/generation). Two models are available: `grok-imagine-video` (v1.0) and `grok-imagine-video-1.5` (xAI's recommended default, with native 1080p text-to-video). Both support **text-to-video and image-to-video**; 1.5 adds **reference-to-video**, and both drive **video editing and extension**. Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. +Based on the [xAI video generation API](https://docs.x.ai/developers/model-capabilities/video/generation). Two models are available: `grok-imagine-video` (v1.0) and `grok-imagine-video-1.5` (xAI's recommended default, with native 1080p text-to-video). Both support **text-to-video and image-to-video**; 1.5 adds **reference-to-video**. **Video editing and extension** are `grok-imagine-video` only — 1.5 has no video input. Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long. Text-to-video: @@ -754,14 +754,14 @@ const { jobId } = await generateVideo({ }); ``` -Video editing and extension — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total. Because the output inherits the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` in both modes (and `duration` in edit mode) instead of sending fields the API ignores: +Video editing and extension (`grok-imagine-video` only) — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total. Because the output inherits the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` in both modes (and `duration` in edit mode) instead of sending fields the API ignores. The adapter rejects a source-video part or `mode` on `grok-imagine-video-1.5`. ```typescript import { generateVideo } from "@tanstack/ai"; import { grokVideo } from "@tanstack/ai-grok"; const { jobId } = await generateVideo({ - adapter: grokVideo("grok-imagine-video-1.5"), + adapter: grokVideo("grok-imagine-video"), prompt: [ { type: "text", content: "The camera keeps panning right across the bay" }, { diff --git a/packages/ai-grok/src/adapters/video.ts b/packages/ai-grok/src/adapters/video.ts index 6b193dcbad..f2af4e25cf 100644 --- a/packages/ai-grok/src/adapters/video.ts +++ b/packages/ai-grok/src/adapters/video.ts @@ -4,8 +4,10 @@ import { toRunErrorPayload } from '@tanstack/ai/adapter-internals' import { getGrokApiKeyFromEnv, withGrokDefaults } from '../utils/client' import { GROK_VIDEO_MAX_REFERENCE_AUDIOS, + GROK_VIDEO_MAX_REFERENCE_IMAGES, getGrokVideoDurationOptions, isGrokVideoReferenceModel, + isGrokVideoSourceModel, parseGrokVideoSize, validateVideoSize, } from '../video/video-provider-options' @@ -26,7 +28,7 @@ import type { GrokVideoModelInputModalitiesByName, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, - GrokVideoProviderOptions, + GrokVideoRuntimeOptions, } from '../video/video-provider-options' import type { GrokClientConfig } from '../utils/client' @@ -99,7 +101,8 @@ function buildGrokVideoUsage( * * Both models support text-to-video and image-to-video; * `grok-imagine-video-1.5` is xAI's documented default and adds native - * 1080p text-to-video plus reference-to-video inputs. + * 1080p generation plus reference-to-video inputs. Source-video edit + * and extend are `grok-imagine-video` only. * * The Imagine video endpoints are not part of the OpenAI SDK surface (and * xAI rejects the SDK's multipart paths), so requests are plain JSON calls @@ -116,9 +119,10 @@ function buildGrokVideoUsage( * `metadata.role: 'reference'` or `'character'` (→ `reference_images`) * and preset voices via `modelOptions.reference_audios` * (grok-imagine-video-1.5 only) - * - Video editing / extension via a source `video` prompt part and - * `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / - * `/v1/videos/extensions`; in extend mode `duration` is the added tail) + * - Video editing / extension on `grok-imagine-video` via a source + * `video` prompt part and `modelOptions.mode: 'edit' | 'extend'` + * (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode + * `duration` is the added tail) * - Usage reporting: billed seconds (`unitsBilled`) and exact cost */ export class GrokVideoAdapter< @@ -197,7 +201,7 @@ export class GrokVideoAdapter< // arrives as deserialized JSON, so the adapter handles the widest option // surface (the 1.5 shape) uniformly and gates by model at runtime. const { mode, ...wireOptions } = (modelOptions ?? - {}) as GrokVideoProviderOptions + {}) as GrokVideoRuntimeOptions // `mode` is typed 'edit' | 'extend' but reaches us untrusted from JSON // callers. An unrecognised value must not fall through to the @@ -222,9 +226,21 @@ export class GrokVideoAdapter< ) } - // A video prompt part is the source clip for edit / extension mode; the - // mode must be chosen explicitly because the two endpoints have different - // semantics (edit rewrites the clip, extend appends `duration` seconds). + // A video prompt part is the source clip for edit / extension mode. + // Those endpoints are grok-imagine-video only — 1.5 has no video input. + if ( + !isGrokVideoSourceModel(model) && + (mode !== undefined || resolved.videos.length > 0) + ) { + throw new Error( + `${this.name}: ${model} does not support video editing or extension. ` + + `Use 'grok-imagine-video' for /v1/videos/edits and /v1/videos/extensions.`, + ) + } + + // The mode must be chosen explicitly because the two endpoints have + // different semantics (edit rewrites the clip, extend appends + // `duration` seconds). if (resolved.videos.length > 1) { throw new Error( `${this.name}: ${model} accepts at most one source video; received ${resolved.videos.length}.`, @@ -326,17 +342,43 @@ export class GrokVideoAdapter< `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_AUDIOS} reference voices; received ${referenceAudioCount}.`, ) } + const referenceImageCount = + wireOptions.reference_images?.length ?? referenceImages.length + if (referenceImageCount > GROK_VIDEO_MAX_REFERENCE_IMAGES) { + throw new Error( + `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_IMAGES} reference images; received ${referenceImageCount}.`, + ) + } // Image-to-video: the single image prompt part becomes the starting frame // and the prompt text describes the desired motion. URL sources are // fetched by xAI's servers; data sources are sent as base64 data URIs. const [startFrame] = startFrames + // xAI rejects `image` + `reference_images` / `reference_audios` as a + // 400: only one of image-to-video or reference-to-video can be active. + const hasReference = + referenceImages.length > 0 || + wireOptions.reference_images !== undefined || + wireOptions.reference_audios !== undefined + if (startFrame && hasReference) { + throw new Error( + `${this.name}: image-to-video and reference-to-video cannot be combined. ` + + `Use a starting-frame image, or reference images / voices, not both.`, + ) + } + // The generic `size` option carries an "aspectRatio_resolution" template // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / // `resolution` parameters; explicit modelOptions win over the template // (including `reference_images`, which replaces the part-derived list). const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined + const resolvedResolution = wireOptions.resolution ?? parsedSize?.resolution + if (hasReference && resolvedResolution === '1080p') { + throw new Error( + `${this.name}: reference-to-video is capped at 720p on ${model}.`, + ) + } const request = { model, prompt: resolved.text, @@ -377,10 +419,10 @@ export class GrokVideoAdapter< mode: 'edit' | 'extend' sourceVideo: VideoPart resolved: ReturnType - wireOptions: Omit + wireOptions: Omit size: string | undefined genericDuration: number | undefined - logger: VideoGenerationOptions['logger'] + logger: VideoGenerationOptions['logger'] }): Promise { const { model, mode, sourceVideo, resolved, wireOptions, logger } = args const endpoint = mode === 'edit' ? '/videos/edits' : '/videos/extensions' @@ -455,7 +497,7 @@ export class GrokVideoAdapter< request: Record, context: { model: string - logger: VideoGenerationOptions['logger'] + logger: VideoGenerationOptions['logger'] logLine: string }, ): Promise { diff --git a/packages/ai-grok/src/index.ts b/packages/ai-grok/src/index.ts index 6f245ceb7f..e1c935f734 100644 --- a/packages/ai-grok/src/index.ts +++ b/packages/ai-grok/src/index.ts @@ -47,7 +47,9 @@ export { export type { GrokVideoMode, GrokVideoBaseProviderOptions, + GrokVideoSourceProviderOptions, GrokVideoProviderOptions, + GrokVideoRuntimeOptions, GrokVideoModelProviderOptionsByName, GrokVideoModelSizeByName, GrokVideoModelDurationByName, diff --git a/packages/ai-grok/src/model-meta.ts b/packages/ai-grok/src/model-meta.ts index fd5ff45f60..35feac479f 100644 --- a/packages/ai-grok/src/model-meta.ts +++ b/packages/ai-grok/src/model-meta.ts @@ -154,10 +154,11 @@ const GROK_IMAGINE_IMAGE_2_0 = { // // Both models support text-to-video and image-to-video (a starting-frame // image is optional). grok-imagine-video-1.5 is the documented default: it -// adds native 1080p text-to-video plus reference-to-video inputs -// (`reference_images` / `reference_audios`; reference output is capped at -// 720p). Both models also drive video editing (`/v1/videos/edits`) and -// extension (`/v1/videos/extensions`) via a source video prompt part. +// adds native 1080p for text-to-video / image-to-video plus +// reference-to-video (`reference_images` / `reference_audios`; reference +// output is capped at 720p). Source-video edit (`/v1/videos/edits`) and +// extend (`/v1/videos/extensions`) are grok-imagine-video only — xAI's +// 1.5 model page lists text+image input, not video. const GROK_IMAGINE_VIDEO = { name: 'grok-imagine-video', supports: { @@ -178,7 +179,7 @@ const GROK_IMAGINE_VIDEO = { const GROK_IMAGINE_VIDEO_1_5 = { name: 'grok-imagine-video-1.5', supports: { - input: ['text', 'image', 'video'], + input: ['text', 'image'], output: ['video', 'audio'], }, pricing: { diff --git a/packages/ai-grok/src/video/video-provider-options.ts b/packages/ai-grok/src/video/video-provider-options.ts index 4a8745577f..3223a72e4d 100644 --- a/packages/ai-grok/src/video/video-provider-options.ts +++ b/packages/ai-grok/src/video/video-provider-options.ts @@ -36,6 +36,14 @@ export type GrokVideoAspectRatio = */ export type GrokVideoResolution = '480p' | '720p' | '1080p' +/** + * Resolutions accepted by grok-imagine-video (v1.0). Native 1080p is a + * grok-imagine-video-1.5 generation feature. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoResolutionV1 = '480p' | '720p' + /** * Size strings for grok-imagine video models. The Imagine API is * aspect-ratio based rather than pixel-size based; like the grok-imagine @@ -49,6 +57,15 @@ export type GrokVideoSize = | GrokVideoAspectRatio | `${GrokVideoAspectRatio}_${GrokVideoResolution}` +/** + * Size strings for grok-imagine-video (v1.0) — 1080p is not in the type. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoSizeV1 = + | GrokVideoAspectRatio + | `${GrokVideoAspectRatio}_${GrokVideoResolutionV1}` + const GROK_VIDEO_ASPECT_RATIOS: ReadonlyArray = [ '1:1', '16:9', @@ -82,6 +99,16 @@ export function parseGrokVideoSize( return { aspectRatio, ...(resolution !== undefined && { resolution }) } } +/** + * Models that accept native 1080p on text-to-video and image-to-video. + * Reference-to-video stays capped at 720p even on these models. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoNative1080pModel(model: string): boolean { + return model === 'grok-imagine-video-1.5' +} + /** * Validate the `size` template for a given grok video model. * @@ -109,6 +136,12 @@ export function validateVideoSize( `Supported resolutions: ${GROK_VIDEO_RESOLUTIONS.join(', ')}`, ) } + if (parsed.resolution === '1080p' && !isGrokVideoNative1080pModel(model)) { + throw new Error( + `Resolution "1080p" is not supported by model "${model}". ` + + `Use 'grok-imagine-video-1.5' for native 1080p text-to-video / image-to-video.`, + ) + } } /** @@ -168,7 +201,8 @@ export function getGrokVideoDurationOptions( * (modify the source clip in place); `'extend'` posts to * `/v1/videos/extensions` (continue the source clip — `duration` is the * length of the **added tail**, not the total). Both require exactly one - * video prompt part carrying the source clip. + * video prompt part carrying the source clip, and both are + * `grok-imagine-video` (v1.0) only. * * Output geometry (aspect ratio / resolution) is inherited from the source * clip in both modes, capped at 720p, and edit outputs also inherit the @@ -180,22 +214,13 @@ export function getGrokVideoDurationOptions( export type GrokVideoMode = 'edit' | 'extend' /** - * Provider options shared by both grok-imagine video models. Apart from - * `mode` (a routing hint stripped before the request is sent), these map + * Provider options shared by both grok-imagine video models. These map * directly onto the Imagine API request body and take precedence over the * generic `size` / `duration` options when both are provided. * * @experimental Video generation is an experimental feature and may change. */ export interface GrokVideoBaseProviderOptions { - /** - * Selects the request mode for a source-video prompt part: `'edit'` - * (`/v1/videos/edits`) or `'extend'` (`/v1/videos/extensions`). Required - * when the prompt carries a video part; not valid without one. Omit for - * plain generation (`/v1/videos/generations`). - */ - mode?: GrokVideoMode - /** * Output aspect ratio. Generation only — edit / extend outputs inherit * the source clip's geometry and the adapter rejects this in those modes. @@ -205,6 +230,8 @@ export interface GrokVideoBaseProviderOptions { /** * Output resolution tier. Generation only — edit / extend outputs inherit * the source clip's geometry and the adapter rejects this in those modes. + * `1080p` is grok-imagine-video-1.5 generation only; reference-to-video + * is capped at 720p. */ resolution?: GrokVideoResolution @@ -216,6 +243,22 @@ export interface GrokVideoBaseProviderOptions { duration?: number } +/** + * Provider options for grok-imagine-video (v1.0), which is the only model + * that accepts a source-video edit / extend job. + * + * @experimental Video generation is an experimental feature and may change. + */ +export interface GrokVideoSourceProviderOptions extends GrokVideoBaseProviderOptions { + /** + * Selects the request mode for a source-video prompt part: `'edit'` + * (`/v1/videos/edits`) or `'extend'` (`/v1/videos/extensions`). Required + * when the prompt carries a video part; not valid without one. Omit for + * plain generation (`/v1/videos/generations`). grok-imagine-video only. + */ + mode?: GrokVideoMode +} + /** * Provider options for grok-imagine-video-1.5, which adds the * reference-to-video inputs on top of the shared options. @@ -242,11 +285,26 @@ export interface GrokVideoProviderOptions extends GrokVideoBaseProviderOptions { reference_audios?: Array<{ voice_id: string }> } +/** + * Widest option surface. Used when `modelOptions` arrives as deserialized + * JSON and the adapter must validate fields the per-model map already + * hides at compile time. + * + * @experimental Video generation is an experimental feature and may change. + */ +export type GrokVideoRuntimeOptions = GrokVideoSourceProviderOptions & + GrokVideoProviderOptions + /** * Maximum reference voices accepted by the Imagine video endpoint. */ export const GROK_VIDEO_MAX_REFERENCE_AUDIOS = 3 +/** + * Maximum reference images accepted by the Imagine video endpoint. + */ +export const GROK_VIDEO_MAX_REFERENCE_IMAGES = 7 + /** * Models that support reference-to-video inputs (`reference_images` / * `reference_audios`). The per-model provider-options map hides the fields @@ -269,14 +327,35 @@ export function isGrokVideoReferenceModel(model: string): boolean { return GROK_VIDEO_REFERENCE_MODELS.has(model) } +/** + * Models that accept a source-video prompt part for `/v1/videos/edits` + * and `/v1/videos/extensions`. xAI lists video input only on + * grok-imagine-video (v1.0). + * + * @experimental Video generation is an experimental feature and may change. + */ +const GROK_VIDEO_SOURCE_MODELS: ReadonlySet = new Set([ + 'grok-imagine-video', +]) + +/** + * True when the model accepts edit / extend source-video jobs. + * + * @experimental Video generation is an experimental feature and may change. + */ +export function isGrokVideoSourceModel(model: string): boolean { + return GROK_VIDEO_SOURCE_MODELS.has(model) +} + /** * Type-only map from model name to its specific provider options. Only - * grok-imagine-video-1.5 exposes the reference-to-video fields. + * grok-imagine-video-1.5 exposes the reference-to-video fields. Only + * grok-imagine-video (v1.0) exposes `mode` for edit / extend. * * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoModelProviderOptionsByName = { - 'grok-imagine-video': GrokVideoBaseProviderOptions + 'grok-imagine-video': GrokVideoSourceProviderOptions 'grok-imagine-video-1.5': GrokVideoProviderOptions } @@ -286,7 +365,7 @@ export type GrokVideoModelProviderOptionsByName = { * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoModelSizeByName = { - 'grok-imagine-video': GrokVideoSize + 'grok-imagine-video': GrokVideoSizeV1 'grok-imagine-video-1.5': GrokVideoSize } @@ -296,11 +375,11 @@ export type GrokVideoModelSizeByName = { * part as the starting frame; image parts with `metadata.role: 'reference'` * or `'character'` become `reference_images` (grok-imagine-video-1.5 only). * A `video` prompt part carries the source clip for edit / extension mode - * (`modelOptions.mode: 'edit' | 'extend'`). + * on grok-imagine-video only (`modelOptions.mode: 'edit' | 'extend'`). * * @experimental Video generation is an experimental feature and may change. */ export type GrokVideoModelInputModalitiesByName = { 'grok-imagine-video': readonly ['image', 'video'] - 'grok-imagine-video-1.5': readonly ['image', 'video'] + 'grok-imagine-video-1.5': readonly ['image'] } diff --git a/packages/ai-grok/tests/realtime-token.test.ts b/packages/ai-grok/tests/realtime-token.test.ts index 4aa74fc3db..e8c291d237 100644 --- a/packages/ai-grok/tests/realtime-token.test.ts +++ b/packages/ai-grok/tests/realtime-token.test.ts @@ -19,7 +19,10 @@ afterEach(() => { } }) -function makeSessionResponse(expiresAt: number) { +function makeSessionResponse( + expiresAt: number, + model = 'grok-voice-think-fast-2.0', +) { return { ok: true, status: 200, @@ -27,7 +30,7 @@ function makeSessionResponse(expiresAt: number) { Promise.resolve({ id: 'sess_1', object: 'realtime.session', - model: 'grok-voice-fast-1.0', + model, modalities: ['audio', 'text'], instructions: '', voice: 'eve', @@ -52,7 +55,9 @@ describe('grokRealtimeToken request body', () => { it('wraps the model under the `session` key per xAI /v1/realtime/client_secrets schema', async () => { const fetchMock = vi .fn() - .mockResolvedValue(makeSessionResponse(1_700_000_000)) + .mockResolvedValue( + makeSessionResponse(1_700_000_000, 'grok-voice-think-fast-1.0'), + ) globalThis.fetch = fetchMock as unknown as typeof fetch await realtimeToken({ @@ -70,11 +75,12 @@ describe('grokRealtimeToken request body', () => { .mockResolvedValue(makeSessionResponse(1_700_000_000)) globalThis.fetch = fetchMock as unknown as typeof fetch - await realtimeToken({ adapter: grokRealtimeToken() }) + const token = await realtimeToken({ adapter: grokRealtimeToken() }) const init = fetchMock.mock.calls[0]![1]! const body = JSON.parse(init.body as string) as Record expect(body).toEqual({ session: { model: 'grok-voice-think-fast-2.0' } }) + expect(token.config.model).toBe('grok-voice-think-fast-2.0') }) }) diff --git a/packages/ai-grok/tests/video-adapter.test.ts b/packages/ai-grok/tests/video-adapter.test.ts index caaa0f051e..dbe6ccc02b 100644 --- a/packages/ai-grok/tests/video-adapter.test.ts +++ b/packages/ai-grok/tests/video-adapter.test.ts @@ -45,6 +45,17 @@ function adapterWithFetch( }) } +function sourceAdapterWithFetch( + fetchMock: ( + input: string | URL | Request, + init?: RequestInit, + ) => Promise, +) { + return createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) +} + /** * Builds a text + starting-frame-image prompt for the request-shape / * status / error tests. @@ -316,6 +327,23 @@ describe('Grok Video Adapter', () => { expect(body.resolution).toBe('1080p') }) + it('rejects 1080p on grok-imagine-video (v1.0)', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = sourceAdapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + // Native 1080p is 1.5-only; the v1 size map excludes the suffix. + // @ts-expect-error 1080p is not a grok-imagine-video size + size: '16:9_1080p', + logger: testLogger, + }), + ).rejects.toThrow(/1080p/) + expect(fetchMock).not.toHaveBeenCalled() + }) + it('rejects unsupported sizes before calling the API', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -480,6 +508,25 @@ describe('Grok Video Adapter', () => { expect(body.reference_audios).toEqual([{ voice_id: 'eve' }]) }) + it('rejects more than 7 reference images before calling the API', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { + reference_images: Array.from({ length: 8 }, (_, i) => ({ + url: `https://example.com/${i}.png`, + })), + }, + logger: testLogger, + }), + ).rejects.toThrow(/at most 7 reference images/) + expect(fetchMock).not.toHaveBeenCalled() + }) + it('rejects more than 3 reference voices before calling the API', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) @@ -502,33 +549,67 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) - it('sends both a starting frame and reference images when combined', async () => { - const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ref-3' })) + it('rejects combining a starting frame with reference images', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = adapterWithFetch(fetchMock) - await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', - prompt: [ - { type: 'text', content: ' enters from the left' }, - { - type: 'image', - source: { type: 'url', value: 'https://example.com/start.png' }, - metadata: { role: 'start_frame' }, - }, - { - type: 'image', - source: { type: 'url', value: 'https://example.com/person.png' }, - metadata: { role: 'reference' }, - }, - ], - logger: testLogger, - }) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' enters from the left' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/start.png' }, + metadata: { role: 'start_frame' }, + }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + ], + logger: testLogger, + }), + ).rejects.toThrow(/cannot be combined/) + expect(fetchMock).not.toHaveBeenCalled() + }) - const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) - expect(body.image).toEqual({ url: 'https://example.com/start.png' }) - expect(body.reference_images).toEqual([ - { url: 'https://example.com/person.png' }, - ]) + it('rejects combining a starting frame with reference_audios', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: i2vPrompt(' narrates'), + modelOptions: { reference_audios: [{ voice_id: 'eve' }] }, + logger: testLogger, + }), + ).rejects.toThrow(/cannot be combined/) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('rejects 1080p on reference-to-video', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: ' walks' }, + { + type: 'image', + source: { type: 'url', value: 'https://example.com/person.png' }, + metadata: { role: 'reference' }, + }, + ], + size: '16:9_1080p', + logger: testLogger, + }), + ).rejects.toThrow(/capped at 720p/) + expect(fetchMock).not.toHaveBeenCalled() }) it('lets explicit modelOptions.reference_images replace the part-derived list', async () => { @@ -649,10 +730,10 @@ describe('Grok Video Adapter', () => { it("posts to /videos/edits in 'edit' mode with the source video", async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'edit-1' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) const result = await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [ { type: 'text', content: 'make the sky stormy' }, sourceVideoPart, @@ -663,12 +744,12 @@ describe('Grok Video Adapter', () => { expect(result).toEqual({ jobId: 'edit-1', - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', }) const [url, init] = fetchMock.mock.calls[0]! expect(url).toBe('https://api.x.ai/v1/videos/edits') expect(JSON.parse(String(init?.body))).toEqual({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: 'make the sky stormy', video: { url: 'https://example.com/clip.mp4' }, }) @@ -676,10 +757,10 @@ describe('Grok Video Adapter', () => { it("posts to /videos/extensions in 'extend' mode with the added-tail duration", async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-1' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [ { type: 'text', content: 'the camera keeps panning right' }, sourceVideoPart, @@ -692,7 +773,7 @@ describe('Grok Video Adapter', () => { const [url, init] = fetchMock.mock.calls[0]! expect(url).toBe('https://api.x.ai/v1/videos/extensions') expect(JSON.parse(String(init?.body))).toEqual({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: 'the camera keeps panning right', video: { url: 'https://example.com/clip.mp4' }, duration: 5, @@ -701,10 +782,10 @@ describe('Grok Video Adapter', () => { it('sends a base64 data source as a data URI source video', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [ { type: 'text', content: 'p' }, { @@ -722,10 +803,10 @@ describe('Grok Video Adapter', () => { it('snaps the extend duration passed via modelOptions', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-2' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], modelOptions: { mode: 'extend', duration: 2.6 }, logger: testLogger, @@ -737,10 +818,10 @@ describe('Grok Video Adapter', () => { it('snaps an out-of-range generic extend duration', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'ext-3' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'keep going' }, sourceVideoPart], duration: 20, modelOptions: { mode: 'extend' }, @@ -753,11 +834,11 @@ describe('Grok Video Adapter', () => { it("rejects a duration in 'edit' mode — inherited from the source", async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], duration: 5, modelOptions: { mode: 'edit' }, @@ -766,7 +847,7 @@ describe('Grok Video Adapter', () => { ).rejects.toThrow(/'edit' mode does not accept a duration/) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], modelOptions: { mode: 'edit', duration: 99 }, logger: testLogger, @@ -777,20 +858,20 @@ describe('Grok Video Adapter', () => { it('rejects size / aspect_ratio / resolution in edit and extend modes', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], - size: '16:9_1080p', + size: '16:9_720p', modelOptions: { mode: 'edit' }, logger: testLogger, }), ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], modelOptions: { mode: 'extend', aspect_ratio: '9:16' }, logger: testLogger, @@ -798,9 +879,9 @@ describe('Grok Video Adapter', () => { ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], - modelOptions: { mode: 'edit', resolution: '1080p' }, + modelOptions: { mode: 'edit', resolution: '720p' }, logger: testLogger, }), ).rejects.toThrow(/does not accept size \/ aspect_ratio \/ resolution/) @@ -809,11 +890,11 @@ describe('Grok Video Adapter', () => { it('rejects an unknown mode instead of misrouting to generations', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], // Mode arrives untrusted from JSON callers; a typo must not fall // through to /videos/generations with a source-video body. @@ -827,11 +908,11 @@ describe('Grok Video Adapter', () => { it('rejects a video prompt part without an explicit mode', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], logger: testLogger, }), @@ -841,11 +922,11 @@ describe('Grok Video Adapter', () => { it('rejects a mode without a source video part', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: 'p', modelOptions: { mode: 'extend' }, logger: testLogger, @@ -856,11 +937,11 @@ describe('Grok Video Adapter', () => { it('rejects image prompt parts in edit / extend mode', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [ { type: 'text', content: 'p' }, sourceVideoPart, @@ -878,14 +959,17 @@ describe('Grok Video Adapter', () => { it('rejects reference inputs in edit / extend mode', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], modelOptions: { mode: 'edit', + // Per-model map hides reference fields from v1.0; JSON callers + // still hit the runtime gate. + // @ts-expect-error reference_audios is 1.5-only reference_audios: [{ voice_id: 'eve' }], }, logger: testLogger, @@ -896,11 +980,11 @@ describe('Grok Video Adapter', () => { it('rejects more than one source video', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) - const adapter = adapterWithFetch(fetchMock) + const adapter = sourceAdapterWithFetch(fetchMock) await expect( adapter.createVideoJob({ - model: 'grok-imagine-video-1.5', + model: 'grok-imagine-video', prompt: [ { type: 'text', content: 'p' }, sourceVideoPart, @@ -912,6 +996,36 @@ describe('Grok Video Adapter', () => { ).rejects.toThrow(/at most one source video/) expect(fetchMock).not.toHaveBeenCalled() }) + + it('rejects edit / extend on grok-imagine-video-1.5', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: [ + { type: 'text', content: 'p' }, + // 1.5 has no video input on generateVideo(); createVideoJob still + // accepts the part so JSON callers hit the runtime gate. + sourceVideoPart, + ], + // @ts-expect-error mode is grok-imagine-video only + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support video editing or extension/) + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + // @ts-expect-error mode is grok-imagine-video only + modelOptions: { mode: 'extend' }, + logger: testLogger, + }), + ).rejects.toThrow(/does not support video editing or extension/) + expect(fetchMock).not.toHaveBeenCalled() + }) }) describe('getVideoStatus', () => { @@ -1090,7 +1204,12 @@ describe('Grok Video Adapter', () => { it('validates sizes', () => { expect(() => validateVideoSize('m', '16:9')).not.toThrow() - expect(() => validateVideoSize('m', '2:3_1080p')).not.toThrow() + expect(() => + validateVideoSize('grok-imagine-video-1.5', '2:3_1080p'), + ).not.toThrow() + expect(() => + validateVideoSize('grok-imagine-video', '16:9_1080p'), + ).toThrow(/1080p/) expect(() => validateVideoSize('m', undefined)).not.toThrow() expect(() => validateVideoSize('m', '9:19.5')).toThrow(/not supported/) expect(() => validateVideoSize('m', 'auto')).toThrow(/not supported/) diff --git a/packages/ai/skills/ai-core/media-generation/SKILL.md b/packages/ai/skills/ai-core/media-generation/SKILL.md index 2e6ac5eb54..074c30bffc 100644 --- a/packages/ai/skills/ai-core/media-generation/SKILL.md +++ b/packages/ai/skills/ai-core/media-generation/SKILL.md @@ -295,18 +295,19 @@ with `allowUrlFetch: true` on the adapter config **Provider support matrix:** -| Provider | `generateImage` image parts | `generateVideo` image parts | -| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| OpenAI | gpt-image-2 / gpt-image-1 / -mini → `images.edit()` (up to 16). dall-e-2 → edit (1). dall-e-3 throws. | Sora-2 / -pro → `input_reference` (single). Throws if >1. | -| Gemini | Native (gemini-\*-flash-image, "nano-banana") → multimodal `contents`. Imagen throws. | Veo → first un-roled / `'start_frame'` image is the input image; `'end_frame'` → `lastFrame`; `'reference'` / `'character'` → `referenceImages`. Omni Flash sends image/video parts as interaction content blocks (no role routing). | -| fal | Per-endpoint field names from a generated map (`pnpm generate:fal-image-fields`). Defaults: 1 input → `image_url`; >1 → `image_urls`; roles → `mask_url` / `control_image_url` / `reference_image_urls`. | Per-endpoint map (e.g. Kling i2v start frame → `image_url`). Defaults: 1 input → `image_url`; `start_frame`/`end_frame` → `start_image_url`/`end_image_url`; `reference` → `reference_image_urls`. | -| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | Un-roled / `'start_frame'` image → starting frame; `'reference'` / `'character'` → `reference_images` (1.5). A `video` part + `modelOptions.mode: 'edit' \| 'extend'` routes to `/videos/edits` / `/videos/extensions`. | -| OpenRouter | Prompt parts map 1:1 onto multimodal `text` / `image_url` content parts, preserving interleaved order. | Dedicated async API (`openRouterVideo`): `start_frame`/`end_frame` → `frame_images[]` (`first_frame`/`last_frame`); `reference`/`character` → `input_references[]`; an unroled image defaults to the start frame. Frame roles validated against the model's `supported_frame_images` metadata. | -| Anthropic | n/a (no image generation API). | n/a | +| Provider | `generateImage` image parts | `generateVideo` image parts | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| OpenAI | gpt-image-2 / gpt-image-1 / -mini → `images.edit()` (up to 16). dall-e-2 → edit (1). dall-e-3 throws. | Sora-2 / -pro → `input_reference` (single). Throws if >1. | +| Gemini | Native (gemini-\*-flash-image, "nano-banana") → multimodal `contents`. Imagen throws. | Veo → first un-roled / `'start_frame'` image is the input image; `'end_frame'` → `lastFrame`; `'reference'` / `'character'` → `referenceImages`. Omni Flash sends image/video parts as interaction content blocks (no role routing). | +| fal | Per-endpoint field names from a generated map (`pnpm generate:fal-image-fields`). Defaults: 1 input → `image_url`; >1 → `image_urls`; roles → `mask_url` / `control_image_url` / `reference_image_urls`. | Per-endpoint map (e.g. Kling i2v start frame → `image_url`). Defaults: 1 input → `image_url`; `start_frame`/`end_frame` → `start_image_url`/`end_image_url`; `reference` → `reference_image_urls`. | +| Grok | grok-imagine models → `/v1/images/edits` JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws. | Un-roled / `'start_frame'` image → starting frame; `'reference'` / `'character'` → `reference_images` (1.5). Starting frame and reference inputs cannot be combined. A `video` part + `modelOptions.mode: 'edit' \| 'extend'` routes to `/videos/edits` / `/videos/extensions` on `grok-imagine-video` only. | +| OpenRouter | Prompt parts map 1:1 onto multimodal `text` / `image_url` content parts, preserving interleaved order. | Dedicated async API (`openRouterVideo`): `start_frame`/`end_frame` → `frame_images[]` (`first_frame`/`last_frame`); `reference`/`character` → `input_references[]`; an unroled image defaults to the start frame. Frame roles validated against the model's `supported_frame_images` metadata. | +| Anthropic | n/a (no image generation API). | n/a | Video and audio prompt parts follow the same `metadata.role` convention -for video-to-video and lipsync flows on fal; other providers throw when -they're passed. +for video-to-video and lipsync flows on fal. Grok accepts one source +`video` part on `grok-imagine-video` with `modelOptions.mode: 'edit' | 'extend'` +and rejects audio parts. Other providers throw when those parts are passed. ### 2. Audio Generation (Music, Sound Effects) @@ -546,7 +547,7 @@ Other video adapters: `openaiVideo('sora-2')` (pixel sizes like `'1280x720'`, durations 4/8/12s, single `input_reference` image prompt part), `grokVideo(...)` (`grok-imagine-video` and `grok-imagine-video-1.5` both do text-to-video + image-to-video; 1.5 adds reference-to-video — `'reference'`-roled image parts → `reference_images`, preset -voices via `modelOptions.reference_audios` (max 3) — and both edit/extend a source `video` +voices via `modelOptions.reference_audios` (max 3); only `grok-imagine-video` edits/extends a source `video` prompt part via `modelOptions.mode: 'edit' | 'extend'` (extend `duration` = added tail); aspect-ratio size template like `'16:9_720p'`, integer durations 1-15s, reports `usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — From 148d92b80ee51c268798abac06f4e3bee938c5cb Mon Sep 17 00:00:00 2001 From: Alem Tuzlak Date: Tue, 18 Aug 2026 11:05:20 +0200 Subject: [PATCH 4/5] fix(ai-anthropic): add opus-5 models to the tool-capabilities map --- .changeset/anthropic-opus-5-tool-map.md | 5 +++++ packages/ai-anthropic/src/model-meta.ts | 2 ++ 2 files changed, 7 insertions(+) create mode 100644 .changeset/anthropic-opus-5-tool-map.md diff --git a/.changeset/anthropic-opus-5-tool-map.md b/.changeset/anthropic-opus-5-tool-map.md new file mode 100644 index 0000000000..d91649d8ae --- /dev/null +++ b/.changeset/anthropic-opus-5-tool-map.md @@ -0,0 +1,5 @@ +--- +'@tanstack/ai-anthropic': patch +--- + +Register `claude-opus-5` and `claude-opus-5-fast` on the per-model tool-capabilities map so they match `ANTHROPIC_MODELS`. diff --git a/packages/ai-anthropic/src/model-meta.ts b/packages/ai-anthropic/src/model-meta.ts index b3b248d27e..5dc0c80f18 100644 --- a/packages/ai-anthropic/src/model-meta.ts +++ b/packages/ai-anthropic/src/model-meta.ts @@ -819,6 +819,8 @@ export type AnthropicChatModelToolCapabilitiesByName = { [CLAUDE_OPUS_4_8.id]: typeof CLAUDE_OPUS_4_8.supports.tools [CLAUDE_FABLE_5.id]: typeof CLAUDE_FABLE_5.supports.tools [CLAUDE_SONNET_5.id]: typeof CLAUDE_SONNET_5.supports.tools + [CLAUDE_OPUS_5.id]: typeof CLAUDE_OPUS_5.supports.tools + [CLAUDE_OPUS_5_FAST.id]: typeof CLAUDE_OPUS_5_FAST.supports.tools } /** From 4ae0d3d66c663588a15d85a29e1e4a3a22aa65ec Mon Sep 17 00:00:00 2001 From: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com> Date: Tue, 18 Aug 2026 19:12:08 +1000 Subject: [PATCH 5/5] fix(ai-grok): normalize JSON-null options and name endpoints in failures Re-applies the review residuals on top of the rebased branch: - Treat JSON-serialized null / empty-array values for duration and the reference fields as unset on every path: the generation branch no longer sends "duration": null or empty reference arrays, and null reference fields no longer trip the v1.0 reference gate; edit/extend destructure the reference fields out of the passthrough so no known key can leak into those bodies. - Name the actual endpoint in create-job failure messages so a failed edit/extension no longer reports as a 'video generation' failure. - Type-tie GROK_VIDEO_REFERENCE_MODELS / GROK_VIDEO_SOURCE_MODELS to the per-model options map so a typo or an unlisted future capable model is a compile error. - Sync the media-generation skill with the current contract (v1.0-only edit/extend, reference caps, 720p reference cap, 1080p on 1.5 only, edit/extend option rejections). - Tests: null-duration strip, null/empty reference normalization, endpoint-named edit failure (124 total). --- packages/ai-grok/src/adapters/video.ts | 96 +++++++++++-------- .../src/video/video-provider-options.ts | 32 +++++-- packages/ai-grok/tests/video-adapter.test.ts | 68 ++++++++++++- .../skills/ai-core/media-generation/SKILL.md | 14 ++- 4 files changed, 156 insertions(+), 54 deletions(-) diff --git a/packages/ai-grok/src/adapters/video.ts b/packages/ai-grok/src/adapters/video.ts index f2af4e25cf..90709fdd8f 100644 --- a/packages/ai-grok/src/adapters/video.ts +++ b/packages/ai-grok/src/adapters/video.ts @@ -275,13 +275,24 @@ export class GrokVideoAdapter< validateVideoSize(model, size) + // Pull the specially-handled keys out of the wire options: `duration` + // is folded into the snapped value below, and the reference fields are + // re-added explicitly so a JSON-serialized `null` or empty array reads + // as "unset" instead of leaking onto the wire. + const { + duration: rawOptionDuration, + reference_images: explicitReferenceImages, + reference_audios: referenceAudios, + ...generationOptions + } = wireOptions + // Coerce the requested duration into the model's valid range (1–15s, // integer) instead of rejecting it — `snapDuration` clamps and rounds. // modelOptions wins over the generic `duration`, mirroring the size // precedence below. - const rawDuration = wireOptions.duration ?? options.duration + const rawDuration = rawOptionDuration ?? options.duration const duration = - rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined + rawDuration != null ? this.snapDuration(rawDuration) : undefined // Image parts split by role: un-roled / 'start_frame' images become the // starting frame (image-to-video); 'reference' / 'character' images @@ -322,28 +333,29 @@ export class GrokVideoAdapter< `Use metadata.role: 'reference' for reference-to-video inputs.`, ) } + // Explicit modelOptions.reference_images replaces the part-derived list + // (an explicit empty array means "none"). + const finalReferenceImages = + explicitReferenceImages ?? + (referenceImages.length > 0 ? referenceImages : undefined) + const referenceImageCount = finalReferenceImages?.length ?? 0 + const referenceAudioCount = referenceAudios?.length ?? 0 + const hasReference = referenceImageCount > 0 || referenceAudioCount > 0 + // Reference inputs are a grok-imagine-video-1.5 feature. The per-model // options map already hides the fields from other models at compile // time; this runtime gate covers prompt-part roles and untyped callers. - if ( - !isGrokVideoReferenceModel(model) && - (referenceImages.length > 0 || - wireOptions.reference_images !== undefined || - wireOptions.reference_audios !== undefined) - ) { + if (!isGrokVideoReferenceModel(model) && hasReference) { throw new Error( `${this.name}: ${model} does not support reference-to-video inputs. ` + `Use 'grok-imagine-video-1.5' for reference_images / reference_audios.`, ) } - const referenceAudioCount = wireOptions.reference_audios?.length ?? 0 if (referenceAudioCount > GROK_VIDEO_MAX_REFERENCE_AUDIOS) { throw new Error( `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_AUDIOS} reference voices; received ${referenceAudioCount}.`, ) } - const referenceImageCount = - wireOptions.reference_images?.length ?? referenceImages.length if (referenceImageCount > GROK_VIDEO_MAX_REFERENCE_IMAGES) { throw new Error( `${this.name}: ${model} accepts at most ${GROK_VIDEO_MAX_REFERENCE_IMAGES} reference images; received ${referenceImageCount}.`, @@ -357,10 +369,6 @@ export class GrokVideoAdapter< // xAI rejects `image` + `reference_images` / `reference_audios` as a // 400: only one of image-to-video or reference-to-video can be active. - const hasReference = - referenceImages.length > 0 || - wireOptions.reference_images !== undefined || - wireOptions.reference_audios !== undefined if (startFrame && hasReference) { throw new Error( `${this.name}: image-to-video and reference-to-video cannot be combined. ` + @@ -373,7 +381,8 @@ export class GrokVideoAdapter< // `resolution` parameters; explicit modelOptions win over the template // (including `reference_images`, which replaces the part-derived list). const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined - const resolvedResolution = wireOptions.resolution ?? parsedSize?.resolution + const resolvedResolution = + generationOptions.resolution ?? parsedSize?.resolution if (hasReference && resolvedResolution === '1080p') { throw new Error( `${this.name}: reference-to-video is capped at 720p on ${model}.`, @@ -383,8 +392,11 @@ export class GrokVideoAdapter< model, prompt: resolved.text, ...(startFrame && { image: { url: mediaPartToUrl(startFrame) } }), - ...(referenceImages.length > 0 && { - reference_images: referenceImages, + ...(referenceImageCount > 0 && { + reference_images: finalReferenceImages, + }), + ...(referenceAudioCount > 0 && { + reference_audios: referenceAudios, }), ...(parsedSize && { aspect_ratio: parsedSize.aspectRatio, @@ -392,10 +404,10 @@ export class GrokVideoAdapter< resolution: parsedSize.resolution, }), }), - ...wireOptions, - // Spread after wireOptions so the snapped duration is - // authoritative (modelOptions.duration is folded into `duration` - // via snapDuration above). + // The remaining options spread after the size template so explicit + // aspect_ratio / resolution win over it; duration and the reference + // fields were destructured out above and re-added normalized. + ...generationOptions, ...(duration !== undefined && { duration }), } @@ -433,35 +445,37 @@ export class GrokVideoAdapter< `prompt parts are not supported by ${endpoint}.`, ) } + + // Pull every generation-only key out of the wire options so nothing can + // leak into the edit/extend body via the spread below. JSON-serialized + // `null` values (a common "unset" encoding) are treated as absent; + // actual values are rejected with actionable errors. + const { + aspect_ratio: aspectRatio, + resolution, + duration: modeDuration, + reference_images: referenceImagesOption, + reference_audios: referenceAudiosOption, + ...passthrough + } = wireOptions if ( - wireOptions.reference_images !== undefined || - wireOptions.reference_audios !== undefined + (referenceImagesOption?.length ?? 0) > 0 || + (referenceAudiosOption?.length ?? 0) > 0 ) { throw new Error( `${this.name}: reference inputs are only supported by video ` + `generation, not '${mode}' mode.`, ) } - if ( - args.size !== undefined || - wireOptions.aspect_ratio !== undefined || - wireOptions.resolution !== undefined - ) { + if (args.size !== undefined || aspectRatio != null || resolution != null) { throw new Error( `${this.name}: '${mode}' mode does not accept size / aspect_ratio / ` + `resolution — the output inherits the source clip's geometry ` + `(capped at 720p).`, ) } - - const { - aspect_ratio: _aspectRatio, - resolution: _resolution, - duration: modeDuration, - ...passthrough - } = wireOptions const rawDuration = modeDuration ?? args.genericDuration - if (mode === 'edit' && rawDuration !== undefined) { + if (mode === 'edit' && rawDuration != null) { throw new Error( `${this.name}: 'edit' mode does not accept a duration — the output ` + `inherits the source clip's length. Use mode 'extend' to append ` + @@ -470,7 +484,7 @@ export class GrokVideoAdapter< } // Extend: the snapped duration is the added-tail length (1–15s). const duration = - rawDuration !== undefined ? this.snapDuration(rawDuration) : undefined + rawDuration != null ? this.snapDuration(rawDuration) : undefined const request = { model, @@ -511,15 +525,13 @@ export class GrokVideoAdapter< }) if (!response.ok) { throw new Error( - `grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, + `grok: ${endpoint} request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, ) } const result = (await response.json()) as GrokVideoCreateResponse if (!result.request_id) { - throw new Error( - 'grok: video generation response contained no request_id', - ) + throw new Error(`grok: ${endpoint} response contained no request_id`) } return { jobId: result.request_id, model } } catch (error: unknown) { diff --git a/packages/ai-grok/src/video/video-provider-options.ts b/packages/ai-grok/src/video/video-provider-options.ts index 3223a72e4d..ca279e1575 100644 --- a/packages/ai-grok/src/video/video-provider-options.ts +++ b/packages/ai-grok/src/video/video-provider-options.ts @@ -305,6 +305,18 @@ export const GROK_VIDEO_MAX_REFERENCE_AUDIOS = 3 */ export const GROK_VIDEO_MAX_REFERENCE_IMAGES = 7 +/** + * Model names whose per-model options declare the reference fields. Keeps + * the runtime set below provably in sync with + * {@link GrokVideoModelProviderOptionsByName} — a typo or a new + * reference-capable model missing from the set is a compile error. + */ +type GrokVideoReferenceModel = { + [TModel in GrokVideoModel]: 'reference_images' extends keyof GrokVideoModelProviderOptionsByName[TModel] + ? TModel + : never +}[GrokVideoModel] + /** * Models that support reference-to-video inputs (`reference_images` / * `reference_audios`). The per-model provider-options map hides the fields @@ -314,9 +326,8 @@ export const GROK_VIDEO_MAX_REFERENCE_IMAGES = 7 * * @experimental Video generation is an experimental feature and may change. */ -const GROK_VIDEO_REFERENCE_MODELS: ReadonlySet = new Set([ - 'grok-imagine-video-1.5', -]) +const GROK_VIDEO_REFERENCE_MODELS: ReadonlySet = + new Set(['grok-imagine-video-1.5']) /** * True when the model accepts reference-to-video inputs. @@ -327,6 +338,16 @@ export function isGrokVideoReferenceModel(model: string): boolean { return GROK_VIDEO_REFERENCE_MODELS.has(model) } +/** + * Model names whose per-model options declare `mode`. Same + * provably-in-sync construction as {@link GrokVideoReferenceModel}. + */ +type GrokVideoSourceModel = { + [TModel in GrokVideoModel]: 'mode' extends keyof GrokVideoModelProviderOptionsByName[TModel] + ? TModel + : never +}[GrokVideoModel] + /** * Models that accept a source-video prompt part for `/v1/videos/edits` * and `/v1/videos/extensions`. xAI lists video input only on @@ -334,9 +355,8 @@ export function isGrokVideoReferenceModel(model: string): boolean { * * @experimental Video generation is an experimental feature and may change. */ -const GROK_VIDEO_SOURCE_MODELS: ReadonlySet = new Set([ - 'grok-imagine-video', -]) +const GROK_VIDEO_SOURCE_MODELS: ReadonlySet = + new Set(['grok-imagine-video']) /** * True when the model accepts edit / extend source-video jobs. diff --git a/packages/ai-grok/tests/video-adapter.test.ts b/packages/ai-grok/tests/video-adapter.test.ts index dbe6ccc02b..d31188b07e 100644 --- a/packages/ai-grok/tests/video-adapter.test.ts +++ b/packages/ai-grok/tests/video-adapter.test.ts @@ -425,10 +425,58 @@ describe('Grok Video Adapter', () => { logger: testLogger, }), ).rejects.toThrow( - /video generation request failed \(400.*Duration must be between 1 and 15 seconds/, + /\/videos\/generations request failed \(400.*Duration must be between 1 and 15 seconds/, ) }) + it('strips a JSON-null duration instead of sending it', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + const adapter = adapterWithFetch(fetchMock) + + await adapter.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: i2vPrompt(), + // Serializers commonly encode "unset" as null; it must read as + // absent, not go on the wire as `"duration": null`. + // @ts-expect-error runtime handling of a JSON-null option + modelOptions: { duration: null }, + logger: testLogger, + }) + + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body).not.toHaveProperty('duration') + }) + + it('treats null / empty reference fields as unset', async () => { + const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) + + // Empty array on 1.5: nothing sent. + const adapter15 = adapterWithFetch(fetchMock) + await adapter15.createVideoJob({ + model: 'grok-imagine-video-1.5', + prompt: 'p', + modelOptions: { reference_audios: [] }, + logger: testLogger, + }) + let body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body).not.toHaveProperty('reference_audios') + + // JSON-null on v1.0: reads as unset — no reference-gate throw, no + // field on the wire. + const adapter10 = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + await adapter10.createVideoJob({ + model: 'grok-imagine-video', + prompt: 'p', + // @ts-expect-error runtime handling of a JSON-null option + modelOptions: { reference_images: null }, + logger: testLogger, + }) + body = JSON.parse(String(fetchMock.mock.calls[1]![1]?.body)) + expect(body).not.toHaveProperty('reference_images') + }) + it('throws when the response carries no request_id', async () => { const fetchMock = mockFetch(() => jsonResponse({})) const adapter = adapterWithFetch(fetchMock) @@ -888,6 +936,24 @@ describe('Grok Video Adapter', () => { expect(fetchMock).not.toHaveBeenCalled() }) + it('names the edit endpoint in failure messages', async () => { + const fetchMock = mockFetch(() => + jsonResponse({ code: 'invalid-argument', error: 'bad clip' }, 400), + ) + const adapter = createGrokVideo('grok-imagine-video', 'test-api-key', { + fetch: fetchMock, + }) + + await expect( + adapter.createVideoJob({ + model: 'grok-imagine-video', + prompt: [{ type: 'text', content: 'p' }, sourceVideoPart], + modelOptions: { mode: 'edit' }, + logger: testLogger, + }), + ).rejects.toThrow(/\/videos\/edits request failed \(400.*bad clip/) + }) + it('rejects an unknown mode instead of misrouting to generations', async () => { const fetchMock = mockFetch(() => jsonResponse({ request_id: 'r' })) const adapter = sourceAdapterWithFetch(fetchMock) diff --git a/packages/ai/skills/ai-core/media-generation/SKILL.md b/packages/ai/skills/ai-core/media-generation/SKILL.md index 074c30bffc..15aef21f7e 100644 --- a/packages/ai/skills/ai-core/media-generation/SKILL.md +++ b/packages/ai/skills/ai-core/media-generation/SKILL.md @@ -546,11 +546,15 @@ const edited = await generateVideo({ Other video adapters: `openaiVideo('sora-2')` (pixel sizes like `'1280x720'`, durations 4/8/12s, single `input_reference` image prompt part), `grokVideo(...)` (`grok-imagine-video` and `grok-imagine-video-1.5` both do text-to-video + image-to-video; -1.5 adds reference-to-video — `'reference'`-roled image parts → `reference_images`, preset -voices via `modelOptions.reference_audios` (max 3); only `grok-imagine-video` edits/extends a source `video` -prompt part via `modelOptions.mode: 'edit' | 'extend'` (extend `duration` = added tail); -aspect-ratio size template like `'16:9_720p'`, integer durations 1-15s, reports -`usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — +1.5 adds reference-to-video — `'reference'`/`'character'`-roled image parts → +`reference_images` (max 7), preset voices via `modelOptions.reference_audios` (max 3) — +1.5-only, capped at 720p, and not combinable with a starting-frame image; only +`grok-imagine-video` edits/extends a source `video` prompt part via +`modelOptions.mode: 'edit' | 'extend'` (extend `duration` = added tail). Edit/extend +outputs inherit the source clip's properties, so `size`/`aspect_ratio`/`resolution` +throw in both modes and `duration` throws in edit mode — pass none of them there; +generation uses the aspect-ratio size template like `'16:9_720p'` (1080p is 1.5-only), +integer durations 1-15s, reports `usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance — aspect-ratio size template like `'16:9_720p'`, durations 4-15s on the 2.0 family, 4-12s on 1.5-pro, 2-12s on the 1.0-pro models; reads `ARK_API_KEY`), `openRouterVideo(...)` (OpenRouter's dedicated `POST /api/v1/videos` gateway),