feat(ai-grok): catch up Grok Imagine image, video, and voice modes

.changeset/anthropic-opus-5-tool-map.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-anthropic': patch
+---
+
+Register `claude-opus-5` and `claude-opus-5-fast` on the per-model tool-capabilities map so they match `ANTHROPIC_MODELS`.

.changeset/grok-imagine-catch-up.md

@@ -0,0 +1,9 @@
+---
+'@tanstack/ai-grok': minor
+---
+
+Catch up with the current xAI Imagine / Voice catalog:
+
+- **Image**: add `grok-imagine-image-2.0` (xAI's recommended model, $0.04/image) with its 2.0-only `quality: 'low' | 'medium'` provider option.
+- **Video**: `grok-imagine-video-1.5` now supports text-to-video (the stale image-to-video-only guard is removed). Reference-to-video lands via image prompt parts with `metadata.role: 'reference' | 'character'` (→ `reference_images`) and preset voices via `modelOptions.reference_audios` (max 3) — 1.5-only, typed per model and gated at runtime. Image-to-video and reference-to-video cannot be combined. Video editing and extension land on `grok-imagine-video` only, via a source `video` prompt part plus `modelOptions.mode: 'edit' | 'extend'` (`/v1/videos/edits` / `/v1/videos/extensions`; in extend mode `duration` is the added tail, not the total). Because edit/extend outputs inherit the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` (and `duration` in edit mode) in those modes instead of sending fields the API ignores.
+- **Voice**: add `grok-voice-think-fast-2.0` (current recommended) and the `grok-voice-latest` alias to the realtime models; the realtime token and adapter defaults move off the deprecated 1.0 ids to `grok-voice-think-fast-2.0`.

docs/adapters/grok.md

@@ -181,18 +181,21 @@ const result = await generateImage({
 console.log(result.images);
 ```
 
-The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-quality`)
-are aspect-ratio sized — `size` takes an `aspectRatio_resolution` template
-like `"16:9_2k"` (the `_2k` suffix is optional):
+The grok-imagine models (`grok-imagine-image`, `grok-imagine-image-2.0`,
+`grok-imagine-image-quality`) are aspect-ratio sized — `size` takes an
+`aspectRatio_resolution` template like `"16:9_2k"` (the `_2k` suffix is
+optional). `grok-imagine-image-2.0` is xAI's recommended model and adds a
+2.0-only `quality` provider option (`'low' | 'medium'`, default `'medium'`):
 
 ```typescript
 import { generateImage } from "@tanstack/ai";
 import { grokImage } from "@tanstack/ai-grok";
 
 const result = await generateImage({
- adapter: grokImage("grok-imagine-image"),
+ adapter: grokImage("grok-imagine-image-2.0"),
  prompt: "A futuristic cityscape at sunset",
  size: "16:9_2k",
+ modelOptions: { quality: "medium" },
 });
 ```
 
@@ -238,16 +241,16 @@ Generate short video clips (1–15 seconds, with audio) with the Grok Imagine vi
 
 Available models:
 
-- `grok-imagine-video` (v1.0) — text-to-video and image-to-video, $0.05 per second of video.
-- `grok-imagine-video-1.5` — **image-to-video only**, $0.08 per second of video. A text-only prompt is rejected by the API; the adapter fails fast with a clear error telling you to add a starting-frame image or use `grok-imagine-video`.
+- `grok-imagine-video` (v1.0) — text-to-video, image-to-video, and source-video edit / extend, $0.05 per second of video.
+- `grok-imagine-video-1.5` — xAI's recommended default, $0.08 per second of video. Supports text-to-video (with native 1080p), image-to-video, and reference-to-video. It does not accept a source video.
 
-Text-to-video with the base `grok-imagine-video` model:
+Text-to-video:
 
 ```typescript
 import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
 import { grokVideo } from "@tanstack/ai-grok";
 
-const adapter = grokVideo("grok-imagine-video");
+const adapter = grokVideo("grok-imagine-video-1.5");
 
 // 1. Create the job
 const { jobId } = await generateVideo({
@@ -267,7 +270,7 @@ while (status.status !== "completed" && status.status !== "failed") {
 console.log(status.url); // hosted .mp4 URL
 ```
 
-For image-to-video (required for `grok-imagine-video-1.5`, optional for `grok-imagine-video`), include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame:
+For image-to-video, include an `image` prompt part as the starting frame and describe the desired motion in the text part. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame:
 
 ```typescript
 import { generateVideo } from "@tanstack/ai";
@@ -290,7 +293,78 @@ const { jobId } = await generateVideo({
 });
 ```
 
-Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional.
+Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and (on `grok-imagine-video-1.5` text-to-video / image-to-video only) `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional.
+
+### Reference-to-Video
+
+On `grok-imagine-video-1.5`, image prompt parts with `metadata.role: 'reference'` (or `'character'`) become `reference_images` — they guide subjects and style without locking the first frame, and are addressed from the prompt text as `<IMAGE_0>`, `<IMAGE_1>`, … in request order. Preset TTS voices (up to 3) can be referenced for generated speech via `modelOptions.reference_audios`, addressed as `<AUDIO_0>`, `<AUDIO_1>`, `<AUDIO_2>`. Reference-to-video output is capped at 720p. A starting-frame image and reference inputs cannot be combined — xAI rejects that mix with 400. Reference inputs are a 1.5-only feature — the adapter rejects them on `grok-imagine-video`:
+
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { grokVideo } from "@tanstack/ai-grok";
+
+const { jobId } = await generateVideo({
+ adapter: grokVideo("grok-imagine-video-1.5"),
+ prompt: [
+ {
+ type: "text",
+ content: "<IMAGE_0> walks through a neon-lit alley while <AUDIO_0> narrates",
+ },
+ {
+ type: "image",
+ source: { type: "url", value: "https://example.com/character.png" },
+ metadata: { role: "reference" },
+ },
+ ],
+ size: "16:9_720p",
+ modelOptions: {
+ reference_audios: [{ voice_id: "eve" }],
+ },
+});
+```
+
+### Video Editing and Extension
+
+`grok-imagine-video` (v1.0) can rewrite or continue an existing clip. `grok-imagine-video-1.5` has no video input — the adapter rejects a source-video part or `mode` on that model. Pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`:
+
+- `mode: 'edit'` posts to `/v1/videos/edits` — modifies only what the prompt asks for, keeping the rest of the clip intact. Duration, aspect ratio, and resolution are inherited from the source (capped at 720p), so the adapter rejects `size`, `aspect_ratio`, `resolution`, and `duration` in this mode rather than sending fields the API ignores.
+- `mode: 'extend'` posts to `/v1/videos/extensions` — continues the clip. `duration` is the length of the **added tail**, not the total: extending a 10-second clip with `duration: 5` yields 15 seconds. Output geometry is still inherited from the source, so `size` / `aspect_ratio` / `resolution` are rejected here too.
+
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { grokVideo } from "@tanstack/ai-grok";
+
+const adapter = grokVideo("grok-imagine-video");
+
+// Edit: change the clip in place
+const edit = await generateVideo({
+ adapter,
+ prompt: [
+ { type: "text", content: "Make the sky stormy with distant lightning" },
+ {
+ type: "video",
+ source: { type: "url", value: "https://example.com/clip.mp4" },
+ },
+ ],
+ modelOptions: { mode: "edit" },
+});
+
+// Extend: append 5 more seconds
+const extension = await generateVideo({
+ adapter,
+ prompt: [
+ { type: "text", content: "The camera keeps panning right across the bay" },
+ {
+ type: "video",
+ source: { type: "url", value: "https://example.com/clip.mp4" },
+ },
+ ],
+ duration: 5, // added seconds, not the total
+ modelOptions: { mode: "extend" },
+});
+```
+
+Both return the usual `{ jobId }` and are polled like any other Grok video job.
 
 When the job completes, the adapter reports usage on the result: `usage.unitsBilled` carries the billed seconds of video and `usage.cost` the exact cost in USD, both as returned by the xAI API.
 
@@ -333,7 +407,7 @@ console.log(result.text);
 
 ## Realtime Voice
 
-Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow.
+Grok also exposes a Realtime voice adapter (`grokRealtime`) and a token issuer (`grokRealtimeToken`) for low-latency voice conversations. The default model is `grok-voice-think-fast-2.0` (xAI's current recommended speech-to-speech model); `grok-voice-latest` always points at the newest model. The 1.0 ids remain accepted for compatibility, but xAI has deprecated `grok-voice-think-fast-1.0`. See [Realtime Voice Chat](../media/realtime-chat) for the end-to-end flow.
 
 ## Environment Variables
 

docs/config.json

@@ -443,7 +443,7 @@
  "label": "Video Generation",
  "to": "media/video-generation",
  "addedAt": "2026-04-15",
- "updatedAt": "2026-08-13"
+ "updatedAt": "2026-08-18"
  },
  {
  "label": "Generation Hooks",
@@ -808,7 +808,7 @@
  "label": "Grok (xAI)",
  "to": "adapters/grok",
  "addedAt": "2026-04-15",
- "updatedAt": "2026-06-24"
+ "updatedAt": "2026-08-18"
  },
  {
  "label": "Groq",

docs/media/video-generation.md

@@ -49,7 +49,7 @@ Currently supported:
 
 - **OpenAI**: Sora-2 and Sora-2-Pro models (when available)
 - **Google Gemini**: Veo 3.1 models (via the long-running operations API), and Gemini Omni Flash (via the Interactions API)
-- **Grok (xAI)**: grok-imagine-video (text-to-video + image-to-video) and grok-imagine-video-1.5 (image-to-video only) models
+- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5 (text-to-video, image-to-video; 1.5 adds reference-to-video; v1.0 adds editing and extension)
 - **BytePlus**: Seedance 2.0, 1.5-pro and 1.0-pro models (text-to-video, first/last frame, and multimodal references on 2.0)
 - **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models
 - **OpenRouter**: Seedance, Veo 3.1, Wan, Kling, Sora 2 Pro and others via the dedicated async video API (`POST /api/v1/videos`)
@@ -691,16 +691,16 @@ instead of letting the model infer the task mode).
 
 #### Grok (xAI Imagine) Model Options
 
-Based on the [xAI video generation API](https://docs.x.ai/docs/guides/video-generations). Two models are available: `grok-imagine-video` (v1.0) supports **text-to-video and image-to-video**, while `grok-imagine-video-1.5` is **image-to-video only** (a text-only prompt is rejected by the API; the adapter throws a clear error pointing you at `grok-imagine-video`). Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long.
+Based on the [xAI video generation API](https://docs.x.ai/developers/model-capabilities/video/generation). Two models are available: `grok-imagine-video` (v1.0) and `grok-imagine-video-1.5` (xAI's recommended default, with native 1080p text-to-video). Both support **text-to-video and image-to-video**; 1.5 adds **reference-to-video**. **Video editing and extension** are `grok-imagine-video` only — 1.5 has no video input. Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long.
 
-Text-to-video with the base model:
+Text-to-video:
 
 ```typescript
 import { generateVideo } from "@tanstack/ai";
 import { grokVideo } from "@tanstack/ai-grok";
 
 const { jobId } = await generateVideo({
- adapter: grokVideo("grok-imagine-video"),
+ adapter: grokVideo("grok-imagine-video-1.5"),
  prompt: "A beautiful sunset over the ocean",
  size: "16:9_720p", // aspect ratio: '1:1' | '16:9' | '9:16' | '4:3' | '3:4' | '3:2' | '2:3'
  // resolution (optional suffix): '480p' | '720p' | '1080p'
@@ -713,7 +713,7 @@ const { jobId } = await generateVideo({
 });
 ```
 
-Image-to-video (required for `grok-imagine-video-1.5`) — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame:
+Image-to-video — include an `image` prompt part as the starting frame. URL sources are fetched by xAI's servers (so they must be publicly reachable); use a `data` source for a base64 starting frame:
 
 ```typescript
 import { generateVideo } from "@tanstack/ai";
@@ -733,12 +733,53 @@ const { jobId } = await generateVideo({
 });
 ```
 
+Reference-to-video (`grok-imagine-video-1.5` only, output capped at 720p) — image prompt parts with `metadata.role: 'reference'` or `'character'` become `reference_images` (addressed from the prompt as `<IMAGE_0>`, `<IMAGE_1>`, …), and up to 3 preset TTS voices can be referenced via `modelOptions.reference_audios` (addressed as `<AUDIO_0>`, …):
+
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { grokVideo } from "@tanstack/ai-grok";
+
+const { jobId } = await generateVideo({
+ adapter: grokVideo("grok-imagine-video-1.5"),
+ prompt: [
+ { type: "text", content: "<IMAGE_0> waves at the camera while <AUDIO_0> says hello" },
+ {
+ type: "image",
+ source: { type: "url", value: "https://example.com/character.png" },
+ metadata: { role: "reference" },
+ },
+ ],
+ size: "16:9_720p",
+ modelOptions: { reference_audios: [{ voice_id: "eve" }] },
+});
+```
+
+Video editing and extension (`grok-imagine-video` only) — pass the source clip as a `video` prompt part and pick the mode with `modelOptions.mode`. `'edit'` (`/v1/videos/edits`) modifies only what the prompt asks for and inherits duration / aspect ratio / resolution from the source (capped at 720p); `'extend'` (`/v1/videos/extensions`) continues the clip, with `duration` meaning the length of the **added tail**, not the total. Because the output inherits the source clip's properties, the adapter rejects `size` / `aspect_ratio` / `resolution` in both modes (and `duration` in edit mode) instead of sending fields the API ignores. The adapter rejects a source-video part or `mode` on `grok-imagine-video-1.5`.
+
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { grokVideo } from "@tanstack/ai-grok";
+
+const { jobId } = await generateVideo({
+ adapter: grokVideo("grok-imagine-video"),
+ prompt: [
+ { type: "text", content: "The camera keeps panning right across the bay" },
+ {
+ type: "video",
+ source: { type: "url", value: "https://example.com/clip.mp4" },
+ },
+ ],
+ duration: 5, // 'extend' mode: seconds added to the clip, not the total
+ modelOptions: { mode: "extend" },
+});
+```
+
 Both models accept any whole second in the **1–15** range. A raw `duration` is coerced into that range rather than rejected — values are clamped to `[1, 15]` and rounded to the nearest second. Inspect or pre-snap the range the same way as Veo:
 
 ```typescript
 import { grokVideo } from "@tanstack/ai-grok";
 
-const adapter = grokVideo("grok-imagine-video");
+const adapter = grokVideo("grok-imagine-video-1.5");
 
 adapter.availableDurations(); // { kind: 'range', min: 1, max: 15, step: 1, unit: 'seconds' }
 adapter.snapDuration(2.5); // 3 — clamped/rounded into range

examples/ts-react-chat/src/lib/use-realtime.ts

@@ -42,7 +42,7 @@ const getRealtimeTokenFn = createServerFn({ method: 'POST' })
 
  if (data.provider === 'grok') {
  return realtimeToken({
- adapter: grokRealtimeToken({ model: 'grok-voice-fast-1.0' }),
+ adapter: grokRealtimeToken({ model: 'grok-voice-think-fast-2.0' }),
  })
  }
 

examples/ts-react-media/src/lib/models.ts

@@ -23,6 +23,14 @@ export const IMAGE_MODELS = [
  sizeType: 'aspect_ratio' as const,
  provider: 'xai' as const,
  },
+ {
+ id: 'grok-imagine-image-2.0',
+ name: 'Grok Imagine 2.0 (xAI Direct)',
+ description: 'xAI recommended Imagine model with the quality option',
+ defaultSize: '16:9' as const,
+ sizeType: 'aspect_ratio' as const,
+ provider: 'xai' as const,
+ },
  {
  id: 'grok-imagine-image-quality',
  name: 'Grok Imagine Quality (xAI Direct)',
@@ -149,11 +157,18 @@ export const VIDEO_MODELS = [
  mode: 'text-to-video' as const,
  provider: 'xai' as const,
  },
+ {
+ id: 'grok-imagine-video-1.5',
+ name: 'Grok Imagine Video 1.5 (Text-to-Video)',
+ description:
+ 'xAI recommended video model via the native grokVideo adapter (native 1080p text-to-video)',
+ mode: 'text-to-video' as const,
+ provider: 'xai' as const,
+ },
  {
  id: 'grok-imagine-video-1.5/image-to-video',
  name: 'Grok Imagine Video 1.5 (Image-to-Video)',
- description:
- 'Animate a starting frame via the native grokVideo adapter (1.5 is image-to-video only)',
+ description: 'Animate a starting frame via the native grokVideo adapter',
  mode: 'image-to-video' as const,
  provider: 'xai' as const,
  },

examples/ts-react-media/src/lib/server-functions.ts

@@ -167,6 +167,17 @@ export const generateImageFn = createServerFn({ method: 'POST' })
  size: '16:9',
  })
  }
+ case 'grok-imagine-image-2.0': {
+ // xAI's recommended Imagine model; `quality` is a 2.0-only option
+ // ('low' | 'medium', default 'medium').
+ return generateImage({
+ adapter: grokImage('grok-imagine-image-2.0'),
+ prompt: asImagePrompt(data.prompt),
+ numberOfImages: 1,
+ size: '16:9',
+ modelOptions: { quality: 'medium' },
+ })
+ }
  case 'grok-imagine-image-quality': {
  return generateImage({
  adapter: grokImage('grok-imagine-image-quality'),
@@ -339,6 +350,18 @@ function videoStreamForModel(data: VideoRequest): AsyncIterable<StreamChunk> {
  duration: 5,
  })
  }
+ case 'grok-imagine-video-1.5': {
+ // Direct xAI Imagine API — grok-imagine-video-1.5 is xAI's recommended
+ // default and supports text-to-video with native 1080p.
+ return generateVideo({
+ stream: true,
+ pollingInterval: VIDEO_POLL_INTERVAL_MS,
+ adapter: grokVideo('grok-imagine-video-1.5'),
+ prompt: asTextPrompt(data.prompt),
+ size: '16:9_1080p',
+ duration: 5,
+ })
+ }
  case 'dreamina-seedance-2-0-260128': {
  // BytePlus Seedance via ModelArk (ARK_API_KEY). `size` is a "ratio" or
  // "ratio_resolution" template; durations are 4-15 integer seconds.

packages/ai-anthropic/src/model-meta.ts

@@ -819,6 +819,8 @@ export type AnthropicChatModelToolCapabilitiesByName = {
  [CLAUDE_OPUS_4_8.id]: typeof CLAUDE_OPUS_4_8.supports.tools
  [CLAUDE_FABLE_5.id]: typeof CLAUDE_FABLE_5.supports.tools
  [CLAUDE_SONNET_5.id]: typeof CLAUDE_SONNET_5.supports.tools
+ [CLAUDE_OPUS_5.id]: typeof CLAUDE_OPUS_5.supports.tools
+ [CLAUDE_OPUS_5_FAST.id]: typeof CLAUDE_OPUS_5_FAST.supports.tools
 }
 
 /**

packages/ai-grok/src/adapters/image.ts

@@ -129,7 +129,8 @@ export class GrokImageAdapter<
  throw new Error(
  `grok: model "${model}" does not support image prompt parts. ` +
  `Image-conditioned generation requires an Imagine API model ` +
- `('grok-imagine-image' or 'grok-imagine-image-quality').`,
+ `('grok-imagine-image', 'grok-imagine-image-2.0' or ` +
+ `'grok-imagine-image-quality').`,
  )
  }
  return await this.editImages(options, resolved)