Format-taxonomy example videos (12 subcategories) — generation-quality + analysis-eval fixtures

[johnhenry]

Paused mid-execution to prioritize shepherding platform PRs #15526/#15529/#15530/#15528 to main. Picking this back up: resume from Wave 1 storyboard generation.

Goal

Validate that showrunner — driven by a coding agent — can generate consistently high-quality content for specific, pre-defined video-type pipelines, per the boss's direction: "socialgpt -> showrunner" (or showrunner alone) needs to be proven out before the platform integration (scrollmark/platform#15532) is worth pursuing further.

12 example videos, one per short-form format subcategory (GRWM, storytime, POV, greenscreen reaction, faceless explainer, Reddit-TTS-over-gameplay, recipe, unboxing, ASMR, duet, lip-sync/dance, multi-character skit). Each example is both a generation-quality benchmark (scored against docs/quality-rubric.md) and an analysis-eval fixture (each format targets a specific SocialGPT analysis capability).

Full plan: see the design doc (recreate from conversation history if the local plan file at ~/.claude/plans/as-we-wait-for-rosy-pumpkin.md isn't available) — summary below.

Status so far

• Wave 0 — scaffolding merged to main: examples/taxonomy.md, examples/README.md, examples/storyboards/*.json (6 of 7 Wave-1 storyboards authored: 1.1 GRWM, 1.2 storytime, 1.3 POV, 3.1 recipe, 3.2 unboxing, 4.2 lip-sync), examples/eval-cards/*.md (all 12), examples/results/manifest.json.
• E2 (MiniMax provider modernization + clip normalization) — implemented, tested, PR open: feat(video): modernize MiniMax provider (Hailuo-02) + conform clips to the storyboard #62.
• E1 (per-scene TTS voice) — Scene.voice schema field added on feat/per-scene-voice (uncommitted call-site wiring into generate_all_narrations in both ai_video/assets.py and faceless_explainer/assets.py — schema is committed, wiring is not). Branch is parked, not pushed.
• E3 (local-asset ingestion, file:// scenes) — not started.
• E4 (FFmpeg compositing layer + new composite format: overlay/PiP/chromakey/hstack-vstack) — not started. This is the largest remaining piece.
• E5 (Veo native-audio passthrough) — not started.
• Wave 1 — 7 examples generatable with existing engines once E2 merges: GRWM, storytime, POV, faceless explainer (LLM-planned, no storyboard needed), recipe, unboxing, lip-sync (deliberate negative fixture). Storyboards authored; none rendered yet.
• Wave 2 — 5 examples blocked on E1/E3/E4/E5: multi-character skit (E1), greenscreen reaction (E4), Reddit-TTS (E3+E4), duet (E4), ASMR (E5, on Veo not MiniMax).
• Wave 3 — score all 12 via showrunner analyze --sync, grade against each eval card's target + the quality rubric, iterate failures with refine, fill in examples/README.md results table.

Cost estimate

9 MiniMax ai-video examples ($2.40 each + retries) + 1 Veo ASMR (~$16) + faceless/composite examples (LLM+TTS only) ≈ $50-70 total.

Resuming

1. Merge/land feat(video): modernize MiniMax provider (Hailuo-02) + conform clips to the storyboard #62 (E2) first — it's a soft prerequisite for correct 9:16 output on all ai-video examples.
2. Generate the 7 Wave-1 examples (storyboards already committed to examples/storyboards/).
3. Build E1/E3/E4/E5 (E4 is the real build — new FFmpeg filtergraph module + formats/composite/).
4. Generate the remaining 5 Wave-2 examples.
5. Score everything, iterate, write up examples/README.md results.

[johnhenry]

Resuming. E2 (#62) merged to main at 503c219 — the 9:16 aspect-ratio prerequisite for Wave 1 is in place. Moving to Wave 1: generate the 7 storyboarded examples.

[johnhenry]

Example 1.1 (GRWM) fully generated: 35s, 7 MiniMax clips + kokoro TTS + burned captions.

Getting there surfaced 4 real gaps, now fixed/documented:

1. Real bug (fixed, fix(cli): --storyboard now runs the full pipeline, not just the plan stage #64, merged): `create --storyboard` loaded the plan, printed it, and returned — never ran assets/compose/render. Every storyboard-driven example was silently no-op'ing (exit 0, nothing generated).
2. Real bug (fixed, fix(video): raise on MiniMax base_resp errors instead of silently timing out #65, merged): MiniMax's `base_resp` app-level errors (e.g. a plan/quota rejection) come back HTTP 200 with an empty task_id — the provider never checked `base_resp.status_code`, so a rejected submit polled a bogus empty id for the full 600s timeout instead of failing immediately.
3. Local env gap: this account's MiniMax plan doesn't support 1080P (`status_code 2061`) — set via `.showrunner.yaml`'s `minimax: {resolution: 768P}` (note: per-provider sections are top-level keys named after the provider, not nested under a literal `provider_config:` key — tripped me up once).
4. Local env gaps: `kokoro` needs Python <3.13 (created `.venv312` via mise's Python 3.12.13); Homebrew's default `ffmpeg` formula has no libass, so the `--captions` burn-in silently fails with a confusing "No option name near ..." parser error — installed `ffmpeg-full` (also in homebrew-core, no extra tap) and prepended it to PATH for these runs.

Moving on to the remaining 5 storyboard-driven Wave-1 examples (1.2 storytime, 1.3 POV, 3.1 recipe, 3.2 unboxing, 4.2 lip-sync) with the now-working setup.

[johnhenry]

Wave 1 complete — all 7 examples generated. Final gap found + fixed along the way:

5. Real bug (fixed, fix(render): resolve relative output_path against caller cwd, not work_dir #66, merged): `RemotionRenderProvider.render()` passed a relative `--output` path straight into the `npx remotion render` subprocess, which runs with `cwd=work_dir`. The path resolved against the subprocess's cwd instead of the caller's — the mp4 silently landed inside the ephemeral work_dir instead of the requested path. No error, no crash: the pipeline reports `done`/"Video rendered: ..." as if everything worked. Only hit `faceless-explainer` (the only Remotion-path format) — `ai-video`'s FFmpeg provider doesn't set `cwd=`, so it was unaffected. Fixed by resolving to absolute before the subprocess call.

Results (generation only — Wave 3 scoring/rubric still pending)

#	Example	Clips	Cost
1.1 GRWM	7	(first run, cost not tracked pre-resume)
1.2 Storytime	7	$2.52
1.3 POV	6	$2.16
3.1 Recipe	8	$2.88
3.2 Unboxing	7	$2.52
4.2 Lip-sync	6	$2.16
2.2 Faceless explainer	— (LLM+Remotion)	$0.49

All 7 delivered to the user as mp4s. All in `examples/output/`.

Summary of everything fixed this session (5 real bugs/gaps)

1. `create --storyboard` never ran past the plan stage (fix(cli): --storyboard now runs the full pipeline, not just the plan stage #64)
2. MiniMax `base_resp` errors silently swallowed → 10min timeout instead of immediate failure (fix(video): raise on MiniMax base_resp errors instead of silently timing out #65)
3. Local: MiniMax plan doesn't support 1080P → `.showrunner.yaml`'s `minimax: {resolution: 768P}` (note: NOT nested under a `provider_config:` key)
4. Local: `kokoro` needs Python <3.13 (`.venv312`); Homebrew's default `ffmpeg` has no libass (installed `ffmpeg-full`, prepended to PATH)
5. Remotion render silently wrote to the wrong path for relative `--output` (fix(render): resolve relative output_path against caller cwd, not work_dir #66)

Next

E1/E3/E4/E5 (per-scene voice, asset ingestion, composite format, Veo audio) before Wave 2's 5 remaining examples, then Wave 3 scoring.