ShadowForge

Turn a JSON lesson script into a polished, word-by-word English shadowing video — fully automated, no manual editing.

ShadowForge converts a simple JSON file of sentences into a beautiful 1080p (or any resolution) MP4 with natural AI narration and deterministic, frame-accurate on-screen word highlighting — the kind of video used for English shadowing / ESL practice content.

JSON → Edge-TTS speech → audio duration → Canvas animation → frame capture → FFmpeg → MP4

Important

Video processing can take some time. Rendering and processing time depends on the total number of scenes being generated. Simply provide the JSON story, sit back, and enjoy the process! 🎬

---

✨ Features

• Zero manual timing — scene duration is derived automatically from the generated speech audio. You never type a duration.
• Deterministic rendering — every frame is computed purely from (text, frameTime, duration), so output is 100% reproducible and never depends on real-time playback or browser speed.
• Premium visual style — dark gradient background, glow effects, smooth word-reveal animation, and a progress indicator, built with plain Canvas 2D (no frameworks).
• Natural voices — powered by Microsoft Edge's neural text-to-speech via edge-tts (free, no API key).
• Minimal footprint — the entire project is 3 files. No database, no server, no build step, no config system.
• Direct FFmpeg control — frames → video → audio mux → concat, all via subprocess, with sensible H.264/AAC settings for MP4 playback everywhere.

---

📁 Project structure

shadow-video/
├── main.py # Full pipeline: TTS, duration detection, browser automation, FFmpeg encoding
├── renderer.html # Self-contained Canvas 2D animation (HTML + CSS + vanilla JS)
├── input.json # Lesson script: title, voice, resolution, fps, and scene text
└── output/ # Final rendered video is written here

That's it — no other source files.

---

🚀 Getting started

Prerequisites

Requirement	Notes
Python 3.9+
FFmpeg	Must be on your PATH — both ffmpeg and ffprobe need to run from a terminal
Google Chrome or Chromium	Installed locally; Selenium 4.6+ auto-manages the matching driver, no manual ChromeDriver setup required

Installation

pip install edge-tts selenium

Run

python main.py

You'll see progress printed as the pipeline runs:

[1/5] Reading JSON...
[2/5] Generating speech...
[3/5] Rendering scenes...
[4/5] Encoding video...
[5/5] Complete!
Output: output/shadowing_video.mp4

The finished video appears at output/shadowing_video.mp4.

---

⚙️ Configuration

Everything is controlled from input.json — no other file needs to change for everyday use.

{
"title": "English Shadowing Practice",
"voice": "en-US-AriaNeural",
"fps": 30,
"width": 1920,
"height": 1080,
"scenes": [
{ "text": "I have been studying English for two years." },
{ "text": "Although it was raining, we went outside." }
]
}

Field	Description
title	Lesson title (for your own reference)
voice	Any valid Edge-TTS voice ID, e.g. en-GB-RyanNeural, en-US-GuyNeural
fps	Output frame rate
width / height	Output resolution — read directly by the renderer, no HTML edits needed
scenes	An array of { "text": "..." } objects. No duration field — each scene's length is taken automatically from its generated audio

Adding or editing lesson text

Just add more objects to the scenes array:

{ "text": "She has never seen snow before." }

Changing the voice

Browse available voices with:

edge-tts --list-voices

and set the voice field to any voice ID from that list.

Changing resolution / frame rate

Update width, height, and fps in input.json. The renderer reads these from the page URL that Python generates, so the animation automatically adapts.

---

🧠 How it works

1. Read input.json and validate the scene list.
2. Generate speech for each scene with edge-tts, saved as MP3.
3. Measure exact duration of each MP3 via ffprobe — this becomes the scene's canonical duration.
4. Launch headless Chrome via Selenium and load renderer.html once.
5. For each scene, step through every frame (frameTime = frameNumber / fps) and call window.renderFrame(text, frameTime, duration) in the page.
6. Capture each frame by reading the canvas's exact pixel buffer with canvas.toDataURL() — this guarantees pixel-perfect, correctly-sized frames regardless of OS display scaling.
7. Encode frames to video with FFmpeg (libx264, yuv420p), then mux in the matching audio for that scene.
8. Concatenate all scenes into the final MP4 using FFmpeg's concat demuxer.
9. Clean up all temporary audio/frame/video files, leaving only the final output.

The animation itself is fully deterministic: given the same (text, currentTime, duration) triple, renderFrame() always paints the exact same pixels — which is what makes frame-by-frame rendering (rather than real-time screen recording) possible and reliable.

---

🛠️ Troubleshooting

• ffmpeg/ffprobe not found — install FFmpeg and confirm ffmpeg -version works in your terminal.
• Chrome/driver errors — make sure Chrome is installed; Selenium Manager downloads the driver automatically on first run (requires internet access once).
• Choppy or garbled audio — check your internet connection; edge-tts streams audio from Microsoft's service at generation time.

---

authour:

Mehedi Murshid

📜 License

MIT — do whatever you like with it, attribution appreciated but not required.