MarkSoft AutoShorts

Turn long videos — YouTube, Twitch, Kick VODs or your own files — into short-form clips (Shorts / Reels / TikTok) using only local tools. No cloud APIs for the pipeline, no sign-up. Publish to YouTube / TikTok / Facebook Reels / Instagram Reels when you're done.

---

What it does

• Transcribes with faster-whisper.
• Finds all good moments with a local LLM via Ollama (no fixed count — as many as the video has).
• Cuts, crops to 9:16, and burns captions with FFmpeg.
• Follows the subject — per-frame face/person tracking with temporal smoothing (MediaPipe / YOLOv8), so the 9:16 window pans with the subject instead of sitting on a single static position.
• Publishes to YouTube / TikTok / Facebook Reels / Instagram Reels via their APIs, or opens a browser with the file staged if you'd rather finalize by hand.
• 100% local for generation. Publishing uses the platform APIs only when you explicitly click Publish.

---

Architecture (2026 rewrite)

Three layers — the pipeline is unchanged, the shell around it is new:

┌─────────────────────────────┐
│ web/ React 19 + Vite 6 │ ← user-facing UI
│ + Tailwind 4 │
└──────────────┬──────────────┘
│ /api/*
┌──────────────┴──────────────┐
│ server.py FastAPI │ ← job queue, SSE progress, streaming
│ src/autoshorts/server/ │
└──────────────┬──────────────┘
│ function calls
┌──────────────┴──────────────┐
│ src/autoshorts/pipeline.py │ ← download → transcribe → highlight
│ + tracking / export / │ → follow-crop → export → publish
│ publish / ... │
└─────────────────────────────┘

A legacy Gradio UI (app_gradio.py) is still present during the transition; it will be removed once the React UI reaches feature parity.

---

Quick start

Prerequisites (install once)

Tool	Install
FFmpeg	Windows: download from ffmpeg.org and add to PATH. macOS: brew install ffmpeg. Linux: apt install ffmpeg.
Python 3.11+	https://www.python.org or pyenv / brew / apt.
Ollama	https://ollama.ai, then ollama pull mistral.
Node 20+	Only if you want to rebuild the web UI. https://nodejs.org.

Run it

# one-time
python -m venv .venv
.venv\Scripts\activate # Windows (or: source .venv/bin/activate on mac/linux)
pip install -r requirements.txt
# start the API + web UI (single process)
python server.py

Open http://localhost:8000.

If web/dist/ hasn't been built yet, the root route shows a JSON placeholder telling you to build the frontend — see the next section. The API is live either way at http://localhost:8000/docs.

Build the web UI

cd web
npm install
npm run build

Then re-open http://localhost:8000. In dev you can also run npm run dev against a live Vite server on :5173 — it proxies /api to FastAPI.

CLI (for headless runs)

python cli.py "https://youtube.com/watch?v=VIDEO_ID" -o ./my_shorts -n 10

Outputs my_shorts/short_1.mp4, … The CLI path supports the same pipeline but not the publishing step.

Legacy Gradio UI

Still works during the transition:

python app_gradio.py
# opens http://127.0.0.1:7860

It will be deleted once the React UI hits parity.

---

How clip selection works

1. Transcribe — faster-whisper emits segments (phrase-level start / end / text).
2. Select — Ollama reads the full segment list and returns every moment that would make a good short (hook + payoff, self-contained, duration bounded). There's a user-controlled cap but no floor — if the video only has two good moments, you get two.
3. Align boundaries —
   • Start: walked back to a sentence boundary (capped at 12s) so clips don't start mid-sentence.
   • End: extended to the next .?! (capped at ~8s of extra content + 3s padding) so punch lines aren't cut off.
4. Follow — for single-subject crops, a per-frame face/person tracker builds a time series of subject centers, smoothed with EMA + deadzone + max-speed clamp. The export stage crops the source frame-by-frame so the 9:16 window follows the subject.
5. Export — OpenCV pre-renders the cropped frames into an FFmpeg pipe that adds audio from the original and burns captions in one pass.

---

Publishing

Every generated short can be sent to:

Platform	API mode	Browser mode
YouTube	✅ full (OAuth)	✅ (opens upload page)
TikTok	✅ inbox / direct post	✅
Facebook Reels	✅ (Page access token)	✅
Instagram Reels	✅ (requires public URL host)	mobile only

See docs/PUBLISHING.md for per-platform credential setup.

Credentials live under credentials/ at the project root. That folder is .gitignored.

---

Subject follow-tracking

The 9:16 crop pans with the subject automatically. Knobs:

• Follow mode — auto (face, fall back to person via YOLO), face, person, off (static).
• Smoothing — low (snappy), medium (default), high (very smooth, more lag).

Follow is only active for single-window layouts (Speaker, Auto resolving to Speaker). Stacked layouts (streaming, split-screen) use their existing static composition.

Coverage check: if the tracker finds a subject in less than 35 % of sampled frames, the pipeline falls back to the static crop so we don't pan to nothing.

---

Project layout

AutoAI/
├─ server.py # uvicorn entry: `python server.py`
├─ cli.py # legacy CLI
├─ app_gradio.py # legacy Gradio UI (to be removed)
├─ requirements.txt
├─ src/autoshorts/
│ ├─ pipeline.py # download → transcribe → select → follow → export
│ ├─ download.py # yt-dlp wrapper
│ ├─ transcribe.py # faster-whisper wrapper
│ ├─ highlights.py # Ollama: pick moments + write titles
│ ├─ focus.py # face detection (legacy static focus)
│ ├─ event_focus.py # YOLO-based group/event crop
│ ├─ tracking.py # NEW: per-frame follow + smoothing
│ ├─ export.py # FFmpeg + OpenCV renderer
│ ├─ edit_short.py # natural-language trim
│ ├─ youtube_upload.py # legacy YouTube uploader
│ ├─ publish/
│ │ ├─ base.py, credentials.py, assisted.py
│ │ ├─ youtube.py, tiktok.py, facebook.py, instagram.py
│ └─ server/
│ ├─ app.py # FastAPI app
│ ├─ jobs.py # in-process job registry + SSE
│ └─ models.py # pydantic req/res models
├─ web/ # React 19 + Vite 6 + Tailwind 4
│ ├─ src/
│ │ ├─ App.tsx, main.tsx, index.css
│ │ ├─ api/client.ts # typed fetch + SSE
│ │ └─ components/ # SourceForm, ProgressCard, ShortCard, ShortsGrid, ui
│ ├─ vite.config.ts
│ └─ package.json
├─ docs/
│ ├─ API.md # HTTP endpoints
│ ├─ PUBLISHING.md # per-platform credential setup
│ ├─ FEATURE_PARITY.md
│ └─ TOOLS_AND_SETUP.md
├─ credentials/ # (git-ignored) OAuth clients + cached tokens
├─ downloads/ # (git-ignored) yt-dlp output + staged uploads
└─ generated/ # (git-ignored) <timestamp>/short_N.mp4 + run_metadata.json

---

Roadmap

• S3 (next): trim + publish wired into the web UI, left-rail run history, credentials UI.
• S4: dark-mode polish, keyboard shortcuts, empty/error states, batch publishing.
• C: package as a Tauri desktop app.

---

Troubleshooting

• "No such filter: 'subtitles'" — FFmpeg was built without libass. Reinstall FFmpeg with libass support; shorts still render without burned captions as a fallback.
• Ollama errors — make sure ollama serve is running and you've ollama pulled the model you picked.
• Cropping wrong — try turning Follow off, or swap layout to Event or Streaming. For stream layouts, use manual crop regions (in the legacy Gradio UI until the React manual-regions panel lands in S3).
• Punch line cut off — pick a larger Whisper model for better sentence punctuation, or raise Max clip length.
• Instagram "cannot reach URL" — the Graph API fetches your video from a public URL. Host the generated/ folder somewhere reachable and set public_base_url in credentials/instagram_token.json. See docs/PUBLISHING.md.

---

License

MIT.