A modular, config-driven pipeline for processing the Active Inference Institute video library.
Download · Transcribe · Extract · Export · Browse · Chat
graph LR
subgraph Ingest
A[🎬 YouTube Channel] -->|yt-dlp| B[📥 Download]
B --> C[📝 Transcripts]
B --> D[🎵 Audio]
end
subgraph Process
D -->|mlx-whisper / WhisperX| C
C -->|Cohere AI| E[🧠 Entities & Graph]
C -->|5 formats| F[📄 Export]
end
subgraph Serve
C --> G[🌐 Web Interface]
E --> G
G -->|Ollama RAG| H[Chat]
end
style A fill:#e63946,color:#fff
style G fill:#457b9d,color:#fff
style H fill:#2a9d8f,color:#fff
One command runs the configured application pipeline:
uv run python run.pyOne file controls all options:config.ini— see reference →
Enumerate the Active Inference channel from a saved manifest. Download transcripts, audio, and video (cookie-free by default) with rate limiting and resume. Transcribe locally on Apple Silicon or GPU with speaker diarization. | FastAPI SPA with searchable video library, embedded YouTube player, transcript viewer, and category browser. Ollama-powered RAG chat with automatic context retrieval. |
Batch-export to Markdown, JSON, HTML, PDF, and plaintext — each enriched with metadata headers (title, category, series, speakers, duration, URL, views). | Cohere AI entity extraction (people, concepts, theories, organizations) and relationship mapping into a SurrealDB knowledge graph. |
# 1. Clone & install
git clone https://github.com/ActiveInferenceInstitute/Journal-Utilities.git
cd Journal-Utilities
uv sync --all-extras
# 2. Run the default pipeline (Config → Validate → Export → Test → Serve)
uv run python run.pyuv run python run.py config # Show current configuration
uv run python run.py download # Download from YouTube
uv run python run.py export# Export transcripts to all enabled formats
uv run python run.py test# Run the pytest suite
uv run python run.py serve # Launch web UI at http://localhost:8000
uv run python run.py full # Full pipeline: download → export
uv run python run.py journal-check # Read-only journal integrity gategraph TB
subgraph "CLI Layer"
RUN["run.py — Pipeline Runner"]
S1["scripts/download_channel.py"]
S2["scripts/transcribe_missing.py<br/>scripts/transcribe_worklist.py"]
S3["scripts/scaffold_youtube_courses.py"]
S4["scripts/validate_journal.py"]
S5["scripts/speaker_cues.py<br/>scripts/apply_speaker_names.py"]
end
subgraph "src/journal_utilities/"
direction TB
YT["youtube/<br/>channel · playlist · categorizer"]
DL["download/<br/>downloader"]
TR["transcribe/<br/>mlx-whisper · WhisperX"]
EX["export/<br/>exporter (5 formats)"]
DATA["data/<br/>database · importer"]
RAG["rag/<br/>extractors · graph · models"]
IF["interface/<br/>app · chat_engine · data_loader"]
RN["render/<br/>renderer"]
end
subgraph "External Services"
OL["Ollama (LLM)"]
DB["SurrealDB"]
CO["Cohere AI"]
YT_API["YouTube (yt-dlp)"]
end
RUN --> EX & IF & DL & S4
S1 --> YT & DL
S2 --> TR
S3 --> RN
DL --> YT_API
RAG --> CO & DB
DATA --> DB
IF --> OL
style RUN fill:#e63946,color:#fff
style IF fill:#457b9d,color:#fff
style EX fill:#2a9d8f,color:#fff
Full directory tree
Journal-Utilities/
├── src/journal_utilities/ # Main package
│ ├── youtube/ # Channel enumeration, categorizer
│ ├── download/ # yt-dlp download engine
│ ├── transcribe/ # MLX-Whisper + WhisperX
│ ├── data/ # SurrealDB client + Coda importer
│ ├── interface/ # FastAPI SPA + Ollama chat
│ ├── rag/ # Entity extraction pipeline
│ ├── render/ # Course scaffolding
│ ├── export/ # Multi-format transcript export
│ └── utils/ # Shared utilities
├── scripts/ # CLI tools
├── tests/ # Pytest suite
├── data/ # Input, output, database storage
├── docs/ # 10 module guides
├── run.py # Pipeline runner
├── config.ini # All configuration
└── pyproject.toml # Python 3.12+
All technical detail lives in docs/. The README you're reading is the overview and entry point.
See docs/JOURNAL_SCHEMA.md for the ActiveInferenceJournal v2 schema
and docs/REFACTOR_READINESS.md for the refactor pipeline
(scripts/refactor_journal.py).
| Guide | What You'll Find |
|---|---|
| Configuration | config.ini sections, environment variables, pipeline step control |
| YouTube | Channel enumeration, playlist parsing, title categorization |
| Download | Cookie auth, 403 troubleshooting, download strategies |
| Transcription | MLX-Whisper (Mac), WhisperX (GPU), model selection |
| Subtitles → Translations | Hosted OpenRouter subtitle translation |
| Export | Format details, metadata enrichment, library API |
| Web Interface | API endpoints, SPA frontend, development server |
| Chat Engine | Ollama RAG, prompt engineering, model auto-discovery |
| RAG & Graph | Cohere extraction, entity schema, knowledge graph |
| Data & Database | SurrealDB schema, Coda import, audit trails |
| Render | Playlist → course scaffolding, module.md format |
| Agent Guide | Architecture, code patterns, agent development rules |
uv run pytest tests/ -v --cov=src # Full suite with coverage
uv run python run.py journal-check # Journal corpus integrity gateTest counts and coverage are intentionally not embedded in this README because they change as the suite and corpus evolve. The commands above and CI output are the live status.
Journal-Utilities is the code-side source of truth for the generated ActiveInferenceJournal layout. The maintenance sequence is explicit and safe to repeat:
uv run python scripts/enrich_metadata.py --journal ../ActiveInferenceJournal
uv run python scripts/enrich_metadata.py --journal ../ActiveInferenceJournal --apply
uv run python scripts/repair_split_transcripts.py --journal ../ActiveInferenceJournal --utilities .
uv run python scripts/generate_journal_indexes.py --journal ../ActiveInferenceJournal
uv run python run.py journal-checkEnrichment is dry-run by default; only the command with --apply writes metadata.
The final journal-check command is read-only and blocks handoff when metadata,
indexes, transcript identities, duplicate handling, coverage, or the main
branch's no-audio/no-credentials boundary is inconsistent.
Journal transcripts follow a raw vs derived design (see
docs/JOURNAL_SCHEMA.md): transcript.json is the
immutable raw WhisperX diarization (SPEAKER_NN labels, never rewritten),
human speaker names live only in metadata.jsonparts[].speakers, and
transcript.txt is regenerated from the two. Private/unlisted videos are
never transcribed. All commands are dry-run by default.
# corpus status: diarized / captions-only / excluded / scheduled
uv run python scripts/transcription_status.py
# WhisperX + diarization for items missing transcripts (GPU, resumable)
uv run python scripts/transcribe_worklist.py # plan
uv run python scripts/transcribe_worklist.py --run
# identify speakers: timestamped YouTube links per SPEAKER_NN
uv run python scripts/speaker_cues.py # items still needing names
uv run python scripts/speaker_cues.py --item TextbookGroup/Namjoshi2026/Cohort_1/Session_024
# record parts[].speakers in the item's metadata.json, then regenerate transcript.txt
uv run python scripts/apply_speaker_names.py --item <item> --applyRequired in .env:
| Variable | Purpose |
|---|---|
HUGGINGFACE_TOKEN | WhisperX speaker diarization |
COHERE_API_KEY | Entity extraction (RAG) |
CODA_API_TOKEN | Coda session data |
OLLAMA_MODEL | Chat model (default: gemma3:4b) |
OLLAMA_BASE_URL | Ollama API URL (default: http://localhost:11434) |
See CONTRIBUTING.md for the dev setup, gate workflow, and pull-request process. Security and cookie-handling policy: SECURITY.md. If you use or build on this work, please cite it — see CITATION.cff.