EMMA builds an event-driven memory hierarchy for long videos and uses multi-round retrieval to answer video questions. The public release has two independent stages:
Stage 1 · Memory Building video → ASR → STM / MTM / Event Table
Stage 2 · KQA memory + question → multi-round retrieval → answer
The reference configuration uses local Qwen3-VL-8B-Instruct for visual memory
construction and raw-video fallback; DeepSeek-V4-Flash for memory text
reasoning, retrieval planning, and validation; text-embedding-3-large for
text retrieval; and DeepSeek R1 for final answers.
- Python 3.10+
ffmpegonPATH(for example,apt install ffmpeg)- API access to an OpenAI-compatible DeepSeek and text-embedding endpoint
Install a compatible PyTorch build, then install the remaining packages:
pip install -r requirements.txtDownload these local models separately; model weights, videos, and outputs are not included in this repository.
| Model | Role | Required format |
|---|---|---|
| Qwen3-VL-8B-Instruct | VLM memory construction and video fallback | Hugging Face checkpoint |
| whisper-large-v3-turbo | ASR | CTranslate2 / faster-whisper checkpoint |
To convert an existing Hugging Face Whisper checkpoint once (CPU is sufficient):
ct2-transformers-converter --model /path/to/whisper-large-v3-turbo --output_dir /path/to/whisper-large-v3-turbo-ct2 --quantization float16 --copy_files tokenizer.json preprocessor_config.jsonCopy the examples; never commit the resulting .env files.
cp memory/.env.example memory/.env
cp KQA/project/.env.example KQA/project/.envSet your own credentials and OpenAI-compatible endpoint in both files. The
default memory text model is deepseek-v4-flash. The public KQA configuration
is KQA/project/configs/deepseek_v4_flash_r1.yaml;
it uses deepseek-v4-flash, text-embedding-3-large, and
ark-deepseek-r1-250528 respectively. Adjust the model identifiers only if
your provider uses different aliases.
memory/run_memory.py is the supported entry point. It first runs
faster-whisper/CTranslate2 ASR, then lets that process exit before starting a
local Qwen vLLM server. Qwen remains resident across all videos, so the report
separates startup time from per-video steady-state time.
python memory/run_memory.py \
--input-dir /path/to/videos \
--output-root /path/to/memory-results \
--asr-model-dir /path/to/whisper-large-v3-turbo-ct2 \
--qwen-model /path/to/Qwen3-VL-8B-Instruct \
--device-id 0 \
--fail-fastFor a reproducible subset, create a text file containing one absolute video
path (or one video filename) per line and pass --video-list /path/to/list.txt.
Use --rerun-asr when ASR should be measured again rather than reused.
Outputs for each video are written to:
{output_root}/{video_name}/outputs/
├── asr_segments.json
├── chapter_segmentation.json
├── short_term.json
├── medium_term.json
└── event_table.json
Timing reports are written to
{output_root}/_runs/<timestamp>/summary.json and
per_video_timings.json. memory_build and each videos[].seconds value
exclude ASR and Qwen server startup, which is the useful production latency
when Qwen is already resident.
The KQA runner starts one Qwen vLLM server for RAW_VIDEO fallback and runs
questions sequentially. It reads the Stage 1 output tree via --results-root.
python KQA/project/run_batch.py \
--videomme-json /path/to/videomme.json \
--results-root /path/to/memory-results \
--raw-video-root /path/to/videos \
--qwen-model /path/to/Qwen3-VL-8B-Instruct \
--device-id 0Use --question-id <id> or --max-questions 1 for a smoke test. Results,
traces, errors, and a timing summary are written under
KQA/project/outputs/batch_<timestamp>/ unless --out-root is supplied.
EMMA uses a grounded coarse-to-fine route: MTM localization, then STM or
RAW_VIDEO evidence before early termination. After two stalled or insufficient
rounds, it forces the escalation MTM → STM → RAW_VIDEO; a no-evidence run
uses a closest-candidate fallback over 15 MTM and 25 STM memories. This is the
core retrieval-control strategy carried into the public reference configuration.
EMMA/
├── memory/
│ ├── ASR.py # faster-whisper / CTranslate2 ASR
│ ├── run_memory.py # supported memory runner
│ ├── AIO.py # memory construction pipeline
│ ├── chapter.py # chapter segmentation
│ └── .env.example
├── KQA/project/
│ ├── run_batch.py # supported KQA runner
│ ├── configs/deepseek_v4_flash_r1.yaml
│ ├── src/ # retrieval agents, pipeline, and utilities
│ └── .env.example
├── qwen_vl_utils.py
├── requirements.txt
└── LICENSE # Apache-2.0
.env, datasets, model checkpoints, generated outputs, and logs are ignored by Git. Scan your working tree before every public push.- The supplied runner releases Whisper before starting Qwen to reduce model residency pressure.