SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding

SIMS-V generates spatial video-training data from AI2-THOR and ProcTHOR. A simulated Stretch robot tours procedural houses while the pipeline records RGB, depth, segmentation, and structured annotations. Those recordings can then be converted into spatial, temporal, and descriptive video question-answer pairs.

flowchart LR
A[Procedural\nhouse] --> B[Robot\nWalkthrough]
B --> C[Synchronized\nvideos]
B --> D[Frame-level\nannotations]
C --> E[Spatial\nVideo QA]
D --> E

Loading

The associated paper uses these data to study which properties of simulation produce spatial reasoning that transfers to real video. A 7B video-language model trained on 25K simulated examples outperforms substantially larger 72B baselines on VSI-Bench while retaining general video-understanding performance.

Quick start

The locked environment uses Python 3.9 and uv:

uv sync --locked
uv run python -m nltk.downloader wordnet wordnet2022

Trajectory generation does not require CUDA. On systems without CUDA, SIMS-V uses one CPU worker, but AI2-THOR still needs a working graphics backend. macOS uses AI2-THOR's desktop build; Linux requires a Vulkan renderer. An NVIDIA GPU is the tested and strongly recommended setup for practical throughput.

To record one short ProcTHOR walkthrough:

uv run sims-v generate \
--dataset-dir outputs/demo \
--house-dataset procthor \
--split val \
--max-houses 1

The primary video is written to a path such as outputs/demo/val/000000/rgb__0.mp4. Generate metadata and video QA from the recording with:

uv run sims-v qa \
--dataset-dir outputs/demo \
--split val \
--question-types vsi_obj_count

Run uv run sims-v generate --help or uv run sims-v qa --help for the full set of options.

What is generated

With the default modalities, a representative output directory after trajectory generation and QA looks like this:

outputs/demo/
├── constants.yaml # Generation settings and resume guard
├── generation_logs/
│ ├── logs.tsv # Per-house timing and outcomes
│ └── metrics.json # Aggregate success/failure counts
├── val/
│ ├── 000000/ # One successfully generated house
│ │ ├── house_spec.json # Complete procedural scene specification
│ │ ├── rgb__0.mp4 # Primary RGB walkthrough
│ │ ├── offline_annos__0.jsonl # Per-frame pose and visible-object data
│ │ ├── hdf5_sensors.hdf5 # Other per-frame task/controller sensors
│ │ ├── success.txt # Completion and resume marker
│ │ ├── spatial_metadata.json # Room and 3D-object geometry from QA
│ │ └── qa_pairs_vsi_obj_count__0.jsonl # Per-video generated questions
│ └── combined_qa_pairs.jsonl # QA candidates merged across houses
└── qas/val/rgb/
└── mt1_vsi_obj_count_mc.jsonl # Final model-training conversations

The __0 suffix identifies trajectory 0; additional trajectories use __1, __2, and so on. See the QA pipeline for every video modality and formatting option.

Optionally, generate additional video modalities with:

uv run sims-v generate \
--dataset-dir outputs/demo-with-ablations \
--extra-video-modalities depth semantic_seg edge

Modality	Added file	Contents
depth	depth__N.mp4	Fixed-scale depth visualization
semantic_seg	semantic_seg__N.mp4	Semantic-class masks
instance_seg	instance_seg__N.mp4	Object-instance masks
edge	edge__N.mp4	Binary instance boundaries
colored_edge	colored_edge__N.mp4	Instance-colored boundaries
non_overlapping_colored_edge	non_overlapping_colored_edge__N.mp4	Separated colored boundaries
mean_mask_overlay	mean_mask_overlay__N.mp4	Instances filled with their mean RGB
masked_background	masked_background__N.mp4	RGB with wall/room regions masked

Use --extra-video-modalities all to output all modalities.

The generator supports standard ProcTHOR houses and the public ProcTHOR-Objaverse asset collection used in the paper.

Documentation

• Getting started: installation, datasets, and walkthrough generation
• QA pipeline: question types, formatting, and output layout
• Paper settings: paper generation settings and the source recipe for the 3Q mixture
• Cluster setup: headless NVIDIA and Slurm guidance
• Development: architecture and repository checks

Citation

@article{brown2025simsv,
title = {{SIMS-V}: Simulated Instruction-Tuning for Spatial Video Understanding},
author = {Brown, Ellis and Ray, Arijit and Krishna, Ranjay and Girshick, Ross and Fergus, Rob and Xie, Saining},
journal = {arXiv preprint arXiv:2511.04668},
year = {2025},
}

License

SIMS-V source code is licensed under the Apache License 2.0. Datasets, model weights, simulator assets, and third-party dependencies remain subject to their respective licenses. See LICENSE.