SparseVideoNav: Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation

Hai Zhang^*Siqi Liang^*Li ChenYuxian LiYukuan XuYichao ZhongFu Zhang^†Hongyang Li^†


The University of Hong Kong

📖 Introduction

SparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It achieves sub-second trajectory inference with a sparse future spanning a 20-second horizon, yielding a remarkable 27× speed-up. Real-world zero-shot experiments show 2.5× higher success rate than state-of-the-art LLM baselines and mark the first realization in challenging night scenes.

Developers: Hai Zhang and Siqi Liang

📢 News

Important

🌟 Stay up to date at opendrivelab.com!

• 🎉 2026-02-05: Project Page is now available!
• 🎉 2026-02-06: arXiv preprint is now available!
• 🎉 2026-03-31: Inference code and model checkpoint released!
• 🎉 2026-07-10: SparseVideoNav Dataset is now available!

📌 Table of Contents

• 📖 Introduction
• 📢 News
• 🔥 Highlights
• 🔧 Installation
• 📥 Checkpoint
• 🚀 Usage
• 📝 TODO List
• 📬 Contact
• 📄 License and Citation

🔥 Highlights

• We investigate beyond-the-view navigation tasks in the real world by introducing video generation model to this field for the first time.
• We pioneer a paradigm shift from continuous to sparse video generation for longer prediction horizon.
• We achieve sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart.
• We achieve the first realization of beyond-the-view navigation in challenging night scenes with a 17.5% success rate.

🔧 Installation

Requirements

• Linux (tested on Ubuntu)
• Python 3.10
• NVIDIA GPU with ≥ 16 GB VRAM
• uv ≥ 0.7

# 1. Clone the repository
git clone https://github.com/OpenDriveLab/SparseVideoNav.git
cd SparseVideoNav
# 2. Create virtual environment and install dependencies
uv sync --all-groups
# 3. Activate the environmentsource .venv/bin/activate

--all-groups also installs flash-attn. Building it from source takes a few minutes on first install.

📥 Checkpoint

Download the SparseVideoNav pipeline checkpoint and place it under models/SparseVideoNav-Models/:

Component	Download
SparseVideoNav pipeline checkpoint	🤗 HuggingFace

Expected directory layout after download:

models/SparseVideoNav-Models/
├── google/
│ └── umt5-xxl/
│ ├── special_tokens_map.json
│ ├── spiece.model
│ ├── tokenizer.json
│ └── tokenizer_config.json
├── models_t5_umt5-xxl-enc-bf16.pth
├── Wan2.1_VAE.pth
└── svn_ckpt/
├── config.json
└── diffusion_pytorch_model.safetensors

If you place the checkpoint elsewhere, update ckpt_path in config/inference.yaml or override it on the command line.

🚀 Usage

1. Command-line inference

python inference.py video_path=/path/to/input.mp4 'prompt=turn right'

Results are written to outputs/<timestamp>_<video_name>/:

• predicted_video.mp4 — generated future video

Key overrides:

Parameter	Default	Description
video_path	—	Input video path (required)
prompt	—	Language instruction (required)
output_path	outputs	Root output directory
ckpt_path	models/SparseVideoNav-Models	Pipeline checkpoint directory
inference.device	cuda:0	Target device
inference.denoise_steps	4	Denoising steps (higher → better quality)

Example with overrides:

python inference.py \
video_path=/path/to/input.mp4 \
'prompt=walk forward and turn left' \
ckpt_path=/path/to/checkpoint \
inference.device=cuda:0 \
inference.denoise_steps=8

2. Gradio web demo

python gradio_interface.py

Opens a local demo at http://0.0.0.0:7860. Upload a video, enter a navigation instruction, and click Run Prediction.

Common options:

Flag	Default	Description
--ckpt_path	from config	Override checkpoint directory
--device	cuda:0	Target device
--port	7860	Server port
--share	False	Create a public Gradio share link

3. Python API

fromomegaconfimportOmegaConffrominferenceimportSVNPipelinecfg=OmegaConf.load("config/inference.yaml")
cfg.ckpt_path="/path/to/checkpoint"cfg.inference.device="cuda:0"pipeline=SVNPipeline.from_pretrained(cfg)
# Returns np.ndarray (T, H, W, C) uint8video=pipeline(video="/path/to/input.mp4", text="turn right")

For direct access to the latent-space model:

fromsparseVideoNav.svn_modelimportSVNModelmodel=SVNModel.from_pretrained("/path/to/checkpoint/svn_ckpt")

📝 TODO List

• SparseVideoNav Paper Release.
  • arXiv preprint is now available!
• SparseVideoNav Code Release.
  • Inference code of distilled video generation model and model checkpoint.
  • Inference code of continuous action head and model checkpoint (Estimate 2026 Q3).
• SparseVideoNav Dataset Release
  • ~140h real-world VLN data.

📬 Contact

For further inquiries or assistance, please contact zhanghenryhai12138@gmail.com or liangsiqi@connect.hku.hk

📄 License and Citation

All the data and code within this repo are under CC BY-NC-SA 4.0.

• Please consider citing our work if it helps your research.

@article{zhang2026sparse,
title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},
author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},
journal={arXiv preprint arXiv:2602.05827},
year={2026}
}