Zifan Wang · Ziang Ren · Pengyang Shi · Zirui Wang · Chenghuai Lin
Tianze Wang · Zekun Qi · Liangliang Zhao · He Wang · Li Yi
✨ ECCV 2026 ✨
From spoken words to expressive humanoid gestures.
RoboGesture · Inference code, trained checkpoints and MuJoCo demos
中文 · Quick Start · Inference Guide · Training Core · Robot Deployment
RoboGesture brings speech and body language together for humanoid interaction. This repository provides its deployed inference workflow: turn a WAV into motion, preview the result in MuJoCo, or run English dialogue on a Unitree G1 with BrainCo Revo2 hands. Explore the research and real-world demonstrations on the project page.
- Speech in, motion out. Generate 30 FPS humanoid gestures directly from a WAV, without preparing transcripts.
- Dialogue with gestures. Connect English ASR, Qwen + LoRA and TTS to the field-validated G1 interaction pipeline.
- Demos ready to explore. Start with 15 paired audio–motion examples and render synchronized MP4s using the bundled robot assets.
- A self-contained setup. One checkout, its own environment and a sibling model directory; no additional code checkout is required.
- Explore how it learns. Run two-stage training-core examples with synthetic tensors, without downloading a training dataset or model checkpoint.
This inference release includes the deployed 54-class motion checkpoint. Training-core examples are available. Full training workflows, preprocessing and private datasets are not included; this is not a full reproduction package for the paper's experiments.
- [2026.08.27] The RoboGesture paper, accepted at ECCV 2026, is available on arXiv.
Start with a video. The bundled examples need no neural model downloads. On a configured workstation, run the following from your checkout:
cd RoboGesture
bash scripts/setup.sh
.venv/bin/robogesture render examples/sample_01.npz \
--output outputs/sample_01.mp4Open outputs/sample_01.mp4 to see the paired motion and speech. The renderer
automatically finds sample_01.wav next to the NPZ.
⚒️ Environment and system prerequisites
The tested GPU workstation uses Linux x86-64, Python 3.10 and an RTX 4090 (24 GB) with a CUDA 12.1-compatible NVIDIA driver. Git LFS, Python venv support, PortAudio and NVIDIA EGL must be available on the host.
On Ubuntu, install missing common system packages with:
sudo apt-get install git git-lfs python3.10-venv libportaudio2 libegl1 libgl1 libglib2.0-0The setup script creates .venv/ inside this checkout and installs PyTorch
2.5.1 + CUDA 12.1 and a pinned Unitree SDK revision. Direct dependencies are in
requirements.txt; tested transitive versions are fixed in
constraints.txt. DDS and FFmpeg are supplied by packages inside
this environment. The G1 audio service has a separate
Python 3.8 setup.
Two trained inference artifacts are included through Git LFS. Public base models are downloaded separately from their publishers.
| Component | Role | Get it |
|---|---|---|
| Motion checkpoint | Audio-conditioned, 41-joint motion generation | Git LFS · checkpoints/motion/ |
| Qwen LoRA adapter | Action-tagged English replies | Git LFS · checkpoints/qwen_lora/ |
| Mimi | Audio tokenization | Official download · CC-BY-4.0 |
| Qwen3-4B-Instruct-2507 | Dialogue base model | Official download · Apache-2.0 |
For WAV-to-motion inference, retrieve the trained artifacts and download Mimi:
git lfs install --local
git lfs pull
.venv/bin/python scripts/download_models.py --only mimiFor the complete setup, including local dialogue generation and installation checks, download both public models:
.venv/bin/python scripts/download_models.py
.venv/bin/python scripts/check_install.py --require-cudaDownloads use pinned revisions and default to ../weights/. Public base-model
parameters are not redistributed in Git or Git LFS. See the
checkpoint guide for exact revisions, checksums and LFS
troubleshooting.
Give the model a WAV, then watch the generated gesture:
.venv/bin/robogesture infer examples/sample_06.wav \
--output outputs/generated.npz
.venv/bin/robogesture render outputs/generated.npz \
--output outputs/generated.mp4Replace examples/sample_06.wav with your own recording to try new speech.
Inference writes a 30 FPS NPZ and a duration-aligned 24 kHz WAV; rendering
combines them into a 960×720 H.264/AAC MP4 by default.
Want to inspect a frame or explore more clips? Save a preview with
--preview outputs/frame.png, or follow the 15-demo guide.
The inference guide covers audio conversion, NPZ layout,
rendering options and custom model paths.
Offline output uses smoothing and collision filtering in a fixed-base scene with
zero leg angles. It is suitable for inspecting motion, not a replacement for a
live robot preflight. Existing outputs are protected; use --overwrite only
when replacement is intended.
The interactive entry point connects English speech recognition, Qwen + LoRA, speech synthesis and gesture playback on the validated G1 + BrainCo Revo2 setup.
# Show the disconnected entry point; no devices are opened.
.venv/bin/robogesture streamTo deploy, follow the robot guide: configure your own Volcengine ASR/TTS credentials, prepare the robot audio service, and run a supervised preflight before enabling motion.
About streaming: this entry point uses a field-validated whole-utterance workflow. Each reply's audio and motion are generated and collision-filtered before synchronized playback; it does not play motion incrementally as speech arrives. The motion sampler itself processes audio in chunks.
See the learning objectives in action, from prepared tensors through backpropagation and an optimizer update:
.venv/bin/python -m training_examples.demoThis CPU-friendly example uses a smaller configuration of MotionModel,
random initialization and synthetic batches. It demonstrates the two-stage
losses, parameter freezing, history masking and optimizer updates without reading
data or loading/saving checkpoints. It is a training-core demonstration, not a
full trainer or a convergence benchmark. The CLI provides three commands:
infer, stream and render. Run the training example as a Python module.
See the training-core guide for the full-size GPU option, tensor contract, objective specification and implementation notes. Dataset preprocessing and Qwen/LoRA training are not included.
On the tested workstation, the offline suite passes 40 tests and renders 16 complete videos (15 bundled examples + 1 generated motion). Validation covers real-model inference, full audio/video decoding and execution without network access, other code checkouts or installation caches.
To check your installation after downloading the models:
.venv/bin/python scripts/verify_offline.py --output acceptance-outputUse an empty output directory. See the verification report for the tested versions and scope. These automated checks do not call cloud services or operate a physical robot; live deployment requires your own credentials and a supervised G1 preflight.
Training-core checks are described in the training-core verification notes.
parent/
├── weights/ # Official models, downloaded separately
│ ├── mimi/
│ └── Qwen3-4B-Instruct-2507/
└── RoboGesture/
├── .venv/ # This checkout's environment
├── robogesture/ # Inference, dialogue, robot control and renderer
│ └── assets/ # G1/BrainCo XML and meshes
├── checkpoints/ # Trained motion model and LoRA adapter
├── examples/ # 15 paired NPZ/WAV demos
├── training_examples/ # Prepared tensors -> loss -> backward -> update
├── scripts/ # Setup, downloads and verification
├── docs/ # Inference and deployment guides
└── tests/
We thank the Qwen, Kyutai, Unitree, BrainCo and MuJoCo projects for the models and tools used here. The README presentation is inspired by SemTalk. Upstream notices are preserved in THIRD_PARTY_NOTICES.md.
If RoboGesture is useful to your research, please cite the paper:
@misc{wang2026robogesture,
title={RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction},
author={Zifan Wang and Ziang Ren and Pengyang Shi and Zirui Wang and Chenghuai Lin and Tianze Wang and Zekun Qi and Liangliang Zhao and He Wang and Li Yi},
year={2026},
eprint={2608.28693},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2608.28693}
}Project-owned code uses the MIT License. Third-party models, SDKs and robot assets retain their own licenses. The 68 bundled meshes match pinned official Unitree sources, and the composite robot XML is documented as a modified description; see the asset provenance and modification notice.