Motion Patches

Code of the paper "Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches" (CVPR 2024).

Framework

Requirements

• Python 3.11
• PyTorch 2.0.1+

Using Poetry (Recommended)

poetry install

Using Conda

conda create -n MoPa python=3.11
conda activate MoPa
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt

Data Preparation

Download HumanML3D Dataset and KIT-ML Dataset from the repository of HumanML3D. Unzip and locate them in the data folder.

The whole directory should be look like this:

MotionPatches
│ README.md
│ requirements.txt
| ...
|
└───conf
└───scripts
└───...
│ └───data
└───HumanML3D
| └───new_joint_vecs
| └───new_joints
| └───...
│ └───KIT-ML
└───new_joint_vecs
└───new_joints
└───...

Then calculate the mean and variance of each dataset by:

python scripts/cal_mean_var.py

Pre-trained Model

Download pre-trained model from huggingface and put them in checkpoints/pretrained/.

Evaluate the model with HumanML3D via retrieval:

python scripts/test.py dataset=HumanML3D exp_name=pretrained

Evaluate the model with KIT-ML via retrieval:

python scripts/test.py dataset=KIT-ML exp_name=pretrained

Using scripts/test_batch.py can get the result of small batches with 32 samples.

Training

Train the model with HumanML3D:

python scripts/train.py dataset=HumanML3D

Train the model with KIT-ML:

python scripts/train.py dataset=KIT-ML

Evaluation

Evaluate the model with HumanML3D via retrieval:

python scripts/test.py dataset=HumanML3D

Evaluate the model with KIT-ML via retrieval:

python scripts/test.py dataset=KIT-ML

Using scripts/test_batch.py can get the result of small batches with 32 samples.

Citation

@InProceedings{yu2024exploring,
title={Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches},
author={Yu, Qing and Tanaka, Mikihiro and Fujiwara, Kent},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}

License

CC BY-NC 4.0

Additionally, this repository contains third-party software. Refer NOTICE.txt for more details and follow the terms and conditions of their use.