🎤 MeanVC2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Guobin Ma^1,*, Yuxuan Xia^1,*, Yuepeng Jiang¹, Dake Guo¹, Hanke Xie¹, Jingbin Hu¹, Yanbo Wang², Lei Xie^1,**, Pengcheng Zhu^3,**

¹ Audio, Speech and Language Processing Group (ASLP@NPU), School of Software, Northwestern Polytechnical University, China
² The University of New South Wales, Australia
³ WeNet Open Source Community, China

🎥 Demo Video

Tutorial	Demo

📖 Introduction

MeanVC2 is a robust, low-latency streaming zero-shot voice conversion (VC) system built upon the diffusion-based conditional flow matching (CFM) framework. It addresses key limitations of its predecessor MeanVC, including training inefficiency, quality degradation under small-chunk settings, and sensitivity to low-quality reference audio.

By introducing Future-Receptive Chunking (FRC) and a Universal Timbre Token Encoder (UTTE), MeanVC2 achieves high-fidelity voice conversion with an end-to-end pipeline latency of only 110 ms — nearly halving the 211 ms latency of MeanVC(160ms) — while maintaining superior speaker similarity and audio naturalness even with a 40 ms chunk size. It operates under a recognition-synthesis paradigm, where a streaming ASR module extracts content representations (BNFs), and a DiT-based decoder generates target mel-spectrograms conditioned on timbre-aware features retrieved via UTTE.

MeanVC2 supports speaker-specific fine-tuning: using the provided training scripts with a pretrained safetensors checkpoint as initialization, you can fine-tune the model on a target speaker's data for improved conversion quality. See Training for details.

✨ Key Features

• Ultra-low latency streaming: 110 ms end-to-end first-packet latency with 40 ms chunk size; full pipeline RTF < 0.633 on single CPU core.
• Future-Receptive Chunking (FRC): Enables stable short-chunk conversion by explicitly scheduling past/future receptive fields across DiT layers, eliminating clean-chunk teacher forcing and reducing peak GPU memory by ~60%.
• Universal Timbre Token Encoder (UTTE): Decouples fine-grained timbre extraction from direct reference mel-spectrograms, using global speaker embeddings + cross-attention to improve robustness under low-quality references and enhance zero-shot speaker similarity.
• Mean Flows + 1-NFE inference: Single-step ODE solving for high-quality mel-spectrogram synthesis, balancing efficiency and fidelity.
• Lightweight yet powerful: Only 18M parameters — comparable to MeanVC (14M) and far smaller than competing streaming VC systems.
• End-to-end & streaming-ready: Supports both file-based conversion and real-time microphone streaming with pre-extracted or on-the-fly features.

🚀 Quick Start

1. Clone and Install Dependencies

git clone https://github.com/ASLP-lab/MeanVC2.git
cd MeanVC2
# Create conda environment
conda create -n meanvc2 python=3.11
conda activate meanvc2
# Install PyTorch (CUDA 12.1)
pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
# Install dependencies
pip install -r requirements.txt

2. Download Pretrained Models

# Download all models (preprocessing + VC + vocoder)
python initialization.py --task all

Or download only what you need:

python initialization.py --task preprocess # BN + SpkEmb extraction only
python initialization.py --task train_120ms # preprocess + 120ms VC + vocoder
python initialization.py --task train_40ms # preprocess + 40ms VC + vocoder

FunASR models (Paraformer, VAD, punctuation) will auto-download from ModelScope at first use.

💿 Standalone Executables

Pre-built JIT-accelerated Windows executables (CPU-only, single .exe) are available on Google Drive:

Executable	Latency	Speaker Input	Size
40ms_40ms.exe	40ms chunk + 40ms future = 80ms	WAV file (WavLM + ECAPA-TDNN)	Full
120ms_40ms.exe	120ms chunk + 40ms future = 160ms	WAV file (WavLM + ECAPA-TDNN)	Full
40ms_40ms_npy.exe	40ms chunk + 40ms future = 80ms	Pre-extracted NPY file	~1.2 GB smaller
120ms_40ms_npy.exe	120ms chunk + 40ms future = 160ms	Pre-extracted NPY file	~1.2 GB smaller

Audio Routing: Input from microphone or VB-CABLE Output (capture PC playback); output to headphones/speakers or VB-CABLE Input (send to other apps, e.g., set as mic in Tencent Meeting).

Requirements: Windows 10+, 8 GB RAM, ~4 GB free disk space. See intro.txt in the drive folder for details.

📁 Data Preparation

Step 1: Extract Mel Spectrograms

python preprocess/extract_mel.py --input_dir /path/to/wavs --output_dir /path/to/mels

Step 2: Extract Content Features (BN)

# 80ms chunk JIT (fastu2pp_80ms.pt, 11-frame window, stride=8)
python preprocess/extract_bn_80ms.py --input_dir /path/to/wavs --output_dir /path/to/bns
# 160ms chunk JIT (fastu2pp_160ms.pt, 19-frame window, stride=16)
python preprocess/extract_bn_160ms.py --input_dir /path/to/wavs --output_dir /path/to/bns

Step 3: Extract Speaker Embeddings

python preprocess/extract_spk_emb.py --input_dir /path/to/wavs --output_dir /path/to/xvectors

Step 4: Create Training Filelist

python scripts/create_filelist.py \
--bn-dir /path/to/bns \
--mel-dir /path/to/mels \
--xvector-dir /path/to/xvectors \
--output train.list

Filelist format (one line per utterance):

utt_id|/path/to/bn/utt_id.npy|/path/to/mel/utt_id.npy|/path/to/xvector/utt_id.npy

🎵 Inference

Zero-Shot (Non-Streaming)

# 120ms chunk + 40ms future (recommended for quality)
python src/infer/infer_zero_shot.py \
--model-config src/config/config_120ms_40ms.json \
--ckpt-path ckpts/pretrained_models/meanvc2_120ms_40ms.safetensors \
--vocoder-ckpt-path ckpts/vocos/vocos.pt \
--output-dir output/ \
--file-scp test.lst \
--bn-path /path/to/bn \
--spk-emb-path /path/to/spk_emb \
--chunk-size 12 --steps 3

End-to-End (Single Script)

No pre-extracted features needed — input two wavs, output converted audio:

# 120ms+40ms model (recommended for quality)
python src/infer/infer_e2e.py --model 120ms \
--source-wav /path/to/source.wav \
--target-wav /path/to/target.wav \
--output-wav output.wav --steps 3
# 40ms+40ms model (lower latency)
python src/infer/infer_e2e.py --model 40ms \
--source-wav /path/to/source.wav \
--target-wav /path/to/target.wav \
--output-wav output.wav --steps 3

Real-Time Streaming

# File modecd runtime
python run_rt.py --mode file --input in.wav --output out.wav --model 120ms
# Microphone mode
python run_rt.py --mode realtime --model 40ms

🏗️ Model Architecture

MeanVC2 consists of five core components:

Component	Description
Streaming ASR Encoder	Fast-U2++ (WeNet) extracts bottleneck features (BNFs) from source waveform; 80 ms chunk size for streaming inference
Speaker Encoder	ECAPA-TDNN + WavLM upstream extracts a global speaker embedding from reference audio
Universal Timbre Token Encoder (UTTE)	Transforms global speaker embedding into K key-value UTT pairs; BNFs serve as queries in cross-attention to retrieve fine-grained, pronunciation-aware timbre cues
DiT-based CFM Decoder	4-layer DiT (hidden dim 512, 2 heads) with Future-Receptive Chunking (FRC); trained with mean flows objective for 1-NFE mel-spectrogram generation
Vocoder	Vocos converts mel-spectrograms to 16 kHz high-fidelity speech waveforms

Total parameters: ~18M

🏋️ Training

# 120ms+40ms training (recommended for quality)
bash scripts/train_120ms_40ms.sh 0 # single GPU
bash scripts/train_120ms_40ms.sh "0,1,2,3,4,5,6,7"# 8 GPUs# 40ms+40ms training (lower latency)
bash scripts/train_40ms_40ms.sh 0 # single GPU
bash scripts/train_40ms_40ms.sh "0,1,2,3,4,5,6,7"# 8 GPUs# Override dataset and experiment name
DATASET_PATH=/path/to/train.list EXP_NAME=my_exp bash scripts/train_120ms_40ms.sh "0,1,2,3"

The dataset path expects a .list file (line items: utt_id|/path/to/bn.npy|/path/to/mel.npy|/path/to/xvector.npy), generated by scripts/create_filelist.py.

🙏 Acknowledgements

This work builds upon the following open-source projects:

• F5-TTS — DiT-based CFM backbone
• Vocos — Neural vocoder
• WavLM — Speaker encoder upstream
• ECAPA-TDNN — Speaker embedding model
• WeNet — ASR encoder for BN extraction
• s3prl — Self-supervised speech models
• FunASR — ASR for CER evaluation
• Emilia — Training data

📜 License & Disclaimer

MeanVC2 is released under the Apache License 2.0. This open-source license allows you to freely use, modify, and distribute the model, as long as you include the appropriate copyright notice and disclaimer.

MeanVC2 is designed for research and legitimate applications in voice conversion technology. Users must obtain proper consent from individuals whose voices are being converted or used as references. We strongly discourage any malicious use including impersonation, fraud, or creating misleading audio content. Users are solely responsible for ensuring their use cases comply with ethical standards and legal requirements.

📄 Citation

If you find our work helpful, please cite:

@article{ma2026meanvc2,
title={MeanVC2: Robust Low-Latency Streaming Zero-Shot Voice Conversion},
author={Ma, Guobin and Xia, Yuxuan and Jiang, Yuepeng and Guo, Dake and Xie, Hanke and Hu, Jingbin and Wang, Yanbo and Xie, Lei and Zhu, Pengcheng},
journal={arXiv preprint arXiv:2606.09050},
year={2026}
}

✉️ Contact

For questions or collaborations, please contact: guobin.ma@mail.nwpu.edu.cn or lxie@nwpu.edu.cn

You’re welcome to join our WeChat group for technical discussions and updates.

⭐ Star History