🕵️Community Testing Request: MTP Speculative Decoding Feedback🤗

[JamePeng]

MTP speculative decoding community testing

Text-only MTP speculative decoding is now available in llama-cpp-python, with support for both:

• Built-in MTP/NextN heads included in the target GGUF
• External MTP draft GGUF models

The current implementation has primarily been tested with the Qwen3.5, Qwen3.6, and Qwen3.8 model families. Testing on additional hardware, backends, quantizations, and workloads is needed before the feature can be considered broadly validated.

Multimodal support is not available yet. Cache handling, rollback behavior, and decoding-efficiency optimizations are still in progress.

Feedback requested

Community testing would be especially helpful for:

• Built-in versus external MTP performance
• Different draft_n_max values
• CUDA, CPU, Vulkan, ROCm, SYCL, Metal, and other backends
• Different model sizes and quantization formats
• Large-vocabulary models
• Hybrid or recurrent models
• Long generations and repeated requests
• Acceptance rates, rollback frequency, and checkpoint overhead
• Output correctness or unexpected divergence
• Model loading, shutdown, memory usage, or runtime errors

For Qwen3.8 27B, draft_n_max=2 has performed well in current testing and is the recommended starting point. However, the optimal value can vary significantly with the GPU, backend, quantization, prompt, sampling configuration, and whether MTP is built in or external. Please benchmark multiple values when possible.

Suggested benchmark

The included benchmark can compare ordinary decoding, built-in MTP, and external MTP:

python -m examples.high_level_api.high_level_api_mtp_speculative -h

A broader speculative decoding benchmark is also available:

python -m examples.benchmark.benchmark_speculative -h

For reliable results, please use the same prompt and sampling settings for each mode, perform a warmup, and run multiple measured iterations.

Feedback template

Please include as much of the following information as possible:

llama-cpp-python version or commit:
llama.cpp vendor commit:
Operating system:
Python version:
CPU:
GPU:
Backend and build options:
Target model:
Target quantization:
MTP mode: built-in / external
External MTP model, if applicable:
MTP quantization:
draft_n_max:
draft_p_min:
n_ctx:
n_batch:
GPU layers:
Sampling settings:
Ordinary decoding speed:
MTP decoding speed:
Speedup:
TTFT:
Draft tokens proposed:
Draft tokens accepted:
Acceptance rate:
Mean accepted length:
Rollback count:
Checkpoint mode and timing:
Output match or first divergence:
Peak RAM/VRAM usage:
Warnings or errors:
Additional observations:

Logs containing the final Llama.generate: draft-mtp summary are particularly useful. Please remove private prompts, model paths, usernames, or other sensitive information before posting.

Positive results are welcome, but slower results, crashes, unsupported model reports, and unusual rollback behavior are equally valuable. They help identify where the implementation needs compatibility fixes or further optimization.

Thank you to everyone willing to test and share results.

— JamePeng