A tool for comparing AI model providers and their performance across different endpoints.
Clone the vLLM repository
git clone git@github.com:vllm-project/vllm.git
Create a virtual environment
python3 -m venv venv source venv/bin/activateInstall vLLM
VLLM_USE_PRECOMPILED=1 pip install --editable vllm
Configure API keys Create accounts on the providers you want to test and add API keys to the corresponding CSV files.
Select model configuration Different providers may use different names for the same model and have varying API endpoint formats. Use a CSV file corresponding to the model you want to test.
Run the benchmark script
./compare_providers.sh 10 1 providers.csv meta-llama/Llama-4-Maverick-17B-128E-Instruct