Sampling Proxy

A middleware server for OpenAI-compatible backends with passthrough (OpenAI/Anthropic), Anthropic-to-OpenAI conversion, sampling params override, and mid-generation response validation.

Features

• Passthrough Modes: OpenAI, Anthropic, and Anthropic-to-OpenAI conversion
• Multiple Upstream Servers: Route different models to different backends with glob pattern matching
• Parameter Override: Apply custom sampling parameters per model
• Parallel Request Limits: Limit concurrent requests per model with automatic queueing
• Request Throttling: Configurable cooldown delays between requests per model
• Streaming Support: Both streaming and non-streaming responses
• Garbage Detection: Validate responses and auto-retry when garbage output is detected
• Mid-Stream Validation: Detect garbage during generation (not just at the end)
• Flexible Validator API: Supports Anthropic and OpenAI API formats

Quick Start

# Clone and setup
git clone https://github.com/avtc/sampling-proxy.git
cd sampling-proxy
python -m venv sampling-proxy
# Activate venv and installsource sampling-proxy/bin/activate # Linux/macOS
sampling-proxy\Scripts\activate # Windows
pip install -r requirements.txt
# Configure and run
cp config_sample.json config.json
python sampling_proxy.py

One-line Scripts (auto-activate venv)

./sampling_proxy.sh # Linux/macOS
.\sampling_proxy.ps1 # Windows

Both scripts auto-activate the sampling-proxy venv and run the proxy.

Sample Validator Setup (llama.cpp)

Run a small model as a validator for garbage detection:

llama-server --hf-repo unsloth/Qwen3.5-4B-GGUF --hf-file Qwen3.5-4B-UD-Q6_K_XL.gguf --host 127.0.0.1 --port 1235 -ngl 99 --parallel 2 --jinja -fa on -c 40000 --chat-template-kwargs "{\"enable_thinking\": false}" --temp 1 --min-p 0 --top-p 0.95 --top-k 20 --repeat-penalty 1 --presence-penalty 1.5 --cache-ram 0

Command Line Options

python sampling_proxy.py --help

Option	Description
--config, -c	Path to config JSON file
--host	Proxy server host
--port	Proxy server port
--target-base-url	Backend URL (repeatable, use name=url to target specific upstream)
--model-bindings	Model-to-upstream bindings as JSON string
--debug-logs, -d	Enable debug logging
--enforce-params, -e	Enforce parameters as JSON

Garbage Detection

Enable validation to detect and retry garbage responses.

Detected issue:

• Repetition loops: Same phrase repeated 3+ times

Key Options

Option	Description	Default
validation.enabled	Enable response validation	false
validation.validator_url	Validator endpoint URL	http://127.0.0.1:1235
validation.validator_model	Model name for validation	Qwen3.5-4B-UD-Q6_K_XL.gguf
validation.max_retries	Max retry attempts	1
validation.mid_stream_validation_enabled	Validate during streaming	true
validation.mid_stream_validation_interval_words	Check every N words	300

Mid-Stream Validation

When enabled, validates responses periodically during streaming:

• Catches repetition loops at ~300 words (configurable)
• Interrupts garbage immediately and retries
• Reduces latency by not waiting for full garbage responses

{
"validation": {
"enabled": true,
"mid_stream_validation_enabled": true,
"mid_stream_validation_interval_words": 300
}
}

Logs

Failed validation responses saved to ~/.sampling-proxy/logs/

Parallel Request Limits

Limit concurrent requests per model to prevent backend overload:

{
"parallel_limits": {
"GLM-5": 2,
"GLM-4.7": 3
}
}

When limit is reached, additional requests queue automatically. Logs show queue status:

[INFO] Queueing for GLM-5, 2 requests waiting (limit: 2)
[INFO] Slot acquired GLM-5, used: 2/2
[INFO] Slot released GLM-5, used: 1/2

Request Throttling

Add cooldown delays between requests to prevent backend overload:

{
"throttle": {
"enabled": true,
"global": {
"start_pause_seconds": null,
"end_pause_seconds": null
},
"per_model": {
"GLM-5-turbo": {
"start_pause_seconds": 1.0,
"end_pause_seconds": 5.0
}
}
}
}

Timers:

• start_pause_seconds: Cooldown after sending a request (delays the next request)
• end_pause_seconds: Cooldown after response completes (delays the next request)
• null disables the timer (default)

Per-model settings override global. Both timers default to null (disabled).

API Endpoints

Endpoint	Description
/chat/completions	OpenAI chat completions
/messages	Anthropic messages (converted)
/models	List available models
/	Health check (shows upstreams and bindings)

Multiple Upstream Servers

Route different models to different backend servers. Each upstream has its own URL, timeout, and format support.

Configuration

{
"listen": {
"host": "0.0.0.0",
"port": 8001,
"base_path": ""
},
"upstreams": [
{
"name": "local-openai",
"base_url": "http://127.0.0.1:8000/v1",
"connect_timeout_seconds": 5.0,
"timeout_seconds": 1200.0,
"supports_openai": true,
"supports_anthropic": false
},
{
"name": "zai-anthropic",
"base_url": "https://api.z.ai/api/anthropic",
"connect_timeout_seconds": 5.0,
"timeout_seconds": 1200.0,
"supports_openai": false,
"supports_anthropic": true
}
],
"model_upstream_binding": {
"glm-*": "zai-anthropic",
"*": "local-openai"
}
}

Model Binding Rules

• Exact match takes priority: "glm-5": "zai-anthropic"
• Glob patterns via fnmatch: "glm-*": "zai-anthropic" matches glm-4.7, glm-5-Turbo, etc.
• Wildcard"*" catches everything not matched above
• Patterns are evaluated in config order — first match wins
• If no pattern matches, the proxy returns a 400 error

Legacy Single-Upstream Config

The old server.target_base_url format still works — it auto-wraps as a single upstream:

{
"server": {
"target_base_url": "http://127.0.0.1:8000",
"supports_openai": true,
"supports_anthropic": false
}
}

License

MIT License