OminiX-API

OpenAI-compatible API server for OminiX-MLX models on Apple Silicon.

Features

OpenAI-compatible endpoints - Drop-in replacement for OpenAI API
LLM chat - Chat completions with Qwen3, Mistral, GLM models
Speech-to-text - Audio transcription with Paraformer ASR
Text-to-speech - Voice cloning with GPT-SoVITS (named voices, few-shot, pre-computed codes)
Image generation - Text-to-image with FLUX.2-klein and Z-Image-Turbo
WebSocket streaming TTS - MiniMax T2A compatible protocol with per-message voice switching
Voice registry - Named voices with aliases, configurable via voices.json
Dynamic model loading - Switch models at runtime without server restart
Memory efficient - One model per category, automatic unloading when switching
Pure Rust - No Python dependencies at runtime

Install

curl -fsSL https://raw.githubusercontent.com/OminiX-ai/OminiX-API/main/install.sh | sh

Requires macOS 14.0+ on Apple Silicon (M1/M2/M3/M4). The script will:

Check for Xcode Command Line Tools (needed for Metal GPU acceleration) and prompt to install if missing
Download the latest release binary and verify its SHA-256 checksum
Install ominix-api to /usr/local/bin
Create ~/.OminiX/ config directory with default settings

To pin a version or change install location:

VERSION=1.0.0 INSTALL_DIR=~/.local/bin curl -fsSL https://raw.githubusercontent.com/OminiX-ai/OminiX-API/main/install.sh | sh

Build from Source

For contributors (requires Rust 1.82+, protoc, and Xcode Command Line Tools)

# Clone both repositories
git clone https://github.com/OminiX-ai/OminiX-API.git
git clone https://github.com/OminiX-ai/OminiX-MLX.git
cd OminiX-API
cargo build --release
# Binary: target/release/ominix-api

Quick Start

# Run with LLM only (downloads model automatically)
LLM_MODEL=mlx-community/Qwen3-4B-bf16 ominix-api
# Run with all models
PORT=8080 \
LLM_MODEL=mlx-community/Qwen3-4B-bf16 \
ASR_MODEL_DIR=./models/paraformer \
TTS_REF_AUDIO=./audio/reference.wav \
IMAGE_MODEL=zimage \
ominix-api
# Run with app manifest validation
ominix-api --app-manifest my-app.ominix.toml

Environment Variables

Variable	Default	Description
`PORT`	`8080`	HTTP server port
`OMINIX_API_HOST`	`0.0.0.0` locally; `127.0.0.1` with CUDA routing	HTTP bind host
`LLM_MODEL`	`mlx-community/Mistral-7B-Instruct-v0.2-4bit`	HuggingFace model ID
`ASR_MODEL_DIR`	(empty)	Path to Paraformer model directory
`ASR_MODE`	`conversational`	ASR batching: `off`, `interactive`, `conversational`, `offline` — see docs/asr-batching.md
`ASR_MAX_BATCH`	(from mode)	Override the batch size chosen by `ASR_MODE`
`ASR_USE_ANE`	M5+ only	Allow ASR encoder work on the Neural Engine. Off on M1–M4 regardless — see Apple silicon generations
`TTS_REF_AUDIO`	(empty)	Path to reference audio for voice cloning
`IMAGE_MODEL`	(empty)	Image model: `zimage` or `flux`
`FLUX_MODEL_DIR`	(auto-download)	Custom path to FLUX.2-klein model
`ZIMAGE_MODEL_DIR`	(auto-download)	Custom path to Z-Image-Turbo model
`QWEN3_TTS_MODEL_DIR`	(empty)	Path to Qwen3-TTS model directory
`VLM_MODEL`	(empty)	VLM model ID
`VOICES_CONFIG`	`~/.dora/models/primespeech/voices.json`	Path to voice registry file
`TTS_VOICES_DIR`	(none)	Allowed directory for voice file path references
`OMINIX_APP_MANIFEST`	(none)	Path to app manifest (`ominix.toml`) for startup validation
`OMINIX_V0_SCHEDULER_URL`	(none)	Authenticated OminiX-SGLang worker-v0 shim base URL; enables explicit CUDA routing
`OMINIX_V0_SCHEDULER_MODELS`	(none)	Comma-separated, case-sensitive public model IDs routed to the shim
`OMINIX_V0_SERVED_MODEL`	(none)	Exact model identity expected from the shim's scheduler
`OMINIX_V0_SCHEDULER_TOKEN`	(none)	Internal bearer token used only for API-to-shim requests
`OMINIX_V0_SCHEDULER_TIMEOUT_SECS`	`1800`	CUDA generation timeout, from 1 to 86400 seconds
`OMINIX_V0_CHAT_TEMPLATE_KWARGS_JSON`	`{}`	Optional JSON object passed to the shim tokenizer's chat template

CUDA LLM routing through OminiX-SGLang

OminiX-API does not guess from GPU availability. Configure exact model IDs so the same public name cannot silently switch between a local MLX checkpoint and a CUDA checkpoint. Mapped requests fail closed when the remote worker is unavailable; only unmapped models use the existing local inference path.

For the C2Rust FP8 + DFlash worker described by the OminiX-SGLang recipe:

:"${OMINIX_C2RUST_WORKER_TOKEN:?set the internal worker token}"
OMINIX_V0_SCHEDULER_URL=http://127.0.0.1:19091 \
OMINIX_V0_SCHEDULER_MODELS=C2Rust-FP8-DFlash \
OMINIX_V0_SERVED_MODEL=C2Rust-FP8-DFlash \
OMINIX_V0_SCHEDULER_TOKEN="$OMINIX_C2RUST_WORKER_TOKEN" \
OMINIX_V0_SCHEDULER_TIMEOUT_SECS=1800 \
OMINIX_V0_CHAT_TEMPLATE_KWARGS_JSON='{"enable_thinking":false}' \
ominix-api

The URL must point to the authenticated worker-v0 HTTP/SSE shim, not directly to SGLang's gRPC port. Plain HTTP is accepted only for loopback; use HTTPS for a worker on another host. The API validates the configured served-model identity through /get_model_info. /health reports process liveness, while /readyz returns 503 until the configured CUDA model is reachable and ready.

OminiX-API remains a macOS/Apple-Silicon process. When OminiX-SGLang runs on a remote CUDA host, either expose the shim through an authenticated HTTPS service or carry its loopback listener over a managed SSH tunnel:

ssh -o ExitOnForwardFailure=yes -N -L 19091:127.0.0.1:19091 cuda-worker.example

With that tunnel, keep OMINIX_V0_SCHEDULER_URL=http://127.0.0.1:19091. The shim must run in gRPC mode, report the exact OMINIX_V0_SERVED_MODEL, and reject generation envelopes for any other model identity.

When CUDA routing is enabled, OminiX-API binds to loopback by default. This server does not yet implement public-client authentication; expose it through an authenticated TLS reverse proxy. Set OMINIX_API_HOST=0.0.0.0 only when that boundary (or an equivalently trusted private network) is in place. The worker token authenticates API-to-shim traffic and is not a public API key. Wildcard CORS is also disabled in CUDA-routing mode; configure browser origins at the authenticated proxy rather than exposing the loopback server directly.

The initial remote route supports plain-string text chat, including true streaming. It rejects multipart/multimodal content and structured tool calls because worker-v0 currently exposes text/token deltas but not OpenAI tool-call deltas.

Endpoints

Endpoint	Method	Description
`/health`	GET	Health check
`/readyz`	GET	Readiness check, including configured OminiX-SGLang models
`/v1/version`	GET	Server capabilities and version info
`/v1/models`	GET	List loaded models
`/v1/models/status`	GET	Get current model status for each category
`/v1/models/report`	GET	Model availability report (scanned from config + hub caches)
`/v1/models/load`	POST	Load/switch model dynamically
`/v1/models/unload`	POST	Unload model to free memory
`/v1/models/quantize`	POST	Quantize FLUX transformer to 8-bit
`/v1/chat/completions`	POST	Chat completions (LLM)
`/v1/audio/transcriptions`	POST	Speech-to-text (ASR)
`/v1/audio/speech`	POST	Text-to-speech (TTS)
`/v1/images/generations`	POST	Image generation (txt2img / img2img)
`/ws/v1/tts`	WebSocket	Streaming TTS with per-message voice switching
`/v1/voices`	GET	List registered voices
`/v1/voices/train`	POST	Start voice cloning training
`/v1/voices/train/status`	GET	Get training task status
`/v1/voices/train/progress`	GET	SSE stream of training progress
`/v1/voices/train/cancel`	POST	Cancel active training task

Text-to-Speech (TTS)

The TTS engine uses GPT-SoVITS for voice cloning. It supports three quality tiers, selected automatically based on what's available for each voice:

┌─────────────────────────────────────────────────────┐
│ Voice Quality Tiers │
├─────────────────────────────────────────────────────┤
│ │
│ 1. Pre-computed codes (best quality) │
│ ref_audio + ref_text + codes_path │
│ │
│ 2. Few-shot / HuBERT (good quality) │
│ ref_audio + ref_text (HuBERT extracts codes) │
│ │
│ 3. Zero-shot (baseline) │
│ ref_audio only (mel spectrogram matching) │
│ │
└─────────────────────────────────────────────────────┘

Voice Registry (`voices.json`)

Named voices are configured in a JSON file. The server loads this at startup from $VOICES_CONFIG or the default path ~/.dora/models/primespeech/voices.json.

{
"default_voice": "doubao",
"models_base_path": "~/.dora/models/primespeech",
"voices": {
"doubao": {
"ref_audio": "moyoyo/ref_audios/doubao_ref_mix_new.wav",
"ref_text": "这家resturant的steak很有名",
"codes_path": "gpt-sovits-mlx/doubao_mixed_codes.bin",
"speed_factor": 1.1,
"aliases": ["default"]
},
"luoxiang": {
"ref_audio": "moyoyo/ref_audios/luoxiang_ref.wav",
"ref_text": "复杂的问题背后也许没有统一的答案...",
"codes_path": "gpt-sovits-mlx/codes/luoxiang_codes.bin",
"speed_factor": 1.1,
"aliases": ["luo"]
}
}
}

Field	Required	Description
`default_voice`	No	Default voice name (used when no voice specified)
`models_base_path`	No	Base directory for resolving relative paths
`voices.<name>.ref_audio`	Yes	Reference audio file (WAV, relative to base path)
`voices.<name>.ref_text`	Yes	Transcript of the reference audio
`voices.<name>.codes_path`	No	Pre-computed semantic codes (`.bin`, best quality)
`voices.<name>.speed_factor`	No	Speaking speed multiplier
`voices.<name>.aliases`	No	Alternative names for this voice

Paths can be absolute or relative to models_base_path. Tilde (~) is expanded to $HOME.

Voice Resolution

When a voice name is provided (via HTTP voice field or WebSocket voice_id), it resolves in this order:

voice: "luo"
│
├─ 1. Registry lookup (case-insensitive)
│ ├─ Direct name match ("luo"?) → no
│ └─ Alias match ("luo" in luoxiang.aliases?) → yes
│ │
│ ├─ Has codes_path? → pre-computed codes mode
│ ├─ HuBERT available? → few-shot mode
│ └─ Otherwise → zero-shot mode
│
└─ 2. File path fallback (if not in registry)
└─ Validated for path traversal, then used as zero-shot reference

Voice switching is on-demand. The base models (BERT, T2S, VITS, HuBERT) stay loaded; only the reference conditioning changes per voice. Switching overhead is ~100-700ms depending on the voice.

HTTP TTS API

POST /v1/audio/speech

Synthesize speech from text. Supports dynamic voice switching per request.

Request:

{
"input": "Hello, this is a test.",
"voice": "luoxiang",
"model": "gpt-sovits",
"response_format": "wav",
"speed": 1.0,
"instruct": "用兴奋激动的语气说话，充满热情和活力"
}

Field	Type	Default	Description
`input`	string	(required)	Text to synthesize
`voice`	string	(none)	Voice name, alias, or file path
`model`	string	(ignored)	Accepted for OpenAI compatibility
`response_format`	string	`"pcm"`	Audio format: `"pcm"` (streaming) or `"wav"` (buffered)
`speed`	float	`1.0`	Speaking speed (0.25 - 4.0)
`reference_audio`	string	(none)	Base64-encoded reference audio for voice cloning
`language`	string	(none)	Language: chinese, english, japanese, korean
`instruct`	string	(none)	Optional natural-language style/emotion instruction for Qwen3-TTS

Response formats:

Default (PCM streaming): Raw PCM audio (16-bit signed LE, mono, 24kHz) with Content-Type: audio/pcm and Transfer-Encoding: chunked. Chunks are sent as they're generated (~800ms per chunk). Headers include X-Audio-Sample-Rate: 24000, X-Audio-Channels: 1, X-Audio-Bits-Per-Sample: 16.
WAV (buffered): Complete WAV file with Content-Type: audio/wav. Triggered by ?format=wav query parameter, "response_format": "wav", or when reference_audio is present (voice cloning always returns WAV).

For Qwen3-TTS endpoints, instruct enables speaker+instruct mode for preset voices and clone+instruct mode for voice cloning. JSON clients may also send prompt as a compatibility alias.

Examples:

# Use a named voice from the registry
curl http://localhost:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "你好世界", "voice": "doubao"}' \
-o doubao.wav
# Use a voice alias
curl http://localhost:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "你好世界", "voice": "luo"}' \
-o luoxiang.wav
# Switch voices between requests (no reconnection needed)
curl http://localhost:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "主持人说话", "voice": "marc"}' -o host.wav
curl http://localhost:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "嘉宾回应", "voice": "luoxiang"}' -o guest.wav

Python example with dynamic voice switching:

importrequestsAPI="http://localhost:8080/v1/audio/speech"dialogue= [
("marc", "大家好，欢迎来到今天的节目。"),
("luoxiang", "谢谢主持人，很高兴来到这里。"),
("yangmi", "我也很开心参加今天的讨论。"),
]
fori, (voice, text) inenumerate(dialogue):
resp=requests.post(API, json={"input": text, "voice": voice})
withopen(f"line_{i+1}_{voice}.wav", "wb") asf:
f.write(resp.content)
print(f"[{voice}] {len(resp.content)} bytes")

WebSocket TTS API

GET /ws/v1/tts (upgrade to WebSocket)

Streaming TTS over a persistent WebSocket connection. Follows the MiniMax T2A protocol with an extension for per-message voice switching.

Protocol Flow

Client Server
│ │
│ ──── WebSocket Connect ────────────► │
│ ◄──── connected_success ─────────── │
│ │
│ ──── task_start ───────────────────► │
│ ◄──── task_started ─────────────── │
│ │
│ ──── task_continue (text) ─────────► │
│ ◄──── task_progress (audio chunk) ─ │
│ ◄──── task_progress (audio chunk) ─ │
│ ◄──── task_progress (final chunk) ─ │
│ │
│ ──── task_continue (text) ─────────► │ (repeat)
│ ◄──── task_progress ... ─────────── │
│ │
│ ──── task_finish ──────────────────► │
│ ◄──── connection closed ─────────── │

1. Connection

Connect to the WebSocket endpoint. The server sends a confirmation:

{"event": "connected_success"}

2. Task Start

Configure voice and audio settings for the session:

{
"event": "task_start",
"voice_setting": {
"voice_id": "marc",
"speed": 1.0
},
"audio_setting": {
"format": "wav"
}
}

Field	Type	Default	Description
`voice_setting.voice_id`	string	(none)	Default voice for this session
`voice_setting.speed`	float	`1.0`	Speaking speed
`audio_setting.format`	string	`"wav"`	Audio format

Server responds:

{"event": "task_started"}

3. Task Continue (Synthesize)

Send text to synthesize. Optionally override the voice per message:

{
"event": "task_continue",
"text": "要合成的文本",
"voice_id": "luoxiang"
}

Field	Type	Required	Description
`text`	string	Yes	Text to synthesize
`voice_id`	string	No	Override voice for this message (falls back to `task_start` voice)

The server synthesizes the audio using native streaming generation and sends PCM chunks as they're produced (~800ms of audio per chunk, hex-encoded):

{"event": "task_progress", "data": {"audio": "52494646..."}, "is_final": false}
{"event": "task_progress", "data": {"audio": "a1b2c3d4..."}, "is_final": false}
{"event": "task_progress", "data": {"audio": "e5f6a7b8..."}, "is_final": true}

The data.audio field contains hex-encoded raw audio bytes. The is_final flag indicates the last chunk for this text segment. Decode with bytes.fromhex(audio_hex).

Processing is sequential: the next task_continue is not processed until the current one finishes streaming all chunks.

4. Task Finish

Close the session:

{"event": "task_finish"}

Error Handling

Errors are sent as:

{"event": "error", "message": "Description of the error"}

Common errors:

"Missing 'text' field" - task_continue without text
"TTS error: ..." - Synthesis failure
"Inference unavailable" - TTS model not loaded
"Invalid JSON" - Malformed message
"Unknown event: ..." - Unrecognized event type

Multi-Participant Example (Python)

Use per-message voice_id to switch speakers within a single connection:

importasyncioimportjsonimportstructimportwebsocketsasyncdefmulti_voice_dialogue():
dialogue= [
("marc", "大家好，欢迎来到今天的节目。"),
("luoxiang", "谢谢主持人。我来谈谈法律方面的问题。"),
("yangmi", "我觉得AI在创意领域也很有潜力。"),
("marc", "两位说得都有道理，感谢参与讨论！"),
]
asyncwithwebsockets.connect("ws://localhost:8080/ws/v1/tts") asws:
awaitws.recv() # connected_success# Configure session (default voice, can be overridden per message)awaitws.send(json.dumps({
"event": "task_start",
"voice_setting": {"voice_id": "marc", "speed": 1.0},
"audio_setting": {"format": "wav"}
}))
awaitws.recv() # task_startedall_pcm=b""forspeaker, textindialogue:
# Send text with per-message voice overrideawaitws.send(json.dumps({
"event": "task_continue",
"text": text,
"voice_id": speaker
}))
# Collect audio chunksaudio=b""whileTrue:
resp=json.loads(awaitws.recv())
ifresp["event"] =="error":
print(f"Error: {resp['message']}")
breakhex_data=resp.get("data", {}).get("audio", "")
ifhex_data:
audio+=bytes.fromhex(hex_data)
ifresp.get("is_final"):
breakprint(f"[{speaker:8s}] {len(audio):>7d} bytes | {text}")
# Save individual linewithopen(f"{speaker}_{text[:10]}.wav", "wb") asf:
f.write(audio)
# Accumulate PCM (skip 44-byte WAV header)iflen(audio) >44:
all_pcm+=audio[44:]
# Write combined WAVwithopen("full_dialogue.wav", "wb") asf:
f.write(b"RIFF")
f.write(struct.pack("<I", 36+len(all_pcm)))
f.write(b"WAVEfmt ")
f.write(struct.pack("<IHHIIHH", 16, 1, 1, 32000, 64000, 2, 16))
f.write(b"data")
f.write(struct.pack("<I", len(all_pcm)))
f.write(all_pcm)
print(f"\nSaved full_dialogue.wav ({len(all_pcm) /2/32000:.1f}s)")
awaitws.send(json.dumps({"event": "task_finish"}))
asyncio.run(multi_voice_dialogue())

Single Voice Example (Python)

importasyncioimportjsonimportwebsocketsasyncdefsimple_tts():
asyncwithwebsockets.connect("ws://localhost:8080/ws/v1/tts") asws:
awaitws.recv() # connected_successawaitws.send(json.dumps({
"event": "task_start",
"voice_setting": {"voice_id": "doubao", "speed": 1.0},
"audio_setting": {"format": "wav"}
}))
awaitws.recv() # task_started# Send multiple sentences (processed sequentially)fortextin ["第一句话。", "第二句话。", "第三句话。"]:
awaitws.send(json.dumps({
"event": "task_continue",
"text": text
}))
audio=b""whileTrue:
resp=json.loads(awaitws.recv())
hex_data=resp.get("data", {}).get("audio", "")
ifhex_data:
audio+=bytes.fromhex(hex_data)
ifresp.get("is_final"):
breakprint(f"{len(audio)} bytes for: {text}")
awaitws.send(json.dumps({"event": "task_finish"}))
asyncio.run(simple_tts())

JavaScript/Node.js Example

constWebSocket=require('ws');constws=newWebSocket('ws://localhost:8080/ws/v1/tts');ws.on('open',()=>{console.log('Connected');});ws.on('message',(data)=>{constmsg=JSON.parse(data);switch(msg.event){case'connected_success':
// Configure voicews.send(JSON.stringify({event: 'task_start',voice_setting: {voice_id: 'doubao',speed: 1.0},audio_setting: {format: 'wav'}}));break;case'task_started':
// Start synthesizingws.send(JSON.stringify({event: 'task_continue',text: '你好，这是一个测试。'}));break;case'task_progress':
constaudioHex=msg.data.audio;constaudioBuffer=Buffer.from(audioHex,'hex');// Process audio chunk (e.g., write to file, play back)console.log(`Received ${audioBuffer.length} bytes (final: ${msg.is_final})`);if(msg.is_final){ws.send(JSON.stringify({event: 'task_finish'}));}break;case'error':
console.error('Error:',msg.message);break;}});

Chat Completions (LLM)

POST /v1/chat/completions

curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{ "model": "qwen3", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"} ], "temperature": 0.7, "max_tokens": 256 }'

Response:

{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1706000000,
"model": "qwen3",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}

Audio Transcription (ASR)

POST /v1/audio/transcriptions

# Encode audio as base64
AUDIO_B64=$(base64 -i audio.wav)
curl http://localhost:8080/v1/audio/transcriptions \
-H "Content-Type: application/json" \
-d "{\"file\": \"$AUDIO_B64\",\"model\": \"paraformer\" }"

Response:

{
"text": "Hello, this is a test.",
"language": "zh",
"duration": 2.5
}

Image Generation

POST /v1/images/generations

# Text-to-image with Z-Image (faster)
curl http://localhost:8080/v1/images/generations \
-H "Content-Type: application/json" \
-d '{ "prompt": "A beautiful sunset over the ocean, digital art", "model": "zimage", "size": "512x512", "n": 1 }'# Text-to-image with FLUX.2-klein (higher quality)
curl http://localhost:8080/v1/images/generations \
-H "Content-Type: application/json" \
-d '{ "prompt": "A cat sitting on a windowsill", "model": "flux", "size": "1024x1024" }'

Response:

{
"created": 1706000000,
"data": [{
"b64_json": "/9j/4AAQSkZJRg...",
"revised_prompt": "A beautiful sunset over the ocean, digital art"
}]
}

Image-to-Image (img2img)

# Encode reference image as base64
IMG_B64=$(base64 -i reference.png)
curl http://localhost:8080/v1/images/generations \
-H "Content-Type: application/json" \
-d "{\"prompt\": \"Make it look like a watercolor painting\",\"model\": \"zimage\",\"size\": \"512x512\",\"image\": \"$IMG_B64\",\"strength\": 0.75 }"

Dynamic Model Management

The server supports one model per category (LLM, ASR, TTS, Image). Models can be loaded, switched, and unloaded at runtime without restarting the server.

┌─────────────────────────────────────────────────────────┐
│ Model Slots │
├─────────────┬─────────────┬─────────────┬───────────────┤
│ LLM │ ASR │ TTS │ Image │
│ (1 slot) │ (1 slot) │ (1 slot) │ (1 slot) │
├─────────────┼─────────────┼─────────────┼───────────────┤
│ Qwen/Mistral│ Paraformer │ GPT-SoVITS │ FLUX/Z-Image │
└─────────────┴─────────────┴─────────────┴───────────────┘

Check Model Status

curl http://localhost:8080/v1/models/status

Response:

{
"status": "success",
"models": {
"llm": "mlx-community/Qwen2.5-7B-Instruct-4bit",
"asr": null,
"tts": null,
"image": "zimage"
}
}

Load/Switch Models

# Load LLM
curl http://localhost:8080/v1/models/load \
-H "Content-Type: application/json" \
-d '{"model": "mlx-community/Qwen2.5-7B-Instruct-4bit", "model_type": "llm"}'# Load ASR
curl http://localhost:8080/v1/models/load \
-H "Content-Type: application/json" \
-d '{"model": "/path/to/paraformer/model", "model_type": "asr"}'# Load TTS (provide reference audio)
curl http://localhost:8080/v1/models/load \
-H "Content-Type: application/json" \
-d '{"model": "/path/to/reference/audio.wav", "model_type": "tts"}'# Load Image model
curl http://localhost:8080/v1/models/load \
-H "Content-Type: application/json" \
-d '{"model": "zimage", "model_type": "image"}'

Unload Models (Free Memory)

# Unload one model type
curl http://localhost:8080/v1/models/unload \
-H "Content-Type: application/json" \
-d '{"model_type": "llm"}'# Unload all models
curl http://localhost:8080/v1/models/unload \
-H "Content-Type: application/json" \
-d '{"model_type": "all"}'

Model Type Reference

Type	Load Parameter	Description
`llm`	HuggingFace model ID	e.g., `mlx-community/Qwen2.5-7B-Instruct-4bit`
`asr`	Path to model directory	Directory with `paraformer.safetensors`
`tts`	Path to reference audio	WAV file for voice cloning
`image`	`zimage` or `flux`	Image generation model

Model Setup

LLM Models

Models download automatically from HuggingFace. Recommended:

Model	HuggingFace ID	Memory
Qwen3-4B	`mlx-community/Qwen3-4B-bf16`	8 GB
Qwen3-8B	`mlx-community/Qwen3-8B-8bit`	8 GB
Mistral-7B	`mlx-community/Mistral-7B-Instruct-v0.3-4bit`	4 GB

ASR Model (Paraformer)

# Download model files
huggingface-cli download funaudiollm/paraformer-large-mlx --local-dir ./models/paraformer

Required files:

models/paraformer/
├── paraformer.safetensors # Model weights
├── am.mvn # CMVN normalization
└── tokens.txt # Vocabulary (8404 tokens)

TTS Model (GPT-SoVITS)

GPT-SoVITS uses few-shot voice cloning. Provide a reference audio file:

# Reference audio requirements:# - WAV format, 16kHz or higher# - 3-10 seconds of clean speech# - Single speaker, minimal background noise
TTS_REF_AUDIO=./audio/reference.wav cargo run --release

The voice registry (voices.json) is loaded automatically if present. See the Voice Registry section for configuration.

Image Models

Image models download automatically from HuggingFace. Choose between:

Model	ID	Steps	Memory	Speed
Z-Image-Turbo	`zimage`	9	~12 GB	~3s
FLUX.2-klein	`flux`	4	~13 GB	~5s

Model Download URLs

FLUX.2-klein (MLX format):

Source	URL
HuggingFace	https://huggingface.co/black-forest-labs/FLUX.2-klein-4B
ModelScope	https://modelscope.cn/models/black-forest-labs/FLUX.2-klein-4B

Z-Image-Turbo (MLX format):

Source	URL
HuggingFace	https://huggingface.co/uqer1244/MLX-z-image

Original Models (for reference):

Model	Original Source
FLUX.2-klein	https://huggingface.co/black-forest-labs/FLUX.1-schnell
Z-Image-Turbo	https://huggingface.co/Zheng-Peng-Fei/Z-Image

Environment Variables for Image Models

# Use specific model (auto-downloads if not present)
IMAGE_MODEL=flux cargo run --release # Use FLUX.2-klein
IMAGE_MODEL=zimage cargo run --release # Use Z-Image-Turbo# Use custom local model path (optional)
FLUX_MODEL_DIR=/path/to/flux-model cargo run --release
ZIMAGE_MODEL_DIR=/path/to/zimage-model cargo run --release

Manual Download

# Download FLUX.2-klein
huggingface-cli download black-forest-labs/FLUX.2-klein-4B --local-dir ./models/flux
# Download Z-Image-Turbo
huggingface-cli download uqer1244/MLX-z-image --local-dir ./models/zimage
# Or using git lfs
git lfs install
git clone https://huggingface.co/black-forest-labs/FLUX.2-klein-4B ./models/flux
git clone https://huggingface.co/uqer1244/MLX-z-image ./models/zimage

Model Directory Structure

models/flux/ # FLUX.2-klein
├── transformer/
│ └── diffusion_pytorch_model.safetensors
├── text_encoder/
│ ├── model-00001-of-00002.safetensors
│ └── model-00002-of-00002.safetensors
├── vae/
│ └── diffusion_pytorch_model.safetensors
└── tokenizer/
└── tokenizer.json
models/zimage/ # Z-Image-Turbo
├── transformer/
│ └── model.safetensors
├── text_encoder/
│ └── model.safetensors
├── vae/
│ └── diffusion_pytorch_model.safetensors
└── tokenizer/
└── tokenizer.json

Versioning & App Manifests

OminiX-API includes a capability versioning system. Each OminiX-MLX model crate declares its capabilities in an ominix.toml manifest. At build time, these are compiled into a registry that apps can query at runtime or validate against at startup.

Version Endpoint

GET /v1/version

Returns all available capabilities, their versions, and currently loaded models:

curl http://localhost:8080/v1/version

{
"ominix_api": "0.1.0",
"capabilities": {
"qwen3-tts": {
"name": "qwen3-tts",
"version": "1.0.0",
"category": "tts",
"description": "Qwen3 TTS with preset voices and voice cloning",
"capabilities": ["streaming", "voice_cloning", "preset_voices"]
},
"qwen3-asr": {
"name": "qwen3-asr",
"version": "0.1.0",
"category": "asr",
"description": "Qwen3 ASR speech recognition",
"capabilities": ["multilingual"]
},
"flux-klein": {
"name": "flux-klein",
"version": "0.1.0",
"category": "image",
"description": "FLUX.2-klein image generation",
"capabilities": ["quantization", "text_to_image"]
}
},
"models_loaded": []
}

App Manifest (`ominix.toml`)

Apps can declare their requirements in an ominix.toml file. OminiX-API validates these at startup when launched with --app-manifest:

ominix-api --app-manifest my-app.ominix.toml
# or via environment variable:
OMINIX_APP_MANIFEST=my-app.ominix.toml ominix-api

Example manifest:

[app]
name = "my-voice-bot"version = "0.1.0"ominix_api = ">=0.1.0"
[requires]
tts = ">=1.0.0"asr = ">=0.1.0"llm = ">=0.1.0"

Requirements are checked by category (tts, asr, llm, image, vlm, ocr). Version constraints use semver syntax.

Advanced: require specific capabilities:

[app]
name = "voice-cloning-app"version = "0.1.0"
[requires.tts]
version = ">=1.0.0"
[requires.tts.capabilities]
voice_cloning = truestreaming = true

If any requirement is not satisfied, the server exits with a clear error message listing what's missing.

OminiX-MLX Capability Manifest

Each model crate in OminiX-MLX has an ominix.toml describing what it provides:

# qwen3-tts-mlx/ominix.toml
[package]
name = "qwen3-tts"version = "1.0.0"category = "tts"description = "Qwen3 TTS with preset voices and voice cloning"
[capabilities]
streaming = truevoice_cloning = truepreset_voices = true

These are read at build time by build.rs and compiled into the binary. When you update a crate's version or capabilities, just rebuild OminiX-API.

Available Capabilities

Category	Crate	Version	Key Capabilities
`tts`	qwen3-tts	1.0.0	streaming, voice_cloning, preset_voices
`asr`	qwen3-asr	0.1.0	multilingual
`llm`	qwen3-llm	0.1.0	streaming, tool_use, thinking
`image`	flux-klein	0.1.0	text_to_image, quantization
`image`	zimage	0.1.0	text_to_image, quantization
`vlm`	moxin-vlm	0.1.0	image_understanding, quantization
`ocr`	deepseek-ocr2	0.1.0	pdf, document, grounding

Project Structure

src/
main.rs Entry point — tracing, channels, thread spawning, server start
config.rs Config struct (from environment variables and CLI args)
state.rs AppState (shared across HTTP handlers)
error.rs render_error() helper for OpenAI-compatible error responses
router.rs Builds the Salvo Router with all endpoint routes
version.rs Capability registry, app manifest parsing, requirement validation
inference/
request.rs InferenceRequest enum — the message protocol between handlers and inference thread
thread.rs inference_thread() — owns all models, processes requests via channel
handlers/
helpers.rs get_state(), send_and_wait() — dedup boilerplate across handlers
health.rs GET /health, /v1/models, /v1/models/status, /v1/models/report
version.rs GET /v1/version — capability registry and version info
chat.rs POST /v1/chat/completions
audio.rs POST /v1/audio/transcriptions, /v1/audio/speech
image.rs POST /v1/images/generations, /v1/models/quantize
models.rs POST /v1/models/load, /v1/models/unload
training.rs Voice training endpoints + GET /v1/voices
ws_tts.rs WebSocket streaming TTS (MiniMax T2A protocol)
engines/
llm.rs LLM inference — Qwen3, GLM-4.7-Flash backends
asr.rs ASR inference — Paraformer, SenseVoice+Qwen backends
tts.rs TTS synthesis — GPT-SoVITS with voice registry
qwen3_tts.rs Qwen3-TTS engine — streaming synthesis, voice cloning (x-vector)
image.rs Image generation — FLUX.2-klein, Z-Image-Turbo
vlm.rs Vision-Language Model inference
types/
chat.rs ChatCompletionRequest/Response, ChatMessage, ChatUsage
audio.rs TranscriptionRequest/Response, SpeechRequest
image.rs ImageGenerationRequest/Response, ImageData
training.rs VoiceTrainRequest, TrainingStage, TrainingProgressEvent
error.rs ApiError, ApiErrorDetail
voice.rs VoiceInfo, VoiceListResponse
model_config.rs Model registry, hub cache scanning, availability checking
training.rs Voice cloning pipeline (5 stages on dedicated thread)
utils.rs Path utilities and security helpers
build.rs Reads OminiX-MLX ominix.toml manifests, generates capability_registry.rs
examples/
voice-bot.ominix.toml Example app manifest for startup validation

Adding a New Model Type

To add a new capability (e.g., video generation):

engines/video.rs — Engine struct with new() and generate()
types/video.rs — Request/response types (add pub use in types/mod.rs)
inference/request.rs — Add Video + LoadVideoModel variants to the enum
inference/thread.rs — Add model slot + match arms (3 lines each with load_model_slot helper)
handlers/video.rs — Handler function (~15 lines using send_and_wait)
router.rs — Add one .push() line

Architecture

┌──────────────────────────────────────────────────────────────┐
│ HTTP/WS Server (Salvo) │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐│
│ │ /chat │ │ /asr │ │ /tts │ │ /images │ │ ws/tts ││
│ │ (REST) │ │ (REST) │ │ (REST) │ │ (REST) │ │ (WS) ││
│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └───┬────┘│
│ │ │ │ │ │ │
│ └───────────┴───────────┴────────────┴──────────┘ │
│ │ │
│ mpsc::channel (bounded: 32) │
│ │ │
└───────────────────────────────┼──────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ Inference Thread (owns all models) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐ │
│ │ LLM Slot │ │ ASR Slot │ │ TTS Slot (legacy) │ │
│ │ Qwen3 / │ │ Qwen3-ASR / │ │ GPT-SoVITS engine │ │
│ │ Mistral / │ │ Paraformer /│ │ + voice registry │ │
│ │ (empty) │ │ (empty) │ │ + on-demand switching│ │
│ └──────────────┘ └──────────────┘ └───────────────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐ │
│ │ Image Slot │ │ VLM Slot │ │ Qwen3-TTS Slot │ │
│ │ Z-Image / │ │ Qwen2.5-VL /│ │ CustomVoice (preset) │ │
│ │ FLUX / │ │ (empty) │ │ Base (voice cloning) │ │
│ │ (empty) │ │ │ │ + streaming synthesis│ │
│ └──────────────┘ └──────────────┘ └───────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

Key Design Points:

Actor Model: MLX models don't implement Send/Sync, so all models run on a dedicated inference thread. HTTP and WebSocket handlers communicate via bounded async channels.
Explicit CUDA Routing: Exact model IDs can bypass the MLX actor and use an authenticated OminiX-SGLang worker-v0 shim. The route is configuration-driven and never falls back locally after a remote failure.
One Model Per Slot: Each category has exactly one model slot. Loading a new model automatically unloads the previous one to free GPU memory.
Shared Inference Path: Both the REST POST /v1/audio/speech and WebSocket ws/v1/tts endpoints use the same InferenceRequest::SpeechStream channel to the inference thread. Voice switching works identically for both.
Streaming TTS: The default response format is raw PCM (16-bit signed LE, mono, 24kHz) streamed via chunked transfer encoding. Each chunk is ~800ms of audio generated by qwen3-tts-mlx's native streaming API (Synthesizer::start_streaming → StreamingSession::next_chunk). This eliminates timeout issues for long text — chunks flow continuously regardless of total audio length. Clients that need WAV can request it explicitly via ?format=wav query parameter or "response_format": "wav" in the JSON body.
TTS Model Auto-Switching: Two Qwen3-TTS model variants exist — Base (with ECAPA-TDNN speaker encoder for voice cloning) and CustomVoice (with preset speakers like vivian, serena, ryan). The inference thread auto-switches between them based on request type: preset voice → CustomVoice, reference audio → Base. Models are discovered dynamically by scanning ~/.OminiX/models/ for directories containing "tts" and "base" or "customvoice" in the name.
Dynamic Loading: Models can be loaded, switched, and unloaded at runtime via /v1/models/load and /v1/models/unload endpoints.
Memory Efficient: Unused model slots remain empty. Unload models you're not using to free GPU memory for other tasks.

Concurrency Model & Limitations

The current architecture uses a single inference thread that processes all requests sequentially via an mpsc channel. This means:

A long TTS generation (30s-2min) blocks all other inference (chat, ASR, image)
Concurrent TTS requests queue up — session B waits until session A finishes
Streaming solves client timeouts but not head-of-line blocking

Workaround: Run separate instances per modality. For production deployments (e.g. crew gateway), run dedicated ominix-api instances for each service:

# ASR instance on port 8080
ASR_MODEL_DIR=~/.OminiX/models/Qwen3-ASR-1.7B-8bit PORT=8080 ./ominix-api
# TTS instance on port 8081
QWEN3_TTS_MODEL=~/.OminiX/models/Qwen3-TTS-12Hz-1.7B-CustomVoice-8bit PORT=8081 ./ominix-api

This ensures ASR and TTS never block each other. No code changes needed.

Roadmap: Dynamic Instance Pool

For high-concurrency TTS workloads, the planned architecture is a dynamic instance pool:

┌──────────────────────────────────────────────────────┐
│ TTS Instance Pool │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Engine 0│ │ Engine 1│ │ Engine N│ (spawned on │
│ │ (always)│ │ (on │ │ demand)│ demand when │
│ │ │ │ demand)│ │ │ RAM allows) │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ └────────────┼────────────┘ │
│ │ │
│ Request Router │
│ (route to idle instance, │
│ spawn new if all busy │
│ and RAM available) │
└──────────────────────────────────────────────────────┘

Each Qwen3-TTS 1.7B 8-bit instance uses ~2GB RAM
On a 128GB machine, 3-5 concurrent TTS instances are feasible
Instances are spawned on demand when all existing instances are busy
Idle instances are reclaimed after a configurable timeout
System RAM is checked before spawning (sysctl hw.memsize - current usage)

Apple silicon generations

The server identifies the host chip at startup and reports it:

INFO Host: chip=M5 gpu_neural_accelerators=true asr_ane=enabled

Generation	GPU Neural Accelerators	ASR on the Neural Engine
M1 – M4	no	not used
M5 and later	yes (Metal 4 tensor ops)	permitted

Every Apple silicon Mac has a Neural Engine, but reaching it requires a Core ML port — MLX has no ANE backend on any generation. Through M4 that port is not worth making for ASR: the ANE cannot beat the GPU path at transformer decode, and running it alongside returns little. M5 changes the arithmetic, because its GPU gained Neural Accelerators and the two became genuinely separate units worth running at once. So ANE use is enabled from M5 onward and off before it.

ASR_USE_ANE=1|0 overrides in either direction, for benchmarking a generation the default excludes. Note that the ANE encoder-offload path is not yet implemented — this setting currently expresses policy and gates what a future offload scheduler is permitted to do.

Performance

ASR — Qwen3-ASR on Apple M5 Max (40 GPU cores, 128 GB)

Measured against LibriSpeech test-clean — 2,620 real utterances, 5.4 h of audio, scored for word error rate so the throughput numbers cannot come from degenerate output. Sessions are open-loop: each simulated speaker produces one second of audio per second of wall clock.

Configuration	Concurrent sessions	p50 latency	WER
1.7B 8-bit, 1 process, no batching	31	754 ms	1.69%
1.7B 4-bit, 1 process, no batching	40	290 ms	1.30%
1.7B 4-bit, 1 process, batching	70	487 ms	1.30%
1.7B 4-bit, 2 processes, batching	90	1,098 ms	1.30%
1.7B 4-bit, 3 processes, batching	105	1,725 ms	1.30%

Recommended operating point: ~95 sessions across 3 processes (p50 598 ms). Saturation is abrupt — latency goes near-vertical within a few extra sessions — so leaving headroom matters more than squeezing out the last five channels. At 90 sessions the GPU runs at 84% mean utilisation (95% p90); memory is 5.1 GB total, nowhere near a constraint on 128 GB.

Batching throughput at 16 concurrent clients:

`ASR_MODE`	Max batch	Throughput	p50	vs `off`
`off`	1	38.0 ×RT	2,391 ms	1.00×
`interactive`	4	66.8 ×RT	1,270 ms	1.76×
`conversational`	8	73.2 ×RT	1,249 ms	1.93×
`offline`	32	79.8 ×RT	981 ms	2.10×

Batching improves latency as well as throughput here: at a fixed number of waiting clients the queue drains roughly twice as fast. Correctness is unaffected — across 48 utterances, transcripts produced in batches of 8 were identical to the same utterances transcribed one at a time.

Model variants, single stream:

Model	Size	Throughput	p50	WER
Qwen3-ASR 1.7B 8-bit	2.3 GB	38.3 ×RT	154 ms	1.69%
Qwen3-ASR 1.7B 4-bit	1.5 GB	46.2 ×RT	103 ms	1.30%
Qwen3-ASR 0.6B 8-bit	0.97 GB	64.2 ×RT	74 ms	2.86%

4-bit is both faster and more accurate than 8-bit on this model — decode is memory-bandwidth-bound, so halving the weight bytes attacks the dominant cost.

Requires MLX 0.32.0 or newer. Earlier builds compute RoPE incorrectly at sequence length 1 with batch > 1, which is exactly the shape batched decode uses; the server probes for the defect at startup and falls back to single-request decoding rather than returning corrupt transcripts. See docs/asr-batching.md.

Other tasks

Benchmarks on Apple M3 Max (128GB):

Task	Model	Throughput	Memory
LLM	Qwen3-4B	45 tok/s	8 GB
ASR	Paraformer	18x real-time	500 MB
TTS	GPT-SoVITS	~3x real-time	2 GB
TTS voice switch	-	~100-700ms overhead	-
Image	Z-Image-Turbo	~3s/image	12 GB
Image	FLUX.2-klein	~5s/image	13 GB

Python Client Example

importopenaiclient=openai.OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed"# No auth required
)
# Chat completionresponse=client.chat.completions.create(
model="qwen3",
messages=[
{"role": "user", "content": "Hello!"}
]
)
print(response.choices[0].message.content)
# Image generationresponse=client.images.generate(
model="zimage",
prompt="A cat in space",
size="512x512"
)
print(response.data[0].b64_json[:50] +"...")

License

MIT OR Apache-2.0

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
docs		docs
examples		examples
qwen-tts-ascend-sys		qwen-tts-ascend-sys
scripts		scripts
src		src
.env-ominix		.env-ominix
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
RELEASING.md		RELEASING.md
build.rs		build.rs
cliff.toml		cliff.toml
install.sh		install.sh
release.toml		release.toml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

OminiX-API

Features

Install

Build from Source

Quick Start

Environment Variables

CUDA LLM routing through OminiX-SGLang

Endpoints

Text-to-Speech (TTS)

Voice Registry (voices.json)

Voice Resolution

HTTP TTS API

WebSocket TTS API

Protocol Flow

1. Connection

2. Task Start

3. Task Continue (Synthesize)

4. Task Finish

Error Handling

Multi-Participant Example (Python)

Single Voice Example (Python)

JavaScript/Node.js Example

Chat Completions (LLM)

Audio Transcription (ASR)

Image Generation

Image-to-Image (img2img)

Dynamic Model Management

Check Model Status

Load/Switch Models

Unload Models (Free Memory)

Model Type Reference

Model Setup

LLM Models

ASR Model (Paraformer)

TTS Model (GPT-SoVITS)

Image Models

Model Download URLs

Environment Variables for Image Models

Manual Download

Model Directory Structure

Versioning & App Manifests

Version Endpoint

App Manifest (ominix.toml)

OminiX-MLX Capability Manifest

Available Capabilities

Project Structure

Adding a New Model Type

Architecture

Concurrency Model & Limitations

Roadmap: Dynamic Instance Pool

Apple silicon generations

Performance

ASR — Qwen3-ASR on Apple M5 Max (40 GPU cores, 128 GB)

Other tasks

Python Client Example

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Voice Registry (`voices.json`)

App Manifest (`ominix.toml`)