Run LLMs on-device with Onde Inference, with first-class support for Apple silicon.
Swift SDK · Kotlin Multiplatform SDK · Flutter SDK · React Native SDK · Website
Onde is already shipping in real apps on the App Store and Google Play. Chat runs fully on-device, so there is no server round trip and no user data leaving the device. For SDK docs, platform notes, and setup details, see https://ondeinference.com/sdk. If you want to test downloads, model selection, or GGUF export before wiring the engine into app code, use Onde CLI.
Siti AI is the flagship open reference app — a private, on-device assistant built on Onde, open source under Apache-2.0. Its source is a complete, readable example of wiring the engine into a shipping Tauri app across macOS, iOS, and Android.
Onde can load GGUF models and UQFF models through the same chat engine. UQFF is mistral.rs' native pre-quantized format. Point model_id at the UQFF export — the repository or local directory holding the shards, residual.safetensors, config.json, and the tokenizer — and name the first shard (or a shorthand such as q4k) in files. Use the UQFF repository, not the original unquantized one; the export is self-contained and everything is resolved relative to it.
use onde::inference::{ChatEngine,UqffModelConfig};let engine = ChatEngine::new();
engine
.load_uqff_model(UqffModelConfig{model_id:"mistralrs-community/gemma-4-E4B-it-UQFF".into(),files:vec!["q4k-0.uqff".into()],display_name:"Gemma 4 E4B (UQFF Q4K)".into(),approx_memory:"~2.5 GB (UQFF Q4K)".into(),chat_template:None,},None,None,).await?;For sharded UQFFs, passing the first shard is enough; mistral.rs discovers sibling shards with the same prefix.
Onde is dual-licensed under MIT and Apache 2.0. You may use it under either license at your option.
| Dependency | License | Author |
|---|---|---|
| mistral.rs | MIT | Eric Buehler |
| UniFFI | MPL-2.0 | Mozilla |
| tokio | MIT | Tokio contributors |
Models downloaded by Onde have their own licenses independent of this crate. By using Onde, you are also subject to the license of the model you load:
| Model | Size | License | Commercial use |
|---|---|---|---|
| Qwen 2.5 1.5B Instruct (GGUF Q4_K_M) | ~941 MB | Qwen Community License | ✅ with conditions¹ |
| Qwen 2.5 3B Instruct (GGUF Q4_K_M) | ~1.93 GB | Qwen Community License | ✅ with conditions¹ |
| Qwen 2.5 Coder 7B Instruct (GGUF Q4_K_M) | ~4.4 GB | Qwen Community License | ✅ with conditions¹ |
| Qwen 3 1.7B (GGUF Q4_K_M) | ~1.3 GB | Apache 2.0 | ✅ |
| Qwen 3 4B (GGUF Q4_K_M) | ~2.7 GB | Apache 2.0 | ✅ |
| Qwen 3 8B (GGUF Q4_K_M) | ~5 GB | Apache 2.0 | ✅ |
| Qwen 3 14B (GGUF Q4_K_M) | ~8.4 GB | Apache 2.0 | ✅ |
| DeepSeek Coder 6.7B Instruct (GGUF Q4_K_M) | ~3.8 GB | DeepSeek License v1.0 | ✅ with conditions² |
¹ Qwen Community License conditions: no training of competing models, attribution required, no misrepresentation of origin. Organisations with more than 100 million monthly active users must obtain a separate commercial licence from Alibaba Cloud.
² DeepSeek License v1.0 conditions: use-based restrictions apply (see Attachment A of the license). Prohibits military use, generation of disinformation, and certain other uses. Governing law is PRC law.
Onde's own license (MIT OR Apache-2.0) is independent of these model licenses. If you build an application on top of Onde, you are responsible for complying with the license of whichever model your users load.
© 2026 Splitfire AB (Onde Inference).