Uh oh!
There was an error while loading. Please reload this page.
asr: add Whisper as an alternative engine, one at a time - #71
Conversation
Parakeet covers 25 European languages, which is the whole story until you dictate in Japanese. BLURT_ASR_ENGINE=whisper (or `blurtd --engine whisper`) swaps in openai/whisper-large-v3-turbo instead: 809M params, ~1.6 GB in bf16, about a hundred languages, and WHISPER_TASK=translate to have any of them typed out in English. Parakeet stays the default and its load path is untouched. One engine per process, chosen at startup. Two resident models would double the VRAM this server exists to keep small, and the choice is a property of the box rather than of the request -- so there is no per-request switch and no reload between dictations. It costs no new dependency. Whisper runs through transformers, which NeMo already pins into the image (4.57.6), on the torch build that is already there. faster-whisper/CTranslate2 is quicker, but its wheels link CUDA 12 cuBLAS/cuDNN while we ship torch cu130 -- a second CUDA runtime in the image to save time on a model that already decodes a dictation segment in a fraction of its duration. transformers is now named in requirements.in rather than inherited silently; the resolved tree is unchanged. The engines are duck-typed, not subclassed: the surface is load / transcribe / release_cache plus what the info messages report, and server/engine.py writes that contract down. Everything downstream keeps holding the single `asr` global in server/app.py and never learns which engine it got. The Wyoming info message now reads the model's description, attribution, version and languages off the engine instead of hardcoding Parakeet's, so Home Assistant's picker lists the ~100 languages when Whisper is loaded and 25 when it isn't. Three details a thinner wrapper gets wrong: Audio over 30 seconds. FINAL_MAX_S is 120 s, so the one-shot final decode of a dictation routinely exceeds Whisper's fixed encoder window. Past it, transcribe() builds untruncated features, an attention mask and timestamps for transformers' sequential long-form decode; without that branch every long dictation would come back silently truncated at 30 s. The .en checkpoints. They have no language or task tokens and raise if handed either, rather than ignoring them, so those arguments are withheld and English is the only language advertised for them. The dtype rename. transformers moved torch_dtype -> dtype around 4.56, and a version in between accepts the new name as an unknown config kwarg and returns fp32 -- twice the VRAM, no error. The loaded dtype is checked rather than trusted, and the old name is still tried on a TypeError. Whisper pads short audio out to the same 30 s window and, on a fraction of a second of near-silence, has a documented habit of filling the rest with a stock phrase from its training data. The VAD gate in front of the model is what prevents that; the defaults were tuned on Parakeet, which is far less prone to it, so README and .env.example say to raise VAD_THRESHOLD and MIN_SEGMENT_S if filler text shows up in a quiet room. Tests are the config and the plumbing, not the model: a new `engine` suite checks engine selection and aliases, that both classes still expose the same surface (a method added to one and not the other is an AttributeError in whichever listener reads it), Whisper's config resolution, and the decode branch itself against a stubbed model and processor -- the long-form argument shape is exactly the thing no CI machine has a GPU to catch. Not verified end to end here: this was written on a machine with no CUDA and no transformers, so the Whisper path has never actually decoded audio. `BLURT_ASR_ENGINE=whisper python scripts/verify_asr.py` on a GPU box is the check that matters. Claude-Session: https://claude.ai/code/session_01RYbzXVVJ6xMugnm6g7MxJE
z33b0t
commented
Aug 18, 2026
Measured in the Docker image, both engines, same boxBuilt this branch to a throwaway tag and ran it through espeak-ng utterances, 1 s of silence padded each end: short 3.44 s, medium 12.06 s, long 61.59 s (deliberately past Whisper's 30 s encoder window). Latency — WebSocket, |
| utterance | parakeet-bf16 | whisper-bf16 |
|---|---|---|
| 3.44 s | 0.062 s | 0.087 s |
| 12.06 s | 0.112 s | 0.177 s |
| 61.59 s | 0.411 s | 0.825 s |
Spread across repeats was ≤11 ms everywhere. Live partials hold the 350 ms cadence on both engines — median inter-partial gap 404 ms (parakeet) vs 407 ms (whisper), so Whisper's decode is not what sets the tick. First partial lands 0.16 s after speech onset on both; the very first decode after load costs 0.53 s on Whisper (0.18 s on Parakeet), once.
/v1/audio/transcriptions, wall clock and RTF:
| audio | parakeet | whisper |
|---|---|---|
| 3.44 s | 0.093 s (RTF 0.027) | 0.124 s (RTF 0.036) |
| 12.06 s | 0.122 s (0.010) | 0.178 s (0.015) |
| 61.59 s | 0.418 s (0.007) | 0.845 s (0.014) |
VRAM — server process off nvidia-smi (weights + CUDA context + activations)
A bare CUDA context in this image measures 612 MiB, so the weight figures below are the process footprint minus that.
| phase | parakeet-bf16 | whisper-bf16 |
|---|---|---|
| idle after load | 1908 MiB (~1296 weights, matches the documented 1.31 GB) | 2172 MiB (~1560 weights, matches 809M × 2 B) |
| typical, mid-dictation (mean) | 1942–2018 MiB | 2387–2399 MiB |
| peak, incl. the 61.6 s one-shot final | 2258 MiB | 2416 MiB |
steady after use (post release_cache) | 1956 MiB | 2336 MiB |
Whisper costs +264 MiB idle / +158 MiB at peak and roughly 1.5–2× the latency. Load from a warm cache: 3.7 s (whisper) vs 13.3 s (parakeet); first ever start, including the HF download, 27.5 s vs 31.1 s.
Correctness
The 61.59 s dictation came back complete on Whisper, ending …The library closes at 6 on weekdays and at 4 on Saturdays. — the long-form branch is doing its job; without it everything past 30 s would be gone. Both engines transcribed all three utterances correctly (Whisper normalises numerals — "30 seconds", "92 degrees" — where Parakeet writes them out).
All three protocols verified against the Whisper engine: WebSocket info reports openai/whisper-large-v3-turbo, /v1/models lists it beside the whisper-1 alias, and Wyoming describe returns Blurt — local dictation server (whisper) / OpenAI Whisper — openai/whisper-large-v3-turbo (bf16) with 100 languages (25 on Parakeet) — the engine-aware info working over the wire. Wyoming round-trip transcript correct at 0.173 s.
Test stack, volume and image tag were removed afterwards and the live server restarted.
The README has always listed `.env` as a config source and `.gitignore` has always excluded it, but nothing read it: the daemon has no dotenv (it reads the environment and nothing else) and Compose was given no `env_file`. So the documented way to configure a container was dead, and the real way was editing a tracked file — which means a local engine choice or auth token shows up in `git status` and rides along in the next diff. Compose now reads `.env` with `required: false`, so the file is genuinely optional: with none present the service still starts, still on 0.0.0.0:25878, still Parakeet. The inline list of common variables stays as a comment, but it points at `.env.example` for the full set rather than duplicating half of it in YAML with different syntax. `HOST` and `PORT` come out of the `environment:` block on the way past. They were redundant — the Dockerfile bakes both as image `ENV` — and worse than redundant: `environment:` silently outranks `env_file:`, so anyone who set `HOST` in their `.env` would have watched it be ignored with no error. With the block gone the precedence runs env_file over image ENV and `.env` is authoritative for every variable `.env.example` names. `.env.example` now says how it is consumed, since "copy this file" is only useful next to where it gets copied to. Verified both ways on the live box: `docker compose config` with a `.env` holding BLURT_ASR_ENGINE=whisper injects it and the container comes up `engine=whisper`, and with the file absent the same compose file validates and injects nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NU7BvwvQBe5UNR1erX71mc
Parakeet covers 25 European languages, which is the whole story until you dictate in Japanese.
BLURT_ASR_ENGINE=whisper(orblurtd --engine whisper) swaps inopenai/whisper-large-v3-turboinstead — 809M params, ~1.6 GB in bf16, ~100 languages, plusWHISPER_TASK=translateto have any of them typed out in English. Parakeet stays the default and its load path is untouched.parakeet(default)whispernvidia/parakeet-tdt-0.6b-v3openai/whisper-large-v3-turbo(any Hub checkpoint viaWHISPER_MODEL)WHISPER_TASK=translateOne engine per process, chosen at startup. Two resident models would double the VRAM this server exists to keep small, and the choice is a property of the box rather than of the request — so no per-request switch, no reload between dictations.
No new dependency. Whisper runs through transformers, which NeMo already pins into the image (4.57.6), on the torch build that is already there. faster-whisper/CTranslate2 is quicker, but its wheels link CUDA 12 cuBLAS/cuDNN while we ship torch cu130 — a second CUDA runtime in the image to save time on a model that already decodes a dictation segment in a fraction of its duration.
Shape
The engines are duck-typed, not subclassed — the surface is
load/transcribe/release_cacheplus what the info messages report, andserver/engine.pywrites that contract down. Everything downstream keeps holding the singleasrglobal inserver/app.pyand never learns which engine it got. Wyoming'sinfonow reads description, attribution, version and languages off the engine instead of hardcoding Parakeet's.Three details a thinner wrapper gets wrong
FINAL_MAX_Sis 120 s, so the one-shot final decode routinely exceeds Whisper's fixed encoder window. Past it,transcribe()builds untruncated features, an attention mask and timestamps for transformers' sequential long-form decode; without that branch every long dictation comes back silently truncated at 30 s..encheckpoints have no language or task tokens and raise if handed either, so those arguments are withheld and English is the only language advertised.torch_dtype→dtypearound 4.56, and a version in between accepts the new name as an unknown config kwarg and returns fp32 — twice the VRAM, no error. The loaded dtype is checked rather than trusted.Known trait
Whisper pads short audio out to the same 30 s window and, on a fraction of a second of near-silence, tends to fill the rest with a stock phrase from its training data. The VAD gate in front of the model is what prevents that; the defaults were tuned on Parakeet, which is far less prone to it, so the docs say to raise
VAD_THRESHOLD/MIN_SEGMENT_Sif filler text shows up in a quiet room.Testing
New
enginesuite inscripts/test_protocol.py(GPU-free, like the rest): engine selection and aliases, interface parity between the two classes, Whisper config resolution, and the decode branch itself against a stubbed model and processor — the long-form argument shape being exactly what no CI machine has a GPU to catch. Full suite passes on 3.12 locally (115 checks).Verified end to end in the Docker image on an RTX 5090, both engines through the same compose config and the same espeak utterances — WebSocket,
/v1and Wyoming. Headline: a 12 s dictation finalises in 0.177 s on Whisper against 0.112 s on Parakeet, a 61.6 s one in 0.825 s against 0.411 s, for +264 MiB of idle VRAM; the 61.6 s transcript comes back complete, so the long-form branch holds. Full numbers in the comment below.🤖 Generated with Claude Code
https://claude.ai/code/session_01RYbzXVVJ6xMugnm6g7MxJE