asr: add a 4-bit nf4 option for the Whisper engine

[z33b0t]

WHISPER_DTYPE=nf4 runs the Whisper engine's weights in bitsandbytes' NF4 — 0.53 GB instead of 1.62 GB, same WER, 1.7× the decode latency. Activations stay bf16, so it needs Ampere or newer like the default does.

Closes the gap the module docstring used to state as fact ("there is no 4-bit option here").

Why NF4 and not the NVFP4 snapshot Parakeet ships

scripts/compare_whisper_quant.py (added here) put four 4-bit options through the same 208-clip corpus, calibration material, and paired bootstrap as the Parakeet precision study. All four hold WER within noise and all four land at 0.53–0.60 GB of weights, so accuracy and memory don't separate them. Latency does:

	WER, all 208	vs bf16	weights / peak	median latency	build
bf16	4.64%	—	1.62 / 1.70 GB	111 ms	6.3 s
nvfp4	4.75%	+0.10 pp [−0.55, +0.72]	0.60 / 0.73 GB	452 ms (4.06×)	39 s
nvfp4-awq	4.34%	−0.31 pp [−0.80, +0.11]	0.60 / 0.74 GB	453 ms (4.07×)	528 s
int4-awq	4.87%	+0.23 pp [−0.51, +0.99]	0.59 / 0.71 GB	453 ms (4.08×)	629 s
nf4	4.36%	−0.28 pp [−0.95, +0.31]	0.53 / 0.61 GB	184 ms (1.66×)	7.8 s

The three nvidia-modelopt formats cost 4.1× because they unpack each weight back to bf16 and call an ordinary GEMM; bitsandbytes fuses dequantization into its matmul. Parakeet doesn't pay that — its encoder is launch-bound, so a 2 s clip and a 36 s clip both take ~27 ms — which is why nvfp4 is the right trade there and the wrong one here, with Whisper's decoder generating a token at a time.

NF4 also needs no snapshot infrastructure. modelopt quantizes from the loaded bf16 weights and peaks at 1.8 GB — above the model it replaces, useless on a card that couldn't hold bf16, which is the whole reason server/nvfp4.py exists. bitsandbytes quantizes layer by layer as the checkpoint streams in and peaks at 0.81 GB. No calibration corpus, no pre-quantized download, no build step.

No variant produced an empty transcript or any runaway hallucination. 4 bits rewrites the exact text of ~18% of clips without moving the aggregate.

What's here

• server/whisper.py — nf4 joins bf16/fp16 (aliases 4bit, int4). _require_device() factored out of load() with an nf4-specific message; _nf4_config() builds the BitsAndBytesConfig (double-quant, bf16 compute, proj_out skipped since it shares storage with the token embedding). The quantized branch skips .to() and the dtype check, and instead fails loudly if bitsandbytes hands back an unquantized model — otherwise it would use full bf16 VRAM while claiming 4-bit.
• scripts/compare_whisper_quant.py — the harness. Reuses compare_precision.py's scoring and paired bootstrap so the two reports are comparable. Refuses to run if the eval and calibration corpora share a clip. Runs each variant in its own subprocess: modelopt's calibration leaks ~0.5 GB of live tensors that no empty_cache reclaims, and in a shared process that lands on whichever variant runs next — which is how the first pass measured NF4 at 1.13 GB.
• requirements.in / .txt — bitsandbytes, lock regenerated (+10 lines, no resolution churn).
• scripts/test_protocol.py — the old test asserted int4 was rejected, now an nf4 alias. Replaced with alias coverage plus a rejection test that still pins nvfp4 as Parakeet-only.
• README / .env.example — new "Whisper on a small card — nf4" section with the numbers and the format rationale.

Verified

• scripts/test_protocol.py — all checks pass
• nf4 via create_asr('whisper') on the GPU: 0.529 GB weights, 0.810 GB build peak, correct transcripts
• bf16 still loads with bitsandbytes absent (the shared path doesn't need the new dep)
• nf4 with bitsandbytes absent: actionable error, raised before the "loading" line
• the hashed lock installs and imports bitsandbytes inside the image

Not done

• No end-to-end image rebuild — the pinned wheel installs and imports in the existing image, but the CUDA layers weren't rebuilt. Worth doing before a release image.
• sm_75/sm_80 untested — measured only on sm_120. Pre-Ampere is refused with a message pointing at fp16; whether NF4's 1.7× holds on Ampere/Ada is unknown.
• English read prose only — LibriSpeech is the only ground truth available, so this says nothing about the ~100 languages that are the reason to run Whisper at all.

🤖 Generated with Claude Code

https://claude.ai/code/session_01376qJiterkkQwTWmN1AC4Q