asr: add a 4-bit nvfp4 option for cards short of VRAM

[z33b0t]

PARAKEET_DTYPE=nvfp4 runs the encoder's weights in NVIDIA's 4-bit float (E2M1 in blocks of 16, each block carrying its own fp8 scale), for 0.78 GB peak VRAM against bf16's 1.43 GB. bf16 stays the default and its load path is unchanged.

It costs latency, not accuracy

Same 208-clip corpus as the fp16 work:

	WER, all 208	vs bf16	weights	RTF (5090)
bf16	2.99%	—	1.31 GB	0.0134
nvfp4	2.86%	−0.13 pp, 95% CI [−0.69, +0.30]	0.51 GB	0.0316

Not significant (p=0.69) — and at 208 clips the corpus could not resolve a difference below ~0.3 pp, so this is "no cost we can measure", not "identical". Four bits does change the exact text of 14% of clips while leaving the aggregate alone, and pushes the encoder output 5x further from fp32 than bf16 (relative L2 0.226 vs 0.044), with no non-finite activation anywhere.

Decode is 2.4x slower. The encoder is bound by kernel launches, not arithmetic — a 2.2 s clip and a 35.6 s clip both take ~27 ms — so cheaper multiplies buy nothing and the extra unpack per layer is overhead. Still ~30x faster than real time. This is a memory trade; don't reach for it expecting throughput. (int8, measured the same way, saved only 0.52 GB — four bits is the better deal here, not the riskier one.)

Why a snapshot rather than a .nemo

4-bit scales come from calibrating on real audio, not a cast, so it ships pre-quantized and is fetched from lightware-dev/parakeet-tdt-0.6b-v3 on first start like the other two. Quantizing at load would defeat the point: the GPU would hold the bf16 model first, peaking at 2.53 GB, which a card that small cannot do. Packed tensors go straight onto the device and nothing is converted there.

No pickles.torch.load reconstructs arbitrary Python objects, so loading one runs its author's code — poor for a file the daemon downloads at startup. Weights are safetensors, the recipe is JSON, and the single class it names resolves by attribute lookup against a modelopt-only allowlist.

Verification

• scripts/build_nvfp4_snapshot.py --verify reloads the finished snapshot and requires the exact transcripts the in-memory model produced (20/20) — a mis-bound scale buffer still yields fluent English, so plausibility proves nothing.
• Cold start from an empty cache downloads only the snapshot (469 MB, not the bf16 .nemo) and transcribes correctly.
• bf16 peak VRAM unchanged at 2.532 GB; both precisions survive release_cache().
• Full scripts/test_protocol.py passes, with 6 new precision checks.

Two adjacent fixes

• The lock's sanity check rejected any nvidia-* package — correct when they all came from the cu130 index, wrong for nvidia-modelopt. It now checks the exclusion list itself.
• The "missing fp16 checkpoint" test was vacuous: with a network the loader downloaded the checkpoint it was told was missing and raised nothing. It only looked green because CI has no torch and skips the block. Pre-existing on main, confirmed by running the suite there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EnL4vH3w19nXYjAUqFT2jF