asr: run on pre-Ampere GPUs with an fp16 checkpoint alongside bf16

[z33b0t]

bf16 needs Ampere. A GTX 1660 or RTX 20xx (sm_75) has no bf16 support at all, so
blurtd refused to start on hardware that is otherwise perfectly capable of running
this model — the image's torch already targets sm_75.

PARAKEET_DTYPE=fp16 now selects a float16 checkpoint instead:

PARAKEET_DTYPE=fp16 ./blurtd

bf16 stays the default and its load path is unchanged. The two checkpoints live
side by side in ~/.cache/blurt/ and are published together in one HF repo,
lightware-dev/parakeet-tdt-0.6b-v3
(env PARAKEET_REPO), told apart by filename. That replaces the two per-precision
repos, which are deleted — so PARAKEET_BF16_REPO and PARAKEET_FP16_REPO collapse
into PARAKEET_REPO.

fp16 costs nothing measurable

208 clips / 25.8 min: LibriSpeech test-clean, the same utterances under white noise
at 10 and 5 dB SNR, babble at 5 dB and near-clipping gain, plus synthetic speech.
Reference is an fp32 run of the same weights through the same pipeline.

	WER, all 208	WER, real speech	Weights / peak VRAM	RTF, RTX 5090
fp32	2.96%	2.40%	2.55 / 2.82 GB	0.0114
bf16	2.99%	2.40%	1.31 / 1.43 GB	0.0133
fp16	2.96%	2.40%	1.31 / 1.43 GB	0.0118

bf16 and fp16 return identical transcripts on 207 of 208 clips. The WER difference
is +0.03 pp (95% bootstrap CI [0.00, +0.08], p=0.74) and exactly zero on real speech.

Numerically fp16 is the closer of the two to fp32 — mean relative L2 error of the
encoder output 0.005 (cosine ≥ 0.995) against bf16's 0.044 (cosine ≥ 0.835), which is
what fp16's 3 extra mantissa bits buy. Its narrower exponent range never bit: no
non-finite activation appeared anywhere, including on the deliberately near-clipping
clips.

Measured on an RTX 5090 (sm_120), which supports both formats — the quality figures
carry to sm_75 (IEEE fp16, fp32 accumulation in cuBLAS), the speed figures do not. A
6 GB GTX 1660 has room for the 1.43 GB peak several times over.

What's here

• server/asr.py — a PRECISIONS table keyed by PARAKEET_DTYPE; resolve_precision
  normalises the obvious spellings and rejects anything else. The bf16 CUDA+bf16 gate is
  unchanged (its message now points at fp16); fp16 requires only CUDA. Loading goes
  through torch.set_default_dtype either way, so no fp32 transient is ever materialised.
• scripts/build_bf16_ckpt.py — --dtype {bf16,fp16}. Each is cast from the upstream
  fp32 weights, never bf16 → fp16, which would inherit bf16's coarser mantissa for nothing.
• scripts/make_eval_corpus.py, scripts/compare_precision.py — build the corpus and
  reproduce the table above (WER/CER, paired bootstrap, encoder divergence, RTF, VRAM).
• scripts/test_protocol.py — a GPU-free precision suite: alias resolution, unknown
  dtype rejected, distinct cache paths, env precedence, actionable missing-checkpoint error.

Verification

• All 7 test_protocol.py suites pass.
• Both published checkpoints verified before upload: 701 float tensors each, all
  float16 / bfloat16, no non-finite values, identical key sets, 627,090,582 params.
  Uploaded LFS hashes match the local files, and the bf16 file is byte-identical to the
  one the old repo served.
• End-to-end in fp16 against a live server: REST /v1/audio/transcriptions on clean,
  noisy and long clips, and ws_client_test.py streaming partials and a final over wss://.

Note

The evaluation corpus is English-only, so it says nothing directly about the other 24
languages the base model supports beyond the general expectation that a pure weight cast
behaves the same across them.

One measurement gotcha worth flagging for anyone extending compare_precision.py: after
NeMo's transcribe() has run once, calling the model directly returns nondeterministic
output, and interleaving the two makes transcribe() itself start returning empty
strings. The harness captures encoder tensors on a never-transcribed model, reloads, then
decodes — and self-checks that the encoder is deterministic before trusting any of it.
blurtd only ever calls transcribe(), so the server is unaffected.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01EnL4vH3w19nXYjAUqFT2jF