Uh oh!
There was an error while loading. Please reload this page.
Formalise the dictation protocol; add Wyoming + OpenAI-compatible listeners - #1
Merged
Conversation
…ners Documents the dictation protocol in docs/protocol.md, adds two optional compatibility surfaces beside it, and backs the lot with a GPU-free regression suite so none of it drifts. Protocol: * start/stop carry a dictation id that every dictation-scoped event echoes, so a late final from a finished dictation can't land in whatever the user is doing next — finalisation is async, so this was reachable by stopping and immediately restarting. * start may declare its audio format; server/pcm.py converts to the canonical 16 kHz mono PCM16 (linear resample with cross-frame interpolation, partial-frame carry) instead of mis-transcribing anything that isn't already 16 kHz. * an info handshake reports model, ready/loading state and canonical format; describe asks for it again. * vad events expose the server-side Silero VAD, so the HUD can show that the whole mic -> network -> server path is live, which the local waveform cannot prove. * partials split into stable `committed` and revisable `live`. Clients render live at full strength: text only commits after a pause, so a single uninterrupted utterance is entirely live, and de-emphasising it would dim the whole transcript for the common case. Lifecycle fixes — each of these previously wedged or silently dropped a dictation: a second start now supersedes the one in flight instead of leaving the session emitting under an id the client has moved on from; stop always yields exactly one final, degrading to the committed text rather than losing a slow decode; a failed decode clears the session so the connection recovers; and malformed frames close at most one connection. Wyoming listener (server/wyoming.py) so Blurt can serve as a Home Assistant STT backend. Off by default: Wyoming has neither auth nor TLS, so an open port would bypass AUTH_TOKEN entirely. WYOMING_PORT=10300 opts in, WYOMING_HOST narrows the bind, and a port conflict — likely, since 10300 is shared with other wyoming-* services — logs a warning instead of aborting startup. OpenAI-compatible transcription API (server/openai_api.py) on the existing TLS port: json, text, srt, vtt, verbose_json and SSE streaming, ffmpeg fallback for compressed input, and size/duration ceilings enforced before the request body is read. Tests: scripts/test_protocol.py stubs the model and the VAD so all five suites run in seconds without a GPU, covering the malformed-input and lifecycle cases as well as the happy path, including a round trip driven by the official wyoming client library that Home Assistant uses. .github/workflows/server.yml runs them across Python 3.11-3.13.
The dictation pill's waveform is fed by the *local* mic, so it ripples identically whether the server is listening or the socket died mid-sentence. The `vad` events added in 27db9fe already carry that answer, but both desktop clients only spent them on the pre-transcript placeholder ("Hearing you…" instead of "Listening…"), which stops rendering the moment the first word lands — exactly when a dead connection starts looking like a quiet room. Bind the meter's opacity to the same signal instead: full strength while the server reports speech, dimmed to 0.4 when it doesn't. Dimmed rather than hidden, because the local meter still has something to say — it's the one control that shows the whole mic -> network -> server path is alive. The value is eased at 0.18/tick at 60 Hz (~270 ms to settle) rather than snapped: VAD toggles on word boundaries, and a hard cut would strobe the pill through every inter-word gap. Sessions start muted — brightness is earned by the server actually reporting speech, not assumed — so reset() zeroes it and startRecording()/StartRecording seeds `hearing(false)` alongside the placeholder. With no tick running there's nothing to ease the value, so setHearing snaps in that case. Mac seeds before show() and Windows after Show(), because the WPF HUD builds its WaveformView lazily in Show() and has nothing to talk to before that. docs/protocol.md: spell out what `vad` is *for* under its message entry — the brightness mapping, the placeholder's narrow window, static/index.html's status line — and note that ignoring it entirely is fine, you just can't tell a quiet room from a dead socket.
z33b0tforce-pushed
the
protocol-wyoming-openai
branch
from
July 25, 2026 09:31
93ec7fb to
8f5ef83CompareUh oh!
There was an error while loading. Please reload this page.
z33b0t added a commit
that referenced
this pull request
Jul 29, 2026
Neither workflow declared a `permissions:` block, so GITHUB_TOKEN was issued with the repository default scope. Both only ever check out — the www deploy authenticates to Cloudflare with its own API token — so `contents: read` is what they need, and stating it means a later change to the repository default can't silently widen them. mac.yml and windows.yml already declare `contents: write` at job level for `gh release create` / `upload`, so they are left alone. Clears CodeQL actions/missing-workflow-permissions alerts #1 and #2. Closes#15 Co-authored-by: z33b0t <z33b0t@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits: the protocol work itself, and the HUD change that consumes one of its new signals.
server:formalise the protocol, add Wyoming + OpenAI-compatible listenersDocuments the dictation protocol in
docs/protocol.md, adds two optional compatibility surfaces beside it, and backs the lot with a GPU-free regression suite.Protocol — dictation ids on
start/stopechoed by every dictation-scoped event (a late final from a finished dictation could previously land in whatever the user was doing next); declared input formats converted to canonical 16 kHz mono PCM16 byserver/pcm.py; aninfohandshake reporting model + ready/loading state;vadevents exposing the server-side Silero VAD; partials split into stablecommittedand revisablelive.Lifecycle fixes — each of these previously wedged or silently dropped a dictation: a second
startnow supersedes the one in flight;stopalways yields exactly one final; a failed decode clears the session; malformed frames close at most one connection.Wyoming listener (
server/wyoming.py) so Blurt can serve as a Home Assistant STT backend. Off by default — Wyoming has neither auth nor TLS, so an open port would bypassAUTH_TOKENentirely.WYOMING_PORT=10300opts in.OpenAI-compatible transcription API (
server/openai_api.py) on the existing TLS port: json/text/srt/vtt/verbose_json + SSE streaming, ffmpeg fallback, size/duration ceilings enforced before the body is read.clients(HUD):drive meter brightness from the server's VADThe waveform is fed by the local mic, so it ripples identically whether the server is listening or the socket died mid-sentence. Both desktop clients only spent
vadon the pre-transcript placeholder, which stops rendering once the first word lands — exactly when a dead connection starts looking like a quiet room. Meter opacity now follows the same signal (full → 0.4 dimmed, eased ~270 ms so word boundaries can't strobe it).Testing
scripts/test_protocol.pystubs the model and the VAD, so all five suites run in seconds without a GPU or a model download — native WebSocket protocol, Wyoming (including a round trip driven by the officialwyomingclient library Home Assistant uses), the OpenAI/v1API, and PCM conversion..github/workflows/server.ymlruns them across Python 3.11–3.13.The Mac and Windows client changes are compiled by
mac.yml/windows.ymlon this PR — they weren't buildable locally (Linux host), so CI is the first check on them.🤖 Generated with Claude Code
https://claude.ai/code/session_01Dcx379h8xKRwhmae1zhN5z