Skip to content

feat(mac): on-device transcription via Apple's SpeechTranscriber - #62

Open
z33b0t wants to merge 3 commits into
mainfrom
mac-local-transcription
Open

feat(mac): on-device transcription via Apple's SpeechTranscriber#62
z33b0t wants to merge 3 commits into
mainfrom
mac-local-transcription

Conversation

@z33b0t

Copy link
Copy Markdown
Contributor

Adds a second transcription engine to the Mac client: macOS 26's on-device
SpeechAnalyzer + SpeechTranscriber, selectable in Settings ▸ Transcription
alongside the existing blurtd path. Server mode is unchanged and stays the default.

Until now the client was a thin pipe to a Parakeet server — great once you have
the GPU box, nothing until you do. Apple's model is in the same league on
accuracy (~2.1% WER clean vs Parakeet TDT v3's ~2.5% on LibriSpeech test-clean),
runs on the Neural Engine, and ships zero bytes with the app.

Reach

TierShareEngine
Apple silicon + macOS 26+~84%local or server
Everything else~16%server, exactly as today

Purely additive — LSMinimumSystemVersion stays 13.0 and nobody loses anything.

Design

AppDelegate already talked to DictationClient through a closure surface, so
that surface became TranscriptionEngine; DictationClient conforms nearly
as-is. Apple's result model lines up with Blurt's, so the HUD needed no changes:

BlurtSpeechAnalyzer
partialLivevolatile result (ReportingOption.volatileResults)
partialCommittedfinalized results (isFinal) accumulated
onVad / "Hearing you…"SpeechDetector(reportResults: true)
onInfo "loading"AssetInventory status / install progress

AudioCapture grows a negotiable output format (the server wants its declared
16 kHz Int16, SpeechAnalyzer names its own) and hands over an
AVAudioPCMBuffer rather than packing Data itself.

Why no #if arch(...) anywhere

SpeechTranscriber.isAvailable answers both halves of "can this Mac do it" —
the API only exists on macOS 26, and even there it reports false without a
Neural Engine, i.e. on every Intel Mac. And because the symbols ship in the
x86_64 slice of the SDK too, the universal build compiles unchanged and
Intel simply takes the server path at runtime.

Settings shows the local option greyed out with the reason rather than hiding
it, and Settings.engine refuses to return .local on a Mac that can't run it,
so a preference restored from an Apple silicon Mac can't strand this one.

Verified

On a MacBookPro16,2 / Intel i7-1068NG7 / macOS 26.5.2 — the negative case,
run against the production source files:

LocalTranscription.isSupported = false
LocalTranscription.unavailable = Needs a Mac with Apple silicon.
stored engine preference = local ← forced
Settings.engine reads back = server ← gate holds

Driving the real SettingsWindowController:

Blurt server (Parakeet, on your GPU box) enabled=yes selected=yes
This Mac (Apple Speech, on-device) enabled=NO selected=no
note: "Needs a Mac with Apple silicon."

Universal build is intact (lipo: x86_64 arm64), signs with Developer ID +
hardened runtime, and the build is warning-free.

Not yet verified — needs an Apple silicon Mac

I only have the Intel machine, so the positive path is unexercised. Before
merging, on Apple silicon + macOS 26:

  • Dictate with Wi-Fi off and no blurtd running — the feature test and the
    privacy proof in one
  • Volatile → final renders correctly in the HUD
  • "Hearing you…" tracks SpeechDetector
  • Esc mid-dictation injects nothing
  • First run in a locale with no installed model shows "Loading model…"
  • Engine switch takes effect next dictation, both directions
  • Confirm whether SpeechAnalyzer needs Speech TCC — NSSpeechRecognitionUsageDescription
    is added defensively; if it's required, Onboarding.swift wants a third
    permission row

CI

mac.yml moves macos-15macos-26: the engine needs the macOS 26 SDK to
compile and the Sequoia image doesn't ship it.

https://claude.ai/code/session_01SUWiz8CtyyAGuxKgAEVZJu

Blurt's Mac client was a thin pipe to a Parakeet server: a great experience
once you have the GPU box, and no experience at all until you do. macOS 26
ships an on-device streaming ASR stack whose accuracy is in the same league
(~2.1% WER clean against Parakeet TDT v3's ~2.5% on LibriSpeech test-clean),
runs on the Neural Engine, and ships zero bytes with the app.
Settings ▸ Transcription now picks between the server and this Mac. Server
mode is unchanged and stays the default.
The result models line up almost exactly, so the HUD needed no changes: a
volatile result is the `live` half of a partial and a finalized one settles
into `committed`. A SpeechDetector rides alongside the transcriber purely so
"Hearing you…" keeps meaning what it means in server mode.
AppDelegate already talked to DictationClient through a closure surface, so
that surface became TranscriptionEngine and DictationClient conforms to it
nearly as-is. AudioCapture grows a negotiable output format — the server wants
its declared 16 kHz Int16, SpeechAnalyzer names its own — and hands over an
AVAudioPCMBuffer rather than packing Data itself.
Gating is one runtime check, SpeechTranscriber.isAvailable, which answers both
halves of the question: the API only exists on macOS 26, and even there it
reports false without a Neural Engine, i.e. on every Intel Mac. Verified on a
MacBookPro16,2 running macOS 26.5 — isAvailable false, supportedLocales empty,
AssetInventory reporting the modules unsupported.
So Intel keeps the server path and loses nothing. The deployment target stays
13.0, the Speech symbols sit behind #available, and because they ship in the
x86_64 slice of the SDK too there is no conditional compilation anywhere — one
universal binary still runs everywhere from Ventura up. Settings shows the
local option greyed out with the reason rather than hiding it, and
Settings.engine refuses to return .local on a Mac that cannot run it, so a
preference restored from an Apple silicon Mac can't strand this one.
CI moves to macos-26: the on-device engine needs the macOS 26 SDK to compile
and the Sequoia image doesn't ship it.
The previous comment claimed Sequoia images don't ship the macOS 26 SDK. They
do: macos-15-arm64 carries Xcode 26.3 (macOS 26.0-26.2 SDKs) next to the
default 16.4, and the workflow's select-newest-stable-Xcode step was already
picking it — the last release off main was built against SDK 26.2. So the
engine would have compiled on macos-15 unchanged.
Pinning is still worth doing, for a different reason: relying on a Sequoia
image to bundle a Tahoe Xcode is an accident of image composition, not a
guarantee. Say that instead.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@z33b0t