Skip to content

Add optional ElevenLabs speech with personal voices - #600

Draft
charleslpan wants to merge 13 commits into
mainfrom
charleslpan/evaluate-elevenlabs-voices
Draft

Add optional ElevenLabs speech with personal voices#600
charleslpan wants to merge 13 commits into
mainfrom
charleslpan/evaluate-elevenlabs-voices

Conversation

@charleslpan

@charleslpancharleslpan commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add optional ElevenLabs speech with encrypted local key setup, personal-voice discovery, persisted selection, and OpenAI fallback.
  • Stream Realtime replies through ElevenLabs Flash v2.5 with single-use tokens, interruption/settling support, bounded diagnostics, and development tracing.
  • Keep long-lived credentials in the main process, block personal voice names from replay, and update privacy, guide, provider-mark, schema, and test coverage.

Evidence

  • Platform-independent checks: ./scripts/check.sh passed
  • macOS Electron verification (./scripts/verify.sh): passed

Automated visual evidence

Download the deterministic macOS evidence · workflow run

  • Commit: 4fcb3f88eafbe51db8756ab30b60037b49b2c7c7
  • Scenario: smoke
  • Physical-notch check: not performed by CI

Physical-device evidence

  • Screenshot or screen recording: not attached
  • Physical-notch check: not performed
  • Device/display configuration: not recorded

Notes

  • Blockers or follow-up verification: fix tool-follow-up audio truncation, stale voice selection after key replacement, and malformed audio-frame handling; then verify live ElevenLabs output, barge-in, tool follow-up, and key replacement/deletion on a physical Mac.

Open in Alchemize

charleslpanand others added 12 commits August 31, 2026 17:57
A failing ElevenLabs socket said only that it had closed, which named a
failure nobody could act on. The two places its own account was being
dropped now carry it:
An error frame states its failure twice, as a sentence in `message` and as
an identifier in `error`, and either may be absent; only `error` was read,
so a frame carrying the sentence alone fell through unread and the turn
ended on the silent close behind it.
A close carries a code and, where the service refused something, its reason.
Both were discarded, and the error event that precedes every close reported
first with neither. Reporting now waits for the close, which is the event
that knows why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A speech socket that says nothing is unreadable from outside: the words are
drawn, no failure is reported, and whether the frames never arrived or
arrived and were never heard cannot be told apart from the panel.
The tap is the realtime call's own, so one reply's speech frames read in
order beside the events that drove them, behind the same unpackaged gate and
recorded nowhere else. Audio is counted rather than carried, and each frame
notes what is scheduled but unheard, because a playhead that never advances
is silence with nothing wrong on the wire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it cannot
The renderer's connect policy named no ElevenLabs host, so the browser
refused the speech socket before it was ever built. The policy now names the
one address the renderer reaches there: a websocket, because the key stays in
the main process and no https request is ever made from this side.
A refused socket throws out of the constructor rather than failing on an
event, and the open runs detached from any caller, so the throw was rejecting
a promise nobody held. The reply kept its words, reported nothing, and owed a
drain that no close and no error would arrive to settle — a reply that simply
stopped. It is now the failure it always was.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A single-use token names the socket it may be spent on, and the dialogue
socket refuses a text-to-speech one by name. The mint now tracks the socket
it is minted for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The settings path moved to Developers, and the address Luke opened was
answering with a permanent redirect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dialogue socket was the expressive model's, and Luke does not need what
it charges for. Flash v2.5 costs half the credits per character and begins
speaking in about a quarter of the time, and Luke speaks unprompted all day,
so both compound.
The socket is a different one, not the same one renamed. The voice moves into
the address, the opening frame is a bare space, a delta is its own text, and
an empty text both flushes and closes — so the reply's last words never need
a flush of their own. The server answers in camelCase here, and its ending is
one field rather than two, a turn having no meaning where there is no
dialogue. A voice id reaching a path segment is encoded, because a separator
inside one would address something else entirely.
The keepalive goes with it: this socket documents no such frame, and the only
gaps it has to survive are between one generated token and the next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercelBot commented Sep 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
ProjectDeploymentActionsUpdated
luke-webSkippedSkippedSep 3, 2026 5:57pm UTC

Request Review

Comment threadapps/desktop/src/renderer/realtime-session.ts
Comment threadapps/desktop/src/main/settings-store.ts
});
this.#ended = true;
this.#armDrain();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Malformed audio frames can throw

Medium Severity

#readFrame passes frame.audio to decodeSpeechAudio, and atob throws on invalid base64. That throw is uncaught, so a bad or final-and-bad frame never reaches the final handling or #fail. The reply can miss its last words, surface a close error, or sit on the settle backstop instead of draining cleanly.

Additional Locations (1)
Fix in CursorFix in Web

Reviewed by Cursor Bugbot for commit 4fcb3f8. Configure here.

…elds
Both the voice list and the token mint failed in the same five ways, so they
now share ELEVENLABS_OUTCOME and explain only the failures. The detail field
neither IPC handler forwarded is gone, as is the outcome the renderer never
read off the voices answer, the cursor-cycle guard the page cap already
covers, and the barrel exports only tests imported. PCM decoding rides
Int16Array instead of hand-rolled sign math.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2798b6f. Configure here.

}

if (page.nextPageToken === undefined || voices.length >= MAXIMUM_VOICES) {
return { outcome: ELEVENLABS_OUTCOME.OK, voices };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Voice list pagination can hang

Medium Severity

Removing the seen-cursor guard means listElevenlabsVoices now stops only when a page has no token or voices hits MAXIMUM_VOICES. A repeating next_page_token on pages that add no valid voices never reaches that cap, so the main-process read loops and the Voice page never leaves refreshing.

Fix in CursorFix in Web

Reviewed by Cursor Bugbot for commit 2798b6f. Configure here.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@charleslpan