Skip to content

feat(ios): watchOS push-to-talk - #693

Merged
dastratakos merged 9 commits into
mainfrom
conductor/apple-watch-push-to-talk
Sep 4, 2026
Merged

feat(ios): watchOS push-to-talk#693
dastratakos merged 9 commits into
mainfrom
conductor/apple-watch-push-to-talk

Conversation

@dastratakos

@dastratakosdastratakos commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a hold-to-talk voice screen to apps/ios/LukeWatch that drives the same LukeKit.RealtimeSession the iPhone's VoiceView uses, with streaming captions on the wrist.

Navigation: Two pages accessible by swiping — voice on the left (default), sessions list on the right. The mic toolbar button on WatchRosterView is removed since voice is now a top-level tab.

Mint timing: The Realtime session is not minted until the developer presses the talk button. WatchVoiceView.task stores the credential reference (prepare) without opening any connection; beginTurn() mints and connects on first press. This keeps parity with iPhone's navigate-then-press gate.

Watch-specific audio:WatchAudioCapturer and WatchAudioPlayer mirror the iPhone classes without .allowBluetoothHFP (watchOS owns its own Bluetooth audio routing) and use AVAudioApplication.shared.recordPermission (watchOS 10 API).

Tool-free:dispatchToolCall always returns {"error":"not authorized"} and contextItems is always empty. The armed-act infrastructure (roster validation, ActClient, ProjectsAnswer) lives on the phone; a future PR may add a bounded act set.

Credential handling:WatchAccountSession.validAccessToken() throws .signedOut near expiry (the watch cannot refresh tokens). onError detects the signed-out state and shows "Open Luke on your iPhone".

Empty state: The Luke face mark (WatchLukeMark — a watchOS copy of FaceArt/LukeMark from apps/ios/Luke/Marks.swift without UIKit) replaces placeholder text.

Chat bubbles: Conversation history accumulates as [WatchVoiceMessage]. onSpokenAsk appends a user bubble (right-aligned, accent); streaming onCaption grows a luke bubble (left-aligned, secondary fill). Bubbles scroll behind the floating mic button.

Test plan

  • Signed-in watch launch: voice tab is default; no network call is made until the talk button is pressed
  • Press and hold talk button: status transitions connecting → listening → thinking → speaking; captions stream in bubbles
  • Release button: session ends turn; speaking → idle
  • Swipe right: sessions list renders with the roster
  • Swipe back to voice: session reconnects on next button press
  • Signed out: "Open Luke on your iPhone" shown
  • Build passes on both iPhone and watchOS simulator targets

🤖 Generated with Claude Code

Automated visual evidence

Download the deterministic macOS evidence · workflow run

  • Commit: 85f781b9915f8eafde68d37bc05b19becd306f6e
  • Scenario: smoke
  • Physical-notch check: not performed by CI

@vercel

vercelBot commented Sep 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated
luke-webReadyReadyPreviewSep 4, 2026 9:35pm UTC

Request Review

Comment threadapps/ios/LukeWatch/WatchVoiceSessionModel.swift Outdated
Comment threadapps/ios/LukeWatch/WatchVoiceSessionModel.swift Outdated

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security verification: unbidden watch voice mint + sessions context

(a) Finding: TRUE. On this PR’s changed watch UI, a signed-in watch launch lands on voice as tab 0; WatchVoiceView.task calls start immediately; that mints a remote Realtime credential and, on channel open, sends the mint’s observed-sessions context to OpenAI — without the user holding the talk button (startWithTurn defaults to false and only gates mic/turn arming, not mint or context send).

(b) Evidence (HEAD d4194d76)

  1. Default tab is voice, not rosterapps/ios/LukeWatch/LukeWatchView.swift:6,14-18 (selectedTab = 0, WatchVoiceView.tag(0), roster .tag(1)).
  2. Appear starts sessionWatchVoiceView.swift:20-21model.start(accountSession:) with default startWithTurn: false.
  3. start always mintsWatchVoiceSessionModel.swift:28-41,87 (VoiceMintClient.mint inside requestConnection, then connect(startWithTurn:)).
  4. connect always mints regardless of startWithTurnRealtimeSession.swift:212-234 (requestConnection() unconditional; startWithTurn only calls beginTurn()).
  5. sessionsContext always sent on channel openRealtimeSession.swift:234,364-375,390-391 (onChannelOpensendText(contextItemJSON(context)) with no startWithTurn / isArmed guard). Context is vault-observed session metadata (titles, branches, repos) from remote mint (apps/web/server/hosted/remote-context.ts).

(c) vs iPhone / regression

  • iPhone also auto-mints on VoiceView appear (.taskstart, same Realtime path) — but Voice is behind NavigationLink / SessionsRoute.voice (SessionsView.swift:51-54,215,222), so mint happens only after navigating to voice.
  • This PR makes voice the watch home screen, so opening the watch app (signed in) is enough. That is a new privacy expansion relative to iPhone’s navigate-to-voice gate, even though mint-on-voice-appear (without talk press) already existed on iPhone.

(d) Severity: HIGH

Unbidden OpenAI mint + observed session roster/context on every signed-in watch open (ambient, frequent) expands the unbidden-send surface beyond intentional voice use. Talk button is not required.

(e) PR description mismatch: YES

PR body still claims: “Adds a mic toolbar button on WatchRosterView that navigates to the voice screen. The roster remains the default home screen.” Code after the swipeable-pages commit does the opposite: voice is default tab 0; roster is tab 1; no mic-nav-from-roster home. Description does not match code.

No fix proposed (verification-only).

Open in WebView Automation

Sent by Cursor Security Agent: Security Reviewer

Comment threadapps/ios/LukeWatch/WatchVoiceSessionModel.swift
@dastratakos
dastratakos added this pull request to the merge queueSep 4, 2026
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Sep 4, 2026
dastratakosand others added 8 commits September 4, 2026 21:26
Adds a hold-to-talk voice screen to LukeWatch that drives the existing
LukeKit RealtimeSession — the same session the iPhone's VoiceView uses.
New files in LukeWatch:
- WatchVoiceView: hold-to-talk button, status glyph, streaming captions
- WatchVoiceSessionModel: @observable@mainactor session model
- WatchAudioCapturer: AVAudioEngine-based mic capture for watchOS
- WatchAudioPlayer: AVAudioPlayerNode-based playback for watchOS
Armed-act discipline: the watch ships tool-free. dispatchToolCall always
returns {"error":"not authorized"} and contextItems is always empty,
because the phone-side armed-act infrastructure (roster validation,
ActClient, ProjectsAnswer, VoiceActContext) is not available on the watch.
A future PR may add a bounded act set when that infrastructure is portable.
Credential handling: requestConnection calls WatchAccountSession
.validAccessToken() and mints; a thrown .signedOut surfaces as "Open Luke
on your iPhone" — the watch never attempts a token refresh.
A mic toolbar button on WatchRosterView navigates to the voice screen.
NSMicrophoneUsageDescription added to the watch Debug and Release build
configurations (GENERATE_INFOPLIST_FILE = YES).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
onCaption delivers String? — nil signals a segment boundary, drain
completion, or session close. Preserving the last non-nil value keeps
the caption visible until the next spoken ask clears it, rather than
erasing Luke's reply mid-sentence.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.defaultToSpeaker is unavailable on watchOS — the watch owns its own
Bluetooth and speaker routing. Drop the option from both WatchAudioPlayer
and WatchAudioCapturer to fix the watchOS simulator build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AVAudioSession.recordPermission was deprecated in watchOS 10 in favour of
AVAudioApplication.shared.recordPermission. The deployment target is
watchOS 10.0 so no availability guard is needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two pages accessible by swiping:
- Left (default): WatchVoiceView — hold-to-talk with chat-bubble history
- Right: Sessions list (existing WatchRosterView in a NavigationStack)
Voice screen changes:
- Replace spokenAsk/captionText strings with a messages: [WatchVoiceMessage]
list; onSpokenAsk appends a user bubble, onCaption updates/appends a
luke bubble so streaming captions grow the last luke bubble in place.
- ZStack layout: scrollable bubble thread behind a floating mic button
(88 pt bottom padding keeps the newest bubble above the controls).
- WatchVoiceBubble mirrors WatchConversationBubble: accent/trailing for
user, secondary-fill/leading for luke.
Navigation change:
- LukeWatchView wraps the signed-in state in a TabView(.page); the
WatchRosterView mic toolbar button is removed since voice is now a tab.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace "Hold the button and speak" text with the Luke face mark,
matching the iOS treatment. WatchLukeMark is a hand copy of FaceArt
and LukeMark from apps/ios/Luke/Marks.swift without the UIKit import
and motion paths (unavailable/unneeded on watchOS).
The mark dims to 40% opacity while connecting so there is visible
feedback without a separate loading state.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Opening the watch app while signed in previously minted a Realtime
credential immediately (WatchVoiceView is the default tab and its .task
called start() unconditionally), sending the connection to the OpenAI
endpoint before the developer touched the talk button.
This moves the mint to the first button press: prepare() stores the
WatchAccountSession reference without opening any connection; beginTurn()
mints and connects with startWithTurn: true when session == nil. After an
idle timeout the session is released and the next press remints cleanly.
Parity with iPhone: VoiceView also mints on appear, but it sits behind a
NavigationLink so appear is gated by deliberate navigation. The watch
equivalent is pressing the talk button.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bug 1 (High): onCaption(nil) closes a caption segment, but the model
tracked only "last message was luke" — so the next segment's first text
overwrote the prior luke bubble instead of appending a fresh one.
Fix: lukeReplyOpen flag is set on the first text of a segment and cleared
on nil, so each segment gets its own bubble.
Bug 2 (Medium): releasing the button while the mint was in flight set
endTurnAfterConnect, but a subsequent press while still connecting left
that flag set. When the mint completed it immediately ended the turn the
developer was actively holding.
Fix: a re-press while connectingForTurn clears endTurnAfterConnect and
returns, so the pending connect still starts the turn as held.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dastratakos
dastratakosforce-pushed the conductor/apple-watch-push-to-talk branch from e8ab11f to d53a7b6CompareSeptember 4, 2026 21:26

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d53a7b6. Configure here.

Comment threadapps/ios/LukeWatch/WatchVoiceSessionModel.swift
…ering
The realtime service delivers onCaption before onSpokenAsk — captions
start arriving before the user's transcript is ready. Appending the user
bubble at the end placed it after Luke's reply rather than before it.
Fix: beginTurn() records turnStartIndex = messages.count (the position
just before any captions for this turn), and onSpokenAsk inserts there
instead of appending. The lukeReplyOpen flag is left intact so the reply
segment that started before the transcript continues growing normally.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dastratakos
dastratakos added this pull request to the merge queueSep 4, 2026
Merged via the queue into main with commit 3059fb8Sep 4, 2026
10 checks passed
@dastratakos
dastratakos deleted the conductor/apple-watch-push-to-talk branch September 4, 2026 21:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@dastratakos