Uh oh!
There was an error while loading. Please reload this page.
feat(ios): watchOS push-to-talk - #693
Conversation
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Security verification: unbidden watch voice mint + sessions context
(a) Finding: TRUE. On this PR’s changed watch UI, a signed-in watch launch lands on voice as tab 0; WatchVoiceView.task calls start immediately; that mints a remote Realtime credential and, on channel open, sends the mint’s observed-sessions context to OpenAI — without the user holding the talk button (startWithTurn defaults to false and only gates mic/turn arming, not mint or context send).
(b) Evidence (HEAD d4194d76)
- Default tab is voice, not roster —
apps/ios/LukeWatch/LukeWatchView.swift:6,14-18(selectedTab = 0,WatchVoiceView.tag(0), roster.tag(1)). - Appear starts session —
WatchVoiceView.swift:20-21→model.start(accountSession:)with defaultstartWithTurn: false. startalways mints —WatchVoiceSessionModel.swift:28-41,87(VoiceMintClient.mintinsiderequestConnection, thenconnect(startWithTurn:)).connectalways mints regardless ofstartWithTurn—RealtimeSession.swift:212-234(requestConnection()unconditional;startWithTurnonly callsbeginTurn()).sessionsContextalways sent on channel open —RealtimeSession.swift:234,364-375,390-391(onChannelOpen→sendText(contextItemJSON(context))with nostartWithTurn/isArmedguard). Context is vault-observed session metadata (titles, branches, repos) from remote mint (apps/web/server/hosted/remote-context.ts).
(c) vs iPhone / regression
- iPhone also auto-mints on VoiceView appear (
.task→start, same Realtime path) — but Voice is behindNavigationLink/SessionsRoute.voice(SessionsView.swift:51-54,215,222), so mint happens only after navigating to voice. - This PR makes voice the watch home screen, so opening the watch app (signed in) is enough. That is a new privacy expansion relative to iPhone’s navigate-to-voice gate, even though mint-on-voice-appear (without talk press) already existed on iPhone.
(d) Severity: HIGH
Unbidden OpenAI mint + observed session roster/context on every signed-in watch open (ambient, frequent) expands the unbidden-send surface beyond intentional voice use. Talk button is not required.
(e) PR description mismatch: YES
PR body still claims: “Adds a mic toolbar button on WatchRosterView that navigates to the voice screen. The roster remains the default home screen.” Code after the swipeable-pages commit does the opposite: voice is default tab 0; roster is tab 1; no mic-nav-from-roster home. Description does not match code.
No fix proposed (verification-only).
Sent by Cursor Security Agent: Security Reviewer
b15af98 to
42edb43CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Adds a hold-to-talk voice screen to LukeWatch that drives the existing LukeKit RealtimeSession — the same session the iPhone's VoiceView uses. New files in LukeWatch: - WatchVoiceView: hold-to-talk button, status glyph, streaming captions - WatchVoiceSessionModel: @observable@mainactor session model - WatchAudioCapturer: AVAudioEngine-based mic capture for watchOS - WatchAudioPlayer: AVAudioPlayerNode-based playback for watchOS Armed-act discipline: the watch ships tool-free. dispatchToolCall always returns {"error":"not authorized"} and contextItems is always empty, because the phone-side armed-act infrastructure (roster validation, ActClient, ProjectsAnswer, VoiceActContext) is not available on the watch. A future PR may add a bounded act set when that infrastructure is portable. Credential handling: requestConnection calls WatchAccountSession .validAccessToken() and mints; a thrown .signedOut surfaces as "Open Luke on your iPhone" — the watch never attempts a token refresh. A mic toolbar button on WatchRosterView navigates to the voice screen. NSMicrophoneUsageDescription added to the watch Debug and Release build configurations (GENERATE_INFOPLIST_FILE = YES). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
onCaption delivers String? — nil signals a segment boundary, drain completion, or session close. Preserving the last non-nil value keeps the caption visible until the next spoken ask clears it, rather than erasing Luke's reply mid-sentence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.defaultToSpeaker is unavailable on watchOS — the watch owns its own Bluetooth and speaker routing. Drop the option from both WatchAudioPlayer and WatchAudioCapturer to fix the watchOS simulator build. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AVAudioSession.recordPermission was deprecated in watchOS 10 in favour of AVAudioApplication.shared.recordPermission. The deployment target is watchOS 10.0 so no availability guard is needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two pages accessible by swiping: - Left (default): WatchVoiceView — hold-to-talk with chat-bubble history - Right: Sessions list (existing WatchRosterView in a NavigationStack) Voice screen changes: - Replace spokenAsk/captionText strings with a messages: [WatchVoiceMessage] list; onSpokenAsk appends a user bubble, onCaption updates/appends a luke bubble so streaming captions grow the last luke bubble in place. - ZStack layout: scrollable bubble thread behind a floating mic button (88 pt bottom padding keeps the newest bubble above the controls). - WatchVoiceBubble mirrors WatchConversationBubble: accent/trailing for user, secondary-fill/leading for luke. Navigation change: - LukeWatchView wraps the signed-in state in a TabView(.page); the WatchRosterView mic toolbar button is removed since voice is now a tab. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace "Hold the button and speak" text with the Luke face mark, matching the iOS treatment. WatchLukeMark is a hand copy of FaceArt and LukeMark from apps/ios/Luke/Marks.swift without the UIKit import and motion paths (unavailable/unneeded on watchOS). The mark dims to 40% opacity while connecting so there is visible feedback without a separate loading state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Opening the watch app while signed in previously minted a Realtime credential immediately (WatchVoiceView is the default tab and its .task called start() unconditionally), sending the connection to the OpenAI endpoint before the developer touched the talk button. This moves the mint to the first button press: prepare() stores the WatchAccountSession reference without opening any connection; beginTurn() mints and connects with startWithTurn: true when session == nil. After an idle timeout the session is released and the next press remints cleanly. Parity with iPhone: VoiceView also mints on appear, but it sits behind a NavigationLink so appear is gated by deliberate navigation. The watch equivalent is pressing the talk button. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bug 1 (High): onCaption(nil) closes a caption segment, but the model tracked only "last message was luke" — so the next segment's first text overwrote the prior luke bubble instead of appending a fresh one. Fix: lukeReplyOpen flag is set on the first text of a segment and cleared on nil, so each segment gets its own bubble. Bug 2 (Medium): releasing the button while the mint was in flight set endTurnAfterConnect, but a subsequent press while still connecting left that flag set. When the mint completed it immediately ended the turn the developer was actively holding. Fix: a re-press while connectingForTurn clears endTurnAfterConnect and returns, so the pending connect still starts the turn as held. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
e8ab11f to
d53a7b6CompareThere was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d53a7b6. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
…ering The realtime service delivers onCaption before onSpokenAsk — captions start arriving before the user's transcript is ready. Appending the user bubble at the end placed it after Luke's reply rather than before it. Fix: beginTurn() records turnStartIndex = messages.count (the position just before any captions for this turn), and onSpokenAsk inserts there instead of appending. The lukeReplyOpen flag is left intact so the reply segment that started before the transcript continues growing normally. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>



Summary
Adds a hold-to-talk voice screen to
apps/ios/LukeWatchthat drives the sameLukeKit.RealtimeSessionthe iPhone'sVoiceViewuses, with streaming captions on the wrist.Navigation: Two pages accessible by swiping — voice on the left (default), sessions list on the right. The mic toolbar button on
WatchRosterViewis removed since voice is now a top-level tab.Mint timing: The Realtime session is not minted until the developer presses the talk button.
WatchVoiceView.taskstores the credential reference (prepare) without opening any connection;beginTurn()mints and connects on first press. This keeps parity with iPhone's navigate-then-press gate.Watch-specific audio:
WatchAudioCapturerandWatchAudioPlayermirror the iPhone classes without.allowBluetoothHFP(watchOS owns its own Bluetooth audio routing) and useAVAudioApplication.shared.recordPermission(watchOS 10 API).Tool-free:
dispatchToolCallalways returns{"error":"not authorized"}andcontextItemsis always empty. The armed-act infrastructure (roster validation,ActClient,ProjectsAnswer) lives on the phone; a future PR may add a bounded act set.Credential handling:
WatchAccountSession.validAccessToken()throws.signedOutnear expiry (the watch cannot refresh tokens).onErrordetects the signed-out state and shows "Open Luke on your iPhone".Empty state: The Luke face mark (
WatchLukeMark— a watchOS copy ofFaceArt/LukeMarkfromapps/ios/Luke/Marks.swiftwithout UIKit) replaces placeholder text.Chat bubbles: Conversation history accumulates as
[WatchVoiceMessage].onSpokenAskappends a user bubble (right-aligned, accent); streamingonCaptiongrows a luke bubble (left-aligned, secondary fill). Bubbles scroll behind the floating mic button.Test plan
🤖 Generated with Claude Code
Automated visual evidence
Download the deterministic macOS evidence · workflow run
85f781b9915f8eafde68d37bc05b19becd306f6esmoke