Uh oh!
There was an error while loading. Please reload this page.
feat(mobile): environment-backed voice transcription - #9028
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| }; | ||
| }); | ||
| export const validateTranscriptionToken = Effect.fn("Transcription.validateToken")(function* ( |
There was a problem hiding this comment.
🟠 Hightranscription/Transcription.ts:75
The same valid transcription URL can be POSTed repeatedly until expiresAt, causing each request to invoke OpenAI and potentially incur another charge. validateTranscriptionToken only checks the stateless signature and expiry; it never records or consumes the token. Add one-shot consumption state (and reject already-consumed tokens) before forwarding the upload to OpenAI.
Also found in 1 other location(s)
apps/server/src/http.ts:401
The route validates only the stateless signature and expiry at
validateTranscriptionToken; it never records or consumes a token. A caller can POST the same minted URL repeatedly until it expires, and every request passes line 401 and invokes OpenAI again. This violates the intended single-upload authorization and allows repeated transcription/billing with one RPC-minted URL.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/transcription/Transcription.ts around line 75:
The same valid transcription URL can be POSTed repeatedly until `expiresAt`, causing each request to invoke OpenAI and potentially incur another charge. `validateTranscriptionToken` only checks the stateless signature and expiry; it never records or consumes the token. Add one-shot consumption state (and reject already-consumed tokens) before forwarding the upload to OpenAI.
Also found in 1 other location(s):
- apps/server/src/http.ts:401 -- The route validates only the stateless signature and expiry at `validateTranscriptionToken`; it never records or consumes a token. A caller can POST the same minted URL repeatedly until it expires, and every request passes line 401 and invokes OpenAI again. This violates the intended single-upload authorization and allows repeated transcription/billing with one RPC-minted URL.
| ): Promise<AtomCommandResult<A, E>> { | ||
| const result = await settleAtomCommandResult(() => command.run(registry, input)); | ||
| const result = await settleAtomCommandResult(() => command.run(registry, input, options.signal)); | ||
| reportAtomCommandResult(result, { ...options, label: options.label ?? command.label }, reporter); |
There was a problem hiding this comment.
🟡 Mediumstate/runtime.ts:292
For singleFlight commands, aborting a second invocation does not return an interrupted result: runAtomCommand passes the signal into command.run, but createAtomCommandScheduler returns the existing promise before observing that caller's signal. The second caller therefore remains blocked until the first invocation finishes; add per-caller abort handling around the shared promise.
🤖 Copy this AI Prompt to have your agent fix this:
In file @packages/client-runtime/src/state/runtime.ts around line 292:
For `singleFlight` commands, aborting a second invocation does not return an interrupted result: `runAtomCommand` passes the signal into `command.run`, but `createAtomCommandScheduler` returns the existing promise before observing that caller's signal. The second caller therefore remains blocked until the first invocation finishes; add per-caller abort handling around the shared promise.
| @@ -2420,11 +2428,13 @@ const makeWsRpcLayer = ( | |||
| ) | |||
| : Stream.empty; | |||
| const settingsUpdates = serverSettings.streamChanges.pipe( | |||
There was a problem hiding this comment.
🟡 Mediumsrc/ws.ts:2430
settingsUpdates is not subscribed until after loadServerConfig emits its snapshot, so a key change during discovery is missed and the client keeps a stale transcriptionServices catalog until another change or reconnect. Acquire the settings subscription before loading/emitting the snapshot, then consume that subscribed stream for live updates.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/ws.ts around line 2430:
`settingsUpdates` is not subscribed until after `loadServerConfig` emits its snapshot, so a key change during discovery is missed and the client keeps a stale `transcriptionServices` catalog until another change or reconnect. Acquire the settings subscription before loading/emitting the snapshot, then consume that subscribed stream for live updates.
| const operation = openAiApiKey.value.length > 0 ? "write-secret" : "remove-secret"; | ||
| yield* ( | ||
| openAiApiKey.value.length > 0 | ||
| ? secretStore.set( |
There was a problem hiding this comment.
🟡 Mediumsrc/serverSettings.ts:669
A failed writeSettingsAtomically leaves the OpenAI secret store changed even though the settings update returns an error, so key rotation can make the running server use an uncommitted key and clearing can disable transcription while settings.json still indicates a key is configured. persistProviderEnvironmentSecrets performs the secretStore.set/remove at lines 669–673 before the file commit at line 811; make the secret update and settings-file update transactional (or restore the previous secret when the file write fails).
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/serverSettings.ts around line 669:
A failed `writeSettingsAtomically` leaves the OpenAI secret store changed even though the settings update returns an error, so key rotation can make the running server use an uncommitted key and clearing can disable transcription while `settings.json` still indicates a key is configured. `persistProviderEnvironmentSecrets` performs the `secretStore.set`/`remove` at lines 669–673 before the file commit at line 811; make the secret update and settings-file update transactional (or restore the previous secret when the file write fails).
| const response = yield* httpClient | ||
| .post(OPENAI_TRANSCRIPTION_URL, { | ||
| headers: { Authorization: `Bearer ${apiKey}` }, | ||
| body: HttpBody.formData(form), | ||
| }) | ||
| .pipe(Effect.exit); |
There was a problem hiding this comment.
Effect.exit swallows the request failure and the cause is never preserved or logged, so the returned 502 carries no diagnostic trail (and an interruption is reported as a request failure too). storeAttachmentUpload keeps the cause via Effect.logError(..., { cause }); consider doing the same here, and for the schemaBodyJson exit below (line 167) whose decode error is also discarded.
| constresponse=yield*httpClient | |
| .post(OPENAI_TRANSCRIPTION_URL,{ | |
| headers: {Authorization: `Bearer ${apiKey}`}, | |
| body: HttpBody.formData(form), | |
| }) | |
| .pipe(Effect.exit); | |
| constresponse=yield*httpClient | |
| .post(OPENAI_TRANSCRIPTION_URL,{ | |
| headers: {Authorization: `Bearer ${apiKey}`}, | |
| body: HttpBody.formData(form), | |
| }) | |
| .pipe( | |
| Effect.tapError((cause)=>Effect.logError("OpenAI transcription request failed.",{ cause })), | |
| Effect.exit, | |
| ); |
Posted via Macroscope — Effect Service Conventions
| ) { | ||
| const [encoded, signature, unexpected] = token.split("."); | ||
| if (!encoded || !signature || unexpected) return null; | ||
| const secret = yield* loadSigningSecret.pipe(Effect.orElseSucceed(() => null)); |
There was a problem hiding this comment.
The signing-key failure is dropped entirely here, so a broken secret store surfaces only as an opaque 404 with no cause anywhere in the logs. The sibling token validators (validateAttachmentUploadToken, AssetAccess) log the cause before falling back — consider matching them so the underlying failure is preserved.
| constsecret=yield*loadSigningSecret.pipe(Effect.orElseSucceed(()=>null)); | |
| constsecret=yield*loadSigningSecret.pipe( | |
| Effect.tapError((cause)=> | |
| Effect.logError("Failed to load the transcription signing key.",{ cause }), | |
| ), | |
| Effect.orElseSucceed(()=>null), | |
| ); |
Posted via Macroscope — Effect Service Conventions
| }); | ||
| throwIfVoiceTranscriptionAborted(transcriptionSignal); | ||
| if (response.status < 200 || response.status >= 300) { | ||
| throw new Error(response.bodyText || `Transcription failed (${response.status}).`); |
There was a problem hiding this comment.
This manufactures a bare Error purely so the catch below can wrap it as cause, and it puts the raw upstream response body into the error message. Throwing the domain error directly at the failure boundary keeps the structural code plus the safe HTTP status and drops the unbounded body text.
| thrownewError(response.bodyText||`Transcription failed (${response.status}).`); | |
| thrownewVoiceTranscriptionError( | |
| "transcription-failed", | |
| `The environment could not transcribe the recording (${response.status}).`, | |
| ); |
Posted via Macroscope — Effect Service Conventions
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit bc83013. Configure here.
| const source = | ||
| selection.selectedSource ?? | ||
| (selection.localTranscriber !== null ? "local" : selection.services[0]?.id); | ||
| if (source === "local" || source === undefined) return selection.localTranscriber; |
There was a problem hiding this comment.
Stale source blocks local transcription
Medium Severity
A persisted environment source is used even when that service is no longer advertised. isAvailable stays true whenever on-device transcription exists, so the mic remains shown, but getTranscriber still builds the environment transcriber and preparation fails. The settings row also disappears once services is empty, so there is no way to switch back to On this device.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit bc83013. Configure here.
| <Input | ||
| aria-label="OpenAI transcription model" | ||
| className="w-full max-w-sm" | ||
| onBlur={() => { | ||
| const model = transcriptionModelDraft.trim(); | ||
| if (model && model !== settings.transcription.model) { | ||
| updateSettings({ transcription: { model } }); | ||
| } else { | ||
| setTranscriptionModelDraft(settings.transcription.model); | ||
| } | ||
| }} | ||
| onChange={(event) => setTranscriptionModelDraft(event.target.value)} | ||
| value={transcriptionModelDraft} | ||
| /> |
There was a problem hiding this comment.
This row rebuilds the shared commit-on-blur field (DraftInput / useCommitOnBlur, already imported in this file) with a local draft plus a useEffect resync. The reconstruction drops Enter-to-commit, which every other server-backed settings text field supports, and the effect-based resync can overwrite an in-progress edit when the optimistic settings push lands, whereas useCommitOnBlur only resyncs while unfocused.
Suggested fix — use the primitive and delete transcriptionModelDraft's useState/useEffect at the top of GeneralSettingsPanel:
<DraftInputaria-label="OpenAI transcription model"className="w-full max-w-sm"value={settings.transcription.model}onCommit={(next)=>{constmodel=next.trim();if(model)updateSettings({transcription: { model }});}}/>Posted via Macroscope — UI Consistency
| <div className="flex w-full max-w-sm gap-2"> | ||
| <Input | ||
| aria-label="OpenAI transcription API key" | ||
| autoComplete="off" | ||
| className="min-w-0 flex-1" | ||
| onChange={(event) => setTranscriptionKeyDraft(event.target.value)} | ||
| placeholder={ | ||
| settings.transcription.openAiApiKey.valueRedacted ? "Key configured" : "sk-..." | ||
| } | ||
| type="password" | ||
| value={transcriptionKeyDraft} | ||
| /> | ||
| <Button | ||
| size="xs" | ||
| variant="outline" |
There was a problem hiding this comment.
The trailing action is sized off the shared control scale used elsewhere: Input (default) is h-8.5 sm:h-7.5, while Button size="xs" is h-7 sm:h-6. Because the wrapper is flex without items-center, the shorter button stretches to flex-start and renders top-aligned and visibly smaller than the field. Every other input+action pair in settings matches sizes (default Input → default Button, compact → compact) and marks the action shrink-0 so the field absorbs the width.
| <divclassName="flex w-full max-w-sm gap-2"> | |
| <Input | |
| aria-label="OpenAI transcription API key" | |
| autoComplete="off" | |
| className="min-w-0 flex-1" | |
| onChange={(event)=>setTranscriptionKeyDraft(event.target.value)} | |
| placeholder={ | |
| settings.transcription.openAiApiKey.valueRedacted ? "Key configured" : "sk-..." | |
| } | |
| type="password" | |
| value={transcriptionKeyDraft} | |
| /> | |
| <Button | |
| size="xs" | |
| variant="outline" | |
| <divclassName="flex w-full max-w-sm items-center gap-2"> | |
| <Input | |
| aria-label="OpenAI transcription API key" | |
| autoComplete="off" | |
| className="min-w-0 flex-1" | |
| onChange={(event)=>setTranscriptionKeyDraft(event.target.value)} | |
| placeholder={ | |
| settings.transcription.openAiApiKey.valueRedacted ? "Key configured" : "sk-..." | |
| } | |
| type="password" | |
| value={transcriptionKeyDraft} | |
| /> | |
| <Button | |
| className="shrink-0" | |
| variant="outline" |
Posted via Macroscope — UI Consistency
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR introduces a substantial mobile-to-environment transcription workflow with new authenticated upload infrastructure, secret-backed OpenAI integration, and sensitive audio handling, while also changing the default transcription model. Unresolved findings include stale source selection and reusable upload URLs that can cause repeated provider calls and charges. Not approved because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
🤖 Generated with Claude Code
Problem
Mobile voice input currently requires Apple's on-device transcription, so it only exists on iOS 26+ iPhones — Android and older iOS have no mic button at all (#8718).
docs/internals/voice-input.md(added in #8614) already specifies how environment-provided transcription should work, but it was unimplemented.Change
Implements that spec as written:
ServerSecretStorewith its own explicit redaction — clients only ever seevalueRedacted: true. Model defaults togpt-transcribe, overridable in settings.transcriptioncapability onExecutionEnvironmentCapabilities, plus a derivedtranscriptionServiceslist (ids and labels only) onServerConfigand thesettingsUpdatedpayload. Older servers expose no remote choices; adding/removing the key updates availability live.packages/client-runtimeimplementing the existingVoiceTranscribercontract, with an injected platform transport (expo-file-system upload on mobile, matching the attachment transport split). Environment, service, and locale are captured when a recording starts; a disconnected environment reports unavailable rather than silently falling back.Web/desktop voice capture remains out of scope, per the spec's boundaries; web only gains the settings section.
Relationship to #5213: that branch covers web/desktop BYOK dictation and has no
apps/mobilecode; this PR covers the mobile surface against the newer internals spec. Fixes#8718.Verification
Screenshots
Web settings (environment key entry):
Implemented by GPT-5.6 Sol (Pi) with orchestration and code review by Claude Fable 5 (Claude Code).
Note
Medium Risk
Touches authenticated upload/transcription HTTP, secret persistence, and OpenAI forwarding with in-memory audio handling; mobile behavior changes when environments advertise transcription.
Overview
Adds environment-backed voice transcription so mobile can transcribe recordings via the connected server (OpenAI) when on-device speech is unavailable, not only on iOS 26+.
Server & contracts: New
transcriptioncapability,transcription.createUrlRPC, and a signed one-shot POST route that validates token/size, buffers audio in memory (no disk), and returns{ text }from OpenAI. OpenAI API key and model live in server settings, stored in the secret store and redacted to clients;transcriptionServicesis advertised on config and settingsUpdated events.Shared client:
createEnvironmentVoiceTranscribermints URLs, uploads via injectable transport, and honors AbortSignal through atom commands. Server state exposes per-environment transcription service lists gated by capability.Mobile: Composers pass
environmentIdintouseVoiceInputController, which resolves local vs remote from persistedvoiceTranscriptionSources. Settings → Environments adds a per-environment Voice transcription picker ("On this device" vs OpenAI). Mic availability becomes local or any advertised environment service.Web: General settings gains Voice input (API key + model). Docs updated for mobile/environment flow.
Reviewed by Cursor Bugbot for commit bc83013. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add environment-backed voice transcription with OpenAI integration
transcriptionCreateUrlWS RPC to get an expiring URL, upload audio, and the server forwards it to OpenAI, returning transcribed text. Token validation enforces exactContent-Lengthand rejects expired or malformed tokens.TRANSCRIPTION_OPENAI_API_KEY_SECRET_NAME; the key is always redacted (value: "",valueRedacted: true) in settings responses. The web General settings panel exposes inputs for the API key and transcription model.environmentIdand selects between on-device transcription and environment-provided services based on user preferences and advertisedtranscriptionServices. A new per-environment row in the environments settings screen lets users pick the transcription source.AbortSignalviaAtomCommandOptions, enabling cancellation-aware transcription flows in the environment transcriber.transcriptionCreateUrlRPC requiresAuthOrchestrationOperateScope; callers without it will be rejected. The transcription POST endpoint atTRANSCRIPTION_ROUTE_PREFIXvalidates tokens server-side — any mismatch in signing key, expiry, orContent-Lengthreturns an error status, so out-of-tree clients must follow the exact upload contract.📊 Macroscope summarized bc83013. 29 files reviewed, 6 issues evaluated, 1 issue filtered, 4 comments posted
🗂️ Filtered Issues
apps/server/src/http.ts — 0 comments posted, 1 evaluated, 1 filtered
validateTranscriptionToken; it never records or consumes a token. A caller can POST the same minted URL repeatedly until it expires, and every request passes line 401 and invokes OpenAI again. This violates the intended single-upload authorization and allows repeated transcription/billing with one RPC-minted URL. [ Cross-file consolidated ]