Skip to content

feat(tui): push-to-talk voice input - #102

Merged
A-x6 merged 9 commits into
devfrom
voice-input
Jul 31, 2026
Merged

feat(tui): push-to-talk voice input#102
A-x6 merged 9 commits into
devfrom
voice-input

Conversation

@A-x6

@A-x6A-x6 commented Jul 31, 2026

Copy link
Copy Markdown

Issue for this PR

Closes#99

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds push-to-talk voice input to the TUI. A /voice slash command (also in the command palette, keybind voice_toggle, default <leader>v) starts recording, the prompt status line shows a ● recording indicator, and running the command again stops recording, transcribes the audio, and inserts the text into the prompt input at the cursor. Nothing is auto-submitted.

Recording (packages/tui/src/voice.ts) spawns the first available external recorder via node:child_process: sox's rec, then sox -d, then arecord (Linux), then ffmpeg (alsa/avfoundation; skipped on Windows since dshow needs a device name). It captures 16 kHz mono WAV to a temp file, caps takes at 5 minutes via recorder args, and stops with SIGINT so the recorder finalizes the WAV header. If no recorder binary is installed, a toast explains what to install. Before uploading, the captured WAV is analyzed client-side and obviously useless takes (shorter than 0.4 s or with no measurable signal) skip the paid transcription call entirely; the analysis fails open so a malformed header never blocks transcription.

Transcription is a new instance endpoint POST /voice/transcribe that accepts base64 audio (base64 JSON rather than multipart because the TUI's default in-process worker transport serializes request bodies as text), validates base64 strictly, and bounds payloads to OpenAI's 25 MB limit. The endpoint is backed by a single pluggable function in packages/opencode/src/voice/transcription.ts with two backends: it prefers a local whisper.cpp install (whisper-cli/whisper-cpp binary plus ~/.local/share/opencode/models/ggml-large-v3-turbo-q5_0.bin, overridable via OPENCODE_VOICE_WHISPER/OPENCODE_VOICE_MODEL), so voice input works offline with no API key, and falls back to OpenAI whisper-1 with a 60 s timeout, resolving the credential from OPENAI_API_KEY or the auth store (auth.get("openai")). The install script now sets this up by default (skipped in CI, opt out with --no-voice): it best-effort installs sox and whisper-cpp via the system package manager and downloads the ~0.6 GB q5 turbo model. Other speech-to-text providers can be added by branching in this one function:

exportconsttranscribe=Effect.fn("VoiceTranscription.transcribe")(function*(input: {audio: Uint8Arraymime: stringlanguage?: string}){constkey=yield*resolveOpenaiKey()if(!key)returnyield*newNoCredentialError({message: "Voice input needs an OpenAI credential. Run `opencode auth login` and add an OpenAI API key.",})// ... multipart POST to api.openai.com/v1/audio/transcriptions via HttpClient})

A missing credential surfaces as a 400 with that message, which the TUI shows as a toast. The legacy JS SDK was regenerated (./packages/sdk/js/script/build.ts) so the TUI calls sdk.client.voice.transcribe(...); the diff there is only the new voice types/method. An exerciser entry covers the new route (invalid-payload 400 path, so no network or credential is needed in CI).

How did you verify your code works?

  • bun typecheck in packages/opencode and packages/tui (pass)
  • bun test in packages/tui (202 pass, 1 pre-existing skip; includes unit tests for the WAV duration/peak analysis)
  • bun run test:httpapi in packages/opencode (coverage/auth/effect modes, 222 pass, 0 missing/skip)
  • bash -n install plus ./install --help for the new --no-voice flag

Screenshots / recordings

Not included: the recording indicator requires a live microphone + recorder binary, which the sandbox used for development lacks.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

Summary by CodeRabbit

  • New Features

    • Added voice input with push-to-talk recording and automatic transcription.
    • Transcribed text is inserted directly into the prompt.
    • Added the default <leader>v shortcut to start or stop voice recording.
    • Added status indicators and notifications for recording, transcription, and errors.
    • Added an API endpoint for submitting audio and receiving transcriptions.
  • Documentation

    • Documented the new voice keybinding in the default keybinding reference.

@vercel

vercelBot commented Jul 31, 2026

Copy link
Copy Markdown

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adevloper152s-projects?upgradeToPro=build-rate-limit

@coderabbitai

coderabbitaiBot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

Voice input

Layer / File(s)Summary
Whisper transcription service
packages/opencode/src/voice/transcription.ts
Adds OpenAI credential resolution, Whisper submission, response validation, MIME handling, and typed transcription errors.
Instance transcription endpoint
packages/opencode/src/server/routes/instance/httpapi/groups/voice.ts, packages/opencode/src/server/routes/instance/httpapi/handlers/voice.ts, packages/opencode/src/server/routes/instance/httpapi/api.ts, packages/opencode/src/server/routes/instance/httpapi/server.ts, packages/opencode/test/server/httpapi-exercise/index.ts
Adds POST /voice/transcribe, registers its schemas and handlers, maps failures to API errors, and tests empty audio validation.
Push-to-talk recording lifecycle
packages/tui/src/voice.ts
Adds recorder discovery, WAV capture, process shutdown, temporary-file cleanup, base64 encoding, and voice status tracking.
TUI command and prompt integration
packages/tui/src/app.tsx, packages/tui/src/component/prompt/index.tsx, packages/tui/src/config/keybind.ts, packages/web/src/content/docs/keybinds.mdx
Adds the voice.toggle command and <leader>v keybind, displays recording status, and inserts transcription into the prompt without submission.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
participant User
participant TUI
participant InstanceHttpApi
participant VoiceTranscription
participant OpenAIWhisper
User->>TUI: Press voice toggle
TUI->>TUI: Start or stop WAV recording
TUI->>InstanceHttpApi: POST /voice/transcribe
InstanceHttpApi->>VoiceTranscription: Transcribe decoded audio
VoiceTranscription->>OpenAIWhisper: Submit multipart audio
OpenAIWhisper-->>VoiceTranscription: Return transcription text
VoiceTranscription-->>InstanceHttpApi: Return validated result
InstanceHttpApi-->>TUI: Return transcribed text
TUI->>TUI: Insert text into prompt
Loading

Suggested reviewers:thdxr

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 28.57% which is insufficient. The required threshold is 80.00%.Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check nameStatusExplanation
Linked Issues check✅ PassedThe changes address issue #99 requirements for recording, fallback handling, transcription, credential errors, pluggability, and prompt insertion without submission.
Out of Scope Changes check✅ PassedThe reviewed changes support the voice-input feature and its endpoint, integration, configuration, and test coverage without unrelated code.
Title check✅ PassedThe title clearly identifies the main change: push-to-talk voice input in the TUI.
Description check✅ PassedThe description follows the template and explains the feature, implementation, verification, screenshots, and checklist status.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch voice-input

Comment @coderabbitai help to get the list of available commands.

@deepsource-io

deepsource-ioBot commented Jul 31, 2026

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in aa22f15...f917149 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

Important

Some issues found as part of this review are outside of the diff in this pull request and aren't shown in the inline review comments due to GitHub's API limitations. You can see those issues on the DeepSource dashboard.

PR Report Card

Overall GradeSecurity

Reliability

Complexity

Hygiene

Code Review Summary

AnalyzerStatusUpdated (UTC)Details
JavaScriptJul 31, 2026 5:12p.m.Review ↗
ShellJul 31, 2026 5:12p.m.Review ↗
SecretsJul 31, 2026 5:12p.m.Review ↗
DockerJul 31, 2026 5:12p.m.Review ↗
PythonJul 31, 2026 5:12p.m.Review ↗
CSSJul 31, 2026 5:12p.m.Review ↗
RustJul 31, 2026 5:12p.m.Review ↗
RubyJul 31, 2026 5:12p.m.Review ↗
SwiftJul 31, 2026 5:12p.m.Review ↗
PHPJul 31, 2026 5:12p.m.Review ↗
LuaJul 31, 2026 5:12p.m.Review ↗
JavaJul 31, 2026 5:12p.m.Review ↗
GoJul 31, 2026 5:12p.m.Review ↗
C & C++Jul 31, 2026 5:12p.m.Review ↗
AnsibleJul 31, 2026 5:12p.m.Review ↗
ApexJul 31, 2026 5:12p.m.Review ↗
ElixirJul 31, 2026 5:12p.m.Review ↗
GroovyJul 31, 2026 5:12p.m.Review ↗
Objective-CJul 31, 2026 5:12p.m.Review ↗
PowerShellJul 31, 2026 5:12p.m.Review ↗
TerraformJul 31, 2026 5:12p.m.Review ↗
VB.NETJul 31, 2026 5:12p.m.Review ↗
SQLJul 31, 2026 5:12p.m.Review ↗
ScalaJul 31, 2026 5:12p.m.Review ↗
PerlJul 31, 2026 5:12p.m.Review ↗
KotlinJul 31, 2026 5:12p.m.Review ↗
HelmJul 31, 2026 5:12p.m.Review ↗
ErlangJul 31, 2026 5:12p.m.Review ↗
DartJul 31, 2026 5:12p.m.Review ↗
C#Jul 31, 2026 5:12p.m.Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

@github-actions

Copy link
Copy Markdown

The following comment was made by an LLM, it may be inaccurate:

Comment on lines +9 to +29
Effect.gen(function* () {
const transcribe = Effect.fn("VoiceHttpApi.transcribe")(function* (ctx: {
payload: typeof TranscribeInput.Type
}) {
const audio = decodeAudio(ctx.payload.audio)
if (!audio) return yield* new InvalidRequestError({ message: "audio must be base64-encoded", field: "audio" })
return yield* VoiceTranscription.transcribe({
audio,
mime: ctx.payload.mime ?? "audio/wav",
language: ctx.payload.language,
}).pipe(
Effect.mapError((error) => {
if (error instanceof VoiceTranscription.NoCredentialError)
return new InvalidRequestError({ message: error.message })
return new UpstreamError({ message: error.message, service: "openai", status: error.status })
}),
)
})

return handlers.handle("transcribe", transcribe)
}),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generator function does not have 'yield'


The generator functions should have a yield keyword.

Comment on lines +32 to +37
function decodeAudio(input: string) {
if (input.length === 0) return undefined
const decoded = Buffer.from(input, "base64")
if (decoded.length === 0) return undefined
return new Uint8Array(decoded)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable


It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.

// Single speech-to-text entrypoint. Alternative transcription providers can be
// added by branching here on the resolved credential/provider instead of
// touching the HTTP surface or the TUI.
export const transcribe = Effect.fn("VoiceTranscription.transcribe")(function* (input: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function has a cyclomatic complexity of 7 with "medium" risk


A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function is a linear happy path with early-error returns; repo style explicitly prefers keeping logic in one function over extracting single-use helpers, and complexity 7 is a minor threshold nit.

Comment on lines +72 to +84
function resolveOpenaiKey() {
return Effect.gen(function* () {
const env = yield* Env.Service
const fromEnv = yield* env.get("OPENAI_API_KEY")
if (fromEnv) return fromEnv
const auth = yield* Auth.Service
const info = yield* auth.get("openai").pipe(Effect.orElseSucceed(() => undefined))
if (info?.type === "api") return info.key
if (info?.type === "oauth") return info.access
if (info?.type === "wellknown") return info.token
return undefined
})
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable


It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive: resolveOpenaiKey is an ESM module-scope helper, not a browser global; module-level helpers below the main export are this repo's documented pattern.

})
}

export * as VoiceTranscription from "./transcription"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Module imports itself


A module should never import itself. This usually happens as a mistake or typo and occurs mostly during refactoring. Self importing might result in unexpected results like wrong functions/variables being used.

function start() {
if (status() !== "idle") return
const command = locate()
if (!command) return "No audio recorder found. Install sox (`rec`), `arecord`, or `ffmpeg` to use voice input."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function 'start' expected no return value


Any code paths that do not have explicit returns will return undefined. It is recommended to replace any implicit dead-ends that return undefined with a return null statement.

})
active = { child, file, exit }
setStatus("recording")
return undefined

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function 'start' expected no return value


Any code paths that do not have explicit returns will return undefined. It is recommended to replace any implicit dead-ends that return undefined with a return null statement.

Comment threadpackages/tui/src/voice.ts Outdated
Comment on lines +66 to +85
async function stop() {
const current = active
if (!current) return undefined
active = undefined
setStatus("transcribing")
// SIGINT lets sox/arecord/ffmpeg finalize the WAV header before exiting.
current.child.kill("SIGINT")
const timer = setTimeout(() => current.child.kill("SIGKILL"), 3000)
await current.exit
clearTimeout(timer)
const bytes = await Bun.file(current.file)
.arrayBuffer()
.catch(() => undefined)
await rm(current.file, { force: true }).catch(() => {})
if (!bytes || bytes.byteLength === 0) {
setStatus("idle")
return undefined
}
return Buffer.from(bytes).toString("base64")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected redeclaration of read-only global variable


It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive: stop shadows the browser global window.stop, but this is a Bun/Node terminal app where no such read-only global applies, and the function is module-local.

Comment threadpackages/tui/src/voice.ts Outdated
const bytes = await Bun.file(current.file)
.arrayBuffer()
.catch(() => undefined)
await rm(current.file, { force: true }).catch(() => {})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected empty arrow function


Having empty functions hurts readability, and is considered a code-smell. There's almost always a way to avoid using them. If you must use one, consider adding a comment to inform the reader of its purpose.

Comment on lines +87 to +89
function reset() {
setStatus("idle")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable


It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive: ESM module scope, not the browser global scope; reset is file-local and only exposed via the voice export object.

A-x6and others added 2 commits July 31, 2026 17:10
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
@vercel

vercelBot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
bolt-cli-appReadyReadyPreviewJul 31, 2026 5:50pm

@coderabbitaicoderabbitaiBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
packages/opencode/src/voice/transcription.ts (1)

72-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use Effect.fnUntraced for the internal resolveOpenaiKey helper.

resolveOpenaiKey is an internal, non-exported helper that returns Effect.gen(function* () {...}) directly instead of using Effect.fnUntraced. As per coding guidelines, "Use Effect.fn(\"Domain.method\") for named or traced effects and Effect.fnUntraced for internal helpers."

♻️ Proposed refactor
-function resolveOpenaiKey() {- return Effect.gen(function* () {- const env = yield* Env.Service- const fromEnv = yield* env.get("OPENAI_API_KEY")- if (fromEnv) return fromEnv- const auth = yield* Auth.Service- const info = yield* auth.get("openai").pipe(Effect.orElseSucceed(() => undefined))- if (info?.type === "api") return info.key- if (info?.type === "oauth") return info.access- if (info?.type === "wellknown") return info.token- return undefined- })-}+const resolveOpenaiKey = Effect.fnUntraced(function* () {+ const env = yield* Env.Service+ const fromEnv = yield* env.get("OPENAI_API_KEY")+ if (fromEnv) return fromEnv+ const auth = yield* Auth.Service+ const info = yield* auth.get("openai").pipe(Effect.orElseSucceed(() => undefined))+ if (info?.type === "api") return info.key+ if (info?.type === "oauth") return info.access+ if (info?.type === "wellknown") return info.token+ return undefined+})
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/opencode/src/voice/transcription.ts` around lines 72 - 84, Update
the internal resolveOpenaiKey helper to wrap its Effect.gen implementation with
Effect.fnUntraced, preserving the existing environment lookup and OpenAI
credential resolution behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/opencode/src/server/routes/instance/httpapi/groups/voice.ts`:
- Around line 9-13: Limit TranscribeInput.audio in
packages/opencode/src/server/routes/instance/httpapi/groups/voice.ts:9-13 with
Schema.maxLength, and enforce the corresponding decoded-byte limit before
creating the upstream file. In packages/tui/src/voice.ts:35-61, add an automatic
maximum recording duration so recordings cannot continue indefinitely; preserve
normal manual-stop behavior for recordings within the limit.
In `@packages/opencode/src/server/routes/instance/httpapi/handlers/voice.ts`:
- Around line 32-37: Update decodeAudio to strictly validate the input’s base64
format before calling Buffer.from, rejecting malformed or mixed valid/invalid
input with the existing “audio must be base64-encoded” error path. Preserve the
current handling for empty input and successfully decoded audio.
In `@packages/opencode/src/voice/transcription.ts`:
- Around line 47-56: Update the OpenAI request in the transcription flow around
client.execute to apply an Effect.timeout of 30 seconds before the existing
Effect.mapError, ensuring pending requests terminate while preserving conversion
of timeout and request failures to TranscribeError.
---
Nitpick comments:
In `@packages/opencode/src/voice/transcription.ts`:
- Around line 72-84: Update the internal resolveOpenaiKey helper to wrap its
Effect.gen implementation with Effect.fnUntraced, preserving the existing
environment lookup and OpenAI credential resolution behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 790c75d2-eb2c-4f13-9a69-be0b09cabe95

📥 Commits

Reviewing files that changed from the base of the PR and between aa22f15 and 4677b34.

⛔ Files ignored due to path filters (2)
  • packages/sdk/js/src/v2/gen/sdk.gen.ts is excluded by !**/gen/**
  • packages/sdk/js/src/v2/gen/types.gen.ts is excluded by !**/gen/**
📒 Files selected for processing (11)
  • packages/opencode/src/server/routes/instance/httpapi/api.ts
  • packages/opencode/src/server/routes/instance/httpapi/groups/voice.ts
  • packages/opencode/src/server/routes/instance/httpapi/handlers/voice.ts
  • packages/opencode/src/server/routes/instance/httpapi/server.ts
  • packages/opencode/src/voice/transcription.ts
  • packages/opencode/test/server/httpapi-exercise/index.ts
  • packages/tui/src/app.tsx
  • packages/tui/src/component/prompt/index.tsx
  • packages/tui/src/config/keybind.ts
  • packages/tui/src/voice.ts
  • packages/web/src/content/docs/keybinds.mdx

Comment threadpackages/opencode/src/voice/transcription.ts
@A-x6
A-x6 merged commit 02fcc1a into devJul 31, 2026
15 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add push-to-talk voice input to the TUI

1 participant

@A-x6