The official JavaScript / TypeScript SDK for the AudarAI platform
Build voice-enabled applications with Text-to-Speech, Speech-to-Text, real-time translation, and AI agent orchestration — all in one SDK.
@audarai/sdk is the official client library for the AudarAI platform — a production-grade audio AI infrastructure supporting:
- Text-to-Speech (TTS) — high-quality voice synthesis with custom speaker cloning
- Speech-to-Text (STT) — accurate transcription via file upload, SSE streaming, or real-time WebSocket
- Audio Translation — end-to-end STT → Translation → TTS pipeline with live streaming
- AI Agent Orchestration — create, manage, and converse with voice-enabled AI agents
- Knowledge Bases — semantic vector search for grounding your agents in domain knowledge
- Tools & Skills — extend agent capabilities with HTTP tools, builtins, MCP, and prompt skills
- Rooms & Sessions — multi-agent voice rooms with LiveKit integration
Designed for both browser and Node.js (18+) environments with first-class TypeScript support.
- Installation
- Quick Start
- Authentication
- Text-to-Speech (TTS)
- Speech-to-Text (STT)
- Audio Translation
- Agent Management
- Knowledge Base
- Tools
- Skills
- Archetypes
- Rooms
- Sessions
- Error Handling
- Token Auto-Refresh
- Node.js Compatibility
- TypeScript Support
- Demo Application
# npm
npm install @audarai/sdk@github:AudarAI/webapp-jssdk
# pnpm
pnpm add @audarai/sdk@github:AudarAI/webapp-jssdk
# yarn
yarn add @audarai/sdk@github:AudarAI/webapp-jssdkOr pin it in package.json:
{
"dependencies": {
"@audarai/sdk": "github:AudarAI/webapp-jssdk"
}
}npm install /path/to/webapp-jssdkimport{createAudaraiClient}from'@audarai/sdk';// 1. Create a clientconstclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',publishableKey: 'pk_your_key_here',});// 2. Synthesize speechconstaudioBuffer=awaitclient.tts.synthesize('Hello, world!',{voice: 'en-US-female',model: 'tts-1-hd',response_format: 'mp3',});// 3. Play in the browserconstblob=newBlob([audioBuffer],{type: 'audio/mpeg'});consturl=URL.createObjectURL(blob);newAudio(url).play();The SDK supports four mutually exclusive authentication modes. Choose exactly one.
| Mode | Field | HTTP Requests | WebSocket |
|---|---|---|---|
| Publishable Key | publishableKey | Auto-exchanged session token | Session token |
| Access Token | accessToken | JWT passed directly | Auto-exchanged session token |
| API Key | apiKey | API key passed directly | Auto-exchanged session token |
| App | appId (+ appSecret) | appid → session token; appid+secret → HTTP Basic | Auto-exchanged session token |
WebSocket endpoints only accept short-lived session tokens (
stk_prefix). The SDK handles the exchange automatically before establishing any connection — no manual handling required.
pk_ keys are safe to include in client-side code. The server validates the request Origin against your configured allowlist.
constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',publishableKey: 'pk_xxx',});Before using this mode, create a publishable key in the dashboard and configure allowed origins:
POST /v1/account/api-keys { "name": "Web App", "key_type": "publishable", "allowed_origins": ["https://yourapp.com"] }
For applications already using Keycloak or another OAuth2 provider. HTTP requests carry the JWT directly; WebSocket connections auto-exchange for a session token.
// Static stringconstclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',accessToken: 'eyJhbGciOiJSUzI1NiJ9...',});// Dynamic function (recommended — supports token refresh)constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',accessToken: async()=>keycloakAdapter.token,});ak_ keys carry full permissions. Never expose them in browser code. Use this mode in Node.js services or local development.
constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',apiKey: 'ak_xxx',});Register an App once to get a pair of credentials:
appId(appid_prefix) — public, frontend uses it alone (safe to embed; restricted by the App's Allowed Origins).appSecret(secret_prefix) — confidential, backend uses it together withappId. Never expose in browser code.
// Frontend — appid only (browser-safe; behaves like a publishable key)constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',appId: 'appid_xxx',});// Backend — appid + secret (authenticates via HTTP Basic base64(appid:secret))constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',appId: 'appid_xxx',appSecret: 'secret_xxx',});Create an App in the dashboard, or:
POST /v1/account/apps { "name": "My App", "allowed_origins": ["https://yourapp.com"] } → { "app_id": "appid_xxx", "secret": "secret_xxx" } // secret shown onceThe
secretis shown only once. Lost it? Reset viaPOST /v1/account/apps/{id}/reset-secret(the old secret is invalidated immediately;appIdstays the same).
constmodels=awaitclient.tts.listModels();// → [{ name: 'tts-flash', display_name: 'TTS Flash', kind: 'tts', is_default: false }, ...]// Use the default for new sessions; let users override via UIconstdefaultModel=models.find((m)=>m.is_default)?.name;constaudioBuffer=awaitclient.tts.synthesize('Hello, world!',{voice: 'en-US-female',model: 'tts-1-hd',// 'tts-1' | 'tts-1-hd' (default: 'tts-1')response_format: 'mp3',// 'mp3' | 'opus' | 'aac' | 'flac' | 'wav' | 'pcm'speed: 1.0,// 0.25 – 4.0provider: 'flash',// 'flash' | 'turbo' | 'pro'});// Play in browserconstblob=newBlob([audioBuffer],{type: 'audio/mpeg'});newAudio(URL.createObjectURL(blob)).play();Receive audio as a stream — ideal for long-form content or low-latency playback.
constresponse=awaitclient.tts.synthesizeStream('Long form content...',{voice: 'en-US-female',response_format: 'mp3',});// Pipe to file (Node.js)import{createWriteStream}from'fs';constwriter=createWriteStream('output.mp3');response.body!.pipeTo(newWritableStream({write: (chunk)=>writer.write(chunk)}));// List available voicesconstvoices: string[]=awaitclient.tts.listSpeakers();// Clone a voice from an audio sampleconstfile=document.querySelector<HTMLInputElement>('input[type=file]')!.files![0];awaitclient.tts.addSpeaker('my-voice',file,'This is the transcript of the recording.',{description: 'Custom voice description',});// Remove a custom voiceawaitclient.tts.deleteSpeaker('my-voice');constmodels=awaitclient.stt.listModels();// → [{ name: 'stt-flash', display_name: 'STT Flash', kind: 'stt', is_default: false }, ...]constdefaultModel=models.find((m)=>m.is_default)?.name;constresult=awaitclient.stt.transcribe(audioBlob,{language: 'en',forced_alignment: false,// Enable word-level timestampsprovider: 'flash',// 'flash' | 'turbo'});console.log(result.text);// Transcribed textconsole.log(result.language);// Detected language codeconsole.log(result.timestamps);// Word-level timestamps (if forced_alignment: true)Receive incremental transcription results as the server processes your audio.
constresult=awaitclient.stt.transcribeStream(audioBlob,{language: 'en',provider: 'flash'},{onChunk: (chunk)=>console.log('Partial:',chunk.text,'index:',chunk.chunk_index),onFinal: (chunk)=>console.log('Final:',chunk.text),onError: (err)=>console.error('Error:',err),},);console.log(result.text);// Full transcriptionconsole.log(result.language);// Detected languageBoth transcribe and transcribeStream downscale the audio to 16kHz mono
before uploading. The backend converts to 16kHz mono regardless, so this does not
change the transcription — it just stops you from sending bytes that are about to
be discarded. A 44.1kHz stereo source shrinks ~5.5x.
This matters because a CDN sits in front of the API and rejects request bodies
over its per-plan cap (100MB on Cloudflare Free/Pro) at the edge, with a
413 Content Too Large the server never sees. In terms of what fits in 100MB:
| Uploaded as | Bitrate | Fits in 100MB |
|---|---|---|
| 44.1kHz stereo 16-bit WAV | 176 KB/s | ~9.5 min |
| 16kHz mono 16-bit WAV (what the SDK sends) | 32 KB/s | ~52 min |
// Default: convert only when it is likely to matter (files over 4MiB).awaitclient.stt.transcribeStream(audioBlob,{language: 'en'});// Always convert, whatever the size.awaitclient.stt.transcribe(audioBlob,{preprocess: 'always'});// Send the exact bytes you passed in.awaitclient.stt.transcribe(audioBlob,{preprocess: 'never'});// Tune the threshold, or the target rate.awaitclient.stt.transcribe(audioBlob,{preprocess: 'auto',transcode: {minBytes: 512*1024,sampleRate: 16000},});Conversion is skipped automatically — and the original bytes uploaded unchanged — when any of these hold, so you never need to branch on environment or format:
- there is no Web Audio implementation (Node, or a very old browser);
- the file is under
transcode.minBytesandpreprocessis'auto'; - the source is already compressed tightly enough that PCM would be larger (an MP3 or Opus file, typically);
- the container cannot be decoded, or decoding fails.
You can run the conversion yourself, e.g. to show the saving before uploading:
import{preprocessForAsr}from'@audarai/sdk';const{ data, applied, reason, originalBytes, bytes }=awaitpreprocessForAsr(file,'always');console.log(applied ? `${originalBytes} → ${bytes} bytes` : `skipped: ${reason}`);Downscaling alone tops out around ~52 minutes of audio. Past that, the SDK routes
the upload around the CDN instead: it mints a presigned URL, PUTs the bytes
straight to object storage, and sends only the resulting upload_id to the
transcription endpoint. The two compose — downscale first, upload the smaller
result.
This is automatic above 80MiB. No code change is needed to benefit:
// Files over 80MiB go via storage; smaller ones use the single-request path.awaitclient.stt.transcribeStream(hugeFile,{language: 'en'});// Force it, whatever the size — errors surface instead of falling back.awaitclient.stt.transcribe(file,{viaUpload: 'always'});// Opt out entirely, or move the threshold.awaitclient.stt.transcribe(file,{viaUpload: 'never'});awaitclient.stt.transcribe(file,{uploadThresholdBytes: 32*1024*1024});Under the default 'auto', if the deployment has no object storage configured
(the endpoint answers 503) or the PUT fails, the SDK falls back to the direct
path rather than erroring — a body under the edge cap still succeeds, so
falling back is never worse than not having tried. 'always' skips the fallback
and surfaces the real failure.
The pieces are also available directly:
constticket=awaitclient.stt.uploadAudio(blob,'meeting.wav');// mint + PUTawaitclient.stt.transcribe(blob,{viaUpload: 'never'});// ...or drive it yourselfawaitclient.stt.deleteUpload(ticket.upload_id);// release early (optional)Uploads stay redeemable until they expire, so a failed transcription can be retried without re-uploading.
Not yet implemented: FLAC / Opus output, which would shrink another 2–8x. Browsers ship no native encoder for either, so it needs a wasm codec or a hand-written Ogg muxer over WebCodecs; this SDK has no runtime dependencies and that is worth keeping. 16kHz mono WAV captures most of the win with universal support. If your audio is long enough for the difference to matter, encode to Opus yourself before calling the SDK and pass
preprocess: 'never'.
For live microphone input with sub-second latency.
conststt=awaitclient.stt.connectWebSocket({language: 'en',provider: 'flash'},{onReady: ({ session_id })=>console.log('Session ready:',session_id),onPartial: ({ text })=>console.log('Live:',text),onSegment: ({ text, segment_index })=>console.log(`Segment ${segment_index}:`,text),onFinal: ({ text })=>console.log('Final:',text),onError: (e)=>console.error('Error:',e),onClose: ()=>console.log('Connection closed'),},);// Send raw PCM audio frames (ArrayBuffer or Int16Array)stt.sendAudio(pcmBuffer);// Signal end of stream — server flushes and closesstt.stop();translate() pushes events for each pipeline stage: STT → Translation → TTS. The method resolves with the final result.
constresult=awaitclient.translation.translate(audioBlob,{target_lang: 'en',source_lang: 'zh',// Optional — auto-detected if omittedtranslation_mode: 'llm',// 'llm' (default) | 'mt' (machine translation)tts_enabled: true,response_format: 'mp3',voice: 'en-US-female',},{onStatus: ({ stage, message })=>console.log(stage,message),onSttPartial: ({ text })=>showSubtitle(text),onSttFinal: ({ text })=>console.log('STT:',text),onTranslationPartial: ({ text })=>showTranslation(text),onTranslationComplete: ({ text })=>console.log('Translation:',text),onTtsChunk: (audio,{ format, sample_rate })=>playAudio(audio),onTtsComplete: ({ total_chunks })=>console.log('TTS complete'),onPipelineComplete: ({ source_text, translated_text })=>console.log(`${source_text} → ${translated_text}`),onError: ({ stage, message })=>console.error(stage,message),},);console.log(result.source_text);// Original textconsole.log(result.text);// Translated textEnd-to-end live translation from microphone input.
constws=awaitclient.translation.connectWebSocket({target_lang: 'en',source_lang: 'zh',tts_enabled: true,translation_mode: 'llm',response_format: 'mp3',},{onReady: ({ session_id })=>console.log('Session:',session_id),onSttPartial: ({ text })=>showSubtitle(text),onSttSegment: ({ text, segment_index })=>console.log('Segment:',text),onTranslationComplete: ({ text, target_lang })=>showTranslation(text),onTtsChunk: (audio,{ format, sample_rate })=>playAudio(audio),onSegmentComplete: ({ source_text, translated_text })=>console.log(`${source_text} → ${translated_text}`),onPipelineComplete: ({ duration })=>console.log(`Done in ${duration}s`),onError: ({ message, stage })=>console.error(stage,message),onClose: ()=>console.log('Disconnected'),},);// Send raw PCM frames from microphonews.sendAudio(pcmBuffer);// End the sessionws.stop();// List agents for the current tenantconstagents=awaitclient.agent.listAgents();// List platform-wide agents (visible to all authenticated users)constplatformAgents=awaitclient.agent.listPlatformAgents();// Create an agentconstagent=awaitclient.agent.createAgent({name: 'Support Assistant',description: 'Voice-enabled customer support agent',system_prompt: 'You are a professional support agent. Be concise and helpful.',voice_id: 'en-US-female',language: 'en',archetype_id: 'archetype-uuid',// Optionalknowledge_bindings: ['kb-uuid'],// Attach knowledge basesskills: ['skill-uuid'],// Attach skillsmemory_policy: {enable_memory: true,num_history_turns: 10,},});// Get / Update / Deleteconstdetail=awaitclient.agent.getAgent(agent.id);awaitclient.agent.updateAgent(agent.id,{name: 'Updated Name'});awaitclient.agent.deleteAgent(agent.id);chat() creates a session and returns { session_id, room_id }. Use getLiveKitToken() to join the voice room.
const{ session_id }=awaitclient.agent.chat(agentId,'Hello!',{voice_id: 'en-US-female',// Optional — overrides the agent default});// Retrieve a LiveKit token for voice connectivityconst{ token, livekit_url }=awaitclient.agent.sessions.getLiveKitToken(session_id);// Connect with the official LiveKit clientimport{Room}from'@livekit/client';constroom=newRoom();awaitroom.connect(livekit_url,token);constkbs=awaitclient.knowledge.list();constkb=awaitclient.knowledge.create({name: 'Product Manual',description: 'Product FAQs and operating instructions',});awaitclient.knowledge.update(kb.id,{name: 'Product Manual v2'});awaitclient.knowledge.delete(kb.id);// Ingest plain text (asynchronous — returns 202 Accepted)awaitclient.knowledge.ingest(kb.id,{source_type: 'text',text: 'The content to be embedded and indexed...',source_label: 'Manual entry',language: 'en',});// Ingest from a URLawaitclient.knowledge.ingest(kb.id,{source_type: 'url',url: 'https://example.com/docs/api',});// Upload a fileconstfile=document.querySelector<HTMLInputElement>('input[type=file]')!.files![0];awaitclient.knowledge.ingestFile(kb.id,file,file.name);constresults=awaitclient.knowledge.search(kb.id,{query: 'How do I reset my password?',top_k: 5,// Number of results (default: 5)language: 'en',});results.forEach(r=>{console.log(`[${r.score.toFixed(3)}] ${r.content}`);});constdocs=awaitclient.knowledge.listDocuments(kb.id);awaitclient.knowledge.deleteDocument(kb.id,docId);// Trigger re-ingestion of all documentsawaitclient.knowledge.reingest(kb.id);Extend your agents with external capabilities: HTTP APIs, built-in tools (web search), and MCP servers.
// HTTP tool — call any REST APIconsthttpTool=awaitclient.tool.create({name: 'Weather API',tool_type: 'http',config: {url: 'https://api.weather.com/v1/current',method: 'GET',headers: {'X-API-Key': 'xxx'},},});// Built-in tool (e.g., web search)constsearchTool=awaitclient.tool.create({name: 'Web Search',tool_type: 'builtin',config: {toolkit: 'web_search'},});// MCP tool (SSE transport)constmcpTool=awaitclient.tool.create({name: 'MCP Tool',tool_type: 'mcp',config: {transport: 'sse',server_url: 'https://mcp.example.com/sse',},});awaitclient.tool.update(httpTool.id,{name: 'Weather API v2'});awaitclient.tool.delete(httpTool.id);constbuiltins=awaitclient.tool.listBuiltins();builtins.forEach(b=>console.log(`${b.toolkit} — ${b.description}`));Skills are Markdown snippets injected into an agent's system prompt. Use them to extend or specialize agent behavior without changing the base prompt.
constskills=awaitclient.skill.list();constskill=awaitclient.skill.create({name: 'Formal Language',description: 'Instructs the agent to always use formal language',content: `## Tone Guidelines\n- Always address the user formally\n- End each response with "Is there anything else I can help you with?"`,});awaitclient.skill.update(skill.id,{content: 'Updated skill content...'});awaitclient.skill.delete(skill.id);Archetypes are reusable base configurations — combining a base system prompt with a default set of skills. Assign an archetype to multiple agents to ensure consistent behavior.
constarchetypes=awaitclient.archetype.list();constarch=awaitclient.archetype.create({name: 'Support Agent Template',description: 'Base configuration for all support agents',base_prompt: 'You are a professional support agent...',});awaitclient.archetype.update(arch.id,{base_prompt: 'Updated base prompt...'});awaitclient.archetype.delete(arch.id);Rooms are persistent containers for multi-turn voice sessions. A room can host multiple agents and multiple concurrent sessions.
constrooms=awaitclient.agent.rooms.list();constroom=awaitclient.agent.rooms.create({name: 'Support Lobby',description: 'Real-time voice support room',talking_style: 'sequential',// 'sequential' | 'moderator_led' | 'freeform'visibility: 'private',// 'private' | 'shared' | 'public'agent_ids: [agentId],});awaitclient.agent.rooms.update(room.id,{name: 'Updated Name'});awaitclient.agent.rooms.delete(room.id);const{ agent_ids }=awaitclient.agent.rooms.listAgents(room.id);awaitclient.agent.rooms.addAgent(room.id,agentId);awaitclient.agent.rooms.removeAgent(room.id,agentId);// Start a new sessionconstsession=awaitclient.agent.rooms.startSession(room.id,{voice_id: 'en-US-female',// Optional — overrides agent default});// List all sessions in a roomconstsessions=awaitclient.agent.rooms.listSessions(room.id);// List sessions (with pagination and status filtering)const{ data, total }=awaitclient.agent.sessions.list({status: 'running',// 'running' | 'paused' | 'ended'page: 1,page_size: 20,});constsession=awaitclient.agent.sessions.get(sessionId);awaitclient.agent.sessions.pause(sessionId);awaitclient.agent.sessions.resume(sessionId);awaitclient.agent.sessions.end(sessionId);constparticipants=awaitclient.agent.sessions.getParticipants(sessionId);// Retrieve conversation historyconst{data: messages}=awaitclient.agent.sessions.listMessages(sessionId,{page: 1,page_size: 50,});// Inject a message into the sessionawaitclient.agent.sessions.appendMessage(sessionId,{role: 'user',content: 'Please check my order status.',speaker_type: 'user',speaker_ref_id: 'user-uuid',});// Get a token for the first participantconst{ token, livekit_url, room_name }=awaitclient.agent.sessions.getLiveKitToken(sessionId,{user_id: 'end-user-123',user_name: 'Alice',});// Join as an additional participantconst{ token, livekit_url }=awaitclient.agent.sessions.join(sessionId,{user_id: 'end-user-456',});// Connect with @livekit/clientimport{Room}from'@livekit/client';constlivekitRoom=newRoom();awaitlivekitRoom.connect(livekit_url,token);Override per-participant configuration at runtime.
awaitclient.agent.sessions.upsertParticipantContext(sessionId,'user-ref-id',{custom_prompt: 'Respond only in Spanish.',variables: {userName: 'Carlos'},});awaitclient.agent.sessions.deleteParticipantContext(sessionId,'user-ref-id');The SDK exports typed error classes for every failure mode.
import{AudaraiError,AuthenticationError,InsufficientBalanceError,RateLimitedError,ApiError,}from'@audarai/sdk';try{constaudio=awaitclient.tts.synthesize('Hello');}catch(err){if(errinstanceofAuthenticationError){// Invalid or expired credentialsconsole.error('Authentication failed — check your credentials.');}elseif(errinstanceofInsufficientBalanceError){// HTTP 402 — account balance depletedconsole.error('Insufficient balance — please top up your account.');}elseif(errinstanceofRateLimitedError){// HTTP 429 — too many requestsconsole.error(`Rate limited — retry after ${err.retryAfter}s`);}elseif(errinstanceofApiError){// Any other HTTP errorconsole.error(`API error ${err.statusCode}: ${err.message}`);}}The SDK proactively refreshes session tokens before they expire (default: 30 seconds before expiry). A mutex prevents redundant concurrent refresh calls. If a 401 response is received, the SDK clears the cached token and retries the request once automatically.
constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',publishableKey: 'pk_xxx',refreshThresholdSeconds: 60,// Refresh 60s before expiry (default: 30)});Node.js 18+ includes native fetch — no extra configuration needed.
For Node.js < 18, pass a custom fetch implementation:
importfetchfrom'node-fetch';constclient=createAudaraiClient({baseUrl: 'https://prod.audarai.com/apiv2',apiKey: 'ak_xxx',fetch: fetchastypeofglobalThis.fetch,});The SDK is written in TypeScript and ships full type declarations out of the box. Every request option, response shape, and callback signature is typed.
import{createAudaraiClient,typeAudaraiClientConfig,typeSynthesizeOptions,typeTranscribeResult,typeAgentResponse,typeSessionResponse,typeKnowledgeResponse,typeTranslationResult,AudaraiError,AuthenticationError,ApiError,}from'@audarai/sdk';A full-featured Vue 3 + Vite demo app is included under the demo/ directory. It provides an interactive UI for every SDK feature, including real-time microphone recording, log viewing, and LiveKit voice sessions.
cd demo
npm install
npm run devOpen http://localhost:5173 and enter your credentials to explore all capabilities interactively.
MIT — see LICENSE for details.
Questions? Open an issue or visit audarai.com.