Voice & Audio

Four cooperating subsystems, all reachable from the Audio console tab and the voice header control:

1. Provider TTS — MiniMax / Qwen synthesis of text (spoken replies, streaming speech).
2. Voice chat (STS) — live speech-to-speech with Qwen Omni realtime models over a local WebSocket proxy on :5002.
3. Wake-word personas — user-defined names that switch the voice used by TTS, browser speech, and STS.
4. Dictation + voice commands — browser Web Speech API dictation with a wake-word command grammar.

Every paid call is priced from model_harness/audio/pricing.py and recorded to the workspace-stats JSONL (record_workspace_stats), so spend shows up in the Usage tab and the Audio dashboard.

HTTP routes (routes_audio.py)

Settings defaults (routes_audio.py:62): empty tts_provider/stt_provider mean "use the browser" (speechSynthesis / Web Speech API); budget_usd: 0 means no cap.

TTS providers (audio/tts.py)

Stdlib urllib only, Bearer key from env (MINIMAX_API_KEY / DASHSCOPE_API_KEY), failures returned as {"ok": False, "error": ...} — never raised (tts.py:113).

• MiniMax — POST https://api.minimax.io/v1/t2a_v2, non-streaming, hex-encoded audio in the JSON response, decoded to bytes (tts.py:189). Default model speech-2.8-turbo, default voice English_Graceful_Lady (tts.py:61).
• Qwen — all Qwen requests are normalized to qwen3-tts-flash-realtime over the international realtime WebSocket (wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime), because the CosyVoice/qwen-audio HTTP endpoints fail on the international region (tts.py:152-170). Voice ids are validated against the Qwen catalog; an unknown voice (e.g. a MiniMax clone id) falls back to Cherry. Protocol: session.update → input_text_buffer.append → input_text_buffer.commit → collect base64 response.audio.delta chunks (tts.py:273-279). The stream is raw PCM; it is wrapped in a 24 kHz/16-bit/mono WAV header so the browser <audio> element can play it (tts.py:312).

Output files are tts_<provider>_<timestamp>.<ext> under <workspace>/audio/ (tts.py:94).

Pricing (audio/pricing.py)

AUDIO_PRICING is the single rate table, USD per 1M units; unit depends on modality: TTS per input character, STT per second, STS per token (pricing.py:8-12).

Omni sessions are billed with the audio-specific rates OMNI_AUDIO_IN_PER_1M = 4.44 / OMNI_AUDIO_OUT_PER_1M = 8.89 (pricing.py:107) via compute_omni_cost(); compute_cost() returns 0.0 for unknown models (fail-safe). The same rates are hardcoded client-side in app.js computeOmniCost (app.js:7212).

Voice catalogs + cloning

voices.VOICES holds verified system ids per provider (MiniMax EN/ZH/Yue/JA; Qwen Qwen3-TTS family Cherry/Ethan/Ryan/…). voices.OMNI_VOICES (voices.py:97) is the full STS catalog — 80+ voices across up to 29 languages, each {id, g, lang, desc}. omni_voices_for_model() enforces the model restriction: Omni-Turbo only supports Chelsie, Serena, Cherry, Ethan; flash and LiveTranslate support the full list (voices.py:182). The JS duplicates this filter in populateVoiceChatVoices (app.js:6885).

Cloning (voiceclone.py) is zero-shot, no training:

• MiniMax: two-step — upload sample to /v1/files/upload → file_id, then POST /v1/voice_clone with the chosen voice_id; sample ~10s–5min (voiceclone.py:106).
• Qwen: DashScope voice-enrollment endpoint with model cosyvoice-clone-v2, the sample as a base64 data URL, optional reference transcript; sample 10–20s (voiceclone.py:152).

voice_id must start with a letter. Successful clones are appended to audio_cloned_voices.json (routes_audio.py:93) and appear in the voice picker; a cloned id is used exactly like a system voice in /api/audio/tts.

Voice chat (STS) over the :5002 proxy

voice_ws_server.py is a standalone asyncio websockets server (default 127.0.0.1:5002, env VOICE_WS_HOST/VOICE_WS_PORT), started once in a daemon thread. It exists because Werkzeug's WS layer intermittently sent frames Chrome rejects as "Invalid frame header" (voice_ws_server.py:1-14). The browser connects to ws://<host>:5002/voice?model=<omni-model> (default qwen3-omni-flash-realtime); the proxy attaches Authorization: bearer $DASHSCOPE_API_KEY and relays frames bidirectionally to DashScope, logging voice changes seen in session.update messages.

Client side (app.js:6809-7231): on connect the browser sends session.update with modalities [text, audio], the resolved persona/picker voice, PCM in/out formats, server_vad turn detection (threshold 0.5, 800 ms silence), input transcription via qwen3-asr-flash-realtime, and a verbosity instruction preset (concise/balanced/detailed; app.js:6944-6950). LiveTranslate models instead send translation.language plus an optional source-language transcription config (app.js:6919-6942). Mic audio is captured by an AudioWorklet (static/js/voice-pcm-processor.js) that decimates to 16 kHz Int16 PCM in ~100 ms chunks, base64-encoded as input_audio_buffer.append. Response audio (response.audio.delta, base64 PCM or binary frames) is scheduled gapless on a 24 kHz AudioContext. Barge-in: input_audio_buffer.speech_started flushes the playback queue (app.js:6995-6999). Usage tokens accumulate from response.done; on disconnect the total is posted to /api/audio/voice-session-end for cost recording (app.js:7170-7181).

flowchart LR
    Mic[Microphone] --> WK[AudioWorklet 16kHz PCM]
    WK -->|input_audio_buffer.append| PX[ws :5002 voice proxy]
    PX -->|bearer DASHSCOPE_API_KEY| UP[DashScope Omni realtime]
    UP -->|response.audio.delta| PX
    PX --> PB[24kHz playback queue]
    UP -->|input transcript deltas| VC[VC.extractWakeName]
    VC -->|persona name heard| SP[app.js setPersona]
    SP -->|session.update voice| PX
    UP -->|response.done usage| CST[computeOmniCost]
    CST -->|POST voice-session-end| FL[Flask routes_audio]

Mid-session persona switch: the user's own speech is transcribed upstream; every transcript delta runs through VC.extractWakeName, and a detected persona name calls setPersona(), which sends a live session.update {session: {voice}} through the open socket (app.js:7006-7018, 6619-6628).

Wake-word personas

Built-in defaults (PERSONA_DEFAULTS, app.js:6579) — user overrides in audio.settings.personas merge over them:

• Names are user-defined: one lowercase word, 3+ letters (/^[a-z]{3,}$/, app.js:6695); add/rename/remove in the Audio tab. VC.setWakeNames() rebuilds the wake regex on every change (app.js:6676).
• computer is always a valid wake word and resets to the neutral (no-persona) voice.
• Voice resolution per surface: personaTtsVoice() maps provider minimax→p.minimax, qwen→p.omni (app.js:6633); resolveOmniVoice() validates against the STS model — on Turbo, personas collapse onto the 4 supported voices (male→Ethan, female→Cherry; app.js:6646-6653); browserVoiceFor() picks an installed English system voice by gender regex (app.js:6658).
• Selection paths: the always-listening wake recognizer (app.js:5388), a dictation utterance starting with a persona name (app.js:4851), and in-STS transcript detection (above). state.activePersona persists in localStorage.

Streaming sentence-level speech (streamSpeaker)

streamSpeaker (app.js:6440-6571) speaks direct-mode responses while they stream: feed() is called on every content SSE delta (app.js:2798), finish() flushes the remainder on done (app.js:2883), cancel() drops everything on error/new query/toggle-off (app.js:2904, 2555). Chunking: buffer until a sentence boundary after ≥40 chars (MIN_CHUNK), force-flush at 500 chars (MAX_CHUNK) on the last space (app.js:6452-6469). With a provider configured, chunks run through a one-at-a-time synthesis pipeline (synthQueue → /api/audio/tts → playQueue → shared audio player, ordered playback); on failure it falls back to the browser voice with one toast per session. Without a provider, chunks go straight to speechSynthesis with the persona browser voice.

Dictation + voice commands

Dictation (app.js:4686-5142) uses the browser Web Speech API (webkitSpeechRecognition): raw finals accumulate in the raw pane, then each chunk is cleaned by POST /api/stt/assist into the draft pane (app.js:4950-4985). Holding Space latches command mode for 2.5 s (app.js:4720-4732). The draft is committed with "apply prompt" or the Use button.

static/js/voice-commands.js is a pure, DOM-free matcher (also require()-able by the node tests). Grammar: <wake-word> [filler punctuation] <command> [payload] — wake regex ^(names)\b[\s,.:;-]*(.*)$, case-insensitive (voice-commands.js:33). The COMMANDS table (~30 entries, voice-commands.js:62-293) defines exact phrases or prefixes with payload: 'rest'; confirm: true commands (stop, new session, clear prompt) ask for a spoken yes first; ack is the spoken confirmation. Notable commands: run query, stop, new session, approve/deny/approve all, inject … (mid-run instruction), read response/read prompt, stop reading, agent mode/direct mode, show/hide trace, analyze trace, apply/append/prepend/clear prompt, review draft + keep/remove phrase lists (parseNumberList handles spoken digits/words), remove duplicate words (dedupeWords), turn on/off mic, reset voice view, show/hide commands, stop listening, and choose option 1–5 for question cards. Unmatched utterances get a "did you mean …?" from suggest() (Dice bigram similarity, threshold 0.45, voice-commands.js:416). The CHEAT_SHEET table (voice-commands.js:310) is the single source for both the "?" popover and the voice column; docOnly rows are documented gestures, and the node tests assert every other row resolves through matchCommand.

verified against code: 2026-08-09