Voice I/O

The chat assistant supports hands-free use: speech-to-text turns what you say into a chat message, and text-to-speech reads the assistant's reply back aloud. The logic lives in apps/web/lib/voice.ts (the VoiceService class) and apps/web/lib/use-voice.ts (the useVoice React hook); the chat page at apps/web/app/app/chat/[[...conversationId]]/page.tsx wires both into the UI.

The two halves have very different requirements:

• Speech-to-text (STT) uses the browser's native Web Speech API — free, no API key, no server round-trip.
• Text-to-speech (TTS) uses ElevenLabs through a server route and needs an API key. It is optional: with no key set, chat stays fully usable and simply doesn't speak replies.

Speech-to-text (browser, free)

STT runs entirely in the browser via SpeechRecognition / webkitSpeechRecognition. It works push-to-talk: tap the mic to start, and it stops when you tap again or after a couple of seconds of silence. When a turn ends, the transcript is auto-sent to the assistant as a normal chat message.

useVoice reports a small state machine — idle, listening, processing, speaking — plus a live currentTranscript while you speak. It also exposes isSupported (detected after mount, so server and client render agree), which the UI uses to hide the mic button in browsers without the Web Speech API.

import { useVoice } from "@/lib/use-voice";

const { toggleListening, voiceState, currentTranscript, isSupported, speak } = useVoice({
  onTurnEnd: (transcript) => {
    // transcript is the finalized speech — send it as a chat message
  },
});

Browser support varies. The Web Speech API is well-supported in Chromium browsers; other browsers may not implement it. isSupported is false there, and the UI degrades to text-only input — no error, just no mic.

Text-to-speech (ElevenLabs, optional)

When enabled, the assistant's reply is sent to POST /api/voice/speak, which proxies to ElevenLabs and streams back audio/mpeg. The API key stays server-side and is never exposed to the client. useVoice handles playback and exposes controls — togglePlayback, seekTo, setPlaybackRate, an audioRef for the <audio> element, and a playback state object (current time, duration, rate, playing).

In the chat UI, TTS is opt-in per session via an "audio" checkbox; when it's on, each completed assistant reply is spoken. Tags and inline chart/image markers are stripped (stripAllTags from @valcraven/core) before the text is sent to TTS, so the assistant never reads a raw marker aloud.

Graceful degradation

TTS is designed to fail soft:

• If ELEVENLABS_API_KEY is unset, POST /api/voice/speak returns 503 ("Voice services not configured") and no audio plays — the rest of chat is unaffected.
• The route is rate-limited and rejects text over 5000 characters.
• Playback errors are caught and logged; they never break the chat turn.

Safari audio unlock. Browsers block audio that isn't tied to a user gesture. VoiceService warms up a silent audio element on the first mic tap (warmUpAudio) so later TTS playback can start without a fresh gesture.

Configuration

STT needs no configuration — it's a browser API.

The TTS request uses the eleven_turbo_v2_5 model with default voice settings; change these in apps/web/app/api/voice/speak/route.ts if you need a different voice profile.

On mobile

The Expo app has its own voice plumbing. Because there's no native rich-block renderer there, mobile (and TTS generally) strips chart/image markers from spoken text.

Related

• AI chat — the assistant voice input feeds into and voice output reads back.