STT & TTS
The STT → Agent → TTS flow turns any existing Agent into a voice agent without changing the agent itself. Speech-to-text is added as a pipeline wrapper; text-to-speech is added as an observer that listens to the model's streamed message chunks.
Provider support#
| Provider | STT | Streaming STT | TTS | Streaming TTS | Audio to Audio |
|---|---|---|---|---|---|
| OpenAI | ✅ | ✅ | ✅ | — | ✅ |
| ElevenLabs | ✅ | — | ✅ | ✅ | — |
| Mistral | ✅ | — | ✅ | ✅ | — |
| Gemini | — | — | — | — | ✅ |
Mixing providers across the two halves is supported and often the best setup: transcribe with OpenAI for live captions, speak with ElevenLabs for low time-to-first-byte.
Note
The Audio to Audio column means a native speech-to-speech endpoint. To get continuous, full-duplex audio out of providers that have none — ElevenLabs voices on any LLM, for instance — use CascadeConfig, which presents an STT + LLM + TTS cascade to LiveAgent as if it were one realtime session.
ElevenLabs classes need the optional dependency and ELEVENLABS_API_KEY in the environment:
Mistral classes need the mistral extra and MISTRAL_API_KEY in the environment (or pass api_key):
MistralRealtimeTranscriber and MistralRealtimeTTSConfig also need the SDK's websocket client:
OpenAI credentials#
The OpenAI voice configs (OpenAIRealTimeConfig, OpenAITranscriber, OpenAITranslationTranscriber, OpenAITTSConfig) accept a keyword-only api_key. Omit it to keep the SDK's OPENAI_API_KEY default:
For a shared client or other SDK options, pass client= instead. Passing both client and api_key raises ValueError.
Audio I/O primitives#
SoundDeviceRecorder captures microphone input and SoundDevicePlayer plays synthesized speech. Both are thin wrappers around the sounddevice library and share the same event stream.
The recorder produces a VoiceInput containing 16-bit PCM bytes plus the sample rate and channel count. The player subscribes to SynthesizedAudioEvent on its context's stream and plays each chunk on a background thread.
Note
Recorder.record(duration=...) is a one-shot, blocking helper for the turn-by-turn flow. For continuous streaming (used by LiveAgent), use the recorder as an async context manager — see LiveAgent.
Speech-to-Text#
OpenAITranscriber (the OpenAI STTConfig class from ag2.live.openai) implements the STTConfig protocol and exposes a .pipe(agent) method that wraps an Agent in a VoicePipeline. Calling pipeline.ask(voice) transcribes the audio and forwards the text to the agent's normal ask() flow.
pipeline.ask(...) returns a VoiceReply that exposes the same surface as AgentReply (.body, .response, .history, .ask(...)) plus .ask(voice_input) for the next voice turn. The agent's history is preserved across turns.
Tip
The transcriber emits TranscriptionChunkEvent and TranscriptionCompletedEvent on the agent's stream as soon as the transcription server starts producing tokens. Subscribe to them to display live captions.
Translation#
If you want the user's speech transcribed into English regardless of input language, swap in OpenAITranslationTranscriber. It has the same API as OpenAITranscriber but uses OpenAI's translation endpoint.
ElevenLabs (Scribe)#
ElevenLabsTranscriber implements the same STTConfig protocol, so it drops into the same .pipe(agent) call:
Optional settings are only sent when you set them — leave language_code unset and Scribe auto-detects. diarize=True asks for speaker labels, and num_speakers caps how many.
Scribe tags non-speech sounds in the transcript by default (for example (laughter)), and the agent sees that text. Pass tag_audio_events=False to drop the tags, and keyterms=[...] to bias recognition toward names and jargon:
Mistral (Voxtral)#
MistralTranscriber uploads the clip as WAV to Mistral's offline transcription endpoint and sends a single TranscriptionCompletedEvent:
language and context_bias (up to 100 words or phrases to favour) are only sent when you set them — leave language unset and the model auto-detects. diarize=True requests speaker diarization (segment timestamps are sent automatically, as the API requires); transcribe() still returns plain text. Pass api_key or a preconfigured client to override the MISTRAL_API_KEY default.
MistralRealtimeTranscriber streams the clip to Voxtral Realtime over a websocket, so the words arrive as they are recognised: a TranscriptionChunkEvent per delta, then a closing TranscriptionCompletedEvent. It keeps the same transcribe(voice, context) -> str contract, so it drops into .pipe() and CascadeConfig unchanged:
The audio is the finished utterance (the turn detector decides where it ends), sent as pcm_s16le at the clip's sample rate, and it must be mono. target_streaming_delay_ms trades latency for accuracy; leave it unset for the model default.
Note
TranscriptionCompletedEvent is sent only with the final transcript Mistral returns once the clip is processed. If the connection drops first, transcribe() raises RuntimeError instead of completing on the partial text. Cancelling it (for example, a CascadeConfig(barge_in=True) interruption) cancels the stream and sends no completed event, so the agent never runs on a half-heard utterance.
Text-to-Speech#
TTSObserver is an observer that listens to ModelMessageChunk events as the agent streams its response, batches them into sentence-sized chunks, calls a TTS provider, and emits SynthesizedAudioEvents onto the stream. A SoundDevicePlayer attached to the same stream then plays them.
Warning
The agent's config must be set up for streaming output (e.g., streaming=True). TTSObserver works at the ModelMessageChunk granularity — if the model emits a single non-streaming ModelMessage, the observer will still synthesize it, but you lose the sentence-level pipelining that keeps latency low.
Never attach it to something that already speaks
TTSObserver synthesizes from the model's text. A LiveAgent running a speech-to-speech config or a CascadeConfig already produces its own audio and emits the matching transcript — so the observer would speak a second copy over it. Two voices, doubled TTS spend. See LiveAgent for the one realtime setup where attaching it is correct.
Voice and speed#
Any config implementing the TTSConfig protocol works with TTSObserver:
ElevenLabsTTSConfig is the buffered mode — required by eleven_v3, which cannot stream. For the flash and turbo families, prefer streaming (below).
Both ElevenLabs TTS configs also take voice_settings, language_code and seed; each is sent only when set. voice_settings is the SDK's VoiceSettings (stability, speed, and so on):
Note
Mistral needs exactly one of voice_id (a saved voice) or ref_audio (a base64 reference clip); there is no default voice. It needs MISTRAL_API_KEY (or api_key=) and pip install "ag2[mistral]". Mistral returns float32 PCM; the configs convert it to the int16 PCM that SoundDevicePlayer plays.
Note
output_format defaults to pcm_24000, matching what SoundDevicePlayer and SoundDeviceRecorder expect. Use a raw PCM format only — the player does not decode mp3 or other containers — and if you change the sample rate, change the player to match.
Streaming TTS#
With a buffered config, TTSObserver waits for each sentence to be fully synthesized before any of it plays. A streaming config removes that wait: audio chunks are forwarded as they are generated, so playback starts at time-to-first-byte.
Note
ElevenLabs and Mistral (MistralStreamingTTSConfig) have streaming TTS configs. OpenAITTSConfig performs a single buffered read and has no streaming equivalent.
Only the config changes — TTSObserver detects that the config can stream and uses that path automatically:
The same works with Mistral — swap in MistralStreamingTTSConfig("voxtral-mini-tts-2603", voice_id="my-saved-voice"). It converts Mistral's float32 PCM to the int16 PCM SoundDevicePlayer expects.
Streaming the reply's text (Mistral realtime)#
A streaming config still waits for a whole sentence: TTSObserver cuts the reply at sentence boundaries (min_chars, default 60) and sends one request per sentence. MistralRealtimeTTSConfig takes the reply's text instead, as the model streams it, over one realtime connection per reply. Mistral decides where to cut the audio, so the voice keeps its context across sentences and the reply pays for one connection instead of one request per sentence. Mistral still holds a short sentence back until the text after it arrives, so the first audio does not come sooner than with sentence buffering:
It works anywhere a TTS config does, including CascadeConfig(tts=...). Each model response is its own reply: when the model says something before calling a tool, that text is spoken on its own connection before the tool runs, and the answer after the tool opens a new one. A non-streaming model config (no message chunks) gets its completed reply spoken. A server error event raises RuntimeError; a connection the server refuses (an unknown voice or model) raises the SDK's RealtimeSpeechException. A CascadeConfig barge-in closes the connection in the background, so the next turn does not wait for the closing handshake.
Note
TTSObserver only sees a reply end through its ModelMessage. If a reply fails mid-stream, its open connection carries over into the next reply.
Streaming directly to the speaker#
ElevenLabsStreamingTTSConfig.stream(...) is usable on its own, outside any agent, when you just want to speak a known string:
player.play(...) only enqueues — a background worker thread writes to the output device, so this loop is never blocked by playback.
Sentence batching#
TTSObserver batches streamed text before synthesizing, since a per-token request would be slow and expensive. min_chars (default 60) is the smallest amount of text worth a request; below it the buffer keeps accumulating even past a sentence boundary.
Lowering it starts the first sentence sooner, at the cost of more requests and more granular sentences.
Combining STT and TTS#
The full round-trip — voice in, voice out — is just both halves wired up at once: pipe the agent through the transcriber, attach a TTSObserver, and share a stream with the player.
The player is scoped to a single turn and attached to a shared MemoryStream, so history carries across turns while each turn's audio is guaranteed to have finished before the microphone opens again.
player.join() does not wait for playback to finish
join() returns as soon as the queue is empty — but the worker thread has already dequeued the final chunk and is still writing it to the device, so audio is very much still coming out of the speaker. With streaming TTS it is worse: the queue legitimately hits zero between chunks arriving from the provider, so join() can return mid-sentence.
If you open the microphone at that point, the recorder captures the assistant's own voice, it gets transcribed as the user's next turn, and the conversation talks to itself:
you: "Hello, are you there?"
bot: Yes — I'm here and ready to help. What can I do for you?
you: "Yes, I'm here and ready to help. What can I do for you?"
Use one player per speaking phase instead. Leaving the async with block calls close(), which pushes a sentinel, lets the worker finish every queued write, and joins the thread — that genuinely waits for the audio to finish.
This is echo suppression by turn-taking, not true echo cancellation
Even with the microphone opened only after playback ends, a loud speaker plus a sensitive mic can still bleed in. There is no AEC in ag2.live — use headphones when testing a speaker-and-mic loop. For genuine overlap handling, LiveAgent has server-side VAD and barge-in.
Mixing providers#
This is an example of how one can use openai STT and elevenlabs TTS together.
Both halves share one stream, so TranscriptionChunkEvent (from the transcriber) and SynthesizedAudioEvent (from the observer) land in the same place — the player only subscribes to the latter, leaving the captions to your own subscriber.
Examples#
examples/stt_tts.py/stt.py— transcription only, with a follow-up turnexamples/stt_tts.py/tts.py—TTSObserverspeaking an agent's replyexamples/stt_tts.py/stt_tts.py— the OpenAI round-tripexamples/stt_tts.py/elevenlabs_stt_tts_streaming.py— streaming TTS, prints time-to-first-byteexamples/stt_tts.py/elevenlabs_stt_tts_sync.py— buffered TTS witheleven_v3
Run the last two back to back to hear the difference on the first sentence.
What's next#
- LiveAgent — drop the turn-by-turn round-trip in favor of a streaming, full-duplex realtime session.
CascadeConfigkeeps the mix-and-match providers from this page while giving you continuous audio and barge-in. - Observers —
TTSObserveris one of many observer patterns; see the harness docs for logging, persistence, and custom observers.