Europe 2026
Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral
Overview
This talk explores the recent architectural trends in text-to-speech (TTS) models, highlighting their convergence with Large Language Model (LLM) approaches. The core thesis is that TTS is increasingly being treated as a sequence modeling problem, similar to how LLMs process text, enabling more natural and lower-latency speech generation, particularly for agentic interfaces.
Who should watch
- AI Engineers interested in TTS advancements
- Product Managers building voice-enabled applications
- Developers working with agents and conversational AI
- Researchers exploring speech generation architectures
Key takeaways
- The dominant trend in TTS is treating speech generation as a sequence modeling problem, leveraging autoregressive decoder backbones inspired by LLMs.
- To reduce latency, TTS models aim to generate audio in patches or tokens rather than sample by sample, allowing for early playback.
- Voice cloning, requiring only a few seconds of audio, is becoming easier and may evolve into a mainstream concept for brand identity.
- The architecture often involves an encoder to convert audio frames into tokens and a decoder to reconstruct audio, with variations in how tokens are processed (e.g., one token per step vs. multiple tokens per step).
- Mistral AI's recently released open-source TTS model uses a diffusion model to generate tokens for each frame simultaneously, differing from some other common patterns.
- Conditioning TTS models on text and voice identity is a key area with various approaches, impacting how context is provided during generation.
- Achieving low latency for real-time agent interactions is crucial, with ongoing research into optimal architectures for streaming text input.
- While the core TTS model is open-source, features like general voice cloning are currently proprietary.
Notable quotes
*The persistent anxiety that fills the rest of my life is calmed for as long as I have the flavor of something good in my mouth.*
*The dominant trend is to generate each sample one after the other. Then another era was generating the whole audio at once, but as we can as we saw, it's very interesting to have the beginning of the audio generated first so that we can start to play it out.*
*My take on this is that you we can go very, very far by just using speech as an interface.*
Unofficial community note. Prefer the recording for nuance.