World's Fair 2025
Optimizing inference for voice models in production - Philip Kiely, Baseten
Overview
This talk focuses on optimizing inference for voice models in production, emphasizing runtime performance and infrastructure considerations. It highlights how the architectural similarity between Text-to-Speech (TTS) models and Large Language Models (LLMs) allows for the application of LLM optimization techniques. The core thesis is that while runtime optimizations are crucial, non-runtime factors like infrastructure and client code implementation can significantly impact overall latency and cost-efficiency.
Who should watch
- AI Engineers working with voice models.
- Product Managers evaluating the feasibility of voice-based AI products.
- Builders looking to deploy and scale TTS models efficiently.
- Those concerned with reducing inference costs for AI workloads.
- Developers seeking to improve the latency of real-time voice applications.
Key takeaways
- TTS models, often built on LLM architectures like Llama, can leverage LLM tooling for optimization.
- Performance metrics for voice models differ from LLMs, prioritizing time to first byte or sentence over time to first token, and focusing on concurrency for cost efficiency.
- Optimization techniques include using TensorRT-LLM, model quantization (e.g., FP8), and optimizing audio decoding pipelines with tools like Torch Compile.
- Dynamic batching, rather than continuous batching, can be effective for streaming audio, and the pipeline may become CPU-bound even with GPU acceleration.
- Achieving low latency requires optimizing not just the model runtime but also infrastructure, such as network overhead and session management in client code.
- Client code pitfalls include sequential request handling and repeated session creation, which can negate runtime gains.
- Voice agent pipelines involve three parts: listening, thinking, and talking, with inter-model infrastructure being critical for overall latency.
- Efficiently serving many voice agents on limited GPU resources, such as using MIG on an H100, can lead to significant cost savings.
Notable quotes
*The most important thing here is again, while you can have great runtimes, the infrastructure to connect these three together is really what's going to determine your latency.*
*As much fun as it is to talk about the runtime stuff and as much work as we do there, the the infrastructure and the client implementation is equally important if not more so.*
Unofficial community note. Prefer the recording for nuance.