Session brief
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Overview
This talk explores the engineering challenges and architectural patterns for building high-quality, low-latency voice agents at scale. It highlights that voice is a natural interface for human-computer interaction, moving beyond customer service to more complex applications. The core of the discussion revolves around the dominant pipeline architecture, its components, and the critical trade-offs involved in achieving real-time, intelligent, and natural-sounding voice interactions.
Who should watch
- AI Engineers
- Product Managers
- Developers building agentic systems
- Those working on real-time conversational AI
- Engineers focused on optimizing latency and quality in AI applications
Key takeaways
- Voice agents require real-time performance, with response times under 500 milliseconds to maintain natural conversation flow.
- Key challenges include achieving low latency, ensuring sufficient intelligence for complex workflows and tool calling, natural-sounding voice output, and scalability for high concurrency.
- The dominant architecture for voice agents is a pipeline: audio streams into speech-to-text (STT), then to a Large Language Model (LLM) for decision-making and tool use, and finally to text-to-speech (TTS) for audio output.
- For STT, word error rate and time to complete a transcript are crucial metrics, with state-of-the-art models achieving around 6% word error rate.
- LLM performance is measured by time to first token (TTFT), ideally around 200-300 milliseconds, influencing model size choices (typically 8-30 billion parameters).
- TTS quality is assessed by time to first audio (TTFA) and real-time factor, with naturalness, pronunciation accuracy, and emotional control being key capabilities.
- Co-locating models and orchestrators within the same data center or building can significantly reduce network latency, yielding substantial performance improvements.
- Pure speech-to-speech models are an emerging alternative to pipelines, offering potential benefits in preserving speech nuances and enabling more fluid, full-duplex communication, though they currently face challenges with instruction following and tool calling.
Notable quotes
*Voice is just this brand new interface to interact with systems and computers.*
*Humans, we learned how to talk before we learned how to read, right? So, this comes very naturally to us.*
*You have to solve every single one of them at the same time, or you're going to be in a little bit of a trouble.*
Unofficial community note. Prefer the recording for nuance.