World's Fair 2025
Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)
Overview
This talk addresses the challenges and advancements in serving voice AI at scale, focusing on the critical need for low latency and high quality in real-time interactive applications. It introduces state space models (SSMs) as a more efficient alternative to transformers for handling long sequences, enabling faster and more natural voice interactions across various devices. The discussion highlights how these advancements are crucial for enterprise voice AI, customer support, gaming, and content creation.
Who should watch
- AI Engineers working on real-time applications.
- Product Managers evaluating voice AI solutions.
- Builders seeking to integrate advanced voice capabilities into their products.
- Developers concerned with model latency and quality.
- Anyone interested in the future of voice interaction and multimodal AI.
Key takeaways
- Traditional cloud-hosted foundation models often struggle with the low-latency requirements of interactive voice applications, where response times must be in milliseconds, not seconds.
- State space models (SSMs) offer a significant advantage over transformer architectures by providing O(1) inference time complexity, enabling consistent low latency regardless of input length.
- Cartisia AI has developed SSM-based models that match or exceed transformer performance in quality while offering superior latency, making them suitable for real-time voice AI.
- Key considerations for voice AI include naturalness, low latency for faster agent reasoning, and controllability for brand alignment through custom voices and accents.
- Voice AI is penetrating diverse markets such as healthcare, customer support, and real-time gaming, with applications ranging from NPC interactions to dynamic narration.
- Edge deployment of voice AI models can offer significantly lower latency than cloud roundtrips, with Cartisia's models running up to five times faster on edge devices.
- While speech-to-speech models are emerging, orchestrated solutions currently provide greater controllability and are better suited for enterprise-grade use cases.
- The development of voice AI relies on rich, high-quality data, but the specific requirements can vary based on the desired model capabilities and use cases.
Notable quotes
*In voice, you don't have seconds to actually give your response back. You have milliseconds.*
*With state based models or SSM generation at inference time is O of one. We maintain a state that you can generate from.*
*Running our models on edge are about five times faster than if you were to roundtrip.*
Unofficial community note. Prefer the recording for nuance.