World's Fair 2025
Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI
Overview
This talk explores the advancements and practical applications of building effective voice agents, moving beyond traditional text-based AI. It highlights the emergence of speech-to-speech technology as a key component of the multimodal AI era, emphasizing that current models are now fast, expressive, and accurate enough for scalable production use. The discussion covers architectural patterns, key trade-offs in development, and best practices for creating robust and engaging audio experiences.
Who should watch
- AI engineers and builders exploring new modalities beyond text.
- Product Managers evaluating the potential of voice interfaces.
- Developers interested in low-latency, expressive audio applications.
- Teams considering customer service or consumer-facing voice solutions.
- Anyone looking to understand the trade-offs in building real-time audio agents.
Key takeaways
- Audio models have reached a tipping point, offering faster, more expressive, and accurate speech-to-speech capabilities compared to older, slower, and more brittle systems.
- Two main architectural patterns exist: the older chained approach (transcribe-LLM-TTS) which can be slow and lossy, and the emerging speech-to-speech architecture (like OpenAI's real-time API) which is more efficient and maintains semantic understanding.
- Building voice agents involves key trade-offs across latency, cost, accuracy, user experience, and integrations, with different priorities for consumer-facing versus customer service applications.
- For voice agents, prompting extends beyond instructions to include controlling voice expressiveness, demeanor, and tone, significantly impacting user experience.
- Effective tool use in voice agents often involves delegation to specialized smaller models for specific tasks rather than overwhelming a single agent with numerous tools.
- Evals for voice agents should include observability, prompt iteration based on labeled data, transcription-based tests for business criteria, and specialized audio evals for tone and pacing.
- Synthetic conversations, where agents interact with simulated customer personas, offer a scalable method for testing and understanding agent performance.
- Guardrails are crucial for safety and confidence in voice agent deployments, with considerations for asynchronous execution and configurable debounce periods.
Notable quotes
*The models are incredibly slow. They're very robotic in how they sound. And they're quite brittle.*
*The models are really, we do believe, at that good enough tipping point where you can start to build much more reliable experiences around it.*
*So, you can see it's much faster. It's incredibly expressive, emotional. You can steer it, you can get it excited, you can slow it down, and it's accurate.*
Unofficial community note. Prefer the recording for nuance.