Session brief
Voice Agents: the good, the bad, and the ugly
Overview
This talk explores the complexities and challenges of building AI voice agents, using a case study of an automated interview system. It highlights that while LLMs offer powerful capabilities, developing robust voice applications requires overcoming issues like transcription errors, conversational flow, and agent behavior. The presentation emphasizes the need for sophisticated architectural patterns beyond simple prompt engineering to achieve reliable and effective voice AI.
Who should watch
- AI engineers and developers working with voice interfaces or conversational AI.
- Product Managers and builders evaluating the feasibility and challenges of voice-based AI products.
- Anyone interested in the practical difficulties of integrating LLMs into real-world applications, particularly those involving audio.
- Individuals facing challenges with LLM hallucination, evaluation, and latency in conversational systems.
Key takeaways
- Voice agents operate on "hard mode," compounding standard LLM challenges like hallucination and evaluation with the complexities of audio processing, transcription, and real-time streaming.
- Initial naive approaches using monolithic prompts for voice agents quickly become insufficient when additional functionality, such as interactive question navigation, is required.
- Introducing tool use, where the agent explicitly signals intent to move to the next question, helps constrain behavior but can lead to reluctance to progress if not managed.
- A "drift detector" agent, operating separately in the text domain, can monitor conversations and strongly signal when the agent is off-track or needs to move to the next question, forcing tool use.
- Defining explicit goals and priorities for interviews, alongside a dedicated "next question agent," provides better guidance for the AI, enabling more informed rephrasing and follow-up questions.
- Transcription errors, particularly from background noise or silence, are a significant UX issue, requiring a separate agent to filter or hide problematic transcript segments.
- Building complex voice AI systems often involves adding multiple agents to address specific issues, leading to complexity in prompt management and debugging.
- Systematic evaluation is critical, even without objective ground truth. Synthetic conversations with personas can be used to measure agent performance across a simulated user population.
Notable quotes
*Voice models are just generally tough to wrangle.*
*When we're dealing with audio and these voice agents everything's on hard mode.*
*The basic kind of call the API do some prompt engineering and get it into a good place is very helpful it gets you a very far in the development process but it's not enough to build your robust app.*
Unofficial community note. Prefer the recording for nuance.