World's Fair 2024
Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy
Overview
This talk introduces a new voice agent API that consolidates speech-to-text, large language model (LLM) processing, and text-to-speech into a single audio-in, audio-out interface. This approach simplifies development by abstracting away the complexities of integrating these components, enabling developers to build real-time voice applications more efficiently. The presentation demonstrates how to create a voice AI agent capable of function calling and discusses strategies for scaling these agents.
Who should watch
- AI Engineers
- Product Managers
- Developers building voice applications
- Those interested in real-time AI agents
- Builders looking to reduce latency in voice bots
- Anyone facing infrastructure challenges with traditional STT-LLM-TTS pipelines
Key takeaways
- The new voice agent API offers a unified interface for audio-in, audio-out processing, simplifying the development of real-time voice bots.
- This API integrates speech-to-text, LLM processing (including function calling), and text-to-speech, reducing the need for developers to manage separate services.
- The system supports multiple LLMs, including Claude, Llama, Mistral, OpenAI, and Groq, with a focus on enabling function calling for dynamic interactions.
- Low latency is a critical factor, and the API aims to minimize delays between speech input and audio output, with self-hosted options offering even lower latency.
- Developers can build more complex AI systems by composing multiple agents, each responsible for a specific task, managed by a routing agent.
- Scaling these AI agent swarms involves considerations for regional deployment, redundancy, and horizontal scaling, with Kubernetes support offered.
- The Deepgram TTS service offers 12 voices with low latency and cost, competing with major cloud providers, and is expanding language support.
- Advanced features like semantic endpointing and handling long pauses are being integrated to improve the naturalness of voice interactions.
Notable quotes
*Building a model is easy, rolling it out into production at a low price point is hard.*
*The vast majority of customers want female voices. I don't know what it is but I guess nobody wants to be mansplained.*
*Distance kills latency.*
Unofficial community note. Prefer the recording for nuance.