World's Fair 2024
Building an AI assistant that makes phone calls [Convex Workshop]
Overview
This talk introduces Floyd, an AI assistant designed to make phone calls, demonstrating how to integrate various technologies to create a more capable personal assistant. The core idea is to leverage real-time speech-to-text, text-to-speech, and AI models to handle conversations with humans, managing tasks like booking appointments or ordering services. The system aims to provide a seamless experience by learning user context and proactively managing interactions.
Who should watch
- AI engineers and builders interested in agentic systems and practical AI applications.
- Product Managers exploring new AI-powered assistant capabilities.
- Developers looking to integrate real-time voice processing and AI models.
- Those interested in building applications with Convex, OpenAI, and Twilio.
- Individuals facing challenges with real-time data streaming and backend development.
Key takeaways
- An AI assistant can be built to handle phone calls by integrating real-time speech-to-text (Google Cloud), text-to-speech (OpenAI), and AI reasoning (GPT-4) with communication platforms (Twilio).
- Convex's reactive database and integrated backend capabilities simplify the development of real-time applications by managing data updates and server logic efficiently.
- User context is crucial for AI assistants; the system learns and stores information about the user over time to better fulfill requests.
- The architecture involves a client capturing voice requests, a server processing them using AI for context and action planning, and then initiating calls via Twilio.
- Real-time transcription and speech synthesis are streamed bidirectionally between the AI and the human on the other end of the call, with updates pushed to the client via Convex.
- Latency is a significant challenge, influenced by prompt length to AI models and the speed of text-to-speech conversion, but ongoing innovation in AI APIs promises future improvements.
- The system prioritizes transparency by informing the other party that they are speaking with an AI, especially when conducting business transactions.
- Convex offers developer ergonomics by unifying backend and frontend code, providing type safety, and eliminating the need for separate infrastructure teams.
Notable quotes
*A lot of the time short of collecting laundry and doing like physical things they're on the phone and they're on email.*
*The technology exists such that we can do all of this in real time.*
*I'm super impressed by the technology that we access to today the fact that this works and isn't painfully slow.*
Unofficial community note. Prefer the recording for nuance.