World's Fair 2025
Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber
Overview
This talk details the experience of hosting an open-source voice AI model, Orpheus, for real-time consumer applications. The core challenge addressed is achieving low latency and high fidelity voice generation at a cost viable for widespread consumer use, which often requires near-free operation. The presentation highlights the technical hurdles in serving these models efficiently and presents solutions involving model fine-tuning, optimized infrastructure, and load balancing.
Who should watch
- AI Engineers
- Product Managers
- Builders focused on real-time AI applications
- Those working with open-source voice models
- Engineers dealing with inference infrastructure and latency
Key takeaways
- Consumer-facing AI applications, such as AI personas, require voice generation costs close to zero to be viable.
- The Orpheus model, trained on extensive voice and text data, outputs audio tokens that need to be processed at approximately 90-100 tokens per second for real-time performance.
- Fine-tuning models using Low-Rank Adaptation (LoRA) can significantly improve voice cloning quality and reduce latency, even with limited training data.
- A major source of latency in voice models can be "head of line silence" at the beginning of generated audio, which can be mitigated through fine-tuning.
- Vast.ai's VLM framework, with features like FP8 dynamic quantization and batch inference with LoRAs, is crucial for efficient GPU utilization and cost savings.
- A consistent hash ring load balancing strategy ensures that sessions are routed to the correct GPU, supporting both streaming input and arbitrarily long audio generation.
- The infrastructure stack involves WebRTC for client connections, WebSockets for GPU communication, and Redis for managing GPU assignments, all within a private network.
- The open-source ecosystem, including models like Orpheus, Llama, and tools like Snack and VLM, is enabling the development of sophisticated real-time AI experiences.
Notable quotes
*Consumer apps need something pretty close to free.*
*We knew that the only way to really get this to execute on our vision of putting these experiences everywhere we had to start bringing more things in house.*
*We were able to host voice models on GPUs and handle that infrastructure, so you can, too.*
Unofficial community note. Prefer the recording for nuance.