Speaker
Philip Kiely
2 sessions in this library.
- Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten
This talk introduces SGLang, an open-source framework designed for high-performance serving of large language models (LLMs) and large vision models (LVMs). It emphasizes SGLang's production readiness, speed, and strong community support, highlighting its ability to provide day-zero support for new model releases and allow users to contribute to its development. The framework is presented as a valuable tool for optimizing LLM inference.
- Optimizing inference for voice models in production - Philip Kiely, Baseten
This talk focuses on optimizing inference for voice models in production, emphasizing runtime performance and infrastructure considerations. It highlights how the architectural similarity between Text-to-Speech (TTS) models and Large Language Models (LLMs) allows for the application of LLM optimization techniques. The core thesis is that while runtime optimizations are crucial, non-runtime factors like infrastructure and client code implementation can significantly impact overall latency and cost-efficiency.