World's Fair 2025
Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten
Philip Kiely , Yineng Zhang , Baseten
Overview
This talk introduces SGLang, an open-source framework designed for high-performance serving of large language models (LLMs) and large vision models (LVMs). It emphasizes SGLang's production readiness, speed, and strong community support, highlighting its ability to provide day-zero support for new model releases and allow users to contribute to its development. The framework is presented as a valuable tool for optimizing LLM inference.
Who should watch
- AI engineers and developers looking to deploy and optimize LLM serving.
- Product Managers and builders evaluating inference solutions.
- Individuals interested in open-source LLM infrastructure and contributing to its development.
- Those seeking to improve inference performance through advanced techniques like speculative decoding and optimized batching.
Key takeaways
- SGLang is a performant, production-ready open-source serving framework for LLMs and LVMs, often used with backends like vLLM or TensorRT-LLM.
- The framework offers excellent performance across various GPUs and provides day-zero support for new model releases from major labs.
- Optimizing inference involves tuning parameters such as `cuda_graph_max_batch_size` to leverage hardware capabilities effectively.
- Speculative decoding algorithms like Eagle 3 can significantly improve performance by increasing token acceptance rates.
- SGLang's codebase is accessible, encouraging community contributions through GitHub issues labeled "good first issue" or "help wanted."
- The framework supports features like constraint decoding and function calling, with an OpenAI-compatible server interface.
- Users can customize and contribute to SGLang, allowing for self-unblocking when specific model support or features are needed.
- The project has seen rapid growth, achieving significant GitHub stars in a short period, indicating strong community adoption.
Notable quotes
*SGLang offers excellent performance on a wide variety of GPUs. It's production ready out of the box.*
*If something is broken in SGLang, if you don't like something, you can fix it.*
Unofficial community note. Prefer the recording for nuance.