World's Fair 2024
System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis
Overview
This talk discusses the system design challenges and future directions for running and training next-generation large language models. It highlights the significant computational and memory bandwidth requirements for inference, particularly for models with trillions of parameters. The presentation also touches upon the massive scale of infrastructure being built for training these models and the associated engineering hurdles.
Who should watch
- AI Engineers
- Product Managers
- Builders working with large language models
- Those interested in the infrastructure behind frontier AI models
- Individuals concerned with the cost and efficiency of LLM inference and training
Key takeaways
- Current large models like GPT-4, while powerful, are considered last-generation, with significantly larger models expected soon.
- LLM inference involves two distinct workloads: pre-fill (prompt processing) which is compute-intensive, and decode (token generation) which is memory bandwidth-intensive.
- Running models with trillions of parameters requires immense memory bandwidth, far exceeding current hardware capabilities for high token generation rates.
- Techniques like continuous batching and disaggregated pre-fill are crucial for efficient inference, especially at scale, to manage varying user request times and workload differences.
- Context caching, as seen in models like Gemini 1.5 Pro, offers a way to leverage long contexts without the prohibitive cost of fine-tuning, though it requires significant memory.
- The trend is towards massive training clusters with tens to hundreds of thousands of GPUs, enabling training of models with tens of trillions of parameters.
- Scaling to such large clusters introduces new challenges, including unreliable networking components (optics) and variations in GPU performance (silicon lottery), which can significantly impact training times.
- Increased compute power from these massive clusters is hoped to address issues like model hallucinations and unlock new capabilities.
Notable quotes
*We haven't seen a big capabilities leap... models that we're using today are largely the same as the models that were trained in 2022.*
*Decode is extremely memory bandwidth intensive... you have to load the whole model from the weights... for decode.*
*Running inference at batch size one is horrendously expensive... it's great to run if you're running it on your own personal devices but if you're running it in the cloud.*
Unofficial community note. Prefer the recording for nuance.