World's Fair 2024
Low Level Technicals of LLMs: Daniel Han
Overview
This talk delves into the low-level technical aspects of Large Language Models (LLMs), focusing on practical implementation details and common pitfalls. The speaker, Daniel Han, shares insights gained from analyzing and debugging open-source models like Gemma and NVIDIA's NeMo, highlighting issues in tokenization, model architecture, and training methodologies. The presentation aims to equip AI engineers with the knowledge to identify and fix bugs, optimize fine-tuning processes, and understand the underlying mathematical principles of LLMs.
Who should watch
- AI Engineers
- ML Engineers
- Researchers
- Anyone interested in the inner workings of LLMs
- Individuals looking to debug and optimize LLM implementations
- Builders working with open-source LLM frameworks
Key takeaways
- **Tokenization Issues:** Different models can have varying tokenization for the same characters (e.g., smiley faces), leading to inconsistencies. The process of tokenization itself can be complex and prone to errors before model training even begins.
- **Model Architecture Analysis:** Understanding the Transformer architecture, including components like attention mechanisms, RoPE embeddings, and MLPs, is crucial for debugging. The talk breaks down the mathematical underpinnings of these components.
- **Fine-tuning Optimization:** Techniques like using Triton kernels and optimizing memory usage can significantly speed up fine-tuning without sacrificing accuracy. The use of efficient GPU resources, like those available on Google Colab, is also discussed.
- **Bug Hunting and Fixing:** The speaker emphasizes the importance of meticulous code analysis to find bugs, citing examples from Gemma implementations where incorrect handling of floating-point types and missing BOS tokens caused issues.
- **Mathematical Foundations:** Concepts like Singular Value Decomposition (SVD) are highlighted as fundamental algorithms in computer science, underscoring the importance of a strong mathematical background for AI engineers.
- **Training Stability:** Techniques such as Layer Normalization (LayerNorm) and upcasting to float32 for critical operations like softmax are essential for stable training and preventing numerical issues like NaNs.
- **GPU Performance:** Understanding GPU specifications, including the impact of sparsity features and different floating-point formats (FP16, BF16, FP8), is key to optimizing performance and cost.
Notable quotes
*The tokenizer is a totally separate beast from language models right tokenization is like extreme extremely annoying.*
*The whole purpose of unso is to make everyone be able to fine tune their language models with very bad gpus.*
*The point is we can only use predict the future words right hello can only predict my name is Daniel and so on right that's the purple the purple component and the blue box is called the attention mechanism.*
Unofficial community note. Prefer the recording for nuance.