World's Fair 2025
[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han
Overview
This talk delves into advanced AI training techniques, focusing on reinforcement learning (RL), quantization, and agent development. It explores the evolution of large language models, from early open-source efforts spurred by leaks to current sophisticated training methodologies. The discussion highlights the critical role of fine-tuning stages, including supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), in transforming base models into capable conversational agents.
Who should watch
- AI Engineers
- Product Managers
- Builders working with LLMs
- Researchers interested in model training and optimization
- Developers looking to improve model performance and efficiency
Key takeaways
- The open-source LLM community has significantly advanced, often catching up to and sometimes rivaling closed-source models in performance.
- Reinforcement learning, particularly RLHF and newer paradigms like RLVR, is crucial for enhancing model capabilities beyond basic pre-training and SFT.
- Training methodologies have evolved from simple pre-training and SFT to more complex stages like mid-training and post-training, including techniques like Direct Preference Optimization (DPO).
- Agent development is closely linked to RL, where the language model acts as an agent interacting within an environment to maximize rewards.
- Efficient training and deployment are paramount, with techniques like quantization, parameter-efficient fine-tuning (PEFT) methods like LoRA, and optimized kernels (e.g., Triton) being key.
- The development of effective reward functions is a critical and challenging aspect of RL, often requiring domain expertise and careful design.
- Frameworks like VLM and Unsloth are highlighted for optimizing LLM serving and fine-tuning, enabling efficient experimentation on accessible hardware.
Notable quotes
*The algorithm is not special. The hard part is actually the reward functions itself and the data that you're going to shove into the model.*
*AI is about optimization. What is more efficient? What is more it's all optimization.*
*The goal of all these algorithms is to somehow force the models not to like overfit to your questions.*
Unofficial community note. Prefer the recording for nuance.