Europe 2026
Everything I Learned Training Frontier Small Models — Maxime Labonne, Liquid AI
Overview
This talk explores the unique challenges and opportunities in training small, frontier AI models, emphasizing that they are not simply scaled-down versions of larger models. The core thesis is that small models, designed for edge deployment, possess distinct characteristics like being memory-bound, having lower knowledge capacity, and being latency-sensitive. Addressing these specific traits requires tailored architectural choices, training methodologies, and problem-solving approaches, particularly for issues like doom-looping.
Who should watch
- AI engineers and researchers focused on efficient model development.
- Developers building applications for edge devices or environments with limited resources.
- Product managers and builders evaluating model performance, latency, and privacy trade-offs.
- Anyone interested in the practicalities of training and deploying smaller, specialized AI models.
Key takeaways
- Small models are memory-bound, leading to lower knowledge capacity and increased sensitivity to latency, necessitating optimized architectures.
- Architectural innovations like gated short convolutions, as seen in LFM 2, offer significant speed and memory advantages over alternatives like sliding window attention.
- Scaling pre-training tokens, even beyond traditional Chinchilla laws, continues to improve performance in small models, making them more compute-optimal than previously thought.
- Supervised fine-tuning (SFT) is most effective when narrowly focused on specific tasks, while preference alignment and reinforcement learning (RL) offer broader improvements and efficiency gains.
- Doom-looping, a repetitive output issue, is a particular challenge for small reasoning models on complex tasks and can be mitigated through preference alignment with LLM juries or RL with verifiable rewards and repetition penalties.
- Small models can excel in agentic tasks when augmented with tools like web search, compensating for their limited inherent knowledge capacity.
- The decision to use small models often hinges on factors like the need for offline functionality, low latency requirements, and strict privacy constraints.
Notable quotes
*Small models are not just scaled on versions of bigger models. They also have their unique challenges.*
*The more narrow you can find it or design it the better it is.*
*Edge models have unique challenges, and they are actually interesting from scientific point of view, and also production point of view.*
Unofficial community note. Prefer the recording for nuance.