Code 2025
Efficient Reinforcement Learning – Rhythm Garg & Linden Li, Applied Compute
Rhythm Garg , Linden Li , Applied Compute
Overview
This talk introduces an efficient reinforcement learning (RL) approach for training large language models, focusing on making the process faster and cheaper for enterprise use cases. The core idea is to break the synchronous dependency between sampling data and training the model, enabling asynchronous operations. This method aims to improve ROI by moving AI beyond simple productivity tasks into real-world automations.
Who should watch
- AI Engineers
- Product Managers
- Builders looking to scale AI model training
- Those interested in optimizing RL training costs and speed
- Individuals working with large language models for specific enterprise tasks
Key takeaways
- Reinforcement learning can be used to imbue models with reasoning and intelligence capabilities for specific enterprise tasks, not just benchmarks.
- Synchronous RL, where sampling and training are locked together, leads to significant GPU underutilization due to long-tail latencies in sample completion.
- Asynchronous RL, particularly pipeline RL, separates sampling and training, allowing for continuous operation and better hardware utilization.
- A key challenge in asynchronous RL is managing "staleness" – when model weights used for sampling are outdated compared to the current training weights, which can destabilize learning.
- Applied Compute focuses on building an RL stack that is fast, cheap, and has low variance in training job duration estimates.
- A systems modeling approach is used to find optimal configurations for asynchronous RL, balancing compute allocation between sampling and training.
- This modeling helps predict performance and optimize GPU usage without running expensive full training jobs, simulating scenarios like increased response lengths.
- The developed system can achieve significant speedups, estimated around 60%, compared to synchronous baselines.
Notable quotes
*We think a lot about how do we push AI beyond productivity into real automations that deliver ROI.*
*The research problem for us that is very business critical is can we build an RL stack that is so efficient.*
*In other words, in synchronous RL, our step times are dictated by whatever sample takes the longest time in order to complete.*
Unofficial community note. Prefer the recording for nuance.