Europe 2026
Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci
Overview
This talk explores engineering reinforcement learning (RL) environments for language models, enabling them to learn through interaction, exploration, and feedback. It highlights how these environments serve as crucial training grounds for LLM agents, allowing them to develop skills in tool use, code execution, and complex task solving. The presentation introduces Verifiers, an open-source library for building such environments, and demonstrates their application through an experiment transforming a basic tic-tac-toe playing model into a master.
Who should watch
- AI engineers and builders interested in advanced LLM training techniques.
- Product Managers and engineers looking to enhance model capabilities beyond supervised learning.
- Developers seeking to create more dynamic and interactive environments for LLM agents.
- Researchers exploring new paradigms for scaling AI intelligence and performance.
Key takeaways
- Reinforcement learning environments allow LLMs to learn by maximizing cumulative rewards through exploration and exploitation, moving beyond imitation learning.
- Verifiers is an open-source library that simplifies the creation and distribution of RL environments as software artifacts, abstracting away infrastructure complexities.
- Environments can be designed for single-turn, multi-turn, or tool-using agents, with Verifiers providing base classes and abstractions for reward functions and response parsing.
- Reinforcement learning with verifiable rewards, where outcomes like correct answers or successful tool calls provide training signals, is a powerful alternative to traditional supervised fine-tuning.
- Training LLMs in RL environments can significantly improve performance on complex tasks, as demonstrated by transforming a small model into a master tic-tac-toe player.
- Careful consideration of hyperparameters, such as batch size and opponent skill variation, is crucial for stable and effective RL training.
- The Environments Hub serves as a community space for sharing RL environments, aiming to combat fragmentation and ensure open-source models have robust training playgrounds.
- Starting with supervised fine-tuning on synthetically generated data can be an effective warm-up before diving into RL training for complex capabilities.
Notable quotes
*These environments let models learn by interacting, exploring, and improving from feedback.*
*They give the LLM an opportunity to actually interact, take actions, see outcomes.*
*Verifiers lets us focus on the task and the rewards rather than the infrastructure.*
Unofficial community note. Prefer the recording for nuance.