Session brief
Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley
Overview
This talk explores the potential of reinforcement learning (RL) to advance AI agents beyond current chatbot and reasoner capabilities. It posits that RL offers a path to developing more autonomous systems that can learn and improve through interaction with their environment, moving beyond static prompt engineering and tool-calling approaches. The discussion highlights emerging trends and open-source efforts in this domain, suggesting a future where RL is integral to agent engineering.
Who should watch
- AI engineers and builders exploring advanced agent capabilities.
- Product Managers and researchers interested in the future of autonomous systems.
- Developers looking to understand how reinforcement learning can enhance agent performance.
- Those curious about the evolution from current LLM applications to more capable agents.
Key takeaways
- Current LLM applications often function as chatbots or reasoners, with agents typically built by chaining these models and employing techniques like prompt engineering and tool calling.
- True autonomous agents, capable of complex, long-duration tasks, are still an emerging area, with few examples like Devon, Operator, and OpenAI's deep research.
- Reinforcement learning, traditionally defined as an agent learning through environmental interaction to achieve goals, is presented as a key mechanism for unlocking new agent capabilities.
- Recent advancements, such as DeepSeek's R1 model, demonstrate that RL can enable models to learn complex reasoning and long chains of thought without explicit programming.
- The concept of "rubric engineering" emerges as a parallel to prompt engineering, focusing on designing reward signals that guide RL agents toward desired behaviors and skills.
- Open-source efforts are crucial for replicating and democratizing RL techniques for agent development, with simple, accessible codebases fostering community engagement.
- While RL offers a path to enhanced autonomy and skill acquisition, challenges remain in areas like reward design, generalization, and preventing reward hacking.
- The future of AI engineering may involve integrating RL, potentially through fine-tuning or specialized API services, to build more robust and autonomous language model-powered systems.
Notable quotes
*This talk is not about things we ship to prod. It's about where we might be headed.*
*Reinforcement learning at the core is really about identifying good strategies for solving problems.*
*The idea of rubric engineering here is that similar to prompt engineering, to have a model do reinforcement learning, it's going to get some reward.*
Unofficial community note. Prefer the recording for nuance.