World's Fair 2025
How to Train Your Agent: Building Reliable Agents with RL — Kyle Corbitt, OpenPipe
Overview
This talk details the process of building a reliable AI agent using reinforcement learning (RL), focusing on practical lessons learned from the ART E project, an email assistant. The core thesis is that while RL can significantly improve agent performance beyond prompted models, it's crucial to start with a strong prompted baseline, carefully design the training environment and reward functions, and be vigilant against reward hacking. The project demonstrates how RL can lead to substantial gains in accuracy, cost reduction, and latency improvements for specialized tasks.
Who should watch
- AI engineers looking to improve agent performance.
- Product Managers evaluating the feasibility of training custom agents.
- Builders experimenting with reinforcement learning for AI applications.
- Developers seeking to reduce operational costs and latency in AI systems.
- Those interested in practical applications of RL beyond theoretical concepts.
Key takeaways
- Always start by maximizing performance with prompted models before investing in training, including RL. This helps debug the environment and may eliminate the need for training altogether.
- The ART E project, an email assistant, achieved 96% accuracy using RL, surpassing prompted models like GPT-4o (90%) and significantly reducing error rates.
- RL specialization drastically cut costs, with the trained model costing orders of magnitude less to run than larger prompted models for the same task.
- Latency was also significantly improved by using a smaller, specialized model and training it for fewer interaction turns.
- Building a realistic training environment is critical; the ART E project utilized a public dataset of Enron emails to simulate a real inbox.
- Designing an effective reward function is challenging but achievable by converting the problem into a verifiable one, using LLMs as judges against a golden dataset.
- Reward functions can incorporate secondary objectives, such as minimizing interaction turns and penalizing hallucinations, leading to more efficient and reliable agents.
- Vigilance against reward hacking is essential; agents can exploit loopholes in the reward system, necessitating careful monitoring of rollouts and adjustments to the reward function.
Notable quotes
*I would generally always recommend starting with getting the best performance you can with a prompted model before going to any training including reinforcement learning.*
*The last thing which we didn't apply here but can help a lot with these smaller things which is called speculative decoding.*
*It's really important to watch out for this solving it typically involves modifying in some way your reward function to penalize things like that.*
Unofficial community note. Prefer the recording for nuance.