World's Fair 2025
OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs
Overview
This talk introduces OpenThoughts 3, a project focused on creating high-quality, open-source reasoning datasets. The core thesis is that while models have shown significant performance gains on reasoning benchmarks, the underlying data recipes for achieving this are often undisclosed. OpenThoughts aims to fill this gap by providing a systematic approach and publicly available datasets to train more capable reasoning models, emphasizing that supervised fine-tuning (SFT) on well-crafted data can be highly effective.
Who should watch
- AI engineers and researchers looking to train or improve reasoning capabilities in language models.
- Builders and product managers seeking to leverage advanced reasoning for specific applications.
- Developers interested in understanding the data creation process for state-of-the-art reasoning models.
- Those exploring alternatives to reinforcement learning (RL) for enhancing model reasoning.
Key takeaways
- Supervised fine-tuning (SFT) on reasoning data is a highly effective method for improving model performance, comparable to or surpassing RL in some contexts.
- Scaling data size significantly boosts accuracy, and SFT makes this scaling more manageable than with RL.
- Sampling multiple reasoning traces per question dramatically increases data efficiency and potential accuracy gains.
- A better-performing model does not automatically make a better teacher model for distillation; factors like response formatting and trace length are crucial.
- Synthetic question generation can be a powerful and scalable source for high-quality reasoning data, often outperforming scraped or manually written data.
- Filtering questions based on perceived difficulty or expected answer length, as judged by a language model, is more effective than embedding-based or fast text classifier approaches for post-training data.
- Focusing on a smaller number of high-quality data sources is more beneficial than aiming for broad diversity across many sources.
- For SFT and distillation, filtering based on answer verification did not significantly improve results, suggesting that the reasoning process itself, even with an imperfect final answer, can be valuable.
Notable quotes
*The missing link is the data recipe.*
*Scaling is difficult, particularly difficult with RL. The good news is for SFT scaling is quite easier.*
*A better model in terms of its own performance on evaluation benchmarks does not necessarily mean it's a better teacher model.*
Unofficial community note. Prefer the recording for nuance.