Europe 2026
Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel
Overview
This talk explores the critical role of task quality in the performance of AI models, particularly in agentic tasks. The core thesis is that data quality and task quality are fundamentally intertwined, and improving task quality directly leads to better model training outcomes and performance uplifts. The research validates this by comparing model performance on high-quality versus low-quality tasks, demonstrating a significant difference in learning efficiency.
Who should watch
- AI Engineers
- Product Managers
- Builders working with agentic systems
- Those focused on improving model training and evaluation
- Individuals concerned with data quality in AI development
Key takeaways
- Task quality is a critical factor influencing model training outcomes, directly correlating with data quality.
- A rigorous definition of task quality includes achievability, non-triviality, functional correctness, and environment reliability.
- Accepted tasks, meeting specific quality criteria, demonstrated twice as many tool calls and required more output tokens compared to rejected tasks.
- Models trained on high-quality tasks showed a 6% performance improvement, a five-fold increase compared to the 1% improvement seen with low-quality tasks.
- Failure modes in accepted tasks are generally cleaner and more indicative of task difficulty, providing better signals for model improvement.
- Task underspecification and implicit dependencies in testing can lead to tasks appearing harder or causing mismatches between requested setups and expected outcomes.
- The Snorkel platform emphasizes human expertise and expert-in-the-loop processes for generating high-quality datasets at scale.
- Future work involves developing benchmarks for less verifiable and more human-centric tasks, exploring multiple possible outcomes and scoring them on a spectrum.
Notable quotes
*The core thesis has been that the quality of data is critical.*
*Task quality and data quality are largely the same thing.*
Unofficial community note. Prefer the recording for nuance.