Ankur Goyal

This talk emphasizes the critical role of robust evaluation (evals) in the development and deployment of AI systems. It argues that evals should be actively engineered, not passively accepted, and that they are essential for both playing offense by identifying new use cases and defense by ensuring quality. The core thesis is that by treating evals as a continuous engineering process, organizations can better adapt to rapid model advancements and ship higher-quality AI products.

This talk discusses the evolution of AI model evaluation (evals), highlighting the shift from manual processes to automated optimization. It introduces Loop, an agent designed to enhance prompts, datasets, and scorers, thereby revolutionizing the eval process. The core thesis is that frontier models, particularly recent advancements, are now capable of significantly improving AI development workflows.