Ara Khan

This talk critiques the current state of AI model evaluations, arguing that while flawed, they remain essential tools for development. The speaker proposes a three-stage approach to using evals effectively: leveraging external evals, using them to improve internal agents, and building custom evals for specific use cases. The core message is to use evals pragmatically, understanding their limitations while still employing them for iterative improvement.

This talk proposes a structured approach to building AI agents, moving beyond the hype to focus on practical maturity. It outlines four distinct levels of agent development, from initial framework experimentation to sophisticated cloud deployment. The core thesis is to slow down, identify solvable problems, and build useful agents by understanding their lifecycle and potential pitfalls.