Session brief
Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil
Overview
This talk addresses the current limitations and challenges in building and evaluating AI agents. While agents are increasingly integrated into products, ambitious visions of their capabilities are far from realized. The core thesis is that AI engineering must prioritize reliability and rigorous evaluation, treating these as first-class concerns to overcome the inherent stochasticity of language models and move beyond misleading benchmarks.
Who should watch
- AI Engineers seeking to understand the practical challenges of agent development.
- Product Managers evaluating the feasibility and reliability of AI-powered features.
- Builders working on AI agents and systems that require robust performance.
- Anyone interested in the gap between AI agent capabilities and real-world reliability.
Key takeaways
- Evaluating AI agents is complex; many products have made unsubstantiated claims, leading to failures and regulatory fines.
- Static benchmarks can be misleading for agents because they require interaction with environments and have open-ended action spaces, unlike static language models.
- Cost must be a first-class metric in agent evaluation alongside accuracy, as the Jevons paradox suggests increased usage with lower costs.
- Over-reliance on benchmarks can be detrimental, as performance on benchmarks does not always translate to real-world success, with examples showing low success rates on actual tasks.
- A critical distinction exists between capability (what a model *could* do) and reliability (consistently getting the answer right), with reliability being paramount for consequential decisions.
- Verifiers, like unit tests, can also be imperfect, leading to false positives that skew performance metrics and hinder true reliability improvements.
- AI engineering should be viewed more as reliability engineering, focusing on system design to work around the stochastic nature of LLMs, rather than solely a modeling problem.
- The historical precedent of early computing reliability issues highlights the need for AI engineers to focus on fixing these inherent problems for user usability.
Notable quotes
*Evaluating agents is genuinely a very hard problem; it needs to be treated as a first-class citizen in the AI engineering toolkit.*
*The challenge for AI Engineers is to figure out what sorts of software optimizations and abstractions are needed for working with inherently stochastic components like LLMs.*
*AI Engineers need a reliability shift in your mindset to think of yourselves as the people who are ensuring that this next wave of computing is as reliable for end users as possible.*
Unofficial community note. Prefer the recording for nuance.