World's Fair 2025
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize
Overview
This talk focuses on building scalable LLM evaluation pipelines. It emphasizes that effective evaluation is crucial for developing high-quality AI products and goes beyond simple "LLM as a judge" approaches. The core thesis is that a comprehensive evaluation strategy involves multiple methods, continuous tuning, and integration into the development lifecycle to accelerate iteration and improve AI system performance.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI applications
- Those responsible for LLM quality and testing
- Teams looking to productionize AI products
- Individuals struggling with LLM evaluation scalability
Key takeaways
- Observability answers what an AI system is doing, while evals measure how well it's performing, acting as a crucial signal for improvement.
- Evaluation methods extend beyond "LLM as a judge" to include user feedback, golden datasets, and heuristic or code-based logic for cost-effectiveness and speed.
- Tuning evaluation pipelines is an ongoing process; the methods used to collect signal should adapt as the AI application evolves.
- Evaluation can be applied at various granularities, from individual LLM calls within a trace to entire agent workflows or multi-turn user sessions.
- Conditional evals are valuable for complex systems, allowing teams to halt evaluation downstream if an earlier component fails, saving resources.
- Customizing evaluation pipelines is essential; out-of-the-box solutions often yield only out-of-the-box results.
- Agent evaluation introduces complexity, requiring analysis of failure modes across aggregate traces and understanding tool usage patterns.
- Trajectory evaluation can assess if an agent follows an expected sequence of components or actions, using LLMs or direct checks against reference paths.
Notable quotes
*Evals are really important in this space because the reality of the fact is if you've ever seen a trace or something like that you're not going to inspect every single trace manually.*
*When we talk about architectures and things like that, um when the industry first started, this was state-of-the-art. Routers, right?*
*My hot take is that don't use out of the box evals. If you get out of the box, if you use out of the box evals, you'll get out of the box results.*
Unofficial community note. Prefer the recording for nuance.