Europe 2026
Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize
Overview
This talk focuses on the practical aspects of evaluating and improving AI agents, moving beyond simple "vibe checks" to implement robust testing frameworks. It emphasizes that effective evaluation is crucial for shipping reliable AI applications, especially agents, which are prone to cascading failures. The session introduces methods for capturing agent behavior through tracing, categorizing failures, and implementing various types of evaluations (code-based, LLM-as-judge, human) to ensure agents perform as expected and to drive iterative improvements.
Who should watch
- AI engineers building or deploying agents.
- Product managers seeking to understand agent performance and reliability.
- Developers looking to implement systematic testing for AI applications.
- Anyone struggling with agent failures or regressions after prompt changes.
Key takeaways
- **Evals are essential for agents:** Unlike simple LLM calls, agents have complex, multi-step decision processes that require rigorous evaluation to catch cascading failures and regressions.
- **Tracing is the foundation:** Capturing detailed logs of agent actions (tool calls, LLM invocations, inputs, and outputs) via tracing is the first step to enabling effective evaluation.
- **Multiple eval types are needed:** A comprehensive evaluation strategy combines deterministic code-based evals for simple checks, LLM-as-judge evals for semantic understanding, and human evaluation for nuanced quality assessment.
- **Custom LLM-as-judge rubrics require care:** Crafting effective prompts for LLM judges involves defining clear criteria, providing labeled examples, and constraining output to ensure reliable judgments.
- **Iterative improvement is key:** Use evaluation results to identify failure patterns, refine prompts, and systematically improve agent performance through experiments and data-driven adjustments.
- **Data quality and prompt engineering are high-impact:** Prioritize fixing data sources and refining prompts over model selection or hyperparameter tuning for the most significant improvements.
- **Eval-driven development is powerful:** Writing evaluations before or alongside feature development provides clear goals and ensures that new capabilities are testable from the outset.
Notable quotes
*Pretty legit is not enough to ship something to production. That is the whole point of evals is I is that the vibe is good, but we want to be doing better than vibes.*
*An eval that you haven't validated is just a fancy way of being wrong at scale.*
*The first time a regression shows up before it reaches your users instead of after, you will have justified the cost of building your evals.*
Up next · Evals first
Watch next
The category error — why unit-test instincts fail for stochastic systems.
Evals Are Not Unit Tests — Ido Pesok, Vercel v0Unofficial community note. Prefer the recording for nuance.