Europe 2026
Why building eval platforms is hard — Phil Hetzel, Braintrust
Overview
Building effective evaluation (eval) platforms for AI agents is a complex systems problem, not just a UI challenge. While starting with simple tools like spreadsheets is a valid first step, maturing requires moving towards more robust solutions that facilitate experimentation and integrate with production data. The core difficulty lies in managing the unique characteristics of AI agent traces, which are often large, unstructured, and high-velocity, demanding specialized data infrastructure.
Who should watch
- AI Engineers
- Product Managers
- Builders developing AI agents
- Those struggling to move AI proofs-of-concept to production
- Teams needing to ensure AI agent reliability and quality
Key takeaways
- LLMs exhibit extreme variability, making rigorous evaluation crucial for agent performance and risk mitigation.
- Initial eval efforts often start with spreadsheets and simple loops, which are good for documentation but limited for experimentation and analytics.
- As maturity grows, teams may "vibe code" custom UIs and databases, improving persistence and accessibility but still often focusing on reporting over iteration.
- True experimentation involves providing sandboxed environments where users can tweak agent configurations and compare outputs, ideally informed by production trace data.
- Integrating observability with evals creates a feedback loop, where production data informs offline testing and agent improvements.
- Agent traces are uniquely challenging due to their large size, semi-structured or unstructured nature, and high velocity, posing significant data infrastructure problems.
- Developing a robust data layer for evals and observability is more critical than the UI, enabling functional requirements like full-text search across millions of traces.
- Future eval platforms should support both human and agent users, potentially incorporating AI agents for self-healing and automated improvements.
Notable quotes
*LLMs have extreme variability. Agents are becoming the norm in how customers are interacting with companies.*
*Evals are important because LLMs have extreme variability.*
*The best way to perform evals is to really think about the failure modes that your agent can fall into and build scoring functions around those failure modes.*
Unofficial community note. Prefer the recording for nuance.