World's Fair 2025
How to run Evals at Scale: Thinking beyond Accuracy or Similarity — Muktesh Mishra, Adobe
Overview
This talk emphasizes the critical role of evaluations (evals) in developing robust AI applications, moving beyond simple accuracy or similarity metrics. It highlights that effective evals are fundamental for aligning applications with system goals, ensuring continuous improvement, and building user trust. The core thesis is that a strategic, data-driven approach to evals, tailored to specific use cases, is essential for scaling AI development.
Who should watch
- AI engineers and developers building AI applications (e.g., RAG chatbots, agents, code generation).
- Product Managers and technical leads responsible for AI product quality and performance.
- Anyone facing challenges in testing non-deterministic AI outputs or measuring application performance.
- Builders looking to establish reliable testing and validation processes for AI systems.
Key takeaways
- Evals are fundamental to AI application development, serving as the primary method for testing and measuring performance.
- Data is the cornerstone of effective evals; start with synthetic data and continuously refine labeled datasets that cover multiple application flows and aspects.
- Evaluation strategies must be adaptive, considering the specific application type (e.g., RAG, code generation, agents) and employing appropriate metrics like functional correctness for code or trajectory evaluation for agents.
- Scaling evals involves caching intermediate results, focusing on orchestration and parallelism, aggregating results, and running evaluations frequently.
- A "measure, monitor, analyze, and repeat" loop is crucial for continuous improvement of AI applications.
- There is a necessary trade-off between speed and fidelity in evals, often requiring a balance between automated testing and human-in-the-loop validation.
- The concept of *eval development*, akin to test-driven development, encourages defining evals based on use cases from the outset.
- Focus on both positive and negative test cases, and prioritize a well-defined process over relying solely on tools, as not everything can be automated.
Notable quotes
*Evals are the most important aspect for AI application.*
*You need to define and establish the process. How do you want to run the EVs?*
Unofficial community note. Prefer the recording for nuance.