Europe 2026
Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI
Overview
This talk introduces GEPA, a method for building calibrated LLM evaluators that align with human annotations. The core idea is to move beyond generic LLM judges, which often fail in production, by optimizing prompts using algorithms like GEPA. This approach aims to accelerate development cycles by providing reliable signals for both offline evaluations and online monitoring, ultimately contributing to the creation of a data flywheel for continuous AI improvement.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI agents and applications
- Those facing challenges with LLM evaluation and reliability
- Individuals interested in prompt optimization techniques
Key takeaways
- Generic LLM judges often fail because they lack calibration with human judgment and specific business goals.
- Calibrated LLM judges are crucial for accurate offline evaluations and effective online monitoring of AI agent performance.
- The GEPA algorithm, similar to genetic algorithms, iteratively samples, evaluates, and filters prompts to optimize LLM judges.
- Key steps in building calibrated judges include designing specific metrics based on use cases, curating and annotating data, optimizing the judge using GEPA, and validating results.
- GEPA employs strategies like prompt mutation and merging, and uses a Pareto frontier to maintain diversity and coverage across test cases.
- The quality and specificity of human annotations, including reasoning behind judgments, are paramount for the optimization algorithm to learn effectively.
- Iterating on the seed prompt and reflection templates, and carefully considering model choices and cost, are important for successful optimization.
Notable quotes
*The bottleneck in this loop is actually the evaluation. How fast can you evaluate?*
*Having calibrated LLM as a judge with a similar quality let's say as human annotator will make your development much faster.*
*The metrics need to come from the use case itself. It does not make sense to have general metrics like hallucination when you're evaluating your AI agent.*
Unofficial community note. Prefer the recording for nuance.