Europe 2026
Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic
Overview
This talk introduces Pydantic AI's agent optimization capabilities, focusing on the Jeppa library for prompt optimization and managed variables for dynamic configuration. The core idea is to systematically improve agent performance by iteratively refining prompts and configurations, demonstrated through a practical example of analyzing political family ties in MP data. The presentation highlights how these tools can lead to more accurate and efficient AI agents.
Who should watch
- AI Engineers looking to improve agent performance and reliability.
- Product Managers seeking to understand how to systematically enhance AI features.
- Builders experimenting with agent frameworks and prompt engineering.
- Developers interested in tools for evaluating and optimizing LLM outputs.
- Anyone facing challenges with prompt consistency and agent behavior.
Key takeaways
- Jeppa is a genetic algorithm-based optimization library that iteratively refines prompts by breeding the best-performing candidates.
- Managed variables allow for dynamic configuration of agent parameters (like prompts, models, and settings) without redeploying the application.
- Pydantic AI integrates with Jeppa and managed variables to provide a framework for optimizing agent behavior and configurations.
- Evals are crucial for measuring agent performance against a golden dataset, enabling data-driven optimization.
- The process involves defining a task, running initial evaluations, using Jeppa to optimize prompts based on eval results, and then applying these optimized prompts via managed variables.
- Optimization is particularly valuable when using smaller, faster, or cheaper models, or when dealing with private data requiring extensive context.
- Managed variables can be updated in production, enabling A/B testing and gradual rollouts of new configurations.
- The ultimate goal is to achieve self-driving optimization where the system autonomously tunes its parameters based on performance metrics.
Notable quotes
*Jeppa is ultimately an optimization library. It optimizes a string. Now, that string can be a simple text prompt or it can be some JSON data.*
*Managed variables... they don't have to be just text. They can be effectively any object that you can define with a Pydantic model.*
*The point is that we can update multiple different values. So, if I instead ask you ask it... we can use this to change variables and ultimately go and like experiment with what new prompts or what new models or temperature or whatever it might be, how they behave in production.*
Unofficial community note. Prefer the recording for nuance.