World's Fair 2026
The Agentic AI Engineer - Benedikt Sanftl, Mutagent
Overview
The Agentic AI Engineer talk introduces a framework for building and iterating on AI agents by applying agentic principles to the development lifecycle itself. This approach aims to automate and optimize the process of agent creation, testing, and deployment, moving beyond manual iteration to a more scalable, agent-driven workflow. The core idea is to create an "agent of agents" that manages the development loop, from initial specification to production monitoring and refinement.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI agents and agentic systems
- Those looking to scale AI agent development
- Teams experiencing bottlenecks in agent iteration and debugging
Key takeaways
- The development lifecycle for AI agents can be broken down into an offline loop (spec, build, evaluate, ship) and an online loop (monitor, diagnose, optimize).
- Manually managing this loop becomes a bottleneck as the number of agents or features increases, necessitating an agentic approach to development.
- Spec-driven development is crucial, defining agent responsibilities, context requirements, integrations, and constraints as a blueprint.
- Eval-driven development, analogous to test-driven development, uses metrics and datasets to verify agent performance, with the eval suite evolving over time through discovery.
- Automated evaluations are essential to avoid human bottlenecks when processing large datasets or numerous agent traces.
- The online loop involves monitoring deployed agents, diagnosing failures through root cause analysis, and feeding insights back into an optimization stage.
- Learned failure modes and code-checkable indicators can streamline the diagnosis of issues in production traces, making the process more efficient.
- The ultimate goal is an autonomous optimization loop where agents are continuously improved and redeployed based on performance data and evaluation results.
Notable quotes
*The bottleneck basically becomes the human review and the human building time. And that you can't scale.*
*The key thing here is basically once you reach a certain number of agents or AI-based features, the human performing this loop again cannot really scale in enough time.*
*The real and the complete eval suite is a product of discovery.*
Unofficial community note. Prefer the recording for nuance.