World's Fair 2026
Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind
Overview
This talk emphasizes the critical need for rigorous evaluation of AI skills before deployment, arguing that shipping skills without proper testing can lead to unpredictable behavior and degraded performance. The speaker highlights the distinction between agents developers use for personal productivity and agents built for consumers, noting that end-users lack the context to troubleshoot skill invocation issues. The core thesis is that comprehensive evaluations are essential for ensuring skill reliability, managing costs, and determining when skills can be retired as models improve.
Who should watch
- AI Engineers building agentic systems
- Product Managers defining agent capabilities
- Developers responsible for shipping AI features
- Anyone working with AI skills and tool-calling functionalities
- Teams struggling with unpredictable agent behavior
Key takeaways
- Human-written skills generally outperform AI-generated ones, which can negatively impact performance.
- Skills should be kept concise, ideally under 500 lines, with detailed information layered in reference files.
- Capability skills, which teach models new functions, are temporary and should be retired when models improve, guided by evals. Preference skills, encoding specific workflows or styles, are more durable and require robust evals for protection.
- Descriptions for model-invoked skills are crucial for guiding the model; they should clearly state the "why" and "how" of skill usage, using directives rather than lengthy explanations.
- Evals should include both positive (when to use) and negative (when not to use) test cases to prevent over-triggering and ensure specificity.
- Testing should be conducted across different agent harnesses and models, as behavior can vary significantly.
- Skills can be retired when models achieve comparable or better performance without them, saving costs and reducing maintenance.
- Simple evaluation harnesses can be built using JSON for test cases and Python scripts for running agents, with regex or LLM-as-a-judge for assertions.
Notable quotes
*Human-written skills are the best we can provide. AI-generated skills can impact performance negatively.*
*Don't ship skills without evals.*
Up next · Evals first
Watch next
Hands-on: what evals look like when the unit of work is an agent, not a completion.
Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, ArizeUnofficial community note. Prefer the recording for nuance.