Europe 2026
Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence
Overview
This talk introduces the concept of spec-driven testing for AI agents, arguing that simply using larger models or extensive datasets is insufficient for ensuring agent safety and reliability. It proposes a more comprehensive approach to defining and testing agent behavior by specifying not just expected outputs but also the context, rules, domain knowledge, and robustness requirements relevant to the agent's intended task.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI agents and systems
- Those concerned with agent safety and reliability
- Individuals involved in testing and validation of AI models
Key takeaways
- Larger models do not automatically equate to safer or better agents; they can sometimes be more susceptible to jailbreaks or have a broader attack surface.
- Defining agent behavior requires more than just a dataset of inputs and outputs; it should include explicit rules, domain knowledge, ontologies, and considerations for user rights and roles.
- Robustness testing is crucial, involving evaluating agent performance under stress, such as with typos, rephrased inputs, or varying environmental conditions.
- Spec-driven validation aims to create task and role-specific benchmarks that go beyond standard evaluation datasets.
- Explicitly defining agent specifications helps identify vulnerabilities and areas where the agent has more power to act within infrastructure.
- The goal is to find a balance where an agent is capable enough to perform its task but not capable of causing arbitrary harm.
- Specifications should be implementation-independent, allowing for flexibility in underlying agent frameworks and infrastructure.
- This approach can inform a continuous loop of testing, iteration, and improvement for agent development.
Notable quotes
*A smarter agent is a better agent, right? So if I have a smarter agent, I'm using a bigger model, uh it's going to be better at doing the job that it's supposed to do.*
*It's not obvious that bigger is safer and it's not obvious that bigger is better.*
*So what you're really seeking is like a model uh an agent that's built on a model that's good enough to perform but it's not capable of doing arbitrary harm.*
Unofficial community note. Prefer the recording for nuance.