Europe 2026
Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests? — Vincent Koc, OpenClaw
Overview
This talk argues that traditional static testing methods are insufficient for evaluating adaptive AI systems. As AI applications become more dynamic and intent-driven, evaluation strategies must evolve to become equally malleable. The core thesis is that static benchmarks fail to capture the emergent behaviors and changing user interactions characteristic of modern AI, necessitating a shift towards more adaptive and continuous evaluation approaches.
Who should watch
- AI Engineers
- Product Managers
- Builders of agentic AI systems
- Those struggling with the limitations of current AI evaluation methods
- Anyone interested in the future of AI testing and quality assurance
Key takeaways
- Traditional software engineering practices like chaos engineering and observability are missing in current AI evaluation.
- Static benchmarks and handcrafted test sets are inadequate for dynamic AI applications, which are increasingly becoming malleable themselves.
- The evolution from prompt engineering to context engineering and now to intent engineering highlights the increasing complexity and adaptability of AI systems.
- Intent engineering focuses on machines self-optimizing based on user intent, requiring evaluations that can adapt to personalized experiences.
- Adaptive evaluation methods could include building rubrics, self-curating test suites from agent traces, and implementing always-on online optimizations.
- Telemetry in the loop, where harnesses are aware of system telemetry and can self-correct, offers a path toward self-healing AI agents.
- The concept of eval calcification suggests that evaluations will become harder to manage unless smarter, adaptive approaches are adopted.
- Evaluations should be treated less like static datasets and more like living, self-optimizing code or agents that grow and adapt over time.
Notable quotes
*Our AI applications are not static, but we're treating them like they're static software.*
*Why are stat benchmarks static? Like why don't we test in a more sort of adaptive manner?*
*People need to start looking at the evals not as this like static data set thing, but actually as like code as like software or as like a a living agent.*
Unofficial community note. Prefer the recording for nuance.