World's Fair 2026
Your Agents Need a Save Button - Hamza Tahir, ZenML
Overview
This talk introduces the concept of a "save button" for AI agents, analogous to document saving, to enable state persistence and replayability. Current agent execution lacks this, with only disconnected traces available. Implementing a save button allows for asking "what if" questions, such as swapping models or mocking tools, to optimize agents for cost, speed, and performance.
Who should watch
- AI engineers building and debugging agents.
- Product managers evaluating agent performance and cost.
- Developers seeking to improve agent reliability and efficiency.
- Anyone frustrated by the inability to understand or alter past agent decisions.
Key takeaways
- Agent execution currently lacks persistent state, with only read-only traces available, disconnected from the runtime.
- A "save button" or checkpointing mechanism allows agents to store their state, enabling replayability and "what if" scenario analysis.
- Replaying agent executions can help answer questions about swapping models, mocking tools, or degrading performance to understand trade-offs.
- Tools like Kitaru, built on ZenML, provide a runtime layer that checkpoints state and connects to traces, facilitating code execution replay.
- Comparing different execution paths side-by-side allows for detailed analysis of changes in tool calls, artifacts, and final outcomes.
- Analyzing cohorts of runs, rather than single instances, is crucial for understanding the broader impact of changes and avoiding false economies.
- Production data can be leveraged for evaluation by building cohorts of relevant runs (e.g., expensive, long, or risky) and replaying them with modifications.
- Automating the loop of checkpoint, replay, diff, and decide, potentially with agents themselves, can significantly improve agent development and deployment.
Notable quotes
*The only thing we have which is closest is a trace. A trace gives you the emitted telemetry data of how an agent calls tools in the input and output of that state.*
*What if it had done a different thing? Would it have been cheaper? Would it have been faster? Well, you can do all these things if your agents have a save button.*
*One replay is just an anecdote and having a cohort analysis is way way way better.*
Unofficial community note. Prefer the recording for nuance.