World's Fair 2026
Your Agent Is Wasting Tokens and You Don't Know It - Erik Hanchett, AWS
Overview
This talk addresses the significant token costs associated with using AI agents, particularly large language models, and provides practical strategies for reducing these expenses. The core thesis is that by implementing specific techniques, developers can optimize agent performance and cost-efficiency without sacrificing functionality.
Who should watch
- AI Engineers
- Product Managers
- Builders working with AI agents
- Those concerned with AI operational costs
- Developers looking to optimize LLM usage
Key takeaways
- Cache system prompts, tool prompts, and messages to reduce redundant data sent to the LLM on subsequent calls.
- Route messages based on task difficulty, using cheaper models for simpler tasks and more advanced models for complex ones.
- Offload large tool results by storing them locally or in the cloud and summarizing them to avoid repeatedly sending large outputs into the agent's context.
- Cap tool loops by setting maximum iterations to prevent infinite loops and excessive token consumption.
- Utilize observability tools to monitor tool call duration and loop counts, identifying inefficiencies.
- Trim conversation history in multi-turn agents by using methods like a sliding window to limit the amount of past dialogue sent with each new message.
- Summarize older conversation history and inject it into the context window when necessary, rather than sending the entire history.
Notable quotes
*You want to use multiple different models based on the use case.*
*If you can find any way that where you have this tool result that you don't necessarily send it on every single call back to the large language model, that will save a lot of tokens for you.*
*So always set a max iterations of how many times it will loop.*
Unofficial community note. Prefer the recording for nuance.