World's Fair 2026
We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco
Overview
This talk addresses the significant cost associated with AI coding tools, which is often driven by excessive context sent to models rather than the model's processing. The core thesis is that by optimizing the input context, substantial cost savings can be achieved, far exceeding savings from output compression. A local code indexing and search layer is proposed as a solution to send only relevant code snippets to AI models.
Who should watch
- AI Engineers
- Product Managers
- Builders concerned with AI tool costs
- Developers using AI coding assistants
- Anyone looking to reduce AI token usage
Key takeaways
- The majority of AI coding tool costs stem from the volume of input tokens (context) sent to the model, not the output generated.
- Attempts to reduce costs by shortening prompts or adjusting model settings like max tokens are ineffective because the cost is incurred before these changes take effect.
- Output compression can save costs, but the impact is limited because output is a smaller portion of the total token usage.
- A local search layer that indexes code and retrieves only necessary pieces significantly reduces input tokens, leading to substantial savings.
- Combining meaning-based and exact-word search methods improves the accuracy of retrieved code snippets, mitigating weaknesses of individual search types.
- A simple scoring formula for relevance, combining meaning, keyword match, and recency, is more effective than complex AI-driven judgment for filtering search results.
- Implementing a shared index across different AI coding tools can provide consistent context and memory, preventing redundant explanations of the codebase.
- Real-world testing showed a 94% reduction in tokens per query by using the local index, translating to significant cost savings.
Notable quotes
*Most of the money was not the AI thinking. Most of it was sending too much context.*
*Fix the input, the model choice matters less than you think.*
Unofficial community note. Prefer the recording for nuance.