World's Fair 2026
How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI
Hanna Lichtenberg , Mixedbread AI
Overview
This talk addresses the significant gap between the rapidly advancing reasoning capabilities of large language models (LLMs) and the slower evolution of retrieval systems. The core thesis is that LLMs are bottlenecked not by their reasoning power, but by their ability to access the correct knowledge. By improving retrieval tools and teaching agents to use them effectively, most of this knowledge gap can be closed, unlocking LLMs for complex tasks beyond coding.
Who should watch
- AI engineers and builders working with LLMs and retrieval systems.
- Product Managers seeking to improve LLM-powered applications.
- Researchers interested in agentic search and knowledge access.
- Anyone facing performance drops in LLM applications due to poor information retrieval.
Key takeaways
- LLM reasoning capabilities have grown exponentially, while traditional search has improved slowly, creating a knowledge gap.
- Benchmarks like BrowseComp Plus and Office QA Pro show a sharp performance drop when LLMs use default retrieval tools compared to oracle performance.
- The queries generated by current agents are often gibberish or keyword-heavy, stemming from training on coding tasks and mimicking human web search patterns.
- Mixedbread AI developed a search agent with a harness that encourages better semantic queries by framing evidence needs before query writing and offering distinct search tools (overview, semantic, filter, grep).
- The agent's harness uses a maximum of four search rounds with parallel searches, deduplicates chunks, and aims for precision, speed, and cost-efficiency.
- Training involves supervised fine-tuning with a teacher LLM and on-policy reinforcement learning with a combined retrieval and trajectory reward.
- The retrieval reward considers metrics like NDCG and LLM judgments on relevance and ranking plausibility.
- The trajectory reward focuses on improving tool queries for efficiency and quality, assessing query naturalness and exploration sufficiency.
Notable quotes
*The bottleneck here is not the reasoning. It's actually the access to the right knowledge it needs to answer this question.*
*Giving the model better search tool we can recover most of the knowledge gap.*
*The agent has to articulate what evidence it needs before writing the query.*
Unofficial community note. Prefer the recording for nuance.