Yuval Belfer

This talk addresses the shortcomings of current Retrieval Augmented Generation (RAG) evaluation methods, arguing that they are fundamentally broken. The presenters contend that most benchmarks rely on simple, local questions with easily identifiable answers within specific text chunks, which does not reflect real-world data complexity. This leads to a cycle of optimizing for flawed benchmarks, resulting in RAG systems that perform poorly when deployed with actual user data.

This talk argues that current large language models (LLMs), despite advancements, still struggle with complex instruction following. The core thesis is that AI agents require robust planning capabilities beyond simple prompting to effectively tackle intricate tasks, emphasizing the need for a lookahead strategy rather than step-by-step execution.