World's Fair 2025
RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot
Overview
This talk addresses the shortcomings of current Retrieval Augmented Generation (RAG) evaluation methods, arguing that they are fundamentally broken. The presenters contend that most benchmarks rely on simple, local questions with easily identifiable answers within specific text chunks, which does not reflect real-world data complexity. This leads to a cycle of optimizing for flawed benchmarks, resulting in RAG systems that perform poorly when deployed with actual user data.
Who should watch
- AI engineers and builders working with RAG systems.
- Product Managers evaluating the performance of RAG-based features.
- Anyone experiencing a disconnect between RAG benchmark scores and real-world application performance.
- Developers seeking to improve the accuracy and reliability of their RAG pipelines.
Key takeaways
- Current RAG benchmarks often use artificial questions with answers localized to specific text chunks, failing to represent the messiness of real-world data.
- A common RAG pipeline, involving chunking, embedding, retrieval, and re-ranking, is insufficient for many complex or aggregative questions.
- Existing benchmarks do not adequately test the entire RAG system, often focusing solely on retrieval or generation in isolation.
- Optimizing RAG systems for these flawed benchmarks creates a vicious cycle, leading to high scores on tests but poor performance in production.
- For certain data types, like financial reports or historical event data, transforming unstructured text into structured formats (e.g., SQL databases) and querying them can yield better results than standard RAG.
- This structured approach involves clustering documents, identifying schemas, populating them, and then using text-to-SQL for querying.
- Challenges remain in schema normalization, handling ambiguity, and managing the trade-offs between ingestion complexity and inference speed.
- RAG is not a one-size-fits-all solution; specific use cases may require moving beyond standard RAG pipelines.
Notable quotes
*Most benchmarks comprise from local questions that has local answers.*
*Existing benchmarks fail to capture these use cases; they are very limited.*
*We should note that the regular pipeline of chunking embedding retrieving reranking is not good enough for many questions.*
Unofficial community note. Prefer the recording for nuance.