World's Fair 2025
open-rag-eval: RAG Evaluation without \"golden\" answers — Ofer Mendelevitch, Vectara
Overview
This talk introduces open-rag-eval, an open-source project designed for scalable Retrieval-Augmented Generation (RAG) evaluation. It addresses the common challenge of RAG evaluation requiring "golden answers" or "golden chunks," which is often impractical and non-scalable. The project, developed in collaboration with the University of Waterloo, offers a research-backed approach to evaluate RAG pipelines without relying on pre-defined correct answers.
Who should watch
- AI engineers building or optimizing RAG systems.
- Product managers evaluating the quality of RAG-based features.
- Developers seeking to improve RAG pipeline performance and reliability.
- Researchers interested in novel RAG evaluation methodologies.
- Anyone facing challenges with traditional RAG evaluation due to the need for golden datasets.
Key takeaways
- open-rag-eval provides a framework for evaluating RAG pipelines without requiring golden answers or chunks, making evaluation more scalable.
- The project includes connectors for popular RAG frameworks like LangChain and LlamaIndex, with support for others.
- Key metrics include Umbrella for retrieval quality, Autonogizer for generation quality, citation faithfulness, and hallucination detection.
- The Umbrella metric, developed with the University of Waterloo, correlates well with human judgment for retrieval quality.
- Autonogizer evaluates generation by creating "nuggets" and using an LLM judge to assess support from the retrieved content.
- Citation faithfulness checks if citations within a response are accurately supported by the retrieved passages.
- A user interface at open-evaluation.ai allows for easy visualization and comparison of evaluation results by uploading output files.
- The project is open-source, encouraging community contributions for new connectors and improvements.
Notable quotes
*It's a open source project that is aiming to solve one of the major problems of rag evaluation. the fact that you require golden answers or golden chunks to do anything which is really nonscalable.*
*The research done by the University of Wateroo lab of Jim Lynn shows that if you use this approach it correlates well with human judgment and that is really really powerful.*
Unofficial community note. Prefer the recording for nuance.