World's Fair 2025
The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes
Overview
This talk details the iterative process of building a Retrieval Augmented Generation (RAG) system, highlighting lessons learned from 37 failed attempts. It emphasizes practical choices for different components of a RAG stack, distinguishing between prototyping and production environments. The core thesis is that careful selection and integration of components like orchestration, embedding models, vector databases, and language models are crucial for effective RAG implementation.
Who should watch
- AI engineers and builders experimenting with RAG.
- Product Managers evaluating RAG solutions.
- Developers seeking to improve RAG system accuracy and scalability.
- Those facing challenges in deploying production-ready generative AI.
Key takeaways
- Prototyping RAG systems can be effectively done in Google Colab using tools like Llama Index or LangGraph, leveraging free hardware accelerators.
- For production environments, especially with financial institutions, Docker is recommended for on-premise or cloud deployment.
- Quadrant is a scalable vector database choice, handling varying document volumes efficiently.
- Open-source embedding models from providers like BAAI or Nvidia can be used, offering an alternative to closed API-based models.
- Monitoring and tracing RAG solutions with tools like Langfuse or Arize Phoenix are vital for troubleshooting and performance analysis.
- Reranking, using cross-encoders, significantly improves accuracy by semantically comparing queries with retrieved documents.
- Evaluating RAG solution quality is essential, with frameworks like Ragas providing a robust method for testing.
- A production RAG stack can be containerized using Docker Compose, integrating components for data ingestion, vector storage, model serving, tracing, and evaluation.
Notable quotes
*Quadrant is an excellent solution because it scales really well.*
*Monitoring and tracing your rag solution is really important.*
Unofficial community note. Prefer the recording for nuance.