Session brief
RAG Agents in Prod: 10 Lessons We Learned — Douwe Kiela, creator of RAG
Overview
This talk addresses the challenges and lessons learned from deploying Retrieval-Augmented Generation (RAG) agents in production environments, particularly within enterprises. The core thesis is that while large language models are powerful, they are often only a small part of a larger system. True value and business transformation come from effectively managing context, specializing AI for domain expertise, and building robust systems that can handle enterprise-scale data and user needs, rather than focusing solely on model capabilities.
Who should watch
- AI Engineers
- Product Managers
- Builders working on enterprise AI solutions
- Those facing challenges in achieving ROI from AI investments
- Teams struggling with scaling RAG systems beyond pilot phases
Key takeaways
- RAG systems are typically 20% model and 80% system; focus on building robust pipelines, not just the language model.
- Enterprise expertise is a key asset; specialization for domain-specific problems yields better results than aiming for general AI (AGI).
- A company's data is its long-term asset and competitive moat; focus on enabling AI to work with noisy, at-scale data.
- Pilots are easy, but production is hard; design for production from day one, considering scale, security, and compliance.
- Speed and iteration are crucial; release functional, not perfect, systems early to gather real user feedback and improve incrementally.
- Empower engineers by abstracting away mundane tasks like chunking strategies, allowing them to focus on delivering business value.
- Make AI solutions easy to consume by integrating them directly into existing enterprise workflows.
- Aim for ambitious goals that drive significant ROI, rather than settling for basic functionalities or gimmicks.
Notable quotes
*Language models are awesome but often they're only 20% of a much bigger system.*
*The data that a company owns, that is the company in the long term.*
*Pilots are very easy, building a moat not very difficult these days.*
Unofficial community note. Prefer the recording for nuance.